Impact and Performance Evaluation of USAID/Macedonia’s Small Business Expansion Project MARCH 2018 This publication was produced for review by the United States Agency for International Development. It was prepared independently by Management Systems International, A Tetra Tech Company. COVER PHOTO Farmers market by the road – Rosoman Credit: GfK Skopje EVALUATION REPORT IMPACT AND PERFORMANCE EVALUATION OF USAID/MACEDONIA’S SMALL BUSINESS EXPANSION PROJECT Prepared by: Molly Hageboeck Rozalija Karchicka-Vasilevska Suzana Ignatovic-Aleksova Brian Price Andrew McNamara Irene Velez Lala Kasimova Contracted under GS-23F-8012H, Task Order AID-165-M-13-00001 Performance Evaluation of YES Network and Impact Evaluation of Small Business Expansion and Civil Society Projects in Macedonia DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP iii ABSTRACT This evaluation reports results from USAID’s $5 million, five-year (2012-2016) Small Business Expansion Project (SBEP) in Macedonia. Focused on regional economic development in two regions (later four with Swiss resources), SBEP facilitated the creation of local economic groups and invested in initiatives including small-scale drip irrigation systems for increasing corn yields; adventure travel; light manufacturing; wild product gathering; and financing for “women in business” ventures in ethnic Albanian communities. Initially envisioned as an impact evaluation of SBEP corn and agribusiness initiatives, the evaluation gathered baseline data through national baseline surveys that USAID expected to be of value for this evaluation and other regional development purposes. An additional targeted survey gathered pre-intervention data from two annual waves of corn farmers that SBEP picked to receive drip irrigation systems and associated training. When the evaluation mid-point implementation fidelity review revealed that the anticipated agribusiness initiative had not gained traction, evaluation endline funds were reprogrammed into performance evaluations of other SBEP initiatives. Endline data from the quasi-experimental impact evaluation of the drip irrigation interventions found a 222 percent increase in treatment farmer production (tons/hectare) compared with a 91 percent average increase among corn farmers without drip irrigation in non-project regions, and a 15 percent increase among non-treatment corn farmers in the four SBEP-supported regions, one of which was Macedonia’s top corn-producing region before SBEP launched. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP iv ACKNOWLEDGMENTS Management Systems International (MSI) wishes to thank the countless men and women in Macedonia who took time away from their firms and farms to provide answers to questions about their enterprises and their experiences through a variety of initiatives under USAID’s Small Business Expansion Project that were designed to help these entrepreneurs take better advantage of market opportunities in the regions where they live. MSI is also grateful to the individuals in government and the private sector at the national, regional and municipal levels who provided information on their exposure to and experience with this project. This study could not have been completed without the exceptional assistance provided by GfK Skopje in Macedonia, the firm that supported this study through its survey research, group interviews and contributions to the analysis of the data collected. Special thanks also goes to the Macedonia members of MSI’s evaluation team, who traveled though the regions gathering data from individuals and organizations involved in the regional development process. At USAID/Macedonia, MSI would like to thank Mr. Ivica Vasev and Ms. Tatjana Markovska for their thoughtful inputs and support over the course of this multi-year evaluation. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP v TABLE OF CONTENTS Abstract............................................................................................................................................................................... iii Acknowledgments............................................................................................................................................................ iv Acronyms and Abbreviations........................................................................................................................................ vii Executive Summary ........................................................................................................................................................ viii Evaluation Purpose and Questions................................................................................................................................1 Evaluation Purpose and Scope ...................................................................................................................................1 Evaluation Questions....................................................................................................................................................2 Project Background...........................................................................................................................................................3 Evaluation Methods and Limitations..............................................................................................................................5 Study Limitations...........................................................................................................................................................6 Findings, Conclusions, and Recommendations...........................................................................................................8 Findings............................................................................................................................................................................8 Evaluation Question 1 .............................................................................................................................................8 Evaluation Question 2 ...........................................................................................................................................12 Evaluation Question 3 ...........................................................................................................................................22 Evaluation Question 4 ...........................................................................................................................................26 Conclusions..................................................................................................................................................................30 Adoption of the “New” ........................................................................................................................................31 Results Monitoring Choices.................................................................................................................................31 Sustainability vs. Business Plans...........................................................................................................................32 Recommendations ......................................................................................................................................................32 Annex A: Evaluation Statement of Work ..................................................................................................................35 Annex B: Approved Evaluation Design Report........................................................................................................46 Annex C: SBEP Evaluation Sub-studies.......................................................................................................................62 Sub-Study 1: Regional Development Support ......................................................................................................63 Sub-Study 2: Grow More Corn Initiative ..............................................................................................................86 Sub-Study 3: Women in Business Initiative........................................................................................................ 119 Sub-Study 4: Adventure Travel Initiative............................................................................................................ 135 Sub-Study 5: Wild-Gathered Products Initiative............................................................................................... 146 Sub-Study 6: Light Manufacturing Initiative ........................................................................................................ 158 Annex D: Data Collection Instruments .................................................................................................................. 170 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP vi Survey Instrument – Farms (main survey) ......................................................................................................... 170 Semi-Structured Interview Instrument – Federation of Farmers of Macedonia (FFM)........................... 190 Survey Instrument – Women in Business.......................................................................................................... 193 Survey Instrument – Tourism ............................................................................................................................... 212 Survey Instrument – Light Manufacturing (Supplier Version)........................................................................ 226 Survey Instrument – Light Manufacturing (Buyer Version)............................................................................ 237 Survey Instrument – Agribusiness........................................................................................................................ 248 Semi Structured Interview – Business and Farmer’s Association or Group Representatives............... 264 Semi Structured Interview – Mayor and LED Representatives..................................................................... 281 Semi-Structured Interview Instrument – LAG Representatives Round 2 (2016)..................................... 298 Semi-Structured Interview Instrument – RDC Representatives Round 2 (2016).................................... 306 Semi-Structured Interview Instrument – Association for Local and Rural Development (ALRD)...... 314 Additional Semi-Structured Interview Instruments......................................................................................... 317 Annex E: Sources of Information.............................................................................................................................. 320 Persons Interviewed................................................................................................................................................ 320 Bibliography of Documents Reviewed ................................................................................................................ 322 Databases................................................................................................................................................................... 324 Annex F: Conflict of Interest Disclosures.............................................................................................................. 325 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP vii ACRONYMS AND ABBREVIATIONS AIRE Association for Innovations in Rural Economies ATTA Adventure Travel Trade Association B2B Business-to-Business BSO Business Support Organization EU European Union FY Fiscal Year GDP Gross Domestic Product GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit (German Society for International Cooperation) HS Harmonized System IDEAS Investment Development and Export Advancement Support IE Impact Evaluation KII Key Informant Interview LAG Local Action Group LEADER Liaison Entre Actions de Développement de l'Économie Rurale LED Local Economic Development Office MAMEI Macedonian Association of Metal and Electrical Industry MCP Macedonia Competitiveness Project MSI Management Systems International MSME Micro, Small, and Medium-Sized Enterprise PMP Performance Management Plan RDC Regional Development Center REG Regional Economic Growth RFP Request for Proposals SBEP Small Business Expansion Project SGE Swiss Global Enterprise SOW Statement of Work UN United Nations USAID United States Agency for International Development USG United States Government EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP viii EXECUTIVE SUMMARY In May 2013, USAID/Macedonia, under Task Order AID-165-M-13-00001, asked Management Systems International (MSI), a Tetra Tech company, to conduct three evaluations. One of the evaluations was envisioned as being an impact evaluation of the Mission’s Small Business Expansion Project (SBEP), a multi-component regional economic development activity initiated in October 2012 with a budget of $5 million with its termination anticipated in September 2016. The project was subsequently modified once to absorb partner funding from the Government of Switzerland in the amount of $1.7 million and a second time to extend its termination to December 2016. Evaluation Purpose and Scope The purpose of the evaluation, as articulated in the task order, was to provide USAID with the most rigorous findings possible on how SBEP affected economic growth in the regions where it was active. USAID/Macedonia and country partner stakeholders were envisioned as the primary audiences for this evaluation, which USAID anticipated would inform future programming decisions. Study scope, which originally included multiple impact evaluations was later scaled back on that aspect of the work and funds were reprogrammed to look at additional SBEP initiatives on a program evaluation basis. The final questions for this evaluation are the following: 1. What have been the measurable effects of SBEP initiatives that operationalize its intervention model on enterprise/farm and community mobilization for economic development as measured by perceptions of economic opportunity, expectations about economic returns and growth, and economic growth-oriented investments at the enterprise/farm and community level? 2. What have been the measurable effects of SBEP initiatives that operationalize its intervention model on enterprise/farm, municipal and regional production volume, product quality, and jobs linked to supply chain requirements in which SBEP-organized training and technical assistance focused? 3. What have been the measurable effects of SBEP initiatives that operationalize its intervention model on economic growth, the emergence of new enterprises, unemployment and other broad economic development measures in municipalities and regions where SBEP was active? 4. What evidence of SBEP project “spill-over” effects or spontaneous replication of SBEP initiatives or aspects of the SBEP model, if any, does evidence collected by the evaluation from regions in which SBEP was not active reveal? Project Background USAID/Macedonia launched SBEP in 2012 as a four-year, $5 million activity, implemented by the CARANA Corporation, to stimulate economic growth, catalyze job creation, and mobilize funds for regional economic development. SBEP worked initially in Polog and Pelagonia, with resources in reserve for a possible third region, Vardar. With an added $1.7 million from the Government of Switzerland, SBEP’s coverage expanded to cover the Northeast regions. Performance monitoring for this project focused on indicators and targets that cut across SBEP, such as jobs created, rather than on the unique nature of these initiatives. Evaluation Methods and Limitations To address USAID’s evaluation questions, MSI constructed and vetted a theory of change that is holistic enough to capture the common intent of all initiatives the SBEP team was exploring, yet open enough to reflect their distinct characteristics. The evaluation team used a mixed-methods approach for evaluating EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP ix each SBEP initiative it was asked to examine. For the impact evaluation portion, a counterfactual was constructed from a nationally representative of corn farmers throughout Macedonia. A more complete description of study methods is in Annex B, a list of persons interviewed is in Annex E, and the interview instruments can be found in Annex D. Issues that may have affected the quality and reliability of data used in this evaluation include: • Missing beneficiary contact information issues; • Respondent fear of possible tax repercussions for answers; • Political campaign season and weather hindrances; and • Number of Grow More Corn Initiative data points. Findings, Conclusions, and Recommendations Findings This section provides answers to each of the four evaluation questions. Empirical data are presented in support of specific findings. Sub-studies that provide a more detailed review of findings for each initiative are provided in Annex C. Evaluation Question 1 What have been the measurable effects of SBEP initiatives that operationalize its intervention model on enterprise/farm and community mobilization for economic development as measured by perceptions of economic opportunity, expectations about economic returns and growth, and economic growth-oriented investments at the enterprise/farm and community level? Question 1 focuses on perceptions and expectations about economic growth and development among SBEP beneficiaries, as well as in the larger communities where these beneficiaries live. Survey data showed that among regional development stakeholders SBEP worked with, those associated with the adventure travel industry were most likely to perceive their businesses as being better off in 2016 than in the past (67 percent), along with private sector representatives in municipalities in SBEP￾assisted regions (70 percent). Municipal public-sector officials in SBEP regions and SBEP Grow More Corn and Women in Business participants were closer to 35 percent on this question, while comparison group farmers (25 percent) and public and private sector representatives from non-SBEP regions (26 percent) were less likely to say they were better off. Response patterns were similar on questions that asked about perceptions of financial prospects for 2017, and SBEP beneficiaries were all more likely to say they planned to invest in their firm, farm, and community economic development in 2017 than were respondents who had not benefited from SBEP assistance. Evaluation Question 2 What have been the measurable effects of SBEP initiatives that operationalize its intervention model on enterprise/farm, municipal and regional production volume, product quality and jobs linked to supply chain requirements in which SBEP-organized training and technical assistance focused? The SBEP intervention model reasoned that to create jobs, the central measure of success for the project, value chains of a variety of relatively large firms would reveal products being imported that local firms might have produced, or could acquire the capacity to produce, at a reasonable cost, in sufficient EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP x quantities and standards to meet the needs of those firms. Substituting local products for imported goods would generate jobs. Production Volume Among SBEP initiatives, only the Grow More Corn Initiative focused explicitly on production volumes. In its first year, SBEP invested in a pilot test at the start of this sub-activity to determine the effect of providing a farmer with a drip irrigation system and agricultural protocols on the output of a single hectare. USAID/Macedonia also contracted with MSI to conduct an impact evaluation to independently answer the same question, applying rigorous methods for selecting a counterfactual, or comparison group. MSI’s study obtained data from treatment farmers who joined SBEP’s program in three annual waves, starting in 2013, and from a comparison group of a nationally representative sample of non￾participants. For the full group, differences between 2013 and 2016 harvests were compared, and MSI compared progress for just the first wave of farmers using drip irrigation to their own performance. Both these comparisons are shown below. The first-wave-only comparison is highlighted, as it perhaps best shows the effects of drip irrigation for farmers starting close to the national average for corn grain and at the low end of production for corn silage. A more complete review of this study is Sub-Study 2. TABLE 1-A. GROW MORE CORN FIRST WAVE COMPARISON Group Corn Grain – 2013 Average Production (t/ha) 2013 Without Drip Irrigation and Agricultural Protocols Average Production (t/ha) 2013 With Drip Irrigation and Agricultural Protocols Percent Increase in Production Between Non-Treatment and Treatment Farmers SBEP pilot treatment (40), comparison (30) 6.3 11.4 80.9 MSI 2014 survey (treatment [41]; comparison B [521]) 5.7 18.71 222.8 MSI 2016 survey (treatment [106]; comparison B [139] to other farmers in non-SBEP regions) 5.0 9.56 91.2 MSI 2016 survey (treatment [106]; Comparison A [201] to other farmers in SBEP regions)* 8.3** 9.56 15.1 * High comparison A group outcome may be driven by Pelagonia’s normally higher-than-national-average performance, spill-over of SBEP agricultural protocols, or higher-than-national-participation in the government’s 50/50 drip irrigation program in SBEP regions. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP xi TABLE 1-B. GROW MORE CORN FIRST WAVE COMPARISON Group Corn Silage – 2013 Average Production (t/ha) 2013 Without Drip Irrigation and Agricultural Protocols Average Production (t/ha) 2013 With Drip Irrigation and Agricultural Protocols Percent Increase in Production Between Non-Treatment and Treatment Farmers SBEP pilot treatment (40), comparison (30) 38.6 63.6 64.7 MSI 2014 survey (treatment [41]; comparison B [521]) 5.6 19.7 251.8 MSI 2016 survey (treatment [106]; comparison B [139] to other farmers in non-SBEP regions) 3,54 20.00 464.9 MSI 2016 survey (treatment [106]; Comparison A [201] to other farmers in SBEP regions)* 11.37 20.00 75.9 * High comparison A group outcome may be driven by Pelagonia’s normally higher-than-national-average performance, spill-over of SBEP agricultural protocols, or higher-than-national-participation in the government’s 50/50 drip irrigation program in SBEP regions. Post-Adoption Production Volumes In evaluation interviews, as elaborated in Sub-Study 2 on the Grow More Corn Initiative, MSI asked farmers about additional hectares planted and queried SBEP’s supplier Magan-Mak and representatives of other stakeholder organizations, including the Ministry of Agriculture, but none of these channels yielded evidence of a high rate of farmers adopting the combination of drip technology and related agricultural protocols relative to the number of farmers interviewed who seemed barely aware of its availability. Job Creation From a survey question about whether these entities had increased their number of employees, stayed the same, or declined over the past two to five years, only modest percentages indicated increased employment – or new jobs created – for Grow More Corn farmers (6.6 percent increased their workforce size) and Women in Business beneficiaries (8.3 percent added jobs). Organizations with which SBEP worked on the Adventure Travel Initiative indicated the highest rate of job growth (28.6 percent.) Evaluation Question 3 What have been the measurable effects of SBEP initiatives that operationalize its intervention model on economic growth, the emergence of new enterprises, unemployment and other broad economic development measures in municipalities and regions where SBEP was active? GDP and Per Capita GDP Across this period, economy-wide growth has translated into positive per capita growth in the regions, but with important differences between them. Using data from Macedonia’s State Statistical Office, MSI calculated the difference between GDP per capita in 2011, at the start of USAID/Macedonia’s Strategic Planning period (2011– 2015), and GDP figures for 2016, at the end of the SBEP project period. Ranking the results showed that Vardar, which became a SBEP-assisted region in 2015, made the most gains at EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP xii this level among the program regions, but two other regions not served by SBEP did better. Further, Northeast, which started receiving SBEP assistance in 2014 barely changed over the period. Unemployment and Enterprise Births Beyond the national level, regions differed in the degree to which unemployment declined between 2012 and 2016. Regions where SBEP was active experienced more reduced unemployment than other regions did. The evaluation team did not find strong trends in enterprise births. The total number of active enterprises in the regions where SBEP worked changed little during the period, and there was no finding of large numbers of new business entities being formed. Evaluation Question 4 What evidence of SBEP project “spill-over” effects or spontaneous replication of SBEP initiatives or aspects of the SBEP model, if any, does evidence collected by the evaluation from regions in which SBEP was not active reveal? The clearest evidence of a “spill-over” effect from SBEP initiatives into regions beyond the four target regions in which the program worked came from Magan-Mak, SBEP’s partner for drip-irrigation equipment and installation, who reported that as of December 2016, nearly 700 hectares of farmland are supported by the company’s drip-irrigation systems. With SBEP reporting that it supplied Magan-Mak drip irrigations to support 280 hectares on more than 300 farms, the firm suggested that the remainder represents the initiative’s “spill-over” effect, wherever they may be in Macedonia. As a result, more than 2.5 times the originally supported farms have implemented drip irrigation systems. Conclusions 1. SBEP was highly effective at opening citizens’ eyes to economic opportunities around them, inspiring them to try new approaches and aim high when it came to what they could accomplish. On farms, among municipal leaders, and in regional forums and ministry and donor circles, the project was well regarded and viewed as adding value through its initiatives, as well as in support of local economic development processes. 2. Among country partner observers of SBEP in Regional Development Centers and municipal governments, SBEP’s assistance leading to the formation of LAGs across four regions was often considered the most significant change the project introduced. Drip irrigation was a close second in this regard. 3. USAID’s decision to rely on bottom-up suggestions on its project’s economic development priorities was much appreciated by mayors, local economic development office representatives, and municipal private sector representatives, and viewed as right for the country’s stage of economic development. 4. The scale of SBEP initiatives may not have been large or gained early enough traction to achieve the kind of “critical mass” of awareness and evidence needed to spark their continued growth and diffusion within or beyond their regions of demonstration. 5. SBEP’s influence on regional development trends may become more detectable and significant over time. 6. The explanation for modest levels of expansion and new farmer adoption of drip irrigation technology may be more cultural than cost-based in nature, given findings concerning other SBEP initiatives suggesting that the uptake of new practices was at times fostered by community participation and ownership of those decisions. Additional research may be warranted. 7. USAID and SBEP’s decision to monitor key outcome indicators only on a project-wide basis, without initiative-specific targets on those metrics, and to otherwise favor monitoring EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP xiii standardized U.S. foreign assistance indictors in lieu of a limited number of initiative-specific, input-level indicators on the intermediate results level may have resulted in USAID not being fully aware of negative developments on regional metrics that are important to the project’s regional economic development purpose. 8. Stronger models may be needed to transition initiatives that are meant to exist beyond the length of the USAID project that creates them. Recommendations Based on this evaluation’s findings and conclusions, the team recommends that USAID/Macedonia: 1. In future large and complex economic growth projects, improve the balance between monitoring for accountability (at the input level) and monitoring to support adaptive management of economic development projects (at the intermediate result and outcome levels). Distribute indicators across the project’s theory of change, whatever it is, to keep USAID informed about progress on each linked hypothesis. Relying exclusively on USAID standard indicators can skew toward an input-level view of results, or cause USAID to not know whether progress toward all aspects of higher-level results is occurring. Implementing partners may not be responsible for achieving results along the entire theory of change, but can be asked to monitor what is happening with respect to higher-level results to which they, and USAID investments, contribute, including regular updates on national trends for which others collect data. 2. When selecting performance indicators, carefully examine what they tell the Mission – not only about a particular result but also about the likelihood of achieving higher-level results. SBEP data on corn farms provided an example of this by demonstrating that farms where USAID could record “new jobs created” may also be losing workers, resulting in unchanged total or net employment, in regions where USAID and the host country government’s ultimate aim is to reduce unemployment. This is the type of situation in which multiple indicators further along a results chain may be more useful for the Mission than multiple input-level indicators. 3. For product-based economic development activities, encourage USAID partners to undertake – while USAID funding is in place - actions that would improve the integration of those products into broader supply chains, including those that reach into international markets on both the import and export sides. Simple steps include using international harmonized system codes as an integral part of activity planning and monitoring, not only by USAID partners but also by local entities, such as the collection centers USAID funded under SBEP and Macedonia’s State Statistical Office. 4. For projects in economic growth and other sectors in which long-term and goal-level success depends on the institutionalization of initiatives in government or their continued existence on a business basis, elevate thinking about sustainability for those which must be self-financing to business plans with milestones that can be monitored and self-sufficiency outcomes to be realized before USAID’s support terminates. Milestones for the institutionalization of initiatives in national institutions may require USAID to participate directly in their achievement. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 1 EVALUATION PURPOSE AND QUESTIONS In May 2013, USAID/Macedonia, under Task Order AID-165-M-13-00001, asked Management Systems International (MSI), a Tetra Tech company, to conduct three evaluations. One was envisioned as an impact evaluation of the Mission’s Small Business Expansion Project (SBEP), a multi-component regional economic development activity initiated in October 2012 with a budget of $5 million and anticipated termination in September 2016. The project was subsequently modified once to absorb partner funding from the Swiss government in the amount of $1.7 million and a second time to extend its termination date to December 2016. The original statement of work (SOW) for the evaluation under this task order is in Annex A. Evaluation Purpose and Scope The purpose of the evaluation, as articulated in the task order, was to provide USAID with the most rigorous findings possible on SBEP’s impact on economic growth in the regions where it was active. USAID/Macedonia and country partner stakeholders were envisioned as the primary audiences for this evaluation, which USAID anticipated would inform future programming decisions. The scope of the evaluation was clarified during a planning visit from May to June 2013, USAID and MSI discussed the scope for the evaluation of SBEP and whether it would be a whole-of-project evaluation or focus more narrowly on a few of the activity’s well-defined interventions that were large enough to support a rigorous evaluation. An early decision about the focus of the evaluation was expected to immediately affect decisions about sampling for the first of two USAID-envisioned nationally representative surveys that would include enough likely SBEP beneficiaries to construct a counterfactual for the SBEP interventions the study would focus on. At the time of this planning visit, SBEP was already engaged in several initiatives, including: • The Grow More Corn Initiative, which was introducing drip irrigation systems on small farms as a means for increasing production, reducing the country’s trade deficit in this commodity, and stimulating demand and job creation by entering the supply chains of Macedonian and other locally based firms that were reliant on imported corn. • A Women in Business Initiative that identified women-owned businesses in the Polog Region and assessed their needs for support. • Interactions with Regional Development Centers and municipalities focused on expanding regional development opportunities, in line with Macedonia’s proposed accession to the European Union, through its LEADER/LAG local and regional development initiative. • Work was also underway in several other areas that had not yet matured into initiatives, including workforce development linked to the construction industry in Polog, an effort to draw investment to the regional level from the diaspora, an effort to commercialize wild product gathering in areas where few job opportunities existed, and other supply chain opportunities, including agribusiness, light manufacturing firms, and rural tourism. Based on these discussions, USAID and MSI prioritized the Grow More Corn Initiative, together with its variations for wheat and tomatoes, along with other possible supply chain interventions, all of which had a common import substitution theory of change. With respect to the national surveys, it was decided that agribusinesses and farmers would be the focus, with other decisions to be made as the evaluation and the SBEP’s initiatives evolved. The design for this evaluation, including a refined set of evaluation questions, was delivered to USAID in June 2013 and the Agency approved it shortly thereafter. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 2 One additional round of changes in the evaluation’s scope occurred early in 2015 after MSI’s Skopje￾based evaluation specialist carried out an implementation fidelity monitoring review in December 2015 and learned that SBEP had abandoned its wheat and tomato interventions. Baseline data collection had taken place and its supply chain interventions focus had shifted away from agribusinesses toward light manufacturing of a much wider variety of products. Those changes lowered the evaluation’s budgetary needs for endline survey research and created an opportunity for USAID to redirect some of the evaluation team’s resources and attention to other SBEP initiatives from a performance evaluation perspective. From among SBEP’s array of sub-activities, USAID selected the Women in Business, Adventure Tourism, Wild￾Gathered Product and Light Manufacturing initiatives, as well as the project’s long-standing involvement with the LEADER/LAG initiative in several of the regions where it worked. Evaluation Questions The final questions for this evaluation, approved by USAID/Macedonia in the June 2013 evaluation design report, are shown below. 1. What have been the measurable effects of SBEP initiatives that operationalize its intervention model on enterprise/farm and community mobilization for economic development as measured by perceptions of economic opportunity, expectations about economic returns and growth, and economic growth-oriented investments at the enterprise/farm and community level? 2. What have been the measurable effects of SBEP initiatives that operationalize its intervention model on enterprise/farm, municipal and regional production volume, product quality and jobs linked to supply chain requirements in which SBEP-organized training and technical assistance focused? 3. What have been the measurable effects of SBEP initiatives that operationalize its intervention model on economic growth, the emergence of new enterprises, unemployment and other broad economic development measures in municipalities and regions where SBEP was active? 4. What evidence of SBEP project “spill-over” effects or spontaneous replication of SBEP initiatives or aspects of the SBEP model, if any, does evidence collected by the evaluation from regions in which SBEP was not active reveal? USAID’S VISION FOR THE SMALL BUSINESS EXPANSION PROJECT (SBEP) During the assessment and design phase of this project, the team concluded that a new approach is required to create jobs in large numbers. A regional approach would offer several distinct advantages over other strategies, such as a sector approach. First, it would define a specific territory for enterprise development that would facilitate substantial involvement of local business and government leaders who instinctively like to focus on concrete opportunities. The business community would be challenged to play a more active role in defining economic development opportunities and supporting implementation of identified initiatives, along with their government and NGO counterparts. Second, it would encourage multiple approaches to economic growth and job creation, involving numerous organizations and employing all tools available for enterprise development: value chain development, clusters, and networks to reduce costs and foster cooperation to compete, business services market development, and varieties of training interventions to upgrade know-how and skills within a defined regional context linked to real market opportunities. Third, it would target the interventions and increase the potential job creation impact of USAID’s limited available resources. Last, the regional approach would complement and support the decentralization efforts and the EU accession agenda for regional economic development. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 3 PROJECT BACKGROUND USAID/Macedonia launched SBEP in 2012 as a four-year activity, implemented by the CARANA Corporation, to stimulate economic growth, catalyze job creation, and mobilize funds for regional economic development. SBEP worked initially in Polog and Pelagonia, with resources for a possible third region in reserve. With added support from the Government of Switzerland, SBEP’s coverage has allowed the project to expand to cover both the Vadar and Northeast regions. Across these regions, SBEP has pursued a range of initiatives, including innovations along cultivated agricultural and wild product value chains, the introduction of adventure tourism into Macedonia’s existing tourism value chain, matchmaking between large domestic and foreign firms working in its target regions and beyond with smaller light manufacturing suppliers, and efforts that help integrate women-owned small firms, sometimes based in households, to regional markets. In parallel, SBEP has provided hands-on assistance to municipal, sub-regional, and regional economic development planning and implementation, including in support of the integration of the EU LEADER/LAG regional development approach in Macedonia. Performance monitoring for this project focused on indicators and targets that cut across SBEP, such as jobs created, rather than on the unique nature of these initiatives. The SBEP team has been generous in sharing this information with the evaluation team. Implementing Partner CARANA Corporation Project Start/End April 2012 – December 2016 Funding Amount $5 million (USD) Project Objective Strengthen private sector capacity to drive regional economic development within selected regions of Macedonia. Expected Results and Impact An effective and replicable model for private-sector led regional enterprise development will be developed and demonstrated in three regions in Macedonia; new jobs created (25 percent for women, 30 percent for youth, 25 percent for ethnic minorities). M&E Considerations The impact evaluation collected baseline data in 2014 and endline data in 2016. A Results Framework exists with four components supporting the project goal and a Performance Management Plan (PMP) exists that tracks 16 standard and custom indicators for the project. SBEP, particularly through its commitment to a regional approach for addressing Macedonia’s chronic employment challenge, actively supported and aimed to foster implementation of the country’s 2007 Law on Balanced Regional Development. Macedonia was in the process of establishing eight Regional Development Centers (RDCs) that would work with municipalities to envision a new mechanism for engaging municipalities, citizen groups, and the private sector in collaborative efforts to draw capital in, create economic resources, and stem the flow of outmigration, particularly among young people. Ideas would emerge from the bottom up, and a key role of the RDCs would be to help channel public sector funds to those most likely to facilitate regionally based, market-led development. SBEP also stood on the shoulders of an array of private sector-led development initiatives over the years, including the Competitiveness Project (2007-2013), AgBiz (2007-2013), and IDEAS (2010-2014). Consistent with USAID’s regionally led vision for SBEP, the SBEP team invested in building networks of public and private sector relationships in Pelagonia and Polog at the start, later expanding with the same philosophy to two other regions, Vardar and Northeast. Through working sessions in the regions, including workshops convened around the SBEP’s own launch activities, it sought ideas for initiatives to emerge locally, such as Polog’s commitment to organizing an initiative that would help local EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 4 businesswomen. In the country’s only region with an Albanian ethnic majority, many of these women operated businesses out of their homes. In the same way, SBEP drew on existing interests and past USAID efforts as it worked with stakeholders who saw possibilities in forest resources for improving the livelihoods of those who had already gathered wild products. Earlier market research, under USAID’s AgBiz project, had identified a supply chain path leading to existing European export markets that SBEP would expand into a full-blown trial through its work with both the public and private sectors. SBEP’s largely organic approach to developing initiatives with regional appeal, no matter where in the territory it was working when an idea emerged, was naturally experimental and unusual compared to the standard USAID project design structures in that a high tolerance for failure was built into the strategy. This allowed the pursuit of ideas for a while with a relatively low level of project resources, including time, committed. Those that already had momentum or around which it grew over time stayed on, while others floated away or were deliberately halted. Notably, in its final report, the SBEP team provided a list of these fallen soldiers; like startup businesses that don’t survive, they were replaced with stronger stuff. Across all of the interventions it explored, the bedrock for their development lay in the network of public and private sector relationships that SBEP built continuously starting in its first days, both horizontally (from municipality to village to municipality) and vertically (from local action groups [LAGs] of multiple municipalities, operationalizing a European model for collaboratively realizing a shared development vision, working from each LAGs through its circle of mayors and up to each region’s RDC and back). At the time this evaluation started, with its scoping visit in June 2013, what would become SBEP’s long￾term operating mode was still evolving into something akin to the central networking planning space around which initiatives orbited and through which they interacted synergistically. The fluidity of the SBEP approach is perhaps what fostered the ease with which it worked with other like-minded actors, notably the office of the German Society for International Cooperation (GIZ) in Tetovo, the Northwest Chamber of Commerce, the Swiss Embassy, and USAID’s Regional Economic Growth Project (REG) over the years. FIGURE 1. SBEP MANAGEMENT DIAGRAM Through this evaluation, MSI examined each of the six interventions shown in Figure 1. For each of these initiatives, the evaluation report includes a sub-study designed to acquaint readers with initiative-specific study findings, while this main portion of the report examines those indicatives collectively to address USAID’s four evaluation questions. These sub-studies are in Annex C. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 5 EVALUATION METHODS AND LIMITATIONS This section summarizes the evaluation methods used to conduct the impact evaluation of SBEP’s Grow More Corn Initiative, as well as those employed for the performance evaluation components that examined several other SBEP initiatives. To address USAID’s evaluation questions, MSI’s team drew on what it was learning from USAID and the SBEP staff to construct a theory of change that is holistic enough to capture the common intent of all the initiatives the SBEP team was exploring, yet open enough to reflect their distinct characteristics. The graphic for this set of linked hypotheses, in Figure 2, was included in the evaluation design submitted in June 2013, and it guided the development of both impact and performance evaluation methodologies for conducting the evaluation. Its development drew heavily on ideas and objectives incorporated in USAID’s initial statement of work for the evaluation. FIGURE 2. SBEP MULTI-INITIATIVE THEORY OF CHANGE Aspects of this theory of change were particularly useful at the start of this evaluation for designing a nationally representative survey instrument that would be used to gather data for what was envisioned as baseline useful for examining the effects of more than one SBEP initiative. It set the stage for developing instruments for extracting data from SBEP quarterly reports, as well as developing key informant interview (KII) protocols later in the project. The evaluation team used a mixed-methods approach for examining each of the SBEP initiatives it was asked to examine. These included document reviews keyed to information about beneficiary and partner perceptions, expectations, production and sales, jobs created, and investments by farmers and business owners. Economy-wide information on growth rates, unemployment, imports, women employers, and crop production all came from Macedonia’s State Statistical Office. The exception was import data for Harmonized System (HS) numbers below the level that the State Statistical Office normally publishes, such as for wild mushrooms and berries, which the evaluation team obtained from the United Nations Comrade Database. The evaluation team also reviewed results information that SBEP reported in its quarterly reports and its performance indicator tracking system. Primary data for the evaluation was collected through interviews. The majority of those interviews were conducted by MSI’s local research partner, GfK, which conducted baseline and endline surveys for the study. The surveys included both closed- and opened-ended questions. Key informant and other beneficiary interviews were conducted by seasoned local evaluators who are fluent in languages used by the country’s ethnic Macedonian and ethnic Albanian populations. Statistical analysis of the study data was supported by GfK, using SPSS software, with the intent to transfer the study’s national data set to USAID, consistent with its open data protocols. Representative survey and purposive interview protocols were all used together with a study-wide informed consent procedure. Respectful of cultural norms in the Polog Region, women conducted the interviews for the Women in Business Initiative. Table 2 contains a summary of interviews conducted. A list of people interviewed is in Annex E and the interview instruments are in Annex D. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 6 TABLE 2. SUMMARY OF INTERVIEWS CONDUCTED FOR THE EVALUATION SBEP Initiative Summary of Interviews Conducted Regional Development – LEADER/LAG Baseline (2014) and endline (2016) interviews with seven RDCs. Baseline interviews with 63 small groups of municipal government representatives and small groups of private sector stakeholder across the country in 2014 and again in 2016. Interviews with 10 representatives of local action groups (LAGs). Key informant interviews with government representatives and other stakeholders (see Annex E for a list of people interviewed). Grow More Corn Survey interviews with 1,799 comparison group farmers in 2014 and interviews with 298 farmers in 2016 who were known to grow corn; many repeated with the same individuals. Survey interviews with 245 agribusinesses in 2014; no follow-up survey was conducted. Both national surveys conducted in 2014 had a confidence level of between 90 and 95 percent, with a confidence interval of +/- 5 percent. 77 farmers who received SBEP assistance in 2014 and 106 farmers in 2016. Government representatives and other stakeholders, including SBEP partners. Women in Business 12 of 25 women business owners/managers who received SBEP assistance. Government representatives and other stakeholders, including SBEP partners. Adventure Tourism Phone interviews with 21 tour operators, travel agencies, and sports-related firms. Government representatives and other stakeholders, including SBEP partners. Light Manufacturing Government representatives and other stakeholders, including SBEP partners. Wild-Gathered Products Government representatives and other stakeholders, including SBEP partners An important difference between SBEP’s Grow More Corn Initiative and the other SBEP initiatives the evaluation team examined is that the Grow More Corn study used the norms for rigorous quasi￾experimental impact evaluations. A large comparison group was constructed and at the baseline stage, and power calculations for this impact evaluation determined the effect of drip irrigation and related agricultural protocols on agricultural outcome variables including hectares planted, crop yield (kilograms per kilometer), production levels (tons per hectare), and sales and profits if enough respondents were willing to share that data. Calculations in the study’s baseline report in July indicated that a power level of .80 was feasible, given the number of interviews it included with farmers who received SBEP assistance. Similarly, baseline data collection for agribusinesses was sufficient to expect that the same standards could be met if an agribusiness impact evaluation opportunity materialized. The full design for this evaluation is in Annex B. Study Limitations Issues that may have affected the quality and reliability of data used in this evaluation include: Beneficiary Identification and Contact: While beneficiary lists with contact information were well organized for most SBEP initiatives, the project kept records of large and small firm “pairs” they had matched to gauge mutual interest on the part of the parties that would yield transactions; this never materialized into a usable form with contact information. Several partial lists lacked information about which names on the list had been matched or up-to-date contact information; these could not be used for the purpose intended. In the end, the team received a list of attendees at a September 2016 Expo conferences, which also could not be used to identify and interview pairs of firms SBEP has “matched.” In the end, the team abandoned the set of interviews planned for this initiative. MSI’s experience trying EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 7 to find a list of matches and contact a small number of pairs did not square well with project statements in its closeout report to the effect that 21 buyer-supplier relationships had been established, i.e., entered the supply chains of lead firms, and that 53 jobs had been established under this initiative. Absent a functional list early in the data collection period, the evaluation team was unable to verify either of those things. In the end, the best the team could do was screen the SBEP quarterly reports and construct what is now a table that catalogues discussions in public documents about matches, of which those documents appear to verify only two or possibly three pairings that resulted in transactions. The table is included in the evaluation’s Sub-Study 6 on the Light Manufacturing Initiative. Political Campaign Season and the Weather: Several seemingly external factors affected study data collection. Repeated changes in the schedule for Macedonia’s election in 2016 conspired with the availability of key actors and the weather to preclude site visits to wild product collection sides where small group interviews with recipients of SBEP assistance might have been possible. The team did, however, complete phone interviews with relevant actors for this intervention. By extending the data collection period past the election, interviews with key government figures and other stakeholders were possible. Only a small number of planned interviews were missed. Fear of Possible Tax Repercussions: Several farmers voiced fears to GfK interviewers that data from SBEP surveys could fall into the hands of government officials and trigger potential tax repercussions. During data analysis, the study team discovered that this concern had translated into selectivity in answering some questions. Low response rates on questions in all interview data sets were therefore examined. In some cases, it turned out that this was a function of answers received early in interviews that eliminated the need to continue a specific question series. On other questions, particularly those relating to sales and revenues, some hesitancy to respond may have existed, but the number of missing cases turned out to be smaller than feedback from the field suggested it might have been. Number of Grow More Corn Initiative Data Points (Farmers): The impact study for the Grow More Corn Initiative included 106 respondents for some questions, but answers for others were available only for a subset of individuals, such as farmers who had used their drip irrigation with corn grain rather than corn silage and could provide harvest data on the former. When the number of respondents was low, such as on yield, it was not prudent to use preferred analysis techniques that are not adequately powered. Accordingly, some study response analyses use analytic techniques that are less sophisticated than desired. The Absence of Intervention-Specific Indicators and Targets: USAID’s whole-of-project indicators, such as jobs and investment mobilized, are useful. However, the absence of initiative-level target levels on those major indicators, as well as the absence of at least one custom indicator for determining whether specific initiatives are on track toward the intermediate results, most likely contributed to achieving the major results, make it difficult to determine each initiative’s absolute as well as relative (to other initiatives) performance in an objective and proactively agreed-upon way. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 8 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Findings This section provides answers to each of the four evaluation questions. Under this evaluation, each SBEP included initiative drew on interviews and documentary sources specific to those efforts. To do justice to each intervention, and simultaneously address the four main evaluation questions that draw on all the initiatives, MSI separated the report into two segments. The first segment addresses the four study questions in a holistic way, drawing as needed on various initiatives. The second segment, which follows the study’s conclusions and recommendations, includes six sub-study reviews, one for each initiative. This device should help keep the length of the main report down for the busy reader, but still focus on the specifics of the initiatives that most interest them. Evaluation Question 1 What have been the measurable effects of SBEP initiatives that operationalize its intervention model on enterprise/farm and community mobilization for economic development as measured by perceptions of economic opportunity, expectations about economic returns and growth, and economic growth-oriented investments at the enterprise/farm and community level? Perceptions and Expectations about Economic Returns and Growth Across the range of SBEP stakeholders the evaluation team interviewed, three questions were consistently posed to reveal how Regional Development Center (RDC) representatives, municipal representatives from both the public and private sectors, farmers, women-business owners, firms, and associations working on the Adventure Travel Initiative viewed the economic situation of their areas, farms, and firms, both retrospectively and prospectively. Most of these stakeholders based their answers on whether they were recipients of SBEP assistance in any of the regions where the program worked. The cluster of questions from which their perceptions and expectations were drawn were variations on the following: • How did the economic situation of your (area, farm, or firm) in 2016 compare to the previous year (2015) or the year just before you received assistance from SBEP? • What do you expect will be the economic situation of your (area, farm, or firm) next year (2017) compared to this year (or a specified prior year)? Answers received across stakeholder groups on these questions are summarized below. Economic Situation in Prior Year (2015) Compared to Current Year (2016) • Small groups of representatives of three out of five RDCs that responded said that the economic situation of their regions was better in 2016 than in the prior year (East, Vardar, and Southeast); two of five small groups said their region’s economic situation was worse (Northeast and Southwest); Pelagonia’s representatives indicated no change from the prior year, and the Polog RDC was not able to arrange to participate in these interviews. As the Skopje Region was not included in the evaluation, no RDC interview was conducted there. • A question for the 10 pre-local action groups (LAGs) that are emerging in the four regions where SBEP worked asked interviewees about the economic situation of their areas of focus in the current year (2016) compared to the past few years. Representatives of five LAGs said their situations had improved and two said theirs had worsened. • Around Macedonia, mayors and their local economic development office heads were also asked a similar question in small groups about their municipalities, as were small groups of private sector EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 9 representatives in the same communities. In other interviews corn farmers, women in Polog in the SBEP program for women business owners, and tourism industry representatives involved in Adventure Travel were all asked similar questions. In the tables that follow each question, their responses are shown along with answers by SBEP program beneficiaries or partners, compared to answers from comparison groups or municipalities that did not receive SBEP assistance. How would you compare your municipality/farm or firm’s financial situation to a couple of years ago (or before you received assistance from SBEP)? Better than Before Worse than Before SBEP ASSISTED Mayors – SBEP Regions 39 14 Private Sector – SBEP Regions 70 10 Farmers – SBEP Assisted 42 15 Women Firms – SBEP Assisted 33 16 Travel Firms working with SBEP 67 5 NOT ASSISTED Mayors – Other regions 25 36 Private sector – Other Regions 27 27 Corn Farmers – Same regions, not assisted 23 15 As the table illustrates, responses from firms, farmers, municipalities, and other regional development organizations were all more positive about their current financial situation than representatives of various types of entities in regions where SBEP was not active were. Financial Prospects for the Coming Year (2017) In answer to a separate question, each group of regional stakeholders described above was asked to consider their firm, farm, or municipality’s likely financial situation in the future, and whether the coming year (2017) was expected to be better or worse for them than the current year (2016). • Representatives of three RDCs (East, Vardar and Southeast) expected that things would improve in the coming year. Pelagonia’s representative expected no change and speaking for Northeast shortly before the 2016 election, one respondent suggested that much might depend on the political results. • Representatives associated with five LAGs said in interviews that they expected their areas to fare better in the following year, while one thought things would be worse for that LAG. • Municipalities, farmers, and representatives of firms were all asked similar questions and their answers, along with those of comparison groups for these types of organizations, are shown. How would you characterize the financial prospects for your firm, farm, or municipality in the coming year (2017)? Better than Before Worse than Before SBEP ASSISTED Mayors – SBEP Regions 64 10 Private Sector – SBEP Regions 40 20 Farmers – SBEP Assisted 42 15 Women Firms – SBEP Assisted 33 16 Travel Firms working with SBEP 67 14 NOT ASSISTED Mayors – Other regions 39 14 Private sector – Other Regions 27 27 Corn Farmers – Same regions, not assisted 23 15 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 10 As with the comparison that representatives of various types of organizations made between the current year and the recent past, these individuals were also more positive about the financial prospects of their firms, farms, and municipalities or regions where they live than were respondents who had no exposure to SBEP initiatives or who lived in other regions. Economic Growth-Oriented Investments at the Enterprise Farm and Community Levels • How would you describe investments you made (in your area’s economic development, your farm, or your firm) this year (2016) compared to the prior year (2015)? • How likely is it that your organization (Regional Development Center (RDC), LAG, municipality, farm, or firm) will make investments in its development during the coming year (2017)? Investments Made During the Current Year (2016) Compared to the Prior Year (2015) Five of the six small groups of RDC representatives (East, Vardar, Pelagonia, Southwest, and Southeast) indicated that their RDCs invested more in their regions in 2016 than in 2015, while the other RDC respondent (Northeast) indicated that that the amount invested was roughly the same as the prior year. Likelihood of Investing Over the Coming Year (2017) RDC representatives for three regions (East, Pelagonia, and Southeast) said it was very likely that they would participate in making investments in the region over the coming year, while two others (Northeast and Vardar) said investments in the coming year were somewhat likely. The representatives for Southwest indicated that they could not yet say for sure. How likely is it that you will make further investments in the economic development or situation of your municipality, farm, or firm, beyond basic operating expenses? Likely / Very Likely Unlikely / Very Unlikely SBEP ASSISTED Mayors – SBEP Regions 61 Private Sector – SBEP Regions 25 Farmers – SBEP Assisted 55 Women Firms – SBEP Assisted 33 Travel Firms working with SBEP 52 NOT ASSISTED Mayors – Other regions 46 Private sector – Other Regions 30 Corn Farmers – Same Regions, not assisted 29 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 11 Corn Farmers – Same Regions, not assisted 29 SBEP INITIATIVES IN PELAGOINA EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 12 Evaluation Question 2 What have been the measurable effects of SBEP initiatives that operationalize its intervention model on enterprise/farm, municipal and regional production volume, product quality, and jobs linked to supply chain requirements in which SBEP-organized training and technical assistance focused? To frame its answer to this question, the evaluation team begins with a brief explanation of the SBEP intervention model. To create jobs, the central measure of success for the project, SBEP reasoned that along segments of the value chains of a variety of relatively large firms already operating in Pelagonia and Polog, the first two regions SBEP would assist, products were being imported that local firms might have or could acquire the capacity to produce at a reasonable cost and in sufficient quantity and at the standards required to replace the imported products with locally made ones. Firms that succeeded would generate jobs and in turn generate their own demands, which could also be met with locally produced products. SBEP’s illustration of this import substitution model involves a farmer who produces corn grain and corn silage, becoming the supplier of choice to both a dairy and a firm that raises chickens. A small version of this diagram appears in Figure 3. SBEP’s first full-blown initiative, Grow More Corn, applied this model quite literally, as explained more fully in Sub-Study 2: Grow More Corn Initiative. The model was also the backbone of numerous light manufacturing possibilities that SBEP pursued under that initiative, including for the production of inputs for airbag cushions, a controlling system for wood stove pellets, tool production, and others listed in Sub-Study 6: Light Manufacturing Initiative. This supply chain penetration model was also evident in other SBEP initiatives in slightly different forms. The “Wedding Cake Supply Chain” text box tells of a The Wedding Cake Supply Chain Ms. Sinani produces cakes and sweets for traditional weddings, the key feature of the local tourism economy. During the summer high season, she has up to 50 orders of cakes a day; she now plans to expand her business to include more local restaurants and shops. To satisfy the increasing demand, she has solicited assistance from the women from the village, many of whom are disadvantaged: single mothers, and women who have lost their husbands and/or have limited opportunities to engage in economic activity due to patriarchal cultural norms in this rural Albanian and predominantly Muslim community. Ms. Sinani also buys milk, butter and eggs from women in the village; thus, as her business expands, actors within her supply chain will benefit. To increase production and respond to market demand, Ms. Sinani is currently preparing her workshop for HACCP certification, which will enable her to make deals with shops and restaurants that wish to sell her products. SBEP Quarterly Report, April-June 2013 FIGURE 3. IMPORT SUBSTITUTION MODEL EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 13 story in Polog that illustrates the SBEP model’s role in investments in women-owned businesses, discussed more fully in Sub-Study 3: Women in Business. Under SBEP’s Wild-Gathered Products Initiative, an export variation on the model gained a foothold when SBEP’s support for wild product collection centers in the Mavrovo National Park produced a sufficiently large and organized supply of mushrooms and berries. This enticed Macedonian export suppliers to the European market to enter into buying contracts with the park for access and to make arrangements with local gatherers that SBEP had trained, under which they paid the cost of commercial gathering certificates for those gatherers in exchange for a steady supply of product, which they then bought from the gatherers (Sub-Study 5: Wild￾Gathered Products). Even SBEP’s Adventure Travel Initiative reflects this SBEP model (Sub-Study 4: Adventure Travel). With domestic tourists accounting for roughly half of the tourist nights spent, whatever the adventure travel industry (which SBEP helped put on the map) does to make vacationing at home more attractive to young experience-seeking Macedonians encourages them to book locally rather than travel abroad. This fits right into an import substitution way of thinking. Moreover, every time a foreign tourist steps off an airplane in Skopje and pays for a cab into town, it rings up as a Macedonian export. Win-win. Regional Production Volume Among the six SBEP initiatives the evaluation team examined, only Grow More Corn focused explicitly on production volumes. However, there are good reasons for treating both wild product gathering and adventure travel as enterprises that yield countable units that, from a job creation perspective, are as important for Macedonia as the tons of corn that farmers literally produce. SBEP’s Light Manufacturing Initiative also focuses on products, but the evaluation team was unable to identify any matchmaking efforts under that initiative resulting in reports of recurrent transactions for specific products on which SBEP collected and reported data over time. The absence of performance reporting in produced volume terms from SBEP was not unique to the Light Manufacturing Initiative; it was common to all SBEP initiatives. No data on corn production over time was collected from the 280 farmer beneficiaries who received drip irrigation systems; SBEP did not document and report the kilos of products that were wild-gathered and sold annually in collection centers for which SBEP purchased scales; and the project did not track and report numbers of tourists and tourist nights at the adventure travel facilities where SBEP arranged financing. Production volume of whatever could be counted under the various initiatives was not a SBEP performance indicator. SBEP invested in a pilot test at the start of the Grow More Corn sub-activity to determine the effect of providing a farmer with a drip irrigation system and agricultural protocols on the output of a single hectare. Almost in parallel, USAID/Macedonia contracted with MSI to conduct an impact evaluation to independently answer the same question, applying rigorous methods for selecting a counterfactual, or comparison group, against which to compare findings from one-hectare plots farmed by SBEP beneficiaries. At USAID’s request, the MSI study drew a nationally representative sample of farmers and from it culled those who grew corn. In both the SBEP trial and the MSI evaluation, beneficiaries (the treatment group) was not randomly assigned, as is required in an experimental design for an impact evaluation. Beneficiaries were selected by SBEP in each of the three years during which the project distributed its drip irrigation and protocols package; this made the MSI evaluation, and SBEPs own trial, quasi-experiments. The motivation behind these research trials in both cases was to be able to answer USAID’s question about production volumes from the Grow More Corn Initiative, using research trial results with respect to corn yield (kilograms/hectare) and production (tons/hectare), and extrapolating from a few cases to all farmers who applied the Grow More Corn package of equipment and protocols to all their farmed hectares, or hectares that others might farm in the future. These metrics also give USAID/Macedonia a way to compare project outcomes, extrapolated to larger numbers of hectares and farmers to data on the same two metrics that Macedonia’s State Statistical Office compiles for the country as a whole and by region, which it publishes each June. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 14 With respect to corn volume improvements achieved with the SBEP combination of drip irrigation and agricultural protocols, SBEP’s pilot found that production rose from 6.3 tons/ha to 11.4 t/ha, a 90 percent increase, and for corn silage it rose from 36.6 t/ha to 63.6 t/ha, a 65 percent increase. See Row 1 in the tables that follow. MSI, using treatment farmer data it collected directly from SBEP beneficiaries who received the SBEP package in waves over three years, and comparison farmer data from the national sample living in regions where SBEP was not active, made identical calculations. See Row 2 in tables 3-A and 3-B. Like SBEPs group of farmers for the Row 2 calculations, MSI’s comparison group was somewhat above the national average. Since the reason for this was not clear, and corn output from MSI’s mixed three-wave treatment group was lower than SBEP’s, MSI wondered if layering waves in setting up that calculation could have merged more seasoned drip irrigation farmers with novices in some way the team could not detect. To get a simpler look at what the data say, MSI went back and drew forward its Round 1 data for the Wave 1 treatment group for 2013. With these data, MSI ran the calculations once more, using Round 1 comparison group data from those same years. The results, in Row 3 below, were surprising, surpassing both SBEP’s findings collected in Pelagonia and MSI’s own findings using treatment farmers from multiple waves across three years of the SBEP program. For farmers selected in a nationally representative sample in 2014, and screened for whether they grew corn, and re-interviews with corn farmers from that set in 2016, MSI’s findings follow, reporting on a common basis with SBEP using data for 2013 for the farmers in Row 2. Additional details for these impact study results are in Sub-Study 2. TABLES 3-A AND 3-B. GROW MORE CORN FIRST WAVE COMPARISON Group Corn Grain – 2013 Average Production (t/ha) 2013 Without Drip Irrigation and Agricultural Protocols Average Production (t/ha) 2013 With Drip Irrigation and Agricultural Protocols Percent Increase in Production Between Non-Treatment and Treatment Farmers SBEP pilot treatment (40), comparison (30) 6.3 11.4 80.9 MSI 2014 survey (treatment [41]; comparison B [521]) 5.7 18.71 222.8 MSI 2016 survey (treatment [106]; comparison B [139] to other farmers in non￾SBEP regions) 5.0 9.56 91.2 MSI 2016 survey (treatment [106]; Comparison A [201] to other farmers in SBEP regions)* 8.3** 9.56 15.1 * High comparison A group outcome may be driven by Pelagonia’s normally higher-than-national-average performance, spill-over of SBEP agricultural protocols, or higher-than-national-participation in the government’s 50/50 drip irrigation program in SBEP regions. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 15 Group Corn Silage – 2013 Average Production (t/ha) 2013 Without Drip Irrigation and Agricultural Protocols Average Production (t/ha) 2013 With Drip Irrigation and Agricultural Protocols Percent Increase in Production Between Non-Treatment and Treatment Farmers SBEP pilot treatment (40), comparison (30) 38.6 63.6 64.7 MSI 2014 survey (treatment [41]; comparison B [521]) 5.6 19.7 251.8 MSI 2016 survey (treatment [106]; comparison B [139] to other farmers in non-SBEP regions) 3,54 20.00 464.9 MSI 2016 survey (treatment [106]; Comparison A [201] to other farmers in SBEP regions)* 11.37 20.00 75.9 * High comparison A group outcome may be driven by Pelagonia’s normally higher-than-national-average performance, spill-over of SBEP agricultural protocols, or higher-than-national-participation in the government’s 50/50 drip irrigation program in SBEP regions. While the results of these trials were relatively close for corn grain, they differ between SBEP and MSI for corn silage on both a pre/post and with/without basis. None of the values in the second table are out of range, however. For 2016, it shows that the national average t/ha for corn silage for individual holdings was 39.7, yet for that same year it reported Bitola at 152.8 t/ha, while Prelip was 5.3, Karbintsi was 5.5, and Novo Selo was 6.1 t/ha for corn silage. Post-Adoption Production Volumes Looking beyond SBEP and MSI research findings about the impact of drip irrigation, SBEP quarterly reports use the number of hectares for which it provided drip irrigation systems to estimate the production levels of these farmers over time, to which they add in extrapolations based on estimates of what would occur if larger-scale farmer trainees expanded the share of their farms planted with drip irrigation, or if the number of these adopters grew. A white paper developed by the SBEP team to help explain the effects of drip irrigation to policymakers discusses the effects of various assumptions on future production volumes. During the life of the project, however, SBEP did not systematically report on the harvests of the farmers it trained or additional hectares they planted with drip irrigation expansions they acquired in other ways. Neither of these metrics were performance indicators for the Grow More Corn Initiative. In evaluation interviews, as Sub-Study 2 elaborates, MSI asked farmers about additional hectares planted and queried SBEP’s supplier Magan-Mak, as well as representatives of other stakeholder organizations, including the Ministry of Agriculture. Through these channels, the team found no evidence of a high rate of adoption of the combination of drip technology and related agricultural protocols relative to the number of farmers interviewed who seemed barely aware of its availability. Based on all the above, MSI’s central finding on production volume for the Grow More Corn Initiative is a confirmation of USAID and SBEP’s intervention “theory of change,” its early “proof of concept” test, and testimonial remarks from farmers and other partners: The application of drip irrigation systems and associated agricultural protocols would dramatically increase corn production where applied, raising prospects for regional development and rural population retention in Macedonia. With respect to goals for future corn production using drip irrigation, MSI’s findings also support the proposition that this initiative could potentially eliminate the need for corn imports, if the technology’s adoption expands along the intended trajectory of first SBEP and, more recently, the Ministry of Agriculture. Evidence demonstrating rapid adoption was not available at the time this study was completed. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 16 To understand corn production on a regional basis compared to research data on SBEP beneficiaries, the evaluation team drew on Macedonia’s State Statistical Office data. The years covered by Table 4 match the period when SBEP was rolling out the Grow More Corn Initiative. In all, SBEP distributed 281 drip irrigation systems with agricultural protocols to farmers in Pelagonia, Polog, Vardar and Northeast, with farmers in Pelagonia receiving 45 percent of these systems. SBEP also worked with the Ministry of Agriculture during these years to facilitate farmer participation by helping them arrange loans to cover their 50 percent. TABLE 4. DRIP IRRIGATION SYSTEMS DISTRIBUTED TO FARMERS Year Pelagonia Polog Vardar Northeast Other Total 2013 27 14 41 2014 35 15 15 15 80 2015 65 10 50 32 157 Total 127 39 65 47 3 281 Percentage 45% 14% 23% 17% 1% At the national level, MSI examined trends in corn grain and corn silage production separately, reaching below this to regions to explain changes over a nearly 10-year period. • Corn grain production has remained fairly stable from 2007 to 2015. At the regional level, however, there have been important changes. In some regions, corn grain production declined. In contrast, production rose from 12,000 tons to 21,000 in Pelagonia over that period, with 5,000 of that increase accruing in the last two years. Production increases in the range of 10,000 tons over this period were also noted for Skopje and Southwest regions, with about half of these increases coming in the past three years. • Production of corn silage spiked significantly in 2014 and 2015. Overall production figures rose from 63,000 tons in 2007 to 120,000 tons in 2015, an increase of 57,000 tons. The primary source of this improvement came from Pelagonia, which increased its production of corn silage by 72,000 tons over that period. Figure 4 shows the change in corn output MSI calculated by region for 2012 to 2016, using State Statistical Office data and the same t/ha basis as the SBEP and MSI research findings above. Corn production on a ton/hectare basis grew for Pelagonia, Northwest and Vardar among the SBEP regions, but it declined in Polog. MSI’s team checked this finding municipality by municipality in the government’s data to make sure that the drop had not come from one location in unusual circumstances. That was not the case; the decline depicted for the region was present in most Polog municipalities that grew corn. FIGURE 4. CHANGE IN CORN GRAIN PRODUCTION (TON/HECTARE) BY REGION, 2012–2016 Source: State Statistical Office data; MSI analysis 9,234 5,798 4,278 3,162 2,297 1,505 -1,298 Pelagonia Northeast Southwest Vardar East Southeast Polog EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 17 Tourism Numbers for 2012–2016 SBEP’s initiative in this field included collaboration with project partner ATTA and USAID’s Regional Economic Growth Project, which has promoted cross-border tourism in the region and collaborated with SBEP and Macedonian stakeholders on specific events and tours. The initiative has made considerable headway in generating excitement about adventure travel domestically and overseas and helped a number of facilities acquire the financing and experience to professionally operate a variety of types of adventure operations. The initiative has gained a foothold in the Ministry of Tourism’s planning process, which itself was moving toward a long-term relationship with ATTA during the evaluation’s fieldwork period in late 2016. Other stakeholders, including the World Bank, are also involved. As suggested, it is relatively easy to see SBEP’s model for increasing jobs in regions of Macedonia through its Adventure Travel Initiative. Neither SBEP nor ATTA seem to have started trying to monitor tourist arrivals or nights stayed for the adventure travel segment of the population. The State Statistical Office, however, reports time series data on both these metrics for municipalities as well as regions. To learn what could be known with existing data on all tourist arrivals for the SBEP years, MSI created Figure 5. This figure reports on the change in tourist arrivals by region using State Statistical Office data and shows that two regions where SBEP worked were at the low end of improvement. The other two, Polog and Pelagonia, lost ground, receiving fewer tourists at the end of SBEP than they had at the start. FIGURE 5. CHANGE IN THE NUMBER OF TOURISTS (2010–2016) Job Creation In the SBEP Performance Monitoring Plan, the top performance indicator was “Number of new jobs created as a result of SBEP linked activities.” The initial project-wide target for this indicator was 5,000 new jobs, but that target was reduced to 2,000 in 2015. This was a recognition that the project was heavily focused on developing regional networks of public and private sector actors and engaging them in efforts to improve the economic development focus on municipal planning processes. It also sought to form 10 local action groups (LAGs) to position these regions for access to EU regional development funding as Macedonia moved further along the path to accession. Sub-elements of SBEP’s job creation performance indicator focused on jobs for women and youth, as well as what the monitoring plan called “jobs retained.” In its final report, in April 2017, SBEP indicated that its target had been slightly exceeded, with 1,673 new jobs and 333 “retained jobs.” Of the new jobs, SBEP reported that 29.4 percent were taken up by women and 19.1 percent were taken up by youth. Table 5 shows how SBEP allocated the creation of new jobs across the various program initiatives, most of which were examined through this evaluation. 70,872 28,114 11,703 10,197 5,328 -2,566 -13,699 Southwest Southeast East Vardar Northeast Polog Pelagonia EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 18 TABLE 5. SBEP’S ATTRIBUTION OF NEW JOBS TO PROJECT INTERVENTIONS SBEP Initiative Number of New Jobs SBEP Attributed to the Initiative Grown More Corn Initiative 644 Wild-Gathered Products Initiative 616 Adventure Travel Initiative 174 Light Manufacturing Initiative 53 Women in Business Initiative 119 LEADER/LAG Initiative 10 Other initiatives and activities 57 Of these, the evaluation did not independently seek information relevant to job creation results, namely SBEP’s work with the LEADER/LAG regional development initiative. This was not a job creation activity, though it aimed to catalyze job growth indirectly. The evaluation team also did not look at job creation in the “other activities” category. Legal Standing and Jobs in the Wild-Gathered Products Initiative SBEP’s final report attributes 616 jobs to SBEP’s Wild-Gathered Products Initiative, but gatherers are not employees of the collection centers that SBEP helped construct. Rather, they are free agents who, pursuant to relatively new government regulations, must have a commercial collection certificate to legally sell what they gather in the forest. At Mavrovo National Park, evaluation team members were told that these cost 1,500 denars per season for commercial gathering in the park and 400 denars for personal use. Under the Wild-Gathered Products Initiative, SBEP trained 1,770 collectors in sound forest product management and collection practices, but did not issue them certificates that made it legal for them to gather and sell forest products commercially. SBEP’s training partner, ProBio, explained to the evaluation team that collectors were referred to another organization that was officially licensed to issue collection certificates. Discussion with Mavrovo National Park indicated that the park also had the authority to issue collection certificates. In the SBEP PMP, “numbers of gatherers trained” was a performance indicator, but “number of trained gatherers that took the next step and obtained an annual commercial collection certificate” was not. Job Creation in the Grow More Corn, Women and Business and Adventure Travel Initiatives Through surveys in 2016, MSI acquired varying levels of information about job creation from sources that included: • 106 of 281 farmers who received drip irrigation systems and protocols under Grow More Corn and were willing to be interviewed, • 12 of 25 SBEP assistance recipients under Women in Business, and • 37 tour operators, travel agencies and associations, and sports-related firms associated with Adventure Travel, of a list of 49 names that SBEP provided. From a survey question about whether these entities had increased their number of employees, stayed the same, or declined over the past two to five years, the percentages indicating increased employment or new jobs created was modest for Grow More Corn farmers and Women in Business beneficiaries. Organizations that SBEP worked with on Adventure Travel indicated the highest rate of job growth. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 19 Has the number of people working at this farm/firm increased, stayed the same, or declined the past two to five years? Group Percentage of Respondents Increased Stayed the Same Declined Corn – Treatment-Beneficiary Group (106) 6.6 86.8 5/7 Corn – Comparison – Same Regions (184) 7.1 78.3 10.9 Corn- Comparison – Other Regions (114) 11.4 71.9 5.3 Women in Business – Beneficiaries (12) 8.3 75.0 16.7 Adventure Travel Firms & Associations (21) 28.6 66.7 4.8 When MSI compared these reported percentages of entities that increased their workforces to SBEP’s performance indicator tracking totals, the two did not seem to agree. That is, applying the percentage of interviewed farmers who said they increased their workforce (106) to the total number of Grow More Corn beneficiary farmers (218) and dividing the resulting number into SBEP’s performance tracking number suggests that each farm that added workers would have had to have added a much larger number of workers than they seemingly could have absorbed. The same applies for Women in Business, although MSI was aware of a few stories in SBEP quarterly reports that indicted specific numbers of jobs created. In one case, shown in the “Job Creation in Dressmaking” text box, a dressmaker added five jobs. SBEP’s total number of new jobs reported among these woman-owned firms suggests that all 25 added the same number of new jobs, although survey data indicated that only 8.3 percent increased their number of employees. For corn farmers who received SBEP assistance (treatment group farmers), MSI collected additional data on employment changes, a question triggered when a farmer who reported a change in number of employees also reported adding or losing more than two employees in the two years before the 2016 survey, as shown below. If the number of people employed by this farm changed (increased/decreased) over the past three to five years (since 2012), please describe how the number of farm employees has changed over this period. 2012 2013 2014 2015 2016 Total Farm Employees By Year This protocol was triggered 13 times in the course of 106 treatment farmer interviews in 2016, giving the team data it could use to determine how net employment changed in any firm that said the size of its workforce had changed. For those 13 cases, the table subtracts the reported number of farm employees in 2016 from the number for 2012. Net employment — the change over the SBEP period — averaged to a net increase of .61 employees, or just over half of a full-time position. Job Creation in Dressmaking Princesha, an artisanal, woman-owned tailoring shop in Gostivar, produces wedding gowns and traditional dresses. SBEP gave the business two sewing machines and professional ironing equipment. In turn, the owner committed to train 15 women and employed five of them during the high season. The business reportedly doubled its production of wedding dresses, from eight to 16 per day. SBEP Quarterly Report EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 20 TABLE 6. FARM EMPLOYMENT CHANGES Group Average Number of People Working on Farm (2016) Average Number of People Working on Farm (2012) Net Employment (Average Change in Number of People Working on Farm) Corn – Treatment-Beneficiary Group (13 responses) 5.69 5.08 .61 Corn – Comparison – Same Regions (33 responses) 4.33 4.30 .03 Corn – Comparison – Other Regions (19 responses) 4.37 3.37 1.0 This finding is important not simply because it contrasts sharply with numbers of new jobs reported for SBEP, but also because it points out the dangers of using “new jobs created” to estimate the employment effects of a USAID project. Taken alone, “new jobs created” ignores job losses in assisted establishments. “Net employment” (average change in the number of people working on farm), on the other hand, considers both new jobs created and job losses over a given period. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 21 SBEP INITIATIVES IN POLOG EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 22 Evaluation Question 3 What have been the measurable effects of SBEP initiatives that operationalize its intervention model on economic growth, the emergence of new enterprises, unemployment and other broad economic development measures in municipalities and regions where SBEP was active? Fieldwork for this evaluation was completed amid Macedonia’s late November 2016 pre-election season, and interviews with officials who had been too busy to meet with the evaluation team wrapped up in January 2017, with a report draft delivered to USAID that month. Since then, considerable additional data for the SBEP period (2012–2016) has emerged, including 2016 corn harvest results. This report uses these data as much as possible, particularly with respect to EQ 4, to provide USAID with a more comprehensive view of the conditions in the four regions SBEP focused on: Polog and Pelagonia from 2012; Northwest Region beginning in early 2014; and expansion to Vardar a year later, made possible through a joint funding arrangement with the Embassy of Switzerland. Regional Development Context in Macedonia and SBEP’s Intervention Model Framed under USAID/Macedonia’s Strategic Plan (2011–2015), SBEP focused on the Mission’s Assistance Objective 3: Increased Job-Creating Private Sector Growth in Targeted Sectors, which already reflected a strong belief that with local public-private leadership with USAID playing a facilitating role, a focus on small- and medium-scale farms and firms could achieve this. Targeting by regions and conceptualizing SBEP’s aims as residing at that level reflected a way out of the chronic unemployment and rural outmigration the government began to tackle through the 2007 Law on Balanced Development. In its design, SBEP’s market-driven, supply-chain-based strategy blended well into this strategic environment and complemented other donors’ investments in Macedonia. GDP and Per Capita GDP Macedonia’s GDP, which moved steadily upward from 2002 and averaged a 3.26 percent growth rate from 2004 through 2014, faltered somewhat between 2014 and 2017 as the country weathered political upheaval until elections moved forward in December 2016 and a new government formed in June 2017. The economy is viewed as moving forward again, though for 2016 its growth rate was 2.4 percent. The World Bank, which initially predicted a higher 2017 performance, stepped back to more conservative expectations. Despite the effects of its political situation on growth, Macedonia will still be well positioned amid its neighboring countries, which continue to experience an improved economic performance. FIGURE 6. GDP, MACEDONIA COMPARED TO NEIGHBORING COUNTRIES The World Bank EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 23 Economy-wide, the regions have experienced positive per capita growth, but with important differences. Using State Statistical Office data, MSI calculated the difference between GDP per capita in 2011, at the start of USAID/Macedonia’s Strategic Planning period (2011–2015), and GDP figures for 2016, at the end of the SBEP period. Vardar, which became a SBEP-assisted region in 2015, made the most gains at this level among program regions, but two non-SBEP regions did better. Northeast, which started receiving SBEP assistance in 2014, barely changed during the period. Given its scale and duration, effects on regional GDP per capita exceeded expectations for this program. Figure 7 shows the range of the economic situation in which SBEP was situated. FIGURE 7. CHANGE IN GDP BY REGION (2012–2015), IN MILLIONS OF DENARS State Statistical Office data; MSI analysis Unemployment As with GDP per capita, changes in regions’ unemployment levels exceeded SBEP’s expectations, but net employment, which the number of new jobs and job losses affects, was SBEP’s directly target, even if not at a scale that would single-handedly change regional employment statistics. SBEP was expected to catalyze job growth through energy generated by successful new job-creation approaches, including by adopting its promoted technologies and market opportunity focus. Over the years SBEP was active, unemployment continued to decline following established trends, as data from Macedonia’s State Statistical Office in Figure 8 shows. FIGURE 8. UNEMPLOYMENT DECLINE CONTINUES 119 1,400 4,929 6,377 8,362 9,166 11,115 Northeast Pelagonia Polog East Vardar Southwest Southeast Macedonia’s unemployment rate dropped to 22.10 percent in September 2017, from the previously reported number of 22.60 percent in June 2017. Macedonia’s unemployment rate is updated quarterly, available from March 2004 to September 2017, with an average rate of 31.70 percent. The data reached an all-time high of 38.70 percent in March 2005 and a record low of 22.10 percent in September 2017. — Macedonia State Statistical Office EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 24 The change in unemployment differed between regions from 2012–2016. Figure 9 shows that, based on the change in unemployment each region experienced, the four regions where SBEP worked reduced their unemployment rates further than regions that did not participate in the program did. SBEP reported that it helped create roughly 2,000 new jobs, which does not explain these differences, nor did it substantially affect them. The project’s focus on regional sources of economic energy and the importance of private-public sector dialog and collaborative planning at the municipal level, as well as in new LAGs the project helped form, may help the regions where it worked continue to reduce unemployment going forward. FIGURE 9. CHANGE IN UNEMPLOYMENT LEVELS (2001–2016), BY REGION At the enterprise level, the evaluation team did not find strong trends. The total number of active enterprises in the regions where SBEP worked changed little over the period, and new business entities did not form in large numbers Tourism, a sector where SBEP was active, grew over this period, but as Figure 5 in the previous section shows, Polog and Pelagonia both lost ground compared to non-SBEP regions. Similarly, with corn production, a major focus of SBEP’s efforts, Figure 4 shows that while three SBEP regions made gains, corn production in Polog declined over the SBEP period. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 25 SBEP INTERVENTIONS VARDAR REGION EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 26 Evaluation Question 4 What evidence of SBEP project “spill-over” effects or spontaneous replication of SBEP initiatives or aspects of the SBEP model, if any, does evidence collected by the evaluation from regions in which SBEP was not active reveal? For this question, the evaluation team looked for evidence that would suggest whether SBEP’s introduced initiatives and interventions would be likely to gain traction and grow through the acquisition of new adopters who take up a technology or practice at their own expense, attracting funding sources that will subsidize their adoption indefinitely or simply provide them with a permanent home. Together, these result in sustained services and outcomes from USAID-funded activities that are meant to function as “seed capital” for longer-term, more widespread development. This section examines each of the SBEP initiatives from this perspective. Grow More Corn Initiative Based on information acquired by the evaluation on the adoption of the type of drip irrigation systems and agricultural practices provided to farmers under Grow More Corn, the number of verified instances of adoption is limited, although the evaluation team heard frequently during data collection about farmers who wanted to expand their SBEP-financed systems or hoped other farmers would do so. • More than two dozen beneficiary farmers stated in interviews that they would like to expand the system they received. Cost was the most frequent reason given for not expanding yet; only about a half-dozen said they had overcome that hurdle. • MSI’s 2016 survey of farmers offers additional evidence of spill-over effects. A diffusion of drip￾irrigation technology exists because farmers from every other region visited SBEP beneficiary farmers to see what they were doing. In Vardar, 91 percent of farmer respondents reported such visits, along with 80 percent in Pelagonia, 43 percent in Polog, and 16 percent in Northeast. Future diffusion of this technology may also occur. • Among farmers who were not Grow More Corn beneficiaries, both within and outside SBEP regions, 25 percent knew about SBEP or the initiative. If SBEP was the main channel through which information about drip irrigation flowed during the project’s life, these results would be self-limiting with respect to uptake. • In addition to the 280 farmers SBEP reportedly provided with free drip irrigation systems, SBEP records show that it worked with 71 farmers to help locate financing for their share of a drip irrigation system that the Ministry of Agriculture co-funded in its 50/50 system. • When the evaluation team asked Magan-Mak, SBEP’s drip irrigation supplier partner, about “spill-over effects” at the end of 2016, the company representative said they could not precisely gauge spread effect because they were not the only firm that sold small-scale drip irrigation systems in Macedonia, adding “We also have competition from Greece.” Nevertheless, Magan￾Mak said from the time it opened in Macedonia in support of SBEP’s project to December 2016, the company’s drip-irrigation systems supported nearly 700 hectares of farmland. SBEP reported supplying Magan-Mak drip irrigation systems to support 280 hectares on more than 300 farms; the firm suggested that the remainder represented the initiative’s “spill-over” effect in Macedonia. Therefore, more than 2.5 times the farms that originally received support had implemented drip-irrigation systems by the end of 2016. • The Ministry of Agriculture allocated additional funding for its subsidy program for drip irrigation in January 2015 and shifted the basis from 50/50 to 20/20 to induce further adoption. The evaluation team did not learn the total number of 50/50 program adopters from the ministry as of the end of 2016, nor has it followed up to learn the number of new adopters since its January 17 announcement of new funding and better terms. This information would help USAID understand EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 27 the extent of adoption after SBEP ended operation. Some double counting may exist between the Magan-Mak estimate of additional systems and reports about the 50/50 system. • Additional indications of possible spill-over, or at least strong interest in the Grow More Corn Initiative, came from SBEP’s partner the Federation of Farmers of Macedonia. The organization told the evaluation team that before they started posting stories about this effort in their online magazine, they had about 2,500 readers. After they started running stories about this initiative, the hit rate jumped to 10,000. In addition, the spill-over effect via media has increased among the Albanian-speaking population, including in Kosovo and Albania. As of the end of SBEP, the Grow More Corn Initiative, including its specific package of agricultural practices and protocols and methods to ensure that all drip irrigation farmers learned them, had no permanent institutional home. By early 2017, the Ministry of Agriculture had expanded its subsidy investment in this program and integrated it into the ministry’s forward plan, including references to training. Further, Magan-Mak appears to be staying in Macedonia, having been deemed “a great success story” at home in Israel. Adventure Travel By the time SBEP terminated, the Adventure Travel Initiative had developed a base of institutional support for the adventure travel sector inside the Ministry of Tourism, which had engaged or was about to partner with ATTA, an association that served as the initiative’s U.S.-based international adventure travel promotion partner under SBEP. Firms working on the initiative had also been introduced to USAID’s REG Project, which promotes this tourism niche market on a regional basis and included Macedonia in its activities. Uptake for this initiative depends on growing demand for Macedonia-based adventure travel opportunities and high-quality service in this competitive market that has high safety standards expectations. In the end, this initiative is expected to be able to stand on its own and thrive to persist and grow. Wild-Gathered Products Initiative By the time SBEP terminated, the Wild-Gathered Products Initiative appeared to be viable in Mavrovo National Park and its main collection center. This is based on regulations issued during the SBEP era that require collectors to obtain commercial certificates to legally sell forest products they gather, as well as a process to license firms that buy from collectors at the park collection center. Gaps between achievements and what appears to be commercially sustainable include the following: • SBEP set out to establish four commercially viable forest product collection centers and train many collectors regarding Macedonia’s new regulations on the types of forest products these collectors sell. One of these centers was Mebi Prima, a private firm SBEP worked with at the start of the project and appeared to still be in business at the end of 2016. Three others were in Mavrovo; SBEP apparently bought equipment for all three. MSI was told that only one or two centers had actually opened. The evaluation team was not able to verify this as their site visit, one of the last pieces of fieldwork, was canceled due to weather. • While park officials said the main collection center is doing well, a gap exists in the supply chain for the products collectors bring in because the park has not been able to identify a buyer for medicinal plants that collectors gather. MSI was told that such a market exists in the EU. • The evaluation team did not learn about other donors who are working to grow this aspect of SBEP’s effort, although its legacy organization, the Association for Innovations in Rural Economies (AIRE), may still be engaged in efforts to make this initiative and its training component fully sustainable, as may ProBio, SBEP’s training partner. These are gaps in the evaluation team’s knowledge of the status of this initiative. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 28 LEADER/LAG This initiative, which benefited from SBEP assistance, is an integral part of a larger effort that is tightly linked to progress toward Macedonia’s accession to the EU, but has potential in residual asset value even if that never occurs. The initiative fits well with the Law on Balanced Development and government’s apparent commitment to continue to pursue its implementation. Expertise to support its continued evolution resides in the Rural Development Network of Macedonia, which was already supporting LAG development in 2013 when the evaluation team conducted its scoping visit. SBEP’s legacy organization, AIRE, says it continues to work in on this EU-backed approach to foster regional development. Women in Business While beneficiaries consistently view this initiative as useful and positive, its history has been tumultuous, starting as an informal group of female entrepreneurship development champions for women-owned businesses in Polog. This group evolved into an advisory board and then a fund, with limited financial backing from private and public-sector sources. SBEP then helped organize a nonprofit home for this program called Foundation Egalite, and helped it acquire a short-term grant from another NGO. Thus, the initiative still nominally existed in December 2016, but its advisory board members said in an interview that the fund had expended its resources and could no longer give out assistance grants to women-owned firms. The evaluation team has not formally followed up on what happened to this initiative, but an attempt in January 2018 revealed defunct phone lines. This could simply mean the initiative has a new home. Light Manufacturing SBEP’s Light Manufacturing Initiative appeared to evolve on two tracks, one apparently more successful than the other, which helps explain its status as the evaluation understood it at the end of 2016. The first track this initiative pursued focused on matchmaking between larger and smaller firms to substitute imported small manufactures for locally made products that could fully meet buyer demands. As Sub-Study 6 explains, while the initiative made a large number of contacts with firms, the matchmaking side developed slowly, only occasionally leading to introductions that resulted in trial orders and follow-on contracts. The evaluation team was not able to independently validate the number of transactions reported for this initiative because of difficulties obtaining lists of pairs of firms from SBEP and the contact information needed to conduct interviews. The initiative’s second — and seemingly more successful — track involved the transformation of the list of firms SBEP acquired into an inventory of local manufacturers, which began to draw interest from Invest Macedonia and the Office of the Deputy Prime Minister for Foreign Investment and Exporting, both of which received requests for the names of Macedonian light manufacturing and possible exporters of such products regularly and came to rely on SBEP for answers. SBEP, with sustainability in mind, arranged for the initiative’s new home in the Macedonian Association of Metal and Electrical Industry (MAMEI), a local association it worked with along USAID and others to organize a light manufacturing expo in September 2016, a finale for this effort as a SBEP creation. MSI is unaware of its exact status within MAMEI, or of MAMEI’s own status; one interview with government representatives indicated some concern about MAMEI’s financial footing, but did not include an offer to invest in a way that would guarantee continuation of the matchmaking side of this initiative, or its inventory process over the long term. Arguing for some sustainable form emerging is the continued need described by government foreign direct investment (FDI) and export officials for information about local light manufacturing suppliers. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 29 SBEP INTERVENTIONS NORTHEAST REGION EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 30 Conclusions About a year ago, the evaluation team reached conclusions about the effectiveness of the USAID/Macedonia SBEP program in catalyzing regional economic development, including: 1. SBEP was highly effective at opening citizens’ eyes to economic opportunities around them, inspiring them to try new approaches and aiming high when it came to what they could accomplish. On farms, among municipal leaders, in regional forums, and in ministry and donor circles, the project was well regarded and viewed as adding value through its initiatives, as well as in supporting local economic development processes. 2. Among country partner observers of SBEP in Regional Development Centers and municipal governments, SBEP’s assistance led to the formation of LAGs across four regions. This was often considered the most significant change the project introduced, with drip irrigation a close second. 3. USAID’s decision to rely on bottom-up suggestions on its project’s economic development priorities was much appreciated by mayors, local economic development office representatives, and municipal private sector representatives, and viewed as right for the country’s stage of economic development. 4. The scale of SBEP initiatives may not have been large or gained traction early enough to achieve the critical mass of awareness and evidence needed to spark continued growth and diffusion within or beyond the project’s focus regions. 5. SBEP’s influence on regional development trends may be more detectable and significant in time. Other draft report conclusions have become consolidated into three additional conclusions that emerged somewhat differently, based on the team’s further examination of why some project outcomes were different from what the project anticipated. Analytically, the team’s return to the theory of change, developed for this project in 2013 during the evaluation’s scoping visit, provided new insights that modified and led the team to elaborate further on several conclusions. FIGURE 2. SBEP MULTI-INITIATIVE THEORY OF CHANGE The immediate result of reintroducing the study’s theory of change was reorganizing existing conclusions into three clusters. One of these taps into what blocks and promotes the adoption of new approaches in Macedonia’s context; the second focuses on improving the balance in performance monitoring between reporting for accountability and picking action-forcing indicators that can help USAID; and the third suggests characteristics in SBEP initiatives that were present at the start and made them more or less likely to be sustained. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 31 Adoption of the “New” Across several SBEP initiatives, adoption of the “new” — whether a type of irrigation system, a flat management structure for LAGs that enables sharing ideas from men, women, and young people via Facebook, or figuring out how to integrate rock climbing walls and paragliding into traditional communities — is key to moving from ideas to a demonstration to broad usage to amazing impact. Yet what the evaluation also found is adoption of the “new” didn’t always move quickly. A slower rate of expansion of drip irrigation systems and less new adoption than the project team might have expected is a prime example. As a result, success on single hectare plots had not demonstrably translated into thousands of adoptions and tons increased production at the municipality and then regional levels. Looking across SBEP initiatives for a different result, the experiences of some of the pre-LAGs SBEP had helped, and for Mavrovo National Park, pointed to possible differences in adoption of the “new.” In one municipality, a type of adventure tourism that had decided to set up was the “new” and some in the community weren’t sure how they felt about that. When the adventure people planned an event, however, the town decided to wrap itself around the “new” by putting on a food festival. The result was a “this changes everting moment” that people remember, and told the story to the evaluators. Since then, everyone has worked together for their collective success and their annual even attracts large numbers of visitors including from other countries. A number of other examples of such awakening moments were conveyed to the evaluation team, about trails half grudgingly built for adventure tourism that have become the pride of a community; and a bike tour of vineyards that turned municipalities that thought they saw things differently into a strong team. In theory of change term diagrams, all these experiences had a “communities in motion” element where adoption occurred when the group got behind something. Even more striking were examples of communities working together to obtain resources they shared communally not in just one municipality but across several, such as mushroom and wild plant drying equipment. The Nobel Laureate, Gunnar Myrdal once said, in his slim volume, Objectivity and Social Science Research, said that we all unknowingly bring the biases of what we grew up assuming to be true to evaluations and research we conduct outside our home environments. Under SBEP, USAID generously gave drip irrigation to farmers sometimes scattered at a distance from each other. Their successes, while lauded by the project, appear from answers to open-ended study questions to have been individual or single￾family triumphs, but may not have energized whole communities to adopt this technology. If community agreement on the desirability of adopting the “new” is central to decision-making in rural Macedonia, what might have happened differently if USAID had selected communities instead of individual farmers and given out batches of 25 drip irrigation systems, placing them on every third or fifth farm in an area you could get around on a bicycle. LAGs could offer an opportunity for further hypotheses testing, and serve as the “community” element for triggering group adoption, if indeed that is a critical factor. Results Monitoring Choices The SBEP Performance Monitoring Plan included 16 indicators, several of which tracked important outcome measures, including jobs created. However, these indicators were not formally disaggregated by SBEP’s interventions, nor were targets set at the intervention level for outcome level metrics. The absence of initiative specific targets limited the evaluation team’s ability to make clear calls as to what worked well and what did not, and may have affected USAID in the same way during implementation. Further, half of the indicators monitored for SBEP were the equivalent of input indicators; they would not likely have sent signals to lead the Mission and the project team to adaptive management decisions that would have improved the likelihood of achieving important regional outcomes that the project could have monitored, even if their achievement lay beyond the SBEP team’s “manageable interest.” EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 32 Two key examples are tourist arrivals and corn production, both of which declined in Polog while SBEP was operational. Similarly, project monitoring did not provide regular information about further adoption of drip irrigations systems or other intermediate results, such as the number and value of contracts that emerged from light industry matchmaking efforts. Sustainability vs. Business Plans Several SBEP initiative entered the post-project period without secure institutional homes or predictable resource flows to continue the work USAID began, regardless of how successful the work was in beneficiary terms. Women in Business and Light Manufacturing seemed to the evaluation team to be the most vulnerable. Notably, while Light Manufacturing has an organizational home, it may not have the resources to replicate the apparently successful expo that SBEP organized in 2016 or make it an annual light manufacturing promotion and networking event. Similarly, for lack of financial underwriting, the Women in Business Initiative seemed to have lost its ability to provide support in the form of equipment to other women-owned firms. Given the resources USAID invests to introduce solutions to important country partner problems, vague sustainability plans may not be sufficient. For economic growth projects that help local businesses succeed, a disciplined business plan might be more likely to result in success in sustaining solid outcomes of well-conceived and well-implemented projects. This problem is not unique to SBEP or a criticism of its management team. Instead, this conclusion suggests that USAID/Macedonia, through its programming documents, can elevate norms, as some other USAID operating units are trying to do and to share what they are learning.1 Recommendations Based on this evaluation’s findings and conclusions, the team recommends that USAID/Macedonia: 1. In future large and complex economic growth projects, improve the balance between monitoring for accountability (at the input level) and monitoring to support adaptive management of economic development projects (at the intermediate result and outcome levels). Distribute indicators across the project’s theory of change, whatever it is, to keep USAID informed about progress on each linked hypothesis. Relying exclusively on USAID standard indicators can skew toward an input-level view of results, or cause USAID to not know whether progress toward all aspects of higher-level results is occurring. Implementing partners may not be responsible for achieving results along the entire theory of change, but can be asked to monitor what is happening with respect to higher-level results to which they, and USAID investments, contribute, including regular updates on national trends for which others collect data. 2. When selecting performance indicators, carefully examine what they tell the Mission – not only about a particular result but also about the likelihood of achieving higher-level results. SBEP data 1 See: https://www.fantaproject.org/research/exit-strategies-ffp. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 33 on corn farms provided an example of this by demonstrating that farms where USAID could record “new jobs created” may also be losing workers, resulting in unchanged total or net employment, in regions where USAID and the host country government’s ultimate aim is to reduce unemployment. This is the type of situation in which multiple indicators further along a results chain may be more useful for the Mission than multiple input-level indicators. 3. For product-based economic development activities, encourage USAID partners to undertake – while USAID funding is in place - actions that would improve the integration of those products into broader supply chains, including those that reach into international markets on both the import and export sides. Simple steps include using international harmonized system codes as an integral part of activity planning and monitoring, not only by USAID partners but also by local entities, such as the collection centers USAID funded under SBEP and Macedonia’s State Statistical Office. 4. For projects in economic growth and other sectors in which long-term and goal-level success depends on the institutionalization of initiatives in government or their continued existence on a business basis, elevate thinking about sustainability for those which must be self-financing to business plans with milestones that can be monitored and self-sufficiency outcomes to be realized before USAID’s support terminates. Milestones for the institutionalization of initiatives in national institutions may require USAID to participate directly in their achievement. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 34 ANNEXES EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 35 ANNEX A: EVALUATION STATEMENT OF WORK SMALL BUSINESS EXPANSION PROJECT (SBEP) IMPACT EVALUATION 1. PURPOSE OF THE IMPACT EVALUATION OF THE SMALL BUSINESS EXPANSION PROJECT The overall purpose of this Impact Evaluation will be to provide baseline information for USAID/Macedonia, collect counterfactual data for comparison during the implementation period, collect final data, and produce a comprehensive evaluation report. The evaluation report should present a set of findings through comparing data from the initial phase of the project with the data in the completion phase thus drawing conclusions for the four-year USAID Small Business Expansion Project (formerly known as Regional Small Business Development Project) and the impact it has had on the economic growth in the selected regions. 2. BACKGROUND ON THE SMALL BUSINESS EXPANSION PROJECT The official unemployment rate in Macedonia stated to be around 32 percent is among the highest in the region, with dominant youth unemployment. USAID’s previous and current work on economic development of the country has yielded increased employment in assisted companies and industries (such as Information and Communication Technology), as well as less rigid labor legislation and improved overall business environment. However, this has not yet translated into a lower unemployment rate. USAID conducted a workforce assessment in March 2009 which revealed a lack of connection between the education system and the market economy, causing a significant skills mismatch between the workforce and business needs. It further concludes that even if the demand and supply of the labor market was perfectly aligned, the opportunities for employment would fall far short of the need, as the private sector is not developed enough to absorb both new entrants and the current unemployed into the labor force. Current government and donor approaches are not generating the investment to create enough jobs to adequately address the problem, much less carry out appropriate action to meet future need. The existing approaches to job creation remain fragmented with responsibilities allocated to organizations and institutions, such as the Employment Services Agency and the Agency for Support of Entrepreneurship, which are neither well-equipped nor appropriately structured to deliver results. Problems addressed by SBEP: In response to these challenges, USAID has designed a four year, $5 million Small Business Expansion Project. The goal of the project is to strengthen private sector capacity to drive regional economic development and job creation. This goal will be achieved through the following: a) Engaging business leaders, local government and other institutions to identify market opportunities for growth and job creation; b) Upgrading the capacity of micro, small and medium enterprises to respond to new market opportunities and grow; c) Strengthening the capacity of business service providers to better identify and serve the needs of the MSMEs; d) Building the necessary workforce capacity to support the growth of the MSMEs; and e) Encouraging women and youth participation in business. During the assessment and design phase of this project, the team concluded that to create jobs in large numbers, a new approach is required that would: EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 36 • Adopt a regional approach to economic development (the term regional refers to geographic regions within Macedonia); • Harness the power and commitment of local business leaders and local officials to help design business solutions to create new jobs; • Re-orient the business support framework towards market-led MSME growth; • Link start-ups coming out of entrepreneurship training programs and Government’s active labor measures with real market opportunities and support their growth; • Leverage and channel public and private sector investment for local economic development; and • Create a business development model that is appropriate for regional replication within Macedonia. The project will target the new and existing micro, small and medium companies with significant growth potential. Micro-entrepreneurs whose sole objective is to provide their livelihood and do not have growth aspirations will not be the focus of this project. A regional approach would offer several distinct advantages over other strategies, such as taking a sector approach. First, it would define a specific territory for enterprise development that would facilitate substantial involvement of local business and government leaders who instinctively like to focus on concrete opportunities. The business community would be challenged to play a more active role in defining economic development opportunities and supporting implementation of identified initiatives, along with their government and NGO counterparts. Second it would encourage multiple approaches to economic growth and job creation, involving multiple organizations and employing all the tools available for enterprise development: value chain development, clusters and networks to reduce costs and foster cooperation to compete, business services market development, and varieties of mentoring and training interventions to upgrade know-how and skills within a defined regional context linked to real market opportunities. Third it would target the interventions and increase the potential job creation impact of USAID’s limited available resources. Lastly the regional approach would complement and support the decentralization efforts and the EU accession agenda for regional economic development. Regional models for MSME development and job creation developed and tested in several regions can then be rolled out in other Macedonia regions in line with Europe’s new region-based development perspective. The regional approach will also allow for more in-depth assistance targeted to different ethnic groups (especially ethnic Albanians) which are traditionally disproportionally represented when the assistance is provided nation-wide. Objective 1: Regional market opportunities for MSME development identified and pursued The core idea underlying the new Small Business Expansion project is to identify and pursue regional business initiatives leading toward job creation, based on identified market opportunities. These market opportunities will be identified and mapped in each operational region with active participation of the business community and municipal officials. Once opportunities are identified and prioritized for their employment creation potential as well as their ability to give relatively quick results, the project will develop action plans in conjunction with business leaders and interested entrepreneurs. The regional entrepreneurs (both aspiring and existing) will be mobilized to take advantage of these opportunities while support will be provided to help them succeed, such as the services of local business service providers and business connections they need, leading to the creation of a mutually self-reinforcing co-development of MSMEs, business service providers and business and municipal leaders centered on realistic market opportunities. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 37 Illustrative Activities: • Establishing an Opportunity Taskforce in each region that will engage the public and private sector in planning regional economic development • Mapping regional job-creating opportunities (focus on opportunities tailored to women and youth) Action plans developed and implemented (innovation, technology, legal reform, market linkages, value chains, workforce development, etc.) • Establishing Performance Measurement systems in the local institutions Illustrative Results for Objective 1: • Number of business leaders and public officials devoting substantial time and energy to identifying opportunities for local economic development • Number of initiatives or % of the items in the developed action plans implemented by the project and by the other stakeholders • Value of public and private investment in economic growth of the regions. Objective 2: Assisted MSMEs engage in identified market opportunities The Small Business Expansion project will employ the large pool of available enterprise development tools (value chain development, cluster strategies, development of business services, etc.) to link the identified opportunities to interested enterprises—medium, small, micro—as the opportunities will vary greatly in size and complexity. This may include activities such as forging cooperation between microenterprises to reduce costs, helping small businesses set up new points of sale in untapped areas, and helping medium-sized lead firms develop their supply chain by upgrading small producers. Every intervention will be based on specific and real market opportunities. Illustrative Activities: • Mobilize interested MSMEs and link with identified market opportunities • Facilitate provision of needed services and technical assistance to MSMEs (such as market analysis, building value chains, access to technology, ICT & Internet use, access to finance, training, collaborative strategies, franchising) • Encourage lead firms and suppliers to provide embedded business services to MSME clients • Provide follow-on support to start-ups coming out of other government and donor projects (GoM self-employment programs, business incubators, student graduates) • Provide special support to women and young entrepreneurs Illustrative Results for Objective 2: • Turnover of MSMEs • # of new jobs in companies and linked MSMEs • # of new formal businesses • # of MSMEs linked in value chains • # of MSMEs introducing new technologies Objective 3: Business support organizations provide quality services to MSMEs to pursue the identified market opportunities Rather than becoming a direct service provider, the project will play a proactive facilitative role bringing together different types of service providers, micro-lending institutions, government institutions, business support organizations, schools and other relevant members of the MSME support network. Small Business Expansion project would carry out a number of important catalyzing functions that EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 38 are not currently performed by the existing business support structures or regional economic development institutions. Small Business Expansion project will also upgrade the capacity of business service providers and most importantly the entrepreneurs and companies that must grow and develop to create the needed jobs. Illustrative Activities: • Building capacity of Business Support Organizations to identify and provide needed services to MSMEs • Building relationships between BSOs & MSME clients • Fostering competition and quality on the BSO market • Encourage use by providing a co-funding mechanism for new business services to MSME Illustrative Results for Objective 3: • # of MSME receiving service • # of new business services developed • $ MSMEs invest in services Objective 4: Assisted MSMEs build a workforce necessary to sustain their growth In situations where there are clear workforce development needs, such as upgrading the skills of new and existing workers needed for companies to take advantage of new opportunities, the Small Business Expansion project will facilitate appropriate technical, soft skills or business management training for employees and /or managers of assisted enterprises. On the long run, the Small Business Expansion project will demonstrate how the private sector should communicate with the formal education system to improve the quality of the workforce. Illustrative Activities: • Identify skill needs of the assisted enterprises • Link MSMEs with capable training providers • Encourage lead firms and suppliers to integrate training in their relationship with MSMEs • Facilitate provision of technical training • Improve management capacity in MSMEs • Long term: Communicate skill gaps with the education system Illustrative Results for Objective 4: • Number of employees and managers trained in assisted MSMEs • Number of MSMEs with staff development plans • Number of new tailored training courses offered to MSMEs • $ value MSMEs invest (pay for) in staff training • # of requests for curriculum modification and skill needs communicated by the private sector through a public private dialogue mechanism Objective 5: Women and Youth in Enterprise The project will implement a special component aimed at fostering women and youth entrepreneurship and employment. Issues related to employment and economic opportunities are far and away the most frequently cited gender inequality facing Macedonian women. Women are also more likely than men to work in the informal economy. Youth (age 15 – 24) are also disproportionately unemployed compared to other age groups. The project will proactively seek market opportunities where women and youth are more likely to be engaged both as employees and entrepreneurs and prosper economically. To achieve EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 39 this, the Opportunity Taskforce responsible for identifying market opportunities and creating strategies for developing small businesses will have to include at least 25% female members and some youth representatives that will be able to better represent the targeted population. Female business leaders and successful young entrepreneurs will be motivated through the Opportunity Taskforce to identify obstacles to women and youth entrepreneurship, point out suitable areas for engagement of women and youth, and lead by setting positive examples of successful women and youth entrepreneurs and managers. The project will target specific value chains for development (identified in the opportunity mapping) that can employ women and youth such as the agribusiness and services sectors. Certain activities in IR 2 and IR 4 will be specifically tailored to stimulate female and youth participation in business. Illustrative Results for Objective 5: • Number of women and youth trained in technical and business skills • % of new job openings in assisted MSMEs filled by women and youth • Number of women and young entrepreneurs supported to open and grow their own business End of project results: • Developed and demonstrated an effective and replicable model for business-led regional enterprise development in three regions of Macedonia; • Number of new jobs (25% for women; 30% for youth; 25% for ethnic minorities); For more details, attached is the latest quarterly report which gives an insight into ongoing pilot activities of the project. 3. TASKS FOR THE IMPACT EVALUATION OF SMALL BUSINESS EXPANSION PROJECT The contractor will employ mixed methods, including quasi-experimental impact evaluation design, to evaluate elements of the Small Business Expansion Project. In particular, the evaluation will focus on examining the effectiveness of Small Business Expansion Project in mobilizing public private dialogue and partnerships boosting local economic development, stimulating growth of small businesses and job creation by: a) Engaging business leaders, local government and other institutions to identify market opportunities for growth and job creation; b) Upgrading the capacity of micro, small and medium enterprises to respond to new market opportunities and grow; c) Strengthening the capacity of business service providers to better identify and serve the needs of the MSMEs; d) Building the necessary workforce capacity to support the growth of the MSMEs; and e) Encouraging women and youth participation in business. The specific tasks for this evaluation are: a) Conduct a baseline data collection of the Small Business Expansion Project to measure whether or not it is achieving its primary objectives: 1) Key public and private stakeholders are actively engaged in identifying and pursuing regional market opportunities for MSME development; 2) MSMEs engage in identified market opportunities, 3) Business Support Organizations provide quality services to MSMEs to pursue identified market opportunities, 4) Assisted MSMEs build a workforce necessary to sustain their growth, and 5) women and youth are more engaged in enterprise. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 40 b) Baseline data to be re-constructed for the regions where SBEP has already started implementing activities and collect counterfactual data from regions where the project does not work (the selection of those regions will be based on the characteristics of the regions where SBEP is present with its activities). c) Examine the data collected close to the end of the project and compare with the data collected under 1. and 2. in a comprehensive final evaluation report that would show USAID the impact that the project had in achieving the objectives listed above. 4. EVALUATION METHODOLOGY – IMPACT EVALUATION SBEP The contractor will utilize a variety of data collection and analysis methods determined in part by data collection challenges including cost, time, and issues of validity and reliability. Impact Evaluation: The contractor is required to conduct an impact evaluation of SBEP activities aimed at creating a replicable business-led model for regional economic development, supporting the growth of small enterprises and creating new jobs in the selected regions. Through the review of project agreements, work-plans and reports, and from interviews with USAID and partner staff (implementing partner and regional counterparts), the contractor will determine the project outcomes to be evaluated and associated outcome indicators to be measured. The contractor shall then design a quasi￾experimental approach to measure the outcome indicators. For the evaluation of the SBEP, the contractor will contract for surveys in two to three regions (the exact number to be determined during the design), and in two to three comparison/control regions. (The contractor should identify the means for selecting the comparison units such that they are as similar as possible to the target units). Sample size will likely be in the range comparable to the units of measurement used for internal Monitoring and Evaluation of SBEP. The contractor is encouraged to utilize the most rigorous quasi-experimental design(s) appropriate for this task. Evaluation Questions: This Statement of Work provides tentative questions subject to revision after discussions between the contractor and the USAID team. a. Are private sector representatives actively engaged in a dialogue and partnership with the public sector on the topic of local and regional economic development? • Are the strategies/policies/interventions developed through this kind of public-private dialogue more realistic and responsive to the needs of the region? • Are they more likely to leverage public, private and donor funding and be implemented in a reasonable timeframe? • How sustainable is this public/private relationship beyond the USAID project? b. Are assisted micro and small businesses really experiencing growth as a result of the project activities? • What impact did the SBEP project have on growth and employment generation among small enterprises in the targeted regions? The intended analysis shall use the following criteria for analysis: i. Growth of private sector firms in the form of increased sales ii. Private sector investment iii. Employment generation (formal) iv. Increase in income of formal or informal employees v. New business startups The analysis shall provide a breakdown by assisted sectors and regions. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 41 • Is the growth of the assisted business translating into new formal jobs (as opposed to informal) and increased income of the employees (including individual farmers and seasonal workers)? • Is the growth of these assisted businesses significantly larger than of other businesses in the region that have not benefited from USAID assistance? • Is the individual growth of assisted businesses and their number large enough to have some impact on the overall economic development of the region, as demonstrated through formal growth indicators (such as regional GDP, employment levels, poverty levels)? • What factors affected the performance and impact of SBEP in each region? What were the major implementation problems or constraints that limited progress at the project level and that limited impact at the enterprise level? c. Have assisted BSOs developed new quality services and workforce development programs that respond to market needs? • How likely are these BSOs to continue to provide the same or improved services and training programs on sustainable (commercial) basis after the project support ends? • Do businesses recognize the benefit of business support services and workforce development and are willing to invest in them in the future? d. Are youth and women better represented in the assisted businesses, than in the other ones? • How effective were the projects’ efforts to include women and youth? • To what extent did SBEP contribute to women and youth employment, income generation, and empowerment? • Has this led to a different public perception on the role of women and youth in the enterprises? • Has this led to a more balanced representation of women and youth in the employed population in the targeted regions? e. How sustainable and replicable are the project interventions? • Did any of the initiated activities and services such as training, technical assistance, business development services, public-private dialogue, technology transfer, or other assistance continue after the end of external support? What actions and plans were put in place to ensure sustainability? • What successful activities/models have been or can be scaled up and replicated in other areas of the country? USAID/Macedonia will provide additional key issues/areas to the contractor at the time of the award. The contractor will meet with USAID/Macedonia and SBEP staff for input, as well as their key public and private stakeholders such as Regional Development Centers, mayors and other municipal officials, private sector champions (lead firms), micro and small businesses, business service providers, representatives from key government ministries (Ministry of Economy, Ministry of Agriculture and Deputy Prime Minister’s cabinet). USAID will provide the contractor with a copy of the project contract, Performance Management Plan, Annual Work Plans, etc. Outcomes must be linked to specific activities of the implementing partners. Illustrative examples of outcomes to be evaluated include: The evaluation team shall create a sampling methodology based on best practices for statistically representing the population from which the sample is drawn. The contractor shall propose sample sizes, confidence level, and expected confidence interval for the surveys in its proposal, but the final details will be completed by the contractor after the award is made and upon approval from the COR. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 42 The contractor will hire a survey firm to conduct and assist with the design of the survey (if deemed appropriate for the task). The surveys will be supplemented (or substituted) with field observations, key informant interviews and additional data collection on the activities of the project partners. The evaluation team will meet with USAID/Macedonia within the first weeks of the Task Order to obtain more detailed information about program activities and objectives for the purpose of identifying and finalizing impact indicators and the survey and focus group questionnaires. The implementing partners will be expected to provide self-collected monitoring data and other project documentation to the evaluation team. However, it is expected that the Contractor team conducting the impact evaluation will minimize disruption of the regular activities of the SBEP and not cause any delays of program or monitoring and evaluation activities. 5. DELIVERABLES – IMPACT EVALUATION SMALL BUSINESS EXPANSION PROJECT The deliverables for the impact evaluation of SBEP are as follows. All reports shall be submitted in English. a. Final Work Plan for the impact evaluation of the SBEP A Work Plan for the impact evaluation of the Small Business Expansion Project as part of the comprehensive Work Plan for all three evaluations under this Task Order, as required in Section III.A.7.(a). b. Two briefings for USAID at the end of the first and second weeks after the start of the evaluation effort on the field including written materials for the impact evaluation of the SBEP c. Evaluation Design (including questionnaires) for the impact evaluation of the SBEP In accordance with the milestone schedule indicated in Part I, Section D, the lead evaluator will present to the COR a detailed evaluation design matrix, including the key question(s), and for each question the methods used to address it, sources of information, the type of evaluation design, data analysis plan, and draft questionnaires and other data collection instruments, planned sampling methods including size and locations, arrangements for employing local staff for data collection, data analysis plans, and known limitations to the evaluation design. The final design, including all survey questions, requires COR’s approval. d. Baseline Data Report for the impact evaluation of the SBEP The baseline report will focus exclusively on the data from the baseline survey. The report should include data tables with information by key demographic groups and regions. A narrative description of the data is not required. However, the report should include a narrative introduction and description of the methodology. e. Interim Evaluation Report for the impact evaluation of the SBEP The interim evaluation report should focus on the data from the baseline survey, as well as data from field visits. It should include an introduction, background of the local context and the projects being evaluated, the main evaluation questions, the methodology or methodologies, and findings. f. Interim Report on Project Implementation Fidelity Monitoring inclusive of two impact evaluations EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 43 The impact evaluations that Contractor will conduct for USAID/Macedonia will be designed to determine the effects of USAID-funded activities under two separate projects: the Civil Society Project and the Small Business Expansion Project. Determining the effects of USAID activities requires not only a valid counterfactual comparison, but also an adequate degree of certainty that the activities hypothesized as being likely to cause specific effects, or impacts, were actually delivered. Through limited periodic data collection (including observation, key informant interviews, and project data reviews), Contractor will track whether and to what extent the project activities were implemented as well as what modifications, if any, were made to activity delivery plans (i.e., how, to whom, how much). Contractor’s Interim Report on Project Implementation Fidelity Monitoring will document deviations in activity delivery from original plans and examine the implications of those changes for the conclusions the impact evaluations will be able to reach. g. Updated final year data and reporting plan for the impact evaluation of the SBEP h. End-line data report for the impact evaluation of the SBEP i. Oral presentation on the evaluation effort for the impact evaluation of the Civil Society Projects j. Draft Evaluation Report for the impact evaluation of the SBEP The draft evaluation report should include all of the elements of the final report as described below, in draft form. k. Final Evaluation Report for the impact evaluation of the SBEP The evaluation final report should include an executive summary, introduction, background of the local context and the projects being evaluated, the main evaluation questions, the methodology or methodologies, the limitations to the evaluation, findings, conclusions, recommendations (if applicable) and lessons learned (if applicable). It should incorporate analysis of data from all three surveys and the field data collection. The executive summary should be up to 4 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.). The annex to the report shall include: • The Evaluation Scope of Work • Any “statements of differences” regarding significant unresolved difference of opinion by funders, implementers, and/or members of the evaluation team • All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides • All data collected during the evaluation as it is the property of the United States Agency for International Development • Sources of information, properly identified and listed • Disclosure of conflicts of interest forms for of all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest • The evaluation design EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 44 Upon approval of the final content by USAID/Macedonia, the Contractor will be responsible for editing and formatting the final report, with approximately 30 calendar days to do so. The Contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within 30 calendar days of final approval of the formatted report. A final will be submitted to USAID’s COR. l. Turn over all data collected while conducting the evaluations to USAID Macedonia at the end of the evaluation project. 6. Criteria to Ensure the Quality of the Evaluation Report Draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report: • The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. • Evaluation reports shall address all evaluation questions included in the scope of work. • The evaluation report should include the scope of work as an annex. All modificationsto the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by USAID/Macedonia. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical and specific, with defined responsibility for the • Action. 7. OTHER REQUIREMENTS FOR THE IMPACT EVALUATION OF THE SBEP The Evaluation team shall demonstrate familiarity with USAID’s Human Subject Protection Policy and USAID’s Evaluation Policy (http://www.usaid.gov/evaluation). The Evaluation team shall provide adequate training for its survey staff on survey methodology, USAID’s survey regulations, other relevant regulations, and data collection plan. Safeguarding the rights and welfare of human subjects involved in the survey research supported by USAID is the responsibility of the Contractor. USAID has adopted the Common Federal Policy for the Protection of Human Subjects, Part 225 of Title 22 of the Code of Federal Regulations (http://www.usaid.govb/policy/ads/200/200mbe.pdf). Recipient organizations must familiarize themselves with the USAID policy and provide “assurance” that they will follow and abide by the procedures of the Policy. All records from the evaluation (e.g., interview transcripts or summaries) must be provided to the COR. All quantitative data collected by the evaluation team must be provided in an electronic file in easily readable format agreed upon with the COR. The data should be organized and fully documented for use EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 45 by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 46 ANNEX B: APPROVED EVALUATION DESIGN REPORT Submitted to USAID on June 7, 2013 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 47 INTRODUCTION AND CONTEXT In early 2013, USAID/Macedonia identified three projects for evaluation pursuant to USAID Evaluation Policy and the Automated Directives System (ADS) chapter 203 requirement that USAID operating units evaluate projects with budgets that exceed the average for their unit. The three projects to be evaluated are: • Youth Employability Skills (YES) Network Project, which seeks to improve the interface between business needs for skilled labor and the educational and employment programs that help to prepare Macedonian youth for the work force; • Civil Society Project (CSP), for stimulating citizen activism through training programs for youth and Civil Society Organization leaders as well as through a variety of grant activities; and • Small Business Expansion Project (SBEP), which employs a regional approach in Polog and Pelagonia that is designed to foster farm and small firms integration into regional supply chains. USAID/Macedonia seeks a performance evaluation of the YES Network Project, and impact evaluations of the CSP and SBEP Projects. In May of 2013, under Task Order AID-165-M-13-00001, the Mission contracted Management Systems International (MSI) to conduct all three of these evaluations. This Report presents MSI’s design for the SBEP impact evaluation. Parallel evaluation design reports are being submitted to USAID/Macedonia covering the YES and CSP evaluations. The following matrix situates the overall context of the SBEP project: Implementing Partner Carana Corporation Project Start/End April 2012 – April 2016 Funding Amount (US $) Objective: $5 million Strengthen private sector capacity to drive regional economic development within selected regions of Macedonia. Expected Results and Impact An effective and replicable model for private-sector led regional enterprise development will be developed and demonstrated in three regions in Macedonia; new jobs created (25% for women, 30% for youth, 25% for ethnic minorities). M&E Considerations An impact evaluation will collected baseline data in 2013 and gather endline data in 2015. A Results Framework exists with four components supporting the project goal and a Performance Management Plan (PMP) exists that tracks 16 standard and custom indicators for the project. The impact evaluation design for SBEP described in this document was developed through a scoping visit carried out during the week of May 13-17, 2013 by: Molly Hageboeck, MSI’s Technical Director for this set of three evaluations; Ramon Balestino, Team Leader for the YES evaluation; and Rozalija Vasilevska, Local Evaluation Specialist for the SBEP evaluation. Dr. Steven E. Finkel, MSI’s Impact Evaluation Design Advisor for all three studies, participated virtually in the SBEP design process, which was refined between the scoping visit and a second visit by Ms. Hageboeck to Macedonia from June 3-7, 2013. Mr. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 48 Mario Martinez, MSI’s Team Lead for the SBEP evaluation, will lead the field work for this evaluation commencing towards the end of June 2013. PROJECT BACKGROUND AND CONTEXT USAID/Macedonia has been working with the government and the private sector since 1993 to accelerate economic growth and facilitate Macedonia’s accession into the European Union (EU). Under earlier projects, including its COMPETITIVENESS project from 2008-2012, USAID assisted clusters of export-oriented industries to strengthen their understanding of markets and improve productivity and product quality. The Small Enterprise Expansion Project (SBEP) which started in 2012, builds on earlier work but also involves a shift in focus towards domestic producer supply chain requirements. The project is also regionally focused. It is being implemented in Polog, which is located on the western side of the country and is viewed as lagging behind other regions, and in Pelagonia to the south. The project anticipates expanding to the Northeast Region and may, with other donor support, assists a fourth region that has yet to be determined. USAID’s Results Framework for this project is shown below. SBEP is being undertaken in an evolving economic development environment. Macedonia’s Law for Local Government (2002) called for the devolution of some responsibilities to the municipal level, including responsibilities for local economic development. Pursuant to that law, municipalities have established local economic development units which are for the first time beginning to bring together local stakeholders to develop local economic policies and strategies. Supportive national government economic policy changes, starting in 2008, have also helped interest in economic development efforts at the municipal level. Parallel to these developments, Macedonia’s EU accession process has encouraged government and the private sector to begin preparing to draw on EU funds for local economic development, including funding through the EU LEADER (Links between Actions for Rural Development) programme. LEADER is designed to flow to rural communities through Local Action Groups (LAGs) that have developed strategic plans through a bottom-up process involving a collaboration among local government, the private sector and civil society, for which Macedonia has already undertaken simulation exercises in 21 communities country-wide, with support from the Swiss International Development Agency in 2008- 2011. At the regional level, Macedonia’s Law for Balanced Regional Development (2008) introduced eight statistical regions for statistical and planning purposes. Each of these regions now has its own administrative Regional Development Center, with authority to engage in regional economic develop planning in coordination with municipalities and assistance from the German International Cooperation Agency, GIZ, in 2013-2014. In addition to this range of municipal and regional economic development EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 49 efforts, other stakeholders in Macedonia are also working to promote small and medium enterprise development, including the Macedonian Enterprise Development Foundation (MEDE). EVALUATION PURPOSE USAID is in the process of winding down its multi-sector assistance program to Macedonia. Current expectations include converting the Mission to a Country Office and restructuring some sector programs, including Economic Growth, on a regional basis in 2015, while others would continue for a while longer. In this context, USAID views the impact evaluation of SBEP as an opportunity for rigorously examining the effectiveness of the SBEP project model for private-sector led regional development in light of broad interest in the region and the EU in regional approaches for stimulating economic development. The specific impacts of the SBEP model on small and medium scale enterprises, as well as on women, youth, and ethnic minorities, are also of interest. STRUCTURE OF THE IMPACT EVALUATION MSI’s approach to the SBEP impact evaluation follows USAID’s definition of an impact evaluation as a study that measures the changes in one or a chain of development outcomes “that is attributable to a defined intervention” utilizing an evaluation design based on a clearly articulated model of cause and effect that utilized “a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change” (see USAID Evaluation Policy, January 2011: http://transition.usaid.gov/policy/ads/200/220mab.pdf). To this end, MSI’s initial work with USAID/Macedonia and the SBEP implementing partner focused on the identification of the intervention to be examined and on the measureable development outcomes (or dependent variables) that intervention is expected to affect. These core elements of MSI’s impact evaluation design for SBEP are described below. The evaluation questions to be answered in regards to the project intervention and anticipated development outcomes are provided in the next section of this Report. The Project Intervention: SBEP Initiatives Model The SBEP model for stimulating enterprise expansion and economic growth on a regional basis differs from the approach employed by a number of other USAID small- and medium enterprise development projects worldwide. In many projects of this type, USAID starts (as it has in some export promotion projects) by identifying enterprises that might be able to expand. The enterprises are then provided with training and technical assistance aimed at improving their understanding of markets and enhancing their productivity and product quality. Inverting this model, SBEP starts with larger businesses with well￾defined supply chain needs. It then matches larger business demands to smaller local firms and farms which, with very targeted assistance, can provide those larger firms with the inputs they require. Ideally, this is at a lower cost than those firms currently incur, as a number of the enterprises SBEP is looking to supply currently import products the project seeks to supply locally. The innovative nature of the SBEP intervention makes it a good candidate for a USAID impact evaluation. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 50 In the two regions where it is active, SBEP works primarily at the municipal level. It first establishes contacts with the public sector, normally Mayor or the municipality’s Local Economic Development (LED) officer. Through these initial contacts, it begins a scanning process for economic development opportunities, which expands to include the private sector to identify products in the supply chains of larger industries in a given region that are currently being implemented but could potentially be sourced locally. The project then pursues these “import substitution” opportunities with local suppliers, figuring out product specifications with larger firms that are interested in local sourcing and working on how to meet those specifications by raising local product volume and standards. The project’s “grow more corn” initiative which grew out of the identification of a large egg producing firm’s need for corn is illustrated in Figure 1. This same supply chain, or “import substitution” opportunity model for drawing smaller local firms and farms into regional value chains, is evident in SBEP’s initiatives with products such as industrial tomatoes and light manufacturing. The latter initiative is helping to reduce greenhouse heating costs. Other SBEP initiatives, including its forest products and construction industry initiatives, represent variations on this model – as does its work with women who are trying to expand their home-based businesses. Intended Development Outcomes Project documents and discussions with USAID staff and the project team focus on regional economic development momentum as the outcome toward which SBEP is driving through the public-private sector dialogues it is fostering and the product-specific supply chain initiatives it is pursuing. In impact terms, this model is expected to yield a chain of results, or dependent variables, within specific initiatives and beyond them, along the lines shown in the diagram below. Among the chain of outcomes below, MSI anticipates that the first set of changes, i.e., in the awareness of farmers, small businesses and FIGURE 1: DIAGRAM FOR “GROW MORE CORN” INITIATIVE EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 51 communities of economic opportunities available to them, and thus in their expectations about the future and their willingness to invest in improving product volume and quality, may be the most challenging to capture. FIGURE 2: CHAIN OF OUTCOMES The diagram above identifies three dependent variables, each of which MSI is in the process of further specifying from a measurement perspective. • At the far right, regional economic growth will be operationally defined using statistical data series from the State Statistical Office covering economic indicators, business, entities, agriculture, the labor force, and consumer views. Macedonia’s balance regional development law (2008) suggests what some of those measures will be and MSI has examined a wide variety of existing data series to determine exactly what measures are available for target and non-target regions and municipalities that might be useful in detecting differences at this level, should they emerge. To this end, MSI has begun to construct a database of municipal and regional variables that may be used to measure regional outcomes as well as to identify municipalities in which SBEP is not working that would constitute an appropriate comparison group for the evaluation. Annex A provides an overview of the existing data variables that MSI is organizing into a municipal and regional database for the evaluation. • At the SBEP initiative, MSI’s discussions suggest that while results will be initiative-specific, there will be family resemblances from initiative to initiative, i.e., corn yield and sales levels versus tomatoes and wheat, versus volume of forest products collected, and jobs linked to these initiatives as well as quality standards local producers meet, and a higher percentage of larger firm purchasing from regional sources. MSI has already determined that data series from the State Statistical Office, including agricultural production data series, business enterprise data series, and labor surveys cover many of these variables, which MSI will supplement using project performance monitoring data as well through surveys to be conducted for this evaluation. • To capture information on “communities in motion”, i.e., changes in perceptions and expectations in the business community with respect to general economic tendencies as well as investment and other actions linked to specific initiatives, MSI anticipates that a combination of project performance data and survey research will be required. Previous USAID projects have undertaken business community surveys, which MSI is exploring as a starting point with respect to the kinds of questions and sampling strategies that would be required. Interviews with lead firms in the initiatives, mayors and other economic development actors in regions are also envisioned. MSI is also drawing on The EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 52 PPD Handbook2 to design research tools that will capture information on the role that SBEP’s engagement of public and private sector actors plays in facilitating project outcomes. EVALUATION QUESTIONS As suggested by the chain of outcomes below, the impact evaluation for SBEP will examine the effects of SBEP interventions on each of the outcome variables and compare changes observed in the environments affected by SBEP interventions to comparable municipalities in regions where SBEP was not active. The examinations of hypotheses in the SBEP program theory of change are stated as evaluation questions below. 1. What have been the measurable effects of SBEP initiatives that operationalize its intervention model on enterprise/farm and community mobilization for economic development as measured by perceptions of economic opportunity, expectations about economic returns and growth, and economic-growth oriented investments at the enterprise/farm and community level? 2. What have been the measurable effects of SBEP initiatives that operationalize its intervention model on enterprise/farm, municipal and regional production volume, product quality and jobs linked to supply chain requirements on which SBEP-organized training and technical assistance focused? 3. What have been the measurable effects of SBEP initiatives that operationalize its intervention model on economic growth, the emergence of new enterprises, unemployment and other broad economic development measures in municipalities and regions where SBEP was active? 4. What evidence of SBEP project “spill-over” effects or spontaneous replication of SBEP initiatives or aspects of the SBEP model, if any, does evidence collected by the evaluation from regions in which SBEP was not active reveal? QUASI-EXPERIMENTAL EVALUATION DESIGN To test hypotheses embedded in the SBEP intervention model theory of change, MSI will utilize a quasi￾experimental design that matches assisted municipalities in regions where SBEP is active to unassisted municipalities in regions where SBEP is not active. An experimental design for this evaluation is not possible since municipalities assisted by the SBEP project were not randomly assigned to treatment and non-treatment groups and because the evaluation has been designed roughly a year after the start of the project, making it necessary to reconstruct the baseline situation in treated regions to the degree possible and employ retrospective techniques for constructing the counterfactual. 1. Propensity Score Matching to Construct the Counterfactual To construct as rigorously defined comparison groups as possible, propensity score matching will be used to match assisted municipalities. To support a propensity score matching exercise, MSI has already identified a large number of relevant municipal status variables, as shown in Annex A, for which it has either obtained data dating from 2010 (prior to the start of SBEP) through 2012, or will request municipal-level data for available regional level data series from the State Statistical Office. Data collected from the evaluation’s baseline surveys in 2013 may also factor in the matching process. Additional data will be obtained from these series for 2013 and 2014 to help construct the endline conditions for both the treatment and comparison groups on some variables. 2 Herzberg, Benjamin and Andrew Wright. The PPD Handbook: A Toolkit for Business Environment Reformers. Washington, D.C., The World Bank Group, 2006. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 53 2. Wave Design Elements to Improve the Pre-Intervention Baseline SBEP is currently working in the Polog and Pelagonia statistical regions of Macedonia. Under the project, there are plans for adding a third region and there is also the possibility of buy-in from another donor that would support project expansion into a fourth of the country’s eight regions. Structurally this design can be treated as if regions are receiving assistance on a phased or wave basis. For the two regions in Wave 1, where SBEP is already operational, project presentations make it clear that any measurement effort is likely to find some indications of the “communities in motion” effect the project is expected to yield, for which no baseline data exists or could be accurately reconstructed. However, these Wave 1 regions have not yet experienced the initiative level results the project is anticipating, i.e., a vastly improved yield in corn on project supported pilot hectares, and it should be possible to gather baseline data for both initiative-level results and municipal and regional economic status for the Wave 1 regions. For the next two regions the project will work in, baseline data at all levels on the result chain can, in principle, be collected, as well as for non-treatment municipalities and regions. EVALUATION METHODS To test hypotheses about the effect of SBEP initiatives on outcomes expected based on the project’s theory of change and address the evaluations questions listed in the previous section, MSI will use baseline and endline data on treatment and comparison municipalities obtained from a combination of sources, including: • Existing government statistical data series including: data business entities and agricultural production at the municipal level; labor force statistics; consumer confidence surveys; and economic growth indicators; • Surveys that gather primary data from municipal officials, business associations, individual farmers and small/medium scale business enterprises at the municipal level in all 67 municipalities outside the Skopje region in 2013 and for either this full set of municipalities or a subset of matched municipalities based on propensity score matching in 2015; • Project performance indicator and project survey data for those farms/firms with which SBEP works directly; • Supplementary direct observation and interview data from project locations and target groups as well as Regional Development Centers in seven regions (excluding Skopje); and • Key informant interviews with national government representatives, other donor and local organizations involved in municipal and regional economic development initiatives, USAID staff, and Carana Corporation staff. MSI’s “Getting to Answers” Matrix in Annex B links these methods to the four SBEP evaluation questions and provides additional detail on data sources, data collection, sample sizes for evaluation surveys, and data analysis techniques, as well as gender disaggregation.3 STUDY LIMITATIONS Several challenges have been identified by the evaluation team as potential study limitations, though efforts will be made to overcome them as the evaluation evolves: 3 The original version of this design included an Annex C that contained rough versions of study instruments. That annex has been deleted in favor of Annex D in this report which includes the study instruments the evaluation team acutally used. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 54 • Regions in Macedonia are not only perceived to be significantly different, but verbal reports from USAID staff and local experts suggest that when these statistics regions were created, there was an explicit effort to cluster municipalities by their similarities, which in turn institutionalized their differences. For this reason, and because of the small number of regions, MSI will construct the counterfactual at the municipal level but cannot be certain a priori how close these matches will turn out to be. • Within the budget available for this evaluation, MSI expects to be able to gather primary baseline and endline data that is statistically representative at the regional level at a confidence level of between 90-95% (for enterprises and farms, respectively), with a confidence interval of +/- 5%. At the municipal level, the sample sizes for farms and firms are being set at 30 each per municipality. This sample size, consistent with the Central Limits Theorem, is sufficient to capture and characterize the range and distribution of responses on survey items for these target populations, and even to detect differences in responses over time. However, data from these samples will not be representative or generalizable in that way that survey data from the study will be at the regional level. • SBEP initiatives vary with respect to how closely they reflect the general intervention model the project has articulated. SBEP’s corn and tomato initiatives are the closest matches, and the wheat initiative it is contemplating may also be a close match. The project’s construction labor supply initiative is a step away, in that its focus is on a very different type of supply chain input. The project’s forest products initiative again deals with commodities, but the hook to a strong regional source of demand so obvious in other initiatives is not present. The project’s efforts to help women expand home-based businesses focuses on opportunities within the region, but may not in all instances be linked to strong demand from larger entities with known supply chain needs. While MSI will attempt to capture data across the project’s range of initiatives, some may prove more or less suited to MSI’s impact evaluation approach than others. • Macedonia is not a level playing field with respect to investments in stimulating public-private sector dialogues that figure prominently in SBEP’ s characterization of its intervention strategy. A number of other efforts to stimulate local economic growth through public-private sector dialogues, government entities, stakeholder planning processes and other techniques – particularly in rural municipalities – are being applied across Macedonia in municipalities where SBEP is active as well as where it is not active. These other similar interventions will invariably make it more difficult for MSI to sort out the effects of SBEP interventions on some of the key dependent variable outlined above, even though the evaluation should not have great difficulty attributing other effects such as changes in production volumes and quality for specific products to SBEP initiatives. Accordingly, MSI will utilize supplementary non-experimental techniques to attempt to sort through alternative possible explanations for changes in variables where multiple programs in a given location worked to build public-private sector dialogues on regional or municipal economic development. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 55 EXHIBIT A: ILLUSTRATIVE EVALUATION TRACKING VARIABLES IN EXISTING DATA SERIES Source of data: Government of Macedonia Statistical Office Key: N = national; R = regional; M = municipal. Note that for some variables that are listed as regional the evaluation team is requesting the underlying municipal level data. Business Entities (SSO) • Active business entities by size (2009-11), M4 • Active business entities by sectors (2010-11), M • Number of active business entities by number of persons employed (2008-10), R5 • Number of active business entities by number of persons employed (2008-10), N • Consumer opinion survey, by months (2009-11), N Agricultural Census 2007 (SSO) • Area under corn, R • Area under wheat, R • Area under tomato, R • Number of individual agricultural holdings, R • Area under vegetable – tomato, R • Households members who work at individual agricultural holdings, by age and gender, R • Area under corn, N • Area under wheat, N • Area under tomato, N • Number of individual agricultural holdings, N • Area under vegetable – tomato, N • Households members who work at individual agricultural holdings, by age and gender, N Agricultural review – Annual (SSO) • Wheat production kg/ha, M • Wheat production T, M • Corn production kg/ha, M • Corn production T, M • Corn-fodder production kg/ha, M • Corn-fodder production T, M • Tomato production kg/ha, M • Tomato production T, M • Average yield per hectare, Agricultural enterprises and cooperatives, wheat, N • Average yield per hectare, Individual agricultural holdings and cooperatives, wheat, N • Average yield per hectare, Agricultural enterprises and cooperatives, corn, N • Average yield per hectare, Individual agricultural holdings and cooperatives, corn, N • Average yield per hectare, Agricultural enterprises and cooperatives, tomato, N • Average yield per hectare, Individual agricultural holdings and cooperatives, tomato, N 4 Municipal level of data 5 Regional level of data – it will be requested for municipal level EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 56 Labor Survey (SSO) • Employment, R • Unemployment, R • Employed by age, N • Unemployed by age, N • Employment by sector type agriculture, N • Employment by sector type manufacturing, N • Employment by sector type construction, N • Employment by sector type accommodation/food, N Regional Statistics (SSO) • GDP 2008-2010, R • Population, R • Density, R • Males, R • Females, R • Employment rate, R • Unemployment rate, men, URBAN, R • Unemployment rate, women, URBAN, R • Unemployment rate, men, RURAL, R • Unemployment rate, women, RURAL, R • Bus. Entities, by employees, R • Catering, employees, R • Catering, entities, R • Catering, turnover, R • Number of completed dwellings, R • Value of completed construction works in 1000 denars, R • GDP 2008-2010, N • Population, N • Density, N • Males, N • Females, N • Employment rate, N • Unemployment rate, men, URBAN, N • Unemployment rate, women, URBAN, N • Unemployment rate, men, RURAL, N • Unemployment rate, women, RURAL, N • Bus. Entities, by employees, N • Catering, employees, N • Catering, entities, N • Catering, turnover, N • Number of completed dwellings, N • Value of completed construction works in 1000 denars, N EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 57 EXHIBIT B: GETTING TO ANSWERS MATRIX Program or Project: Small Business Expansion Project (SBEP) Evaluation Questions Type of Answer/ Evidence Needed (Check one or more, as appropriate) Methods for Data Collection, e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey Sampling or Selection Approach, (if one is needed) Data Analysis Methods, e.g., Frequency Distributions, Trend Analysis, Cross-Tabulations, Content Data Source Method Analysis 1. What have been the measurable effects of SBEP initiatives that operationalize its intervention model on enterprise/farm and community mobilization for economic development as measured by perceptions of economic opportunity, expectations about economic returns and growth, and economic-growth oriented investments at the enterprise/farm and community level? Yes/No • Small and medium scale businesses (manufacturing, agricultural, construction, accommodations/ food) • Municipal Mayors, Local Economic Development Officers/Units, Local Chambers of Commerce and Farmer Associations, or equivalent • Regional Development Centers (RDCs) • Key Informants – Project Implementing Partner (Carana), Ministries, RuralNet, and others as appropriate. • Enterprise & farm level surveys – specific items on Q 1 outcome variables; sex and age of respondent collected. • Structured Interviews • Item extraction from State Statistical Office data series on a municipal and/or regional basis; gender disaggregated data extracted where feasible • 2013 – random sample of 30 small/medium scale enterprises in each of 67 municipalities (outside the Skopje region) proportional to enterprise type by municipality (for four enterprise categories listed under data sources of SSO Agri Census 2007); 2015 – repeat above or for a subset of propensity score matched municipalities (assisted and not) from that universe • 2013 – random sample of 30 farms in each of 67 municipalities (stratified by farm size consistent with size distribution proportions in 2007 agricultural census, • Baseline/endline comparisons for SBEP assisted municipalities on outcome variables for Question 1 variables (perceptions, expectations, investment) • Baseline/endline comparisons for unassisted municipalities/regions, as above • Comparisons of the status of outcome variables for Question 1 between assisted and unassisted municipalities/regions, t￾tests; difference in differences, as appropriate. • Content analysis of results of sets of 67 interviews covering Mayors, LEDs, C of Cs, Farmer’s Association; response pattern identification; transformation into X Description X Comparison X Explanation EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 58 Evaluation Questions Type of Answer/ Evidence Needed (Check one or more, as appropriate) Methods for Data Collection, e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey Sampling or Selection Approach, (if one is needed) Data Analysis Methods, e.g., Frequency Distributions, Trend Analysis, Cross-Tabulations, Content Data Source Method Analysis • Key Informants, as above • Documents and Secondary Source Data including State Statistical Office (SSO): consumer surveys (2010- 2015); Agricultural Census- 2007 and annual updates; Business Enterprise surveys Figure K1); 2015 – as above • TBD if feasible within budget – supplementary random sample of verified corn farmer interviews (30 in Polog and 30 in Pelagonia); and 30 verified tomato farmer interviews (Pelagonia) in both 2013 and 2015. • 2013 - Joint Interviews with Mayors & LED Officers in 67 municipalities; 2015 – repeat or matched subset • 2013 - Small group interviews with C of C and Farmer Association representatives in 67 municipalities; 2015 – repeat or matched subset • 2013 – Interviews with seven RDCs; 2015 - repeat quantitative form, and integration with statistical analysis of differences over time and between assisted and unassisted municipalities/regions. • Trend analysis of SSO existing data series for assisted and unassisted municipalities/regions. • Analysis of public-private sector dialogue data from surveys/interviews, generally aligned with approaches used elsewhere (as described in the PPD Handbook, op. cit.) • Gender specific and/or differential impacts analyzed by outcome variable to the degree data permits. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 59 Evaluation Questions Type of Answer/ Evidence Needed (Check one or more, as appropriate) Methods for Data Collection, e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey Sampling or Selection Approach, (if one is needed) Data Analysis Methods, e.g., Frequency Distributions, Trend Analysis, Cross-Tabulations, Content Data Source Method Analysis 2. What have been the measurable effects of SBEP initiatives that operationalize its intervention model on enterprise/farm, municipal and regional production volume, product quality and jobs linked to supply chain requirements on which SBEP￾organized training and technical assistance focused? Yes/No • Enterprises and farms, as above • Municipal representatives, as above • RDCs, as above • Documents and Secondary Source Data including State Statistical Office: Agricultural Census￾2007 and annual updates; Business Enterprise surveys; Labor Force Surveys • Project Implementing Partner (Carana) • Enterprise & farm level surveys – specific items on Q 2 outcome variables • Structured Interviews • Item extraction from State Statistical Office data series on a municipal and/or regional basis; gender and age disaggregated data extracted where feasible • Review & reanalysis of project performance monitoring data – PMP indicators and lower level indicators and surveys • Enterprise and farm survey samples are the same as above, Question 2 draws on different items in the same surveys • Joint and small group interviews, as above, drawing on different interview items • Baseline/endline comparisons for assisted and unassisted municipalities/regions covering survey, interview, PMP and SS0 existing data series • Comparisons between assisted and unassisted municipalities/regions on Question 2 outcomes (production volume, quality and jobs); t-tests; difference in differences, as appropriate. • Gender specific , age specific and/or differential impacts analyzed by outcome variable to the degree data permits. X Description X Comparison X Explanation EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 60 Evaluation Questions Type of Answer/ Evidence Needed (Check one or more, as appropriate) Methods for Data Collection, e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey Sampling or Selection Approach, (if one is needed) Data Analysis Methods, e.g., Frequency Distributions, Trend Analysis, Cross-Tabulations, Content Data Source Method Analysis 3. What have been the measurable effects of SBEP initiatives that operationalize its intervention model on economic growth, the emergence of new enterprises, unemployment and other broad economic development measures in municipalities and regions where SBEP was active? Yes/No • Documents and Secondary Source Data including State Statistical Office: Agricultural Census￾2007 and annual updates; Business Enterprise surveys; Labor Force Surveys • Municipal representatives, as above • RDCs, as above • Project Implementing Partner (Carana) • Item extraction from State Statistical Office data series on a municipal and/or regional basis; gender and age disaggregated data extracted where feasible • Structured Interviews • Joint and small group interviews, as above, drawing on different interview items • Baseline/endline comparisons for assisted and unassisted municipalities/regions covering SS0 existing data series • Comparisons between assisted and unassisted municipalities/regions on Question 2 outcomes (economic growth, new enterprise registration, unemployment and other broad measures); t-tests; difference in differences, as appropriate. • Gender and age specific and/or differential impacts analyzed by outcome variable to the degree data permits. X Description X Comparison X Explanation 4. What evidence of SBEP project “spill-over” effects or spontaneous replication of SBEP initiatives or aspects of the SBEP model, if any, does evidence collected by the evaluation from regions in which SBEP was not active reveal? Yes/No • Enterprises and farms • Documents and Secondary Source Data including State Statistical Office: Agricultural Census￾2007 and updates; Business Enterprise surveys; Labor Force Surveys • Sample surveys, as above, specific items therein • Relevant items from those extracted from SSO data series • Individual and small group interviews, as • Survey sizes, as above • Joint and small group interviews, as above • Comparison group data (surveys, interviews) item analysis of pre-post data for presence of gains that are not consistent with overall baseline/endline picture for unassisted regions, including supplementary “spill￾over”/”cross-over interview analysis in 2014. X Description Comparison Explanation EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 61 Evaluation Questions Type of Answer/ Evidence Needed (Check one or more, as appropriate) Methods for Data Collection, e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey Sampling or Selection Approach, (if one is needed) Data Analysis Methods, e.g., Frequency Distributions, Trend Analysis, Cross-Tabulations, Content Data Source Method Analysis • Municipal representatives, as above • RDCs, as above • Project Implementing Partner (Carana) above for 2013 and 2015 • Supplementary interviews in 2014 specifically focused on “spill-over” and cross￾over effects as well as on changes in the SBEP intervention model over time EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 62 ANNEX C: SBEP EVALUATION SUB-STUDIES EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 63 Sub-Study 1: Regional Development Support LEADER-LAG Initiative, Regional Development Centers, and Municipalities Photos: SBEP (USAID/Macedonia) Introduction Regional development is a hallmark of Macedonian strategy documents developed over the past half￾decade on pre-EU accession, fiscal policy, and economic policy, each of which has renewed its commitment to balanced regional growth. This focus reflects Macedonia’s needs and political commitments, while incorporating the European Union’s approach to economic development and integration that extends across the continent and is central to the accession process. The focus from both these perspectives is development on a subnational scale, based on local priorities. Regional development is valued in the EU, as well as in Macedonia, as a way to preserve traditions, including with respect to local cuisines; foster tourism; encourage the development of industry clusters; and ensure economic opportunity and well-being well beyond capital cities. In developing its Small Business Expansion Project (SBEP), USAID accorded balanced regional development a central role, anticipating that a variety of activities would be incorporated into an implementation plan for achieving this result in the regions where SBEP was active. This review of the SBEP LEADER-LAG initiative in Macedonia and its collaboration with Regional Development Centers (RDCs) and municipalities is based on documentary evidence, primarily in SBEP quarterly reports but also from Macedonia’s State Statistical Office, as well as interviews the evaluation team conducted with RDCs, all 10 pre-LAGs that SBEP assisted, and 63 municipalities across the country. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 64 Evolution of SBEP’s Regional Development Support Recognizing this dual imperative, USAID put regional development at center stage when it initiated SBEP. Carana Corporation drew on its significant previous experience as USAID’s implementing partner for USAID/Macedonia’s Competitiveness Project (2007–2012) in bidding for the contract for SBEP’s implementation. Within months of winning it, Carana initiated an early round of workshops and trainings and agreements with RDCs in Pelagonia and Polog to begin building public-private sector dialogues as a forum to explain and promote the EU’s LEADER/LAG model for catalyzing regional development. In August 2012, SBEP and the Pelagonia RDC signed an agreement to formalize cooperative efforts to create a local action group (LAG) in preparation for accessing EU development funding under the LEADER (Liaison Entre Actions de Développement de l'Économie Rurale) approach and positioning the RDC in each region of Macedonia as an executive body of the LAG. Through the SBEP project life cycle, a steady portion of the overall effort was devoted to advancing regional development using this model. Central to that effort was the formation of LAGs, structured on a micro-region basis and focused on identifying economic opportunities across traditional municipal lines and pre-existing expectations about hierarchies, seniority, gender norms, and the place and role of youth in determining the future of the regions where they live. Context for Local Economic Development Assistance under SBEP Assistance under SBEP to support local economic development built on more than a decade of USAID’s involvement in democratic governance, fiscal decentralization, and private sector capacity building as the engine for economic growth. On paper, Macedonia’s commitment to decentralization reaches back to a 1998 National Strategy for Decentralization, but a practical step in that direction emerged only in the wake of an 2001 political-ethnic crisis, which continues to flare and seek new forms of resolution, as events of the past several years culminated in elections in late 2016 and a new government in mid-2017. The origins of local governance and municipal economic development structures as they look today evolved through a 2002 Law on Local Self￾Governance, which superseded 1995 legislation with the same name. Critical implementing provisions emerged subsequently in the 2004 Law on Local Finance, which became intertwined in political efforts to multiply the number of municipalities and better balance their ethnic distribution, pursuant to agreements in 2001. In this environment, USAID initiated its three-year Make Decentralization Work project, which built municipal capacity for financial management, budgeting, revenue generation, and tax administration; helped institutionalize municipal capital improvement plans; and created Citizen Advisory Boards to articulate priorities. Municipal-level economic development offices established at that time continue to play these roles, including reaching out to the private sector and citizen groups for input. USAID has followed on from its early and important involvement in decentralization and local economic growth through of other projects described in the Project Background section of this report. By 2007, evidence showing that economic growth was increasingly Skopje-centered, and disparities had widened between the capital and the country’s least-developed areas, government passed the Law on Balanced Regional Development, which focuses on these gaps and contributes to multi-level government discussions about how the law allocates 1 percent of national funds back to regions for development purposes. To help implement this law, the statistical (not administrative-political) planning regions it calls for were established in each of eight regions. Regional development centers (RDCs) were established along with them and these entities play a role in organizing each region’s program for economic development — largely, but not exclusively, by gathering recommendations from municipalities that still prepare local economic development strategies and capital investment plans. Regional development programs, in turn, inform the National Strategy for Regional Development, and influence the disposition of the 1 percent of budget that the Law on Balanced Development requires be shared back with the regions. This law and its structures also conform to requirements that Macedonia faces as a function of its status as an EU accession candidate. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 65 SBEP quickly entered the four regions in which it eventually worked and began working at a pace that reflected how quickly buy-in to the concepts evolved and the speed with which trust across municipalities and ethnicities grew in each region. SBEP helped mobilize and launch 10 pre-LAGS, which are poised to take the next step toward full registration as soon as political decisions permit, and to request and manage economic development funds from the EU based on feasible ideas for projects developed locally. While SBEP started working toward the creation of LAGs in 2012, this initiative began to reach its potential late in the SBEP project cycle, with small pre-LAG projects emerging in 2015–2016 and more coming online in 2017. For those, SBEP’s legacy organization AIRE provided continued assistance with Swiss Embassy resources during part of 2017. The sections that follow describe MSI’s findings on regional development under SBEP from various perspectives and levels. These include SBEP’s collaboration with regional development centers (RDCs) Regional Development Centers and Ministries Macedonia’s RDCs are an outgrowth of the Law on Balanced Regional Development of 2007. Since then, Germany’s development agency GIZ has provided assistance to the RDCs, helping them develop their structures, planning protocols, and interfaces with other actors. These actors include municipalities, other donors working in the regions, and, more recently, LAGs that are forming at the micro-region level in line with the EU LEADER/LAG model. SBEP’s relationships with the RDCs were collaborative throughout the project life, rather than focused primarily on assistance to those centers. Given their function as a linkage between government ministries and municipalities on regional development, SBEP organized meetings with RDCs soon after the project started in Pelagonia and Polog, which were core regions for SBEP from the start. With successive regions (Northeast and then Vardar), the RDCs were again an early priority. These were the organizations with which SBEP signed memoranda of understanding documents outlining the scope of its operations in each arena. Throughout, the RDCs provided SBEP an entry into the municipalities they encompass and a range of contacts beyond local governments. SBEP, in turn, centered their launch activities and some of their workshops and trainings of region-wide interest at the centers. As part of this evaluation, MSI’s team interviewed seven of the eight RDCs, excluding Skopje, in 2014 to understand existing regional and municipal planning systems and related activities and actors. The evaluation team re-interviewed six of these RDCs in 2016 when it carried out the second phase of the evaluation’s fieldwork. These interviews provided insights on the way RDCs were working with LAGs in various regions. In regions where SBEP was active, RDC representatives described a close working relationship with SBEP on the development of LAGs and gave Carana credit for taking the lead in mobilizing sentiment to support regional development as a focus for municipalities, which were not accustomed to thinking and working beyond their boundaries at the time SBEP started. In non-SBEP regions, the RDCs explained, they worked in various ways. In the Eastern Region, the RDC played an active role, working with the Regional Development Network of Macedonia (RDN); in the Southwestern Region, an informal action group on agriculture worked on LAGs with GIZ support; and in the Southwestern region, the RDC again collaborated with RDN on start-up efforts. In SBEP regions, RDCs reported a variety of interactions with SBEP beyond collaboration on LAG development. These included participation in SBEP workshops and trainings; working together on various kinds of public-private sector dialogues; and having received some funds from SBEP. All three of those interviewed said they had received publicity as a function of their collaborations with SBEP. Each also said they had occasionally received advice from SBEP on matters pertaining to their own RDC. In contrast, RDCs in other regions were not aware of SBEP, or only vaguely so. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 66 As with other regional development stakeholders (municipalities, LAGs, corn farmers, women in business and firms involved in adventure travel with SBEP), MSI asked RDC representatives in 2016 what they would say was the most significant change or effect of SBEP’s involvement in development in their region. RDC responses to this question, which were numerous, specific, and uniformly positive, are provided below: • We consider that the initiation of public-private dialogue on development issues and establishment of the LAGS represent the “most significant changes.” • Carana’s project “Drop by Drop” and the change in the mindset about the use of this irrigation system were the best marketing for us. • Also, LAGs are excellent (continuous support by USAID about LAGs) • Trainings for wild plants – very good • Paragliding, cycling – this project was successful. It would be great if there is a possibility for an extension. • Experience with Carl Larkins – COP and his concept of strategic thinking. He always used to say, “think strategically not only for now” …this I consider is the biggest significant change for me, personally. • Modernization and usage of new methods and technology in agriculture such as the drip irrigation system. • LAGs establishment as a unique model of bottom-up approach in community development. • Training for Wild Product gatherers • Feasibility studies for irrigation on the whole territory of Pelagonia. Municipalities Networking with municipal officials and local business groups was a key element of SBEP approach to regional development from its inception, and the project was able to build on USAID relationships at this level established in its Make Decentralization Work project (2004-2007), which upgraded introduced financial management systems at the municipal level as well as local economic development planning approaches, still in use, in response to 2004 legislation. Subsequent economic development and municipal governance activities furthered USAID relationships, providing SBEP with a takeoff point for its activities in the four regions where it focused. The evaluation team conducted interviews at the municipal level in 2014 and again in 2016. In both rounds, they used small group interviews (with two to four individuals): one with mayors, where they were available, and their Local Economic Development Office staff, and a second with local business and farming community representatives, usually from associations. These interviews, MSI’s local survey research and focus group interview partner GfK, used a combination of closed- and open-ended questions to gather information on local development practices and plans and impressions of the contributions SBEP had made. Perceptions of Recent Municipal Progress At the municipal level, interviewees included public sector representatives — mayors and local economic development (LED) offices, as well as private sector representatives (such as business and farming association representatives). In these interviews, the evaluation team asked people how they would compare their municipality’s current financial situation and conditions (2016) to the municipality’s circumstances a couple of years earlier (2014). EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 67 • Among public sector representatives, 39 percent in regions where SBEP was active (Polog, Pelagonia, Vardar, and Northeast) said their municipalities were better off than before, while 14 percent said their municipalities were worse off than they were two years prior. • In regions not assisted by SBEP (East, Southeast, and Southwest) 25 percent of public sector representatives said their municipalities were better off currently, while 36 percent said they were worse off. • Among private sector representatives, the contrast was greater: 70 percent of private sector representatives in SBEP-assisted regions viewed their municipal situation positively compared to two years prior, versus 27 percent of private sector representatives in non-SBEP regions. Open-ended questions about why circumstances were better or worse than two years earlier showed that in both SBEP-assisted and other municipalities where differences existed between individual municipalities as well as between regions, some of the same positive and negative views on municipal circumstances were offered in both types of regions. For example, improved infrastructure and new businesses that represented foreign direct investment and jobs were reasons for describing municipalities in all regions as being better off. Financial management issues at the municipal level and the outmigration of young people described worsening municipal circumstances in all regions. In Polog, multiple respondents cited the opening of industrial parks as fostering improvements in that region, and Southwest Region respondents mentioned the opening of industrial zones. This gives a flavor of the degree to which factors associated with improved circumstances cut across the areas where SBEP was involved. On this broad question about the reasons why municipalities were or were not better off than they had been two years earlier, one official cited SBEP for jobs it had created: “With the support of the USAID Carana project, five to six smaller businesses have been established and are still functional. These include boutiques, glasses shop, souvenir shop, and a handmade accessories shop.” — Public sector representative, Northeast Region Greatest Needs and Impediments to Future Municipal Progress To foster greater municipal economic progress in the future, municipal representatives were asked to identify the most challenging needs they face. In both assisted and unassisted regions, the responses to this question from public sector respondents were intertwined. Infrastructure was the most frequently cited impediment to future economic development, with sewage and sanitation systems pinpointed nearly twice as often as roads; municipal financing gaps were cited as a key reason for infrastructure being an obstacle; municipalities need more resources to build and maintain adequate infrastructure than many are bringing in; also noted were industrial zones where the buildout remains incomplete, which relates to the ability to draw domestic and foreign investors and, hence, their occupancy rates and the number of jobs they produce. Along a separate line, skill and experience gaps in the workforce were cited a few times as an impediment to capitalizing on opportunities that do exist, as was the outmigration of young people and its regional workforce impact. On the private sector side, investment was the most pressing concern, with municipal investments in irrigation at the top of that list; more and more ready-to-go industrial zones, and municipalities working harder/smarter to attract and draw in more investment from public sector sources as well as foreign direct investment. Expectations for the Future in Municipalities The evaluation team also asked public and private sector representatives about their expectations for the future and their plans with respect to investing in the economic development of their municipalities. On a question about whether they expected the financial situation and circumstances in their municipality to be better or worse in 2017 than in 2016, 64 percent of public sector representatives in EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 68 SBEP-assisted regions said they expected that their municipalities would be better off in 2017, compared to 39 percent of public sector representatives in non-assisted regions. As to investing in their municipalities in 2017, 61 percent of public sector representatives in assisted regions said they plan such investments, compared to 46 percent in other regions. Among private sector representatives, 40 percent in assisted regions expected their municipalities to be better off in 2017, compared to 27 percent in other regions. Private sector expectations about whether they expected their municipalities would invest in municipal economic development in 2017 were closer together, around 25 to 30 expecting such investments would be made. SBEP’s Involvement and Impact on Municipalities in Assisted Regions In each of the regions in which SBEP worked, it engaged with virtually all municipalities through at least one of SBEP’s initiatives. This section provides answers the evaluation team received from mayors, municipal LED office representatives, and representatives of the business and farming communities on an open-ended question that asked: Given whatever experiences you and your municipalities had with SBEP, what seems to be the “most significant change” that has come about because of SBEP’s presence and involvement in your community and their municipal, sub-regional, and regional economic development activities? Diagrams on these pages show the municipalities in each region by name, with the pre-LAG they belong to where applicable. For each municipality shown, the diagrams expand to indicate the number and types of SBEP initiatives that were active there. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 69 PELAGONIA SBEP was active in nine municipalities in Pelagonia and helped establish three pre-LAGs, through which specific municipalities are linked in joint efforts to develop particular micro￾regions in Pelagonia. Asked about the most significant change or effect of their work with SBEP, public and private sector municipal representatives identified a range of outcomes from their various perspectives. These are included below, separate from the public and private sectors. Public Sector • In relation to promoting adventure tourism, we used to have smaller interest and participants only from the region, but ever since the take off ground was equipped, we organized an event for promoting adventure tourism, promotion of traditional food, and had participants from around the world. That is definitely beneficial for improvement of the municipal economy. • The biggest significant change is enabling farmers to start using modern irrigation systems like drip irrigation that will probably result in better production. Support to LAG activities, agricultural cooperative and “Tractoriada” is also something that will improve our future development. • The municipality itself did not have direct benefits of this project. The citizens who received training and direct technical assistance benefited from it. As a result of the drip irrigation system, the farmers who received it increased their production. • Development of alternative tourism. There are incomes and the feedback from citizens is positive PELAGONIA Bitola (PeLAGonija) Drip Irrigation Light Manufacturing Demir Hisar (PeLAGonija) Drip Irrigation Wild Product Training Dolneni (Agro Lider) Drip Irrigation Krivogastani (Agro Lider) Drip Irrigation Krusevo (Agro Lider) Adventure Paragliding Adventure Biking & Hiking Mogila (PeLAGonija) Drip Irrigation Novaci (PeLAGonija) Prelip (Agro Lider) Adventure Paragliding Light Manufacturing Drip Irrigation Resen (Prespa) Light Manufacturing Drip Irrigation Drip Irrigation EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 70 Private Sector • Although I personally cannot say much about the most significant change that happened as a result of SBEP and its activities and projects, as I was not involved in their activities, I would say that the project contributed to raising awareness about the importance of small and medium￾size enterprises and their contribution to overall municipal economic development. • Increase of corn production per hectare and savings of water. • Positive aspect about using drip irrigation system is that we decreased our work and increased production, as we are in the region with very little rains. Before I started to use this system, I used to produce 7 tons of corn, and after I started using it, I managed to increase the production by 10 tons per hectare. Drip irrigation system enabled irrigation of bigger land with smaller amount of water. It enabled maximum water use without the need to use the water wells. Negative side of using this system is that we increased expenses as it is not cheap to get the drip irrigation system installed. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 71 POLOG Perceptions of the most significant changes or effects of public and private sector representatives from nine municipalities with which SBEP had worked on various initiatives and in efforts to develop pre-LAGs in Polog are provided below, separately from public and private sector interviewees. Public Sector • The most significant change is definitely direct impact of the project activities for the beneficiaries, i.e., businesswomen and farmers primarily. It was not about working on developing yet another set of documents and papers, but specifically on improving business climate. As a municipality, we formed a Foundation for Support of Women and Youth Entrepreneurs, “Egalite,” that is hopefully going to secure longer-term sustainability of SBEP initiatives. Specific change description you will probably get if you talk to businesswomen and farmers who got SBEP assistance. • The farmers and all the others who were included in the project got grants. Women in business got grants in the amount of 2 - 5000 USD for opening small businesses, whereas farmers got 3000 USD for drip irrigation systems. These grants definitely helped all of them improve and increase their production. • Related to collection of wild forest products, all the collectors that have been trained gained invaluable theoretic knowledge. There were series of trainings and now it is easier for them to collect these products as they learnt what products are not dangerous and what are, what specific varieties they collect and similar. In one word, they are more trained and it is much easier for them to do the collection. • Series of workshops/trainings were delivered for LAG strategy development. Also, there was a technical support for LAG registration and financial support for one LAG project idea. Bogovinge (BB LAG) Drip Irrigation Wild Products Training Women in Business Brvenica (BB LAG) Drip Irrigation Women in Business Gostivar Drip Irrigation Wild Products Training Women in Business Adventure Paragliding Jeunovce (Skardus) Drip Irrigation & Rostuse Terace (Skardus) Drip Irrigation Tetovo Women in Business Light Manufacturing Drip Irrigation Vrapciste Drip Irrigation Zelno Women in Business Drip Irrigation POLOG Wild Products & Training Collection Center Mavrovo EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 72 Private Sector • Now that I got a machine for rolling and a big fridge for keeping cakes and cookies, I can say we make more delicious and prettier cakes and cookies. We used to make cakes before as well, but we did not have a big fridge to keep bigger cakes, with several tiers, so we could not make a cake in advance as we did not have a proper place to keep it until the person who ordered it comes to take it. Now we can. Also, we have a kind of seasonal production of cakes for weddings. The season is from June to October, and if we used to be able to make only 10 bigger cakes before, now our capacity improved and we are able to make 50 cakes. The equipment really helped me in increasing my company's profit and to be able to employ five people. • (Farmer) Drip irrigation system is a good system. If you grow vegetables it is very useful. I didn't know about what difference it makes, I heard about it on television. This system enables doubling yield. I improved my production for 40 percent. All you need to do is follow the instructions project people provided to us related to irrigation. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 73 Kratovo (Osogovski Lisec) Drip Irrigation Wild Products Training Kriva Patanka (Osogovski Lisec) Drip Irrigation Light Manufacturing Kumanovo (Aber 2015) Drip Irrigation Light Manufacturing Adventure Kumanovo Museum Lipkovo (Aber 2015) Drip Irrigation Rankovce (Osogovski Lisec) Wild Products Training Drip Irrigation Staro Nagoricane (Aber 2015) Drip Irrigation Wild Product Trainings NORTHEAST In the Northeast, as in other regions, the evaluation team asked public and private sector municipal representatives about the most significant change or effect the presence of SBEP had in their communities. Their responses are presented below, separately for public and private sector respondents. Public Sector • Maybe the most significant change is encouraging young people to start their own, small business, the whole initiative of fostering entrepreneurship of young people. Those 2.500 - 3.000 Euros individual financial support helped open 5-6 new businesses that still work. There is no bigger impact than new businesses. • The key change is probably exposing farmers to modern systems for irrigation. Nevertheless, since only one farmer got drip irrigation system, we may not talk about a bigger result and impact of project activities. • Certificates for wild product gatherers are really important for their future work. The trainings they received were invaluable as they learnt a lot about wild products. The most interesting aspect that these people learnt is that wild products should not be destroyed, but rather nourished. They learnt that poisonous mushrooms should not be cut and destroyed as that can infect the soil and future wild products can easily turn into poisonous as a result. NORTHEAST EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 74 Private Sector • There was no particular change because we were only part of a training for innovation. In terms of system drip irrigation we wanted to apply but we demanded documents issued to us for 8 months and the project ended in three. • Our company received the drip irrigation system at the end of 2015, and we couldn't use 100 percent of it. Before the drip irrigation system, we had produced 3 tons per hectare corn and in 2016 our production increase to 7 tons per hectare. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 75 VARDAR On a question about the most significant change that had come about in their municipalities as a result of SBEP’s initiatives and involvement in Vardar, public and private sector interviewees provided their impressions. These are presented below, separately for the public and private sectors. Public Sector • We do not know of specific significant changes as those results have not been shared at municipal level. We don't know how many farmers got drip irrigation system. LAG is a good idea of municipal working together but we doubt it will have any results as the cooperation with Lozovo is not very good as they cooperate more with Veles. • The cooperation with the Small Business Expansion Project had big impact on our municipality but also on increasing the capacities for development of agriculture in the region. The project also supported development of active tourism by construction and markings of two bicycle mountain trails/paths. These trails/paths remain in the region and will serve numerous tourists who are into active tourism. • Interactions with SBEP increased confidence of business sectors and farmers in institutions. Citizens had benefits from education regarding wild product harvesting and sales. Many young women participated in this event and they could further transmit the knowledge to younger generations in their family. • The municipality does not have any information about the results of this project's activities. The only activity the municipality participated in, at the request of the project, it provided a room and projector for training of farmers. No other information was ever shared with the municipality about the project's results. VARDAR Caska (Vardar Leader) Drip Irrigation Wild Products Training Demir Kapja (Tikvesh Wine Valley) Drip Irrigation Adventure Rock Climbing Gradsko (Vardar Leader) Drip Irrigation Kavadrci (Tikvesh Wine Valley) Drip Irrigation Negotino (Tikvesh Wine Valley) Drip Irrigation Light Manufacturing Rosoman (Tikvesh Wine Valley) Drip Irrigation Veles (Vardar Leader) Wild Products Study Tour Light Manufacturing Adventure Biking Drip Irrigationr EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 76 • Four corn farmers got drip irrigation system and some training on its use, but the municipality does not have specific information on the results and impact of this assistance. From what is being reported on TV and from some hearsay, the corn production has increased, but no other specific data. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 77 The LEADER/LAG Initiative As indicated, outreach to RDCs in each region where SBEP worked was an early step and soon focused on developing formal agreements with the RDCs, under which they would function as the sponsor of SBEP activities focused on developing public-private sector dialogues with municipalities and working with them to identify micro-regions with potential to become LAGs in line with pre-accession steps the country was taking, as well as serving as a new mechanism for the kind of balanced regional development Macedonia seeks domestically. This section examines the results of the LAG formation aspect of these efforts. Status of Pre-LAG Planning and Project Realization As of the end of SBEP, the pre-LAGs it had helped form were at different stages of their development. MSI interviews with each of these 10 pre-LAGs illustrated some of the variation in their level of development. For example, based on their own descriptions: • Two of three pre-LAGs in Pelagonia had written strategies they had approved, one covering 2014-2018 and the other 2015-2019. • Both of Polog’s pre-LAGs had such strategies, one for 2017-2019 and the other for 2015-2019. • In Vardar, one pre-LAG told the team that its approved strategy ran 180 pages. A second Vardar pre-LAG said they have a five-year operation plan but no strategy yet; the final pre-LAG in this region explained that they had a draft strategy as of December 2016. • In the Northeast, both pre-LAGs indicated that they are in the process of developing vision statements and were not yet at the strategy development stage. Substantively, the visions and plans LAG members described to the evaluation team were highly varied as are the micro-regions themselves. Some were highly delineated with detailed tasks described; others less so. Across the pre-LAG strategies described to the evaluation team, there was clear evidence of an understanding of what USAID would call a “results statement,” meaning the ability to identify specific desired outcomes and intermediate steps to achieve them. Styled in a European or U.N. manner, often starting with “to do X to achieve Y,” rather than the more American “done deal” form (as if one is standing at the result and looking backward), they were nonetheless clear, organized and in forms, or moving into forms, that could be targeted and monitored by the pre-LAGs themselves. MSI did not become familiar enough with SBEP’s training modules to say whether these vision, strategy, plan documents reflect USAID’s assistance or pre-dated it for certain, but what was observed is certainly in line with what a seasoned Carana team would have encouraged. With respect to projects mobilized and implemented by pre-LAGs, each of the 10 pre-LAGs had completed at least one activity as a multi-municipality, micro-region investment in line with their LAG’s vision and plans. These were generally small-scale, but had a strong collaborative element, including sharing schemes for equipment purchased, including LAG which was not only widely sharing access to equipment by also charging farmers a percent of earnings for their use which, in turn, was being contributed to the pre-LAG coffers to support its activities. Maps of the regions SBEP assisted, annotated to show the territories each pre-LAG spans, are provided below, along with short descriptions of the activities each pre-LAG had completed as of the end of SBEP in December 2016. SBEP’s Involvement and Impact on LAG Development in Assisted Regions The following pages provide the evaluation team’s mapping of the pre-LAGs SBEP helped form over the life of project, along with “sticky notes” that identify the projects each pre-LAG had undertaken when the project ended. Interspersed on these pages are responses from interviews the evaluation team conducted with small groups from each LAG indicating what these pre-LAG members considered to be the most significant change that emerged from their experiences with SBEP under its LEADER/LAG initiative. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 78 Pre-LAG PeLAGonija Agricultural Fair in Mogila Pre-LAG Agro Lider Paragliding video Thermic visit Pre-LAG Prespa Familiarization (FAM) tour Perceptions of LAG members in Pelagonia Concerning the most significant change or effects of SBEP’s involvement in efforts to identify micro￾regions and form a LAG to advance the development of those new regional economic development structures: • The most significant change is support to the activities and project implementation. It includes training, networking, promotion, organizing fairs. Also, the PR trainings were one of the most significant changes. • The most significant change was that we got positioned as a touristic subject that gathers all entities. • The most significant change was the establishment of LAG, the opportunity different sectors to sit together and to work as a team towards common goals. With other words, networking, multisectorial representation, PPD and possibility to reach consensus among the LAG members. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 79 Pre-LAG BB Lag Technical plans for road Pre-LAG Skardus Fruit & vegetable dryer bought Perceptions of LAG members in Polog Concerning the Most Significant Change or effects of SBEP’s involvement in their efforts to identify micro-regions and form a LAG to advance the development of those new regional economic development structures: • Confidence building for better future, together we can achieve much more than a single municipality by itself. We already have proven this with building of local road. • Understanding of LAG and bottom-up approach, public-private dialogue, active involvement of young people in the society EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 80 Pre-LAG Aver 2015 Solar-powered dryer bought Pre-LAG Osogovski Lisec Discover Osogovski website Perceptions of LAG members in Northeast Concerning the most significant change or effects of SBEP’s involvement in their efforts to identify micro-regions and form a LAG to advance the development of those new regional economic development structures: • Most significant positive impact resulted by our interaction is the experience and knowledge earned in the process of projects implementation, in particular the knowledge that younger members of LAG have gained. • Most significant positive impact resulted by our interaction is: networking among the regional stakeholders, cooperation spirit, and initiation of public-private dialog. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 81 Pre-LAG Vardar Leader Bike and wine tour and trail marking Pre-LAG Tikvesh Wine Valley Bike and wine tour Perceptions of LAG members in Vardar Concerning the most significant change or effects of SBEP’s involvement in their efforts to identify micro-regions and form a LAG to advance the development of those new regional economic development structures: • Borders of LAG expanded – both for participation and for having impact. We can make impact outside the territorial borders of our LAG. Positive opinion from the Mayor of Veles. • I, personally, as a president of the LAG, am very satisfied by the cooperation with Carana SBEP. They are very open, you can reach them 20 times by phone in one hour. Employees are excellent in Carana. Mimi, Martin, Atanas, Hari, Svetle. Both the cooperation with Carana and the creation of the LAG are equally important. Establishment of a public-private dialogue. We received funds for both LAGs - 150.000 MKD for each LAG. This cooperation between two LAGs is unique, the only example. • The collaboration among the CSO and institutions. But, we need office and someone to employ to succeed more. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 82 Pre-LAG Recommendations for Moving Forward Asked about what help they need to move forward – their top recommendations – pre-LAGs highlighted their priorities for the evaluation team, which included: • More support on resources mobilization • Greater public-private sector dialogue at more levels – pre-LAG, municipal, regional, national • More citizen engagement in development activities • Greater inter-municipal cooperation • More practical workshops, not only theoretical, but learning through working • Pilot projects • More financial sources (donor organizations) to finance different groups and activities • Give priority to rural development versus urban, as it is now neglected • Money from national funds are more difficult than the EU, it should be the opposite • For regions, like Polog, where agriculture is a very important industry, more irrigation is needed. The possibilities for organic production exist but are not being realized • Establishing a LAG register in the Ministry of Agriculture and Forestry and Water Economy • LEADER program to start functioning • More steps to prevent the emigration of young people • Develop realistic action plans • Networking with other LAGs • The DIS support for agriculture should continue • More promotion of our regional cities for tourism benefits • Further improving transportation infrastructure • Improvement in LAG planning processes and selection and investing in pilots with the pre-LAGs that are ready to implement them, with funding to be provided from national sources • Support for pre-LAGs that are lagging through a mentoring system • Mechanisms that foster following the strategic plans pre-LAGs have adopted and documented • Stronger institutionalization of pre-LAGs, enough to allow each one to employ one person • More exposure to international experience, in particular how LAGs function and contribute to regional development in EU countries • Perhaps mentoring partnerships with LAGs in European countries could be a proper model. In parallel to its work with nascent pre-LAGs, SBEP engaged in other aspects of the LEADER/LAG initiative, including what it termed Leader “pilot projects” that involved development activities above and beyond what pre-LAGs could mobilize and mange in their initial year or so of operations. These pilots, which were not a focus of the evaluation, are listed in SBEP’s final report, as including: • Vegetable seedling production in Bitola • Kokino tourism destination development in Staro Nagoricane • Osogovo tourism destination development in Rankovce, Kratovo and Kriva Palanka • Prespa tourism destination development in Resen • Paragliding site development in Krivogashtani Somewhere between the small projects that pre-LAGs developed and implemented over this period and the Leader pilot projects listed above lay cluster of LAG fairs at the regional level and a showcase for LAGs that SBEP organized in Skopje. By the time SBEP ended, Vardar had held its fourth regional fair. The popularity of these events is indicated in the collage below. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 83 LEADER/LAG in Other Regions and Sustained Investments across Macedonia While SBEP was starting its work in Pelagonia and Polog in mid-2012, another organization was engaging in the early stages of a similar effort near Skopje, namely The Rural Development Network of the Republic of Macedonia, which was already presenting workshops on the EU LEADER approach and had produced a short volume to introduce rural Macedonian communities to the LEADER/LAG initiative, with support from the European LEADER Association for Rural Development (ELARD). On a smaller scale, USAID engaged this group to help organize LAGs in other regions under its Adaptation to Climate Change in Agriculture (ACCA) project. Further, with follow-on funding from the Swiss Embassy, the SBEP legacy organization AIRE spent half of 2017 helping the pre-LAG SBEP had helped take additional steps, and worked more broadly to help the national government open a registry for pre-LAGs country-wide. As an aide to further development and expansion of LEADER/LAG in Macedonia, AIRE has published a guide on LEADER highlighting the progress of LAGs that USAID, and more recently the Embassy of Switzerland, have supported. In addition, in November 2017, the Embassy of Switzerland committed further funding for regional development, one component of which will fund infrastructure and “innovative projects” in all eight regions. Given investments the Swiss have already made in LEADER/LAG in Macedonia, some of those resources might follow their earlier investments in regional development through LAGs. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 84 Exhibit 1.1 Regional Development Support Chronology October-December 2012: EU Leader initiative launched. LEADER/LAG Info Days and Workshops held in Pelagonia. January-March 2013: First 2 training cycles held. April-June 2013: Third and fourth training cycles held. July-September 2013: A study tour to Croatia was organized for Pelagonia mayors in order to share experiences and learn from the Croatian experience with LEADER/LAG. Letter of Collaboration between USAID SBEP and the RDC Pelagonia for Year 2 signed. October-December 2013: A LEADER/LAG one-day workshop was held on December 11 in Bitola for counterparts from Polog, including representatives from the municipal LED offices, the RDC and the Economic Chamber. January-March 2014: SBEP entered a third region (Northeast) with the LEADER/LAG approach. Three LAG Strategy Development sessions were held in each of the Pelagonia micro-regions. A meeting was held in the Polog region about the possibilities of introducing the LEADER/LAG there. April-June 2014: In Pelagonia, 2 of the 3 LAGs were officially registered with the Central Register of Macedonia. An official Letter of Cooperation was signed with the Regional Development Center of the Northeast Region, and an official launch of the LEADER/LAG initiative for that region was held on June 23. A meeting with the Council of Polog Mayors held about introducing the LEADER/LAG in Polog. PointPro presented the Feasibility Study on “Business Opportunities for Processing Apples” in Oteshevo, Prespa. July-September 2014: In Pelagonia, the last of three LAGs, Agro Lider in Krivogashtani, was officially registered with the Central Register of Macedonia. In the Northeast Region, the process of mobilizing the local population for the LEADER process has started with Info Days and the first round of trainings. A meeting with the Council of Vardar Mayors was held about introducing the LEADER/LAG in Kavadarci. Preparations for launch the LEADER in Polog began. October-December 2014: In the Northeast Region, four rounds of workshops for the LEADER approach took place in each micro-region. In the Polog region, individual meetings were held with representatives of each municipality. A kick-off event will take place in late January or early February 2015. January-March 2015: LEADER kick-off event in the Polog region took place in the village of Vrutok on February 24th, 2015, gathering around 120 attendees, including 24 women. In the Vardar region, the kick-off event for the LEADER initiative happened on March 19th, 2015 in the Stobi Winery in Gradsko. More than 180 local stakeholders, including 67 women, were present. In the Northeast region, LEADER trainings continued: two rounds of workshops were organized in each micro-region, gathering a total of 90 participants, of whom 46 women. In Pelagonia, official events to inaugurate the offices of each of the three LAGs were organized on March 26th, 2015. April-June 2015: Trainings in the Vardar region continued according to schedule in each micro-region, with an average of 27 participants, of whom usually half are women. The EU NIMS (New IPARD Measures) Project held a number of meetings and workshops, at which SBEP representatives and partners took part. The result is a draft LEADER measure, which will be presented to the Ministry, and the DG Agri in Brussels for accreditation. July-September 2015: In the Northeast region, the two pre- LAGs were officially registered this quarter. Osogovski Lisec covers the municipalities of Kratovo, Kriva Palanka and Rankovce, while Aber 2015 includes the municipalities of Kumanovo, Lipkovo and Staro Nagorichane. In the Vardar region, the three pre-LAG groups have started discussing the structure of each of their public-private partnerships, their respective Statutes and chose the name of the LAG. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 85 October-December 2015: In the Northeast region, the two pre-LAGs were officially registered this quarter. Osogovski Lisec covers the municipalities of Kratovo, Kriva Palanka and Rankovce, while Aber 2015 includes the municipalities of Kumanovo, Lipkovo and Staro Nagorichane. In the Vardar region, the three pre-LAG groups started discussing the structure of each of their public-private partnerships, their respective Statutes and chose the name of the LAG. January-March 2016: Representatives of all ten pre-LAGs met for the first time on February 17th to discuss mutual cooperation, sign a Memorandum of Cooperation and agree upon joint activities. In Vardar, all three pre-LAGs held their Founding Assemblies and elected members of the groups’ managing bodies. April-June 2016: SBEP supported the pre-LAGs from Polog, Pelagonia and the Northeast regions to organize fairs to promote what each region has to offer in terms of food, handicrafts, tourist attractions, etc. The Vardar region will showcase its offer in September. The pre-LAGs from the Northeast and the Pelagonia regions continued implementing their respective pilot projects with the support of SBEP. In Vardar, a couple of sessions were held to identify potential pilot projects for each pre-LAG. In Polog, one pre-LAG was officially registered, while the registration of the other is still pending. In the Vardar region, all three pre-LAGs submitted the necessary documentation to the Central Registry, awaiting official registration. July-September 2016: SBEP and AIRE supported the first Bike and Wine Tour in Macedonia, which was organized by the pre-LAGs in Vardar. The fourth regional fair, showcasing products and services from Vardar, took place from September 26-27 October-December 2016: In the project’s final quarter, SBEP organized a LAG fair in Skopje with 17 exhibitors. The project, culminating many months of work to move toward full LAG status for the pre-LAGs it fostered, “facilitated the signing of a letter addressed to the Ministry of Agriculture by mayors of 21 rural municipalities that are members of the pre-LAGs. The mayors are petitioning the Ministry to expedite the establishment of the Register of LAGs, in order to unlock the potential for these public private partnerships to access funding for small projects, which would con-tribute to regional development. The Ministry is yet expected to respond to the petition.” (SBEP, Final quarterly report). EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 86 Sub-Study 2: Grow More Corn Initiative Photos: left, TV ORBIS (Macedonia); center and right, SBEP (USAID/Macedonia). Introduction In Macedonia, the food and agriculture industry comprises about 16 percent of the country’s GDP and employs over 20 percent of the national population.6 Corn is a particularly prominent crop, as approximately 7,643 hectares are cultivated each year regionally. Historically, Macedonia corn yields are relatively low (3.9 kg/HA in 2012 when SBEP started and 4.6 kg/Ha in 2016, compared to an average of 11.9 kg/Ha in neighboring Greece). As a result, Macedonia routinely supplements its production of corn (less than 150 tons per year) with imports approximately 50,000 tons of corn annually, a portion of which was being used as feed by dairies and livestock operations. Modernizing corn production through the introduction of drip irrigation systems was identified as an opportunity by SBEP early in its life cycle. Evolution of the Grow More Corn Initiative Having discussed corn as one of a number of possible agricultural commodity import substitution initiatives, along with tomatoes and wheat which were also imported, SBEP began recruiting farmers who were interested in participating in a “proof of concept” pilot for this initiative with 40 farmers in Pelagonia. SBEP’s selection criteria focused on experienced farmers with at least 5 hectares under cultivation who would apply the protocols taught to them by SBEP’s partner, Magan-Mak, an experienced Israeli owned equipment supplier operating in Macedonia. As a companion effort, SBEP organized a comparison group of 6 SBEP Quarterly Report: January – March 2013 0 10,000,000 20,000,000 30,000,000 40,000,000 50,000,000 60,000,000 70,000,000 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Macedonia Corn Imports (HS 1005) Source: U.N. Comtrade: Net weight: kg EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 87 30 farmers in Polog to whom it provided the agricultural protocols taught to its pilot farmers in Pelagonia, but it did not supply these comparison farmers with drip irrigation systems. Following this initial year of testing, SBEP expanded the initiative to reach a larger number of farmers by providing drip irrigation systems and protocols to additional farmers in 2014 and again in 2015. In all, drip irrigation systems were installed on 281 plots, of which 278 were in the four regions where SBEP worked. The figure at right shows the distribution of these drip irrigation systems by region over the life of the project. In addition to farmers SBEP provided with drip irrigation systems, it worked with other Macedonian farmers to help them obtain these systems. Notable in this regard were 71 farmers it helped apply for loans that would constitute their payment portion under a 50/50 subsidy scheme to encourage the adoption of drip irrigation funded by the Ministry of Agriculture. As a champion of drip irrigation, SBEP encouraged the Ministry of Agriculture to keep increasing the amount it spent on this subsidy program and it encouraged other organization to join it the effort to spread the use of drip irrigation for corn and for other crops, including sunflowers. SBEP’s focus on funding for these systems led to an effort with USAID/Macedonia that resulted in the opening of a Development Credit Authority (DCA) Fund that would stand behind farmer loan applications which opened in 2015. SBEP also used a variety of novel means to promote the adoption of this new to Macedonia approach to corn farming including: • A “Corn Caravan” that organized 15 events in February of 2014 for 80 villages in Pelagonia, attaching 170 farmers • A documentary film produced by a leading agricultural journalist that aired on three regional TV stations disseminating information about increased corn production in Pelagonia in 2013 with drip irrigation and presenting testimonials about this technology from farmers and initiative partners. Beyond its own promotional activities, the Grow More Corn initiative benefitted from “earned” publicity in the form of press coverage which appeared in a variety of media forms over the life of the project. Grow More Corn Initiative Results Key program results of interest to USAID/Macedonia for this evaluation include the effects of SBEP’s initiatives on the perceptions and expectations of project beneficiaries, changes in their production and product quality and jobs created as a result of SBEP assistance. Findings on these anticipated outcomes presented in this section come from evaluation team multiple surveys and their comparative analysis as well as from other interviews and documentary sources including Macedonia’s State Statistical Office and SBEP’s own performance reports. The evaluation’s central finding for the Grow More Corn initiative is that its data confirm USAID and SBEP’s intervention “theory of change,” its early “proof of concept” test, and testimonial remarks from farmers and other partners that the application of drip irrigation systems and associated agricultural protocols would dramatically increase corn production 45% 14% 23% 17% Pelagonia Polog Vardar Northeast EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 88 where applied, raising prospects for regional development and rural population retention in Macedonia. With respect to still future goals, the evaluation’s findings also support the proposition that this initiative could potentially eliminate the need for corn imports, if its adoption continues to expand along first SBEP’s and, more currently, the Ministry of Agriculture’s intended trajectory. These results are not only obvious from interviews with farmers and program partners, they are also supported, in several instances by rigorous impact evaluation analyses that test differences between the group of farmers to whom SBEP directly provided the project’s intervention (treatment), a combination of drip irrigation and agricultural protocols and comparison group farmers to whom that intervention was not provided. The study’s impact evaluation analysis of differences found and their significance is provided in Exhibit 2.3. Perceptions and Expectations More than a few times, the evaluation team heard the proud response, “This is the future” from Macedonian farmers who had received drip irrigation systems from SBEP and see their yields and harvest production levels rise. Farmers who have used drip irrigation to grow their corn uniformly understand it merits and appreciate its benefits. They encouraged the evaluation team to let USAID and the government know that more farmers should be made aware of what drip irrigation can do. At the same time, there was a steady drumbeat in their comments that government needed to distribute these systems for free to boost their use and the gains in production, incomes and jobs that would, almost certainly, follow. (See Exhibit 2.2 which summarizes the findings of the evaluation’s survey of corn farmers and provides numerous direct comments from farmers about the initiative’s effect on SBEP as the Grow More Corn initiative’s pioneer, and a daily observer of its results for five years, sees expansion of the initiative as already being on its way to reducing corn imports, saying in its closeout report, highlighted above: “The farmers have already substituted 11 percent of the imports of this commodity, and if the trend continues, Macedonia could produce enough to fully replace corn imports by 2012, potentially creating 7,000 jobs in all supply chains dependent on corn production.” SBEP Closeout Report, 2017 In Bitola, a representative of the National Extension Agency described the “most significant change” from SBEP’s effort as being a change in mindsets: “Introduction of new technology will probably change a lot of the mentality of farmers and other stakeholders towards water resources management, including eco-friendly production in general” National Extension Agency Staff Magan-Mak, the firm with which SBEP partnered to supply drip irrigation equipment and training for farmers, described SBEP as the most successful project his firm has participated in. “There is a widespread opinion that in Macedonia there are no conditions for growing corn because of an insufficient water supply. Now, with this project we have proved they are wrong.” Magan-Mak Representative In addition to over 100 farmers interviewed and other stakeholders as represented above, the evaluation team met with the Ministry of Agriculture to obtain perceptions of SBEP, including several of its initiatives with which the Ministry was familiar. This interviewee described the “most significant EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 89 changes” resulting from SBEP has having included “the formation of LAGs and the introduction of new technology and watering measures for increasing corn production by drip irrigation.” In addition, the evaluation’s comparison of differences between SBEP’s treatment group of beneficiaries and a comparison group from within the four regions where USAID worked, as well as a second comparison group from other regions in Macedonia, excluding Skopje, showed that: • With respect to the financial situation of their farms, there was a 26 percentage point increase in the proportion of SBEP beneficiaries reporting that their farms’ financial situation was better in 2016 relative to non-participant farmers from the same regions. This effect was statistically significant at the .01 level. • There was an even larger effect, a 30 percentage point increase, relative to farmers in other regions, with respect to whether SBEP beneficiaries and other farmers were optimistic with respect to their farm’s financial prospects for the coming year, but this effect was not statistically significant. • Additionally, the study found that SBEP beneficiary farmers are statistically no more likely to make new investments next year compared to non-participant farmers in the same or other regions. These findings are presented graphically with their difference in differences regression results in Exhibit 2.3. Production Volume and Quality Under the Grow More Corn initiative, all participating farmers received the same size drip irrigation system which covered a single hectare, regardless of how large their farm or how many hectares of corn they normally plant. The consistency of the size plots was useful from an evaluation perspective as it somewhat leveled the playing field, given that SBEP beneficiary groups were not average, but were instead deliberately three to five size the average small farm in Macedonia, while comparison group farms were much closer to that norm. The table below displays these group size differences. AVERAGE HECTARES FARMED BY EVALUATION TREATMENT AND COMPARISON GROUPS Average Hectares Commercial Use Average Hectares Growing Crops Treatment: SBEP Drip Irrigation System Farmers in Four Regions (106) 22.3 20.0 Comparison A: Other Farmers in the Same Regions (184) 5.4 4.9 Comparison B: Other Farmers in Three Additional Regions (114) 3.0 2.0 Treatment Regions: Pelagonia, Polog, Vardar, Northeast Comparison Regions: East, Southeast, Southwest While MSI did not create the SBEP’s structure of regions, they have been useful for this evaluation as the SBEP regions, with 92,000 individual holdings as of 2016, or 52 percent of these individual farms. Individual holdings make up 98 percent of all farms in the country, and beyond the SBEP regions, the three regions in the evaluation’s Comparison Group B account for another 38 percent, with Skopje, which was not included I the study including the final 10 percent. In structuring regions including into the study into two distinct clusters the study team expected that it would be easier to spot differences and similarities between farmers in the same region. In practice the opposite proved to be the case. When you conduct an experiment, as SBEP in effect did, it helps if all is quiet in other parts of the lab, even if the lab is a large and well populated area. In terms of changes in corn production and output, the three regions in Comparison Group B kept doing producing in almost precisely the same way for the whole SBEP period that they had been before it started. Their timelines for corn grain and corn silage yield and EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 90 production are close to being flat, acting like a perfect laboratory counterfactual. Against the backdrop of Comparison B, the results of SBEP’s drip irrigation initiative stand out starkly. Comparison Group A data, on the other hand, tells a completely different story. Perhaps because of SBEP’s helping the government promote the 50.50 program, or simply the proximity of Comparison Group B to great farmers who are already account for a large proportion of corn production in the country, data from Comparison B Group are “noisy” in the sense that it is clear that there are things going on other than SBEP which are driving up the very indicators the evaluation used to test the effect of SBEP’s drip irrigation initiative. The evaluation team checked for but did not see a sea change with respect to the use of fertilizer, mechanization, hybrid seeds, or any type of irrigation system that would explain the extent to which yield and production results higher and rising production levels in Comparison Group than Comparison Group B. Additional graphs that show results for Comparison Group A are provided in Exhibit 2.4. Also included in these graphs are national data for corn grain and corn silage sources from Macedonia’s State Statistical Office which publishes them annually. (Thus, 2017 corn data are expected to be available in late June 2018. However, output will be down for the year due to the ‘Lucifer’ heath wave and drought in August 2017, compared to the highs for 2016’s end of SBEP reported in this volume.) As to the national average for corn grain show in Exhibit 2.4, it runs well below SBEP’s beneficiary line. The opposite is true for corn silage, where the national average runs above both the SBEP beneficiary and comparison groups and is startling floating above the other trend line. Its elevated nature is driven by extremely high corn output (tons/Hectare) in micro-regions. Bitola, for example has averaged over 30 tons/hectare for several years, compared to about 17, which is the highest average SBEP’s beneficiaries achieved collectively, even with strong gains in 2017. Individual farmers among SBEP’s beneficiaries also soared above the treatment group’s average according to SBEP quarterly reports. With all reports of high levels of corn silage production, it is important to point out that all of them, including for the Bitola municipality, are for individual holdings not cooperatives. As data from the evaluation indicate, the majority farmers in all three of the study groups grow corn silage and all and some farmers in all groups grow both corn grain and corn silage as the table below shows. Accordingly, in interviews with farmers that grow both, the study team asked them to go through a 2nd round of questions to report their experiences with each, and most complied, i.e., out of 45 farmers that grew corn grain either as their only corn crop or together, 41 provided the study team with enough data for the Grow More Corn years for the team to be able to calculate yields. This was a pleasant result as quite a few farmers voiced a degree of concern about sharing information that they did not want to end up in the hands of tax officials, on a by name basis, which the team had assured them would not happen. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 91 PERCENTAGE OF FARMERS GROWING AND AVERAGE HECTARES GROWN BY CORN PRODUCT Percentage of Farmers Growing only Corn Grain Percentage of Farmers Growing only Corn Silage Percentage of Farmers Growth Both Products Average Hectares of Corn Grain Grown Average Hectares of Corn Silage Grown Treatment: SBEP Drip Irrigation System Farmers in Four Regions (106) 19 58 23 9.3 9.5 Comparison A: Other Farmers in the Same Regions (184) 38 54 8 2.4 2.2 Comparison B: Other Farmers in Three Additional Regions (114) 6 57 37 .7 1.0 In the following subsections, MSI findings on crop yields and production for both corn grain and corn silage are presented in comparison to values for these variables for farmers from other regions that were exposed to few, if any, aspects of the SBEP Grow More Corn Initiative, perhaps except for news spots and short SBEP videos that were sometimes played on national television. (Comparison Group B). Data from farmers in these regions serve as a solid counterfactual against which to present Grow More Corn results. As indicated about additional data for Comparison Group A, non-participants living in SBEP assisted regions and government yield statistics are added back into these graphs for a more holistic view of study findings. Corn Grain and Corn Silage Production with Drip Irrigation MSI findings on crop production for both corn grain and corn silage are presented in this section. Under SBEP’s Grow More Corn initiative 218 farmers received drip irrigation systems and agricultural protocols in three annual waves of farmers that received drip irrigation and agricultural protocols from the project: Wave 1 – 41 farmers; Wave II – 80 farmers, and Wave III – 160 farmers, for a total of 218 direct beneficiaries of the Grow More Corn initiative. MSI collected data from SBEP’s treatment farmers in 2014 and again in 2016 following protocols set forth in its evaluation design (see Annex B): • In 2014, MSI collected data from all 41 of the Wave 1 farmers. The treatment group for the corn aspect of the evaluation was 1722 farmers drawn in a nationally representative sample as called for in USAID’s evaluation Statement of Work (see Annex A). Of these, 521 responded positively on a question about whether they grow corn (30 percent), and their data were then used to create the study’s comparison group to which production results in SBEP’s treatment group would be compared for the 2013 harvest. • In 2016, MSI gathered data from 106 treatment farmers that agreed to be interviewed (38 percent of the 281 SBEP Grow More Corn initiative beneficiaries). Their distribution, by wave was: Wave 1 – 14; Wave 2 – 34 and Wave 3 – 58. In 2016, MSI relocated and obtained consent to re-interview for 340 farmers in the 2014 comparison group (65 percent). The relatively small number of the 41 Wave 1 beneficiaries the evaluation team located and obtained consent to re-interview (14), or 34 percent of that cluster limited MSI’s ability to undertake some of the statistical analyses contemplated in the study design, even with a large number of repeat comparison group interviews available. Accordingly, more basic comparisons between SBEP treatment farmers and comparison group farmers are reported in this sub-study. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 92 In the analysis stage of the evaluation, MSI elected to separate its relatively large comparison group of 340 into two sub-groups for a finer grain look at differences between SBEP beneficiaries and other farmers. At this step, MSI clustered farmers living in the regions in which SBEP worked, and from which treatment group farmers into Comparison Group A (201 farmers), and farmers from non-SBEP regions into Comparison Group B (139). For a comparison group to represent the counterfactual in the sense USAID intends in an impact evaluation, it will be likely to perform over the period of an evaluation in almost the same way it perform at the start before the treatment group received the intervention the impact evaluation was designed to assess. From this perspective, Comparison Group B, comprised of non-treatment farmers from non-SBEP regions, serve that purpose well. Comparison Group B Production (t/ha) (139 farmers) Year 2013 2014 2015 2016 Corn Grain 5.0 5.7 5.7 6.3 Corn Silage 3.5 4.5 5.1 5.6 It is to this counterfactual that the evaluation team was most certain of the validity of comparing non￾treatment farmers to SBEP’s beneficiary group, for which the next table presents production data over four years for both corn grain and corn silage. SBEP Treatment Group Production (t/ha) (106 farmers) Year 2013 2014 2015 2016 Corn Grain 9.56 9.3 7.6 9.2 Corn Silage 20.0 16.9 16.4 22.9 Based on these two tables, grain corn grown with drip irrigation yielded 9.2 t/ha in 2016 compared to Comparison Group B’s non-treatment farmers from non-SBEP regions, which produced 6.3 t/ha. With drip irrigation SBEP’s performance was 46 percent higher for 2016. For corn silage, SBEP’s treatment group, at 22.9 tons/ha in 2016, was 309 percent higher than Group B’s 5.6 t/ha for 2016. On the other hand, Comparison Group A, comprised of non-treatment farmers, seemed to shadow the performance of treatment group farmers, delivering higher production levels than might have been expected. Several factors could explain this shadowing type data. On is that their results may have been driven by performance in Pelagonia, which is a top corn producer nationally. It is possible that better basic farming practices are ubiquitous across this region. A second explanation might be that there was spill over from the SBEP project to non-beneficiaries, at least in terms of agricultural protocols which were delivered at least in part by SMS messaging. In addition, SBEP treatment farmers indicated that they had been visited by other farmers who were interested in learning about what the treatment farmers were doing. Another possibility is that farmers in SBEP regions participated more heavily in the Government’s 50/50 subsidy program for promoting drip irrigation uptake. As a counterfactual, the MSI team found that results for Group A differed sufficiently from expectations with respect to the likely performance of a comparison group unaffected by an intervention being studies, and the absence of a clear and validated reason for this resulted in the evaluation team’s preference for Comparison Group B as the impact evaluation’s functional counterfactual. Thus, while data for both are presented, the team suggests that USAID also treat Comparison Group B as the appropriate data set to used when making comparisons to the SBEP treatment group’s performance over the full project term. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 93 Comparison Group A Production (t/ha) (201 farmers) Year 2013 2014 2015 2016 Corn Grain 8.8 8.3 7.2 10.1 Corn Silage 11.4 11.3 13.0 14.6 To better understand the performance of SBEP’s corn production performance compared to that of comparison groups and to see how results from SBEP’s pilot study compare to MSI’s findings, the study team has put the full picture together for the year 2013 for which four sets of data are available, namely: • SBEP’s pilot involving 40 farmers from Pelagonia where its Wave 1 farmers received drip irrigation and agricultural protocols; and 30 comparison farmers in Polog who received only agricultural protocols. • MSI’s 2013 data for only SBEP’s Wave 1 beneficiaries (41), and MSI’s comparison for that year to its full set of 521 comparison group members who grew corn in 2013. • MSI treatment group data for 2013 as provided in 2016 by 106 SBEP treatment group farmers from all three Grow More Corn initiative beneficiaries. • MSI Comparison Group B (139 non-treatment, not SBEP region farmers) and Comparison Group A (201 non-treatment, SBEP region farmers) for 2013 as provided in 2016. These data are presented side by side in separate table for corn grain and for corn silage, standardized to the 2013 harvest for all groups. In this table, MSI also shows the percentage increase realized by farmers using drip irrigation and SBEP’s agricultural protocols over farmer who did not receive that combined intervention. Group Corn Grain – 2013 Average Production (t/ha) 2013 Without Drip Irrigation and Agricultural Protocols Average Production (t/ha) 2013 With Drip Irrigation and Agricultural Protocols Percent Increase in Production Between Non-Treatment and Treatment Farmers SBEP pilot treatment (40), comparison (30) 6.3 11.4 80.9 MSI 2014 survey (treatment [41]; comparison B [521]) 5.7 18.71 222.8 MSI 2016 survey (treatment [106]; comparison B [139] to other farmers in non￾SBEP regions) 5.0 9.56 91.2 MSI 2016 survey (treatment [106]; Comparison A [201] to other farmers in SBEP regions)* 8.3** 9.56 15.1 * High comparison A group outcome may be driven by Pelagonia’s normally higher-than-national-average performance, spill-over of SBEP agricultural protocols, or higher-than-national-participation in the government’s 50/50 drip irrigation program in SBEP regions. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 94 Group Corn Silage - 2013 Average Production (t/ha) 2013 Without Drip Irrigation and Agricultural Protocols Average Production (t/ha) 2013 With Drip Irrigation and Agricultural Protocols Percent Increase in Production between non￾treatment and treatment farmers) SBEP pilot treatment (40), comparison (30) 38.6 63.6 64.7 MSI 2014 survey (treatment [41]; comparison B [521]) 5.6 19.7 251.8 MSI 2016 survey (treatment [106]; comparison B [139] to other farmers in non-SBEP regions) 3,54 20.00 464.9 MSI 2016 survey (treatment [106]; Comparison A [201] to other farmers in SBEP regions)* 11.37 20.00 75.9 * High comparison. A group outcome may be driven by Pelagonia’s normally higher than national average performance, spillover of SBEP agricultural protocols, or higher than national participation in government 50/50 drip irrigation program in SBEP regions. Job Creation Job creation, USAID and SBEP’s top outcome level priority for the project as a whole, is also a key aim of the country’s balanced regional growth strategy, which underpinned government’s close collaboration with a number of aspects of the SBEP program and its institutionalization and continued support for aspects of several initiatives, including Grow More Corn. The number of employees that farms examined through this evaluation had differed almost directly in relation to differences in the size of theses farms. Comparison group farmers were drawn from a representative national sample survey population USAID had requested be used when developing the baseline for the evaluation, whereas treatment group farms were deliberately select to be five hectares or more and be run by experienced farmers, as SPEP’s aim was to learn what was the most that introducing drip irrigation system would yield in terms of desirable outcomes. As a result, most comparison group farms were small, with less 77 percent of comparison farmers in the regions SBEP worked having five hectares or less and 82 percent of comparison farmers from other regions falling in that same cluster, compared to only 13 percent of SBEP drip irrigation beneficiaries representing that low end of the farm size scale. In employment terms, SBEP beneficiary farms had, on average, from two to two and a half more employees, in full time equivalent terms, than did beneficiary firms, as the table below shows. Most of this additional labor appears to have fallen into a part time labor category (not displayed here). One farmer described some of these workers as being “part-time all year round,” and the data on family members who worked for farms suggests some part time workers also fell in this category. Among SBEP beneficiary farmers, 42 percent indicated that family member workers were paid, as did 51 percent of comparison farmers from other regions, while only 18 percent of comparison group farmers from SBEP regions reported that they paid family member workers. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 95 Number of Employees Family Members Full Time Employees Seasonal Employees Percent Female Percent Youth Treatment: SBEP Drip Irrigation System Farmers in Four Regions (106) 5.7 2.6 1.6 1.6 30.0 25.7 Comparison A: Other Farmers in the Same Regions (184) 3.2 2.7 1.9 .7 37.5 18.5 Comparison B: Other Farmers in Three Additional Regions (114) 3.7 2.9 1.2 1.5 37.2 19.0 Youth: Under 30 years of age With respect to farm employment changes over time, the evaluation used two study sources: the 2014 baseline survey it had conducted and 2016 survey answers from SBEP beneficiaries, in addition to examining SBEP job creation performance indicator tracking reports which gave the figure 644 for the number of new jobs created by the Grow More Corn Initiative. With respect to new jobs created, 6.6 percent of SBEP drip irrigation system beneficiaries reported that the number of people they had increased over the past 2-5 years. MSI’s statistical analysis of differences between SBEP’s beneficiary group and comparison farmers within and beyond the regions where SBEP worked (see Exhibit 2.3) indicated, however, that the increase in jobs among SBEP beneficiary farms were modest and not statistically significant when compared to the nonparticipant farms in the same regions over time. When compared to nonparticipant farms in other regions, there is a decrease in jobs, but this effect is not statistically significant. In this regard it is notable that 7.1 percent of comparison farmers in regions where worked reported increasing their workforce over the same period, as did 11.4 percent of comparison farmers in other regions. Nevertheless, these and other study data indicate that SBEP beneficiary farms saw an increase of 1.47 total workers relative to other farmers in the same regions as SBEP worked. However, this change was not statistically significant. When considering the net impact of initiatives like Grow More Corn, it is important to recognize that new jobs created are only part of the employment story of such efforts. Job losses need to be considered in tandem with new jobs created to form an accurate basis for projecting an indicative’s impact on employment, or from USAID/Macedonia’s and its country partner’s perspective, on reducing rural unemployment and associated outmigration in the direction of Skopje. Important that regard is the fact that 5.7 percent of the SBEP beneficiary respondents to the evaluations 2016 survey reported job declines, as well as increased employment. With 6.6 percent of those farmers reporting employment gains and 5.7 percent reporting losses over the same period the initiative’s employment effect would appear to drop considerably, to a net positive 0.9 percent change in employment, whatever that translates into on a firm-by-firm hiring and staff loss basis. Reasons for losing employees (offered during open-ended question sections of the 2016 survey and responses, some of which are included at the end of Exhibit 2.2) describe family members becoming too old to work, youth taking off to see what the big city is like, daughters bringing new husbands to work with families on their farms, and many more, including more work to be done in some cases. Further evidence of the importance of considering job losses as well as job gains when assessing the impact of activities that aim to increase employment came from MSI’s comparison of farm employment data in its 2016 to answers from farmers, at least some of whom also participated in the study’s 2014 survey. Compared to the average of 5.7 employees reported by 106 respondents to the 2016 survey, the average number of employees given by the 77 respondents (40 in the drip irrigation group and 30 or so in the SBEP-organized Polog protocols only group) in answering the 2014 survey was 6.28, which indicates a net loss of .58 jobs on farms SBEP helped grow corn over that two-year period. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 96 To try to understand the actual new job implications of its findings, MSI applied these findings to the 400 farms SBEP reports it assisted in growing corn, including the roughly 300 SBEP directly with drip irrigation system, as well as those it helped find loan financing for the 50/50 program, or with farming practices in the first year’s Polog “protocols only” comparison group mobilized along the first 40 drip irrigation recipients. The implications were mixed. If all 400 corn-growing farms SBEP assisted actually hired the 1.47 new employees MSI’s analysis suggested that some might have, the net job gain would have been 588. But if only 6.6 percent of those farmers added one employee each the number of new jobs that emerged would drop to 35, and this does not factor in evidence acquired indicating job losses occurred on beneficiary farms over the years of SBEP’s involvement with the Grow More Corn initiative. While a number of the evaluation team’s interview questions were fairly intrusive, we did not ask farmers to give us a head by head accounting of the exact number of employees that left or were hired. This left the evaluation with a good understanding of important factors influencing net employment from this initiative, but unable, without asking for far more detailed project records, to pinpoint precisely the initiative’s net employment effect, or fully verify the project teams reported number of new jobs. Grow More Corn Initiative Sustainability and Scalability Institutionally, the Grow More Corn initiative and its expansion is centered in Macedonia’s Ministry of Agriculture which announced a shift in government funding for drip irrigation systems from a 50/50 to 80/20 basis. As reported by local press, subsidies for drip irrigation will be sustained in a new VMRP￾DPMNE program aimed at agricultural development, for which 3 million euros will be allocated annually. AIRE and other stakeholders in SBEP’s effort are also reportedly guiding the Grow More Corn initiative into a viable permeant space within the Ministry. In interviews, respondents expressed that the continuation of government subsidies and better terms for them, if not simply making drip irrigation systems free, was on the minds of most farmers. Financial mechanisms through which farmers can participate were also discussed with sustained availability being the focus of thinking in the ministry. If results are to be sustained, and uptake expanded, more than funds will be required, as SBEP’s equipment supplier and main trainer, Magan-Mak explained in the text box on this page. Beyond finance and training, demand is critical. Interview data create a sense that farmers are a mix with respect to their awareness of drip irrigation, the government’s subsidy program and the Grow More Corn Initiative, despite reportedly extensive advertising. Those who do and others were interested in learning more, while others did know what this type of system was, but saw it as unaffordable in study interviews that occurred a month or two before the 80/20 modification to the government’s program was announced. SBEP’s supplier held something of a similar view, reporting that he was aware of a level of “spread effect” from the initiative, but that it was not extensive. Equipment Alone Is Not Enough Education must go together with the equipment we are selling. [Under SBEP} we provided general training for the farmers to explain them how it functions and what they need to do to prepare the land for installation. When the team did the installation, they gave them instructions about the operation and maintenance. Later, the communication was organized through SMS messages and FB. We also followed the weather conditions and development phases of plants. For example - the corn in that region should have 12 leaves and receive this or that amount of water. We gave them advice when to start applying the fertilizers and kind of fertilizers to buy. Also through the FB group they communicated among themselves. Social networks help us a lot in the communication. — Magan-Mak Representative EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 97 “We are not the only producers of this equipment. We also have competition from Greece. Our documentation shows that currently 700 hectares are covered with drip irrigation systems. If you subtract the 280 SBEP provided, that’s a 420 hectare “spill over” effect from the project.” — Magan-Mak Representative Similarly, the Ministry of Agriculture indicated that there was some uptake through the then 50/50 program, but that it was not yet as large a demand as the Ministry hope this technology would stimulate. In its close out report, SBEP talked encouragingly about being 11 percent toward the goal of eliminating corn exports through advances in the uptake of drip irrigation systems. But 89 percent is still a very long way to go. Getting to scale, with any technology, starts when critical mass is reached, at the point in the curve below where people reach the point where something in which they have a passing interest, or even becomes a product they sincerely desire to own, but still do not move at all do speed to acquire, becomes a “must have.” Cell phones are the most obvious current example this, and over 50 percent of Macedonians now carry one around. Study data show that a segment of farmers already know that using drip irrigation works for them in bottom line terms, and requires less time and water than other options. Government, in the evaluation’s interview was focused on finding what would turn on demand and bring it to a tipping point where the rest of the farming sector will sell itself on adoption and utilization. The shift to 80/20 subsidies could contribute, or not. SBEPs supplier too is looking for the key to unlocking demand. But as interviews in which sustainability and scaling-up were a focus of the discussion showed, none of these stakeholders was convinced they had yet found the way to achieve the lift off that would be needed. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 98 Exhibit 2.1: Corn Initiative Chronology • October-December 2012: First four corn farmers for pilot project with corn fields under drip irrigation are identified and signed Memorandums of Understanding. • January-March 2013: 40 farmers selected for drip irrigation pilot. Magan- Mak awarded contract for drip irrigation package. • April-June 2013: Planting of the corn on the demo fields and installing the drip irrigation systems. The 40-plus farmers who make up the pilot groups in Polog and Pelagonija received trainings and consultations about how to grow corn with the drip irrigation system by MaganMak. A total of 42 hectares of land, on 45 different plots, were covered in Polog and Pelagonija • July-September 2013: Initial results from the Grow More Corn campaign’s 2013 season show yields have increased dramatically with potential for large-scale effects in the corn and dairy industries. The preliminary results gathered from the 45 farmers in this pilot group show a 36% increase in yields of corn grain, from an average of 7.9 tons/hectare to 10.8 t/ha, and 30% increase in the yields of silage, from an average of 44.7 t/ha to 58.3 t/ha. • October-December 2013: Corn harvest in Polog and Pelagonija completed. • January-March 2014: 80 corn farmers were identified for this year’s Grow More Corn initiative, in four regions: Polog, Pelagonija, and the Northeast and Vardar regions. A Corn Caravan was organized in Pelagonija. MK Government officially approved a “50/50 measure” for subsidizing drip irrigation for growing corn in the National Program for Financial Support of Agriculture for 2014 - $111.000 allocated. • April-June 2014: Drip irrigation equipment was delivered to the 80 new farmers in this years’ program. Promotional events were organized in all four regions. • July-September 2014: SBEP and PointPro finalized a policy paper on corn “White Paper” and shared it with governmental decision-makers. • October-December 2014: Drip irrigation equipment procured for 160 hectares across the four regions where SBEP operates. SBEP identifies a new group of farmers to participate in the 2015 Grow More Corn program • January-March 2015: SBEP organized educational sessions with the 160 new farmers participating in the 2015 season of the Grow More Corn initiative. The Ministry of Agriculture announced a new round of subsidies for growing corn and sunflower with drip irrigation, allocating approximately US$280.000 (15,000.000MKD) under the so-called “50/50” measure. • April-June 2015: SBEP assisted 39 farmers from across Macedonia to apply for the 50/50 measure. • July-September 2015: Development Credit Authority (DCA) Guarantee Fund established, which helps farmers to access funding through commercial banks, where 50% of the loan will be guaranteed by the Fund. 28 farmers successfully received the subsidies, for a combined 103.9 hectares, amounting to a total investment of 9.224.951 MKD, i.e. US$ 163.273. Since the beginning of SBEP’s Grow More Corn initiative, more than 300 farmers across Macedonia have started using drip irrigation for corn, sunflower and other forage cultures, on nearly 700 hectares. • October-December 2015: SBEP organized a number of educational sessions in six regions across Macedonia, gathering some 150 farmers interested in learning more about the drip irrigation for corn and how to apply for government subsidies for buying the equipment. SBEP provided support to the Ministry of Agriculture in preparing a set of policies and supporting activities related to the so-called 50/50 measure. • January-March 2016: SBEP initiated and organized the first ever meeting of representatives from the institutions involved in processing the farmers’ applications: the Ministry of Agriculture, the Ministry of Environment and the Paying Agency. The parties agreed to improve the flow of EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 99 information to streamline the process and urged for an integrated digital system to be put in place. • April-June 2016: SBEP began cooperation with the Swiss funded Nation Conservation Program implemented by Pharmachem, on a joint project for growing rice with drip irrigation. • August-September 2016: SBEP continued to monitor the Grow More Corn initiative, and participated in several events. Reports released by Israel’s top supplier of irrigation equipment cited SBEP’s work in Macedonia as exemplary and encouraging. • October-December 2016: A one-day fair in Mogila highlighted the impact of the Grow More Corn initiative on that community and “The National Program for the Sup-port of Rural Development was announced in October, with nearly US$ 3.3 million allocated for the purchase of drip irrigation equipment.” EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 100 Exhibit 2.2: Corn Farmer Survey Summary The first part of this survey asked about the respondents’ farms. The second part asked about the respondents’ awareness, experience and perceptions of SBEP. In 2016 the evaluation team interviewed 106 recipients of SBEP assistance including single hectare size drip irrigation systems and protocols as well as farmers living in the same regions (184) and other regions (114) who had not received this assistance. These end line interviews followed up on data collection in 2014 with 77 SBEP assisted drip irrigation farmers and a national random sample of farmers that constituted a comparison group for USAID’s proof of concept drip irrigation assistance intervention known as Grow More Corn. Characteristics of the intervention treatment group at the end line are provided in the text box to the right. Observations on the comparison group farmers profile are noted only where they differ from those of the treatment group. Treatment group farmers that began using drip irrigation in 2013 or 2014 who were interviewed in 2016 had also been interviewed in 2014; whereas those that received drip irrigation systems in 2015 were interviewed for the first time in 2016. All comparison group interviews carried out in 2016 were with farmers who had also be interviewed in 2014. Growing Corn • 91% of treatment group farmers were growing corn in 2016, as were 82% - 91% of comparison farmers. • Farmers who were not growing corn in 2016 did not do so because of family issues or because of the work or water required, or because another crop paid better. • Many treatment farms grew only corn silage (58%) in 2016, which is about 10% more than was seen in 2014, while 19% grew only corn grain in 2016 and 23% grew both. Comparison groups also favored growing only corn silage in 2016 (54%-57%), with other comparison farms in SBEP regions favoring growing corn grain exclusively, and those in non-SBEP regions favoring a mix. Employment on the Farm • 93% of treatment farms employed less than 10 people in 2016, and the share for comparison farms is slightly higher at 94%-98% • Farmers on both treatment and comparison groups said that employment levels on their farms had changed over the past 2-5 years o 7% of treatment farms said employment increased, as did 7%-11% of comparison farms o 87% of treatment farms said employment stayed the same, and so did slightly fewer comparison farms PROFILE OF THE EVALUATION’S CORN FARMER SURVEY RESPONDENTS (TREATMENT FARMERS - 106; COMPARISON FARMERS – 298) Treatment group farmers who received drip irrigation systems from the SBEP Grown More Corn initiative are men (94%) who own part of the land they farm and rent the rest, while comparison groups farmers are more likely to be owners. Treatment group farms are larger (with around 50% holding 6-20 hectares, and another 12% holding 1-5 hectare plots. Among comparison group farmers, 25% farm less than 1 hectare, while 56% have 1-5 hectare plots. Treatment group farmers are slightly younger, on average, than comparison farmers, among whom the average age is higher because of a larger proportion falls into the 60 year + age group. Notably treatment group farmers are also somewhat less likely to have a university degree. Treatment group farms are more likely to be registered 88%, compared to 40-45% in the comparison group. The vast majority of farms in both clusters (92-94%) employ less than 10 people. Farmer membership in associations or cooperatives was low in both groups, and awareness of economic development plans for their area was below 50% in both groups, though higher among treatment than comparison group farmers. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 101 o Comparison farms in SBEP regions were more likely to report declines (11%) than were treatment or other comparison farms (closer to 5%) • Over 50% of employees on treatment and comparison farms in SBEP regions are full time staff, though slightly fewer are in non-SBEP regions • Seasonal employees account for 40% on treatment farms, and between 22% and 49% on comparison farms • 27% if treatment group employees were youth (under the age of 29) in 2016 as were 19% of employees on comparison group farms • 47% are family members on treatment farms, while the percentage on comparison farms is higher (87%-94%) • Between 40% and 50% of family member staff are paid, though that figure was lower for comparison group farms in the SBEP regions than in non-SBEP region comparison farms • Close to 7% of farmers in SBEP regions, on both treatment and comparison farms, reported that the number of people working on the farm had increased over the past 2-5 years; in non-SBEP regions the figure was higher at 11%. Use of Modern Farming Equipment and Methods • On treatment farms, fertilizer was used for growing corn grain by 43% of farmers. 34% had been doing this before 2012, and the rest began using fertilizer since then. Slightly fewer comparison farms started using fertilizer for corn grain as early as did treatment farms. • All treatment group farmers said they use fertilizer for corn silage, as did virtually all comparison group farms. 58% of treatment group farms had started using fertilizer for corn silage before 2012, which was close to the same as comparison farms reported. • All treatment farmers and over 90% of comparison group farmers use mechanization for farming corn grain, with 60% to 75% of farmers in SBEP regions doing so before 2012, and about half that percentage doing so early in non-SBEP regions. Similarly, mechanization is used by more than 95% of all farmers for growing corn silage, which was adopted before 2012 by 40%-60% of all farmers interviewed. • All treatment farmers and 80% of comparison farmers use high-yield seeds when growing corn grain, and 75% of treatment farmers have done so since before 2012, while a lesser share of comparison group farmers started doing so that early. The percentages of treatment and comparison farmers that use high-yield seed for growing corn silage is only slightly lower, and again a somewhat larger share of treatment farmers (51) did this before 2012, with the rest adopting this practice since then. • In 2016, 72% of treatment farmers said they use drip irrigation to grow corn grain, and 42% said they do so for growing corn silage. Surface irrigation and sprinklers were also reportedly used, but less frequently (around 20-%), whereas surface irrigation was more commonly used among comparison farmers, whose use of drip irrigation use was much lower, at around 12%. Subsidies • 80% of treatment group farmers indicated that they received some type of subsidy for planting corn, as did 40% of comparison farmers in SBEP regions and 53% in non-SBEP regions. • Over 50% of all farmers described subsidies for corn farming as being easy or very easy to come by. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 102 Farming Assistance from External Sources • 83% of treatment group farmers said they had received technical advice or assistance from government or foreign donor organizations to improve the volume or quality of their farm production and sales, compared to 13% or fewer comparison group farmers. • Asked whether they knew about a U.S. Government program called the Small Business Expansion Program or SBEP, 94% of treatment farmers said they did, as did 27% of comparison farmers in SBEP regions, and 13% in non-SBEP regions. Among those who knew of the program, virtually all identified it by the name USAID, rather than the project name. • The year in which farmers became aware of SBEP varied considerable o 29% of farmers first knew of SBEP in 2012, as did 10% of comparison farmers in the SBEP region o Another 28% of treatment farmers and 26% of comparison farmers in the same region learned about SBEP in 2013. Only two farmers from other regions had heard of it by then. o Another 14% of treatment farmers and 16% of comparison farmers in SBEP regions knew by 2014. Another four farmers from other regions also knew of it by then. • Separately, farmers were asked if they knew of a national program called “Grow More Corn”, to which only 25% said they did. The Grow More Corn Initiative was then described to farmers and they were asked if they understood the ideas behind it, which 95% of treatment farmers and the majority of comparison farmers did. [This suggests that they may know it by another name or simply associate it with SBEP and USAID.] Record Keeping • Responding to a question farmers were asked whether they keep written planting, harvesting and sales records on a crop by crop basis or for the farm as a whole, 25% of treatment farmers said they keep records for the farm as a whole, but only 13% did so on a crop by crop basis. Among comparison farmers, less than 20% keep either crop by crop or whole farm records in a written form • As to whether they used these records to answer questions in this survey, less than 4% of farmers in SBEP regions said they had done so. In contrast 27% of comparison farmers from non-SBEP regions had provided some of their answers based on their written farm records. [This was the same group of farmers that was most likely to have said that they keep crop by crop farm records.] Corn Production and Sales • Data from the survey, which are reported through graphs in this sub-study indicate confirm SBEP reporting with respect to the impact of drip irrigation on corn production and yields, and quotations at the end of this survey summary describe the impact of these changes on farm income and job creation. • For many farmers assisted by SBEP, drip irrigation has double their corn output, and in some cases driven it even higher as reports of SBEP assisted farmers winning completions (described in SBEP quarterly reports) attest. Use of Corn Grown on the Farm for Animal Feed • Asked whether their use of corn from their own farms as animal feed had changed in the last 2- 5 years, 43% of treatment farmers said their use of their own corn for that purpose had increased, compared to between 15%-30% of comparison farmers. Between 50%-60% of all EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 103 farmers interviewed said that the level of their use of corn they grow to feed animals had stayed the same over this period. • Among farmers who purchase feed for their livestock, close to 75% or more say they purchase it from local sources in the region where they live. Prospects for Wider Use of Drip Irrigation Systems • 89% of treatment farmers indicated that the drip irrigation systems they had received from SBEP covered one hectare of land • 18% said that they had added on to these systems to cover more land • Among farmers who have increased the size of the drip irrigation systems SBEP gave them, 69% said they had done so using farm savings, while 18% said they had obtained a loan to pay for increasing the size of their drip irrigation systems, and 13% said their system expansion was partially subsidized by the government. • On a question that asked farmers whether other farmers come to visit them to observe their farming methods and use of modern farming techniques, including fertilizer, mechanization, high yield seeds and drip irrigation, 75% of treatment group farmers said that such visits had occurred, compared to between 2% and 20% of comparison group farmers who responded to this question. • Based on their knowledge of SBEP’s Grow More Corn initiative, including what others say about it, treatment farmers were asked how likely they thought it would be that drip irrigation and related agricultural planting protocols would be adopted by increasing numbers of farmers even after SBEP ends. Overall 80% thought it was very likely (52%) or somewhat likely (29%). Farm and Family Income and Prospects for the Future • Among treatment group farmers, who own or use a larger number of hectares than do comparison group farmers, the profile of respondents in the text box on this page explained, 70% described earnings from the farm as their only source of family income, compared to comparison group members, many more frequently said that the farm accounted for 50%-75% of their family’s income. • Asked to compare the financial situation of their farms to the previous year, 45% of treatment group farmers said their farm’s situation had improved, compared to 14%-30% of comparison group farmers reporting on doing better in the current year. Most other farmers reported that their farm’s financial situation had remained the same, though between 18%-24% of comparison group farmers said things had gotten worse. • Asked to look forward to next year, 55% of treatment farmers said they expect that things will improve for their farms, compared to 15%-44% of comparison group farmers, among whom those from non-SBEP regions were most optimistic. • As part of the farmer survey, respondents were asked what were most likely to have an impact on the future prospects of their farms. The percentage of farmers saying that each of the factors asked about was “very important” for their farm’s prospects are shown in the table below, ranked in order of treatment group responses EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 104 Factors Affecting Future Business Prospects Treatment Farmers Comparison Farmers in SBEP Regions Comparison Farmers in Other Regions General economic conditions in Macedonia 95% 85% 80% Macedonian government policies 89% 77% 67% Availability of subsidies for businesses/farms 88% 83% 75% Price of business/farm inputs 87% 61% 66% Foreign investment in Macedonia 84% 55% 60% Business/farm access to international markets 84% 21% 50% Business/farm access to domestic markets in throughout Macedonia 84% 21% 50% Access to finance 82% 66% 72% Business/farm cost of production/service delivery 78% 53% 53% Economic conditions in the EU and other large economies 77% 46% 41% Macedonian government leadership 74% 73% 67% Business/service production levels 73% 47% 31% International product standards 69% 43% 55% Farmer Investments in their Farms • Over the past 2-5 years, 70% of treatment group farmers said they had made investments in their farms, as had farmers from non-SBEP regions (65%). Fewer comparison farmers in SBEP regions (38%) did so. • As stated above, 18% of treatment farmers specified that one of their recent investments had been to expand the size of the drip irrigation system SBEP had provided to them. • Asked whether it was likely that they would invest in their farms, beyond normal operating levels, in 2017, 53% of treatment farmers said that it was very likely (22%) or somewhat likely (33%) that they would. By comparison, 29% of comparison farmers in SBEP regions and 43% of comparison farmers in non-SBEP regions said they expected to invest in their farms in 2017. Most Significant Change When asked what they thought was the most significant change that had come from their involvement in SBEP’s Grow More Corn initiatives, treatment group farmers gave a variety of responses, of which the following are illustrative: • Yield is higher with less work, but it requires better hydro systems • With drip irrigation I increased production and bought it for two additional hectares • USAID program inspires me to increase the surface for the processing of corn • This is the future, but with its own water source and continuous pressure • This program contributed to increase the development of my farm using another seed cord for food and feed • The most significant change for my farm was the application of new fertilizers, mechanization, new seeds and the application of drip irrigation for other crops that contribute to increase the income of the farm • SBEP is a good program and a lot of farmers need to use it because you really see the benefit – it facilitates work and increases earnings • Tips on fertilizer and sowing • Using USAID program and the equipment that I got made me buy more land using drip irrigation system • If it is not free, no one will use it EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 105 • More efficient way of growing, economically, and the opportunity for young people to have their own business • If we have the financial capabilities it would be better in the future to increase the surface drip irrigation system • We plan to buy drip irrigation because we increase the income • Device and technical assistance, the method of processing and yield increase • Need to invest in irrigation equipment to maximize yield; best irrigation for any culture but the standard in Macedonia is low and this should be provided free of charge • This is the future, lower costs by 20%, higher yield, in season gets time for another job • We are satisfied with the collaboration with USAID; this organization helped us a lot. Farmer Recommendations for Expanding Adoption of Drip Irrigation in Macedonia While making drip irrigation systems free to all farmers was the most frequent response to this question, farmers the evaluation team interviewed also had a number of other suggestions for USAID, the government and other programs that promote the use of this method: • More farmers need to know that the corn treated this way will give more • It is very difficult to find these things, there is no information; the drip system should be more accessible • Government should subsidize this system and trainings for farmers • Fields should be visited more often and every farmer should be informed about that • Systems should be provided to farmers equally, not selectively • More meetings with farmers • Farmers from small towns and villages should get together an invest in drip irrigation machines • Bigger promotion for the drip irrigation system is needed • The selling price should be reasonable for the systems and, of course, there should be subsidies for cultivating corn • In our region the drip irrigation system was not so promoted, farmers should be informed and there should be bigger initiative for its application in a region like ours • Bigger attention to the regions in which drip irrigation is not so used • More promotion in small towns and rural areas • Additional technical assistance and help from the Ministry of Agriculture • Tax incentives for investing in drip irrigation • More farmer education • Better inspection I the agricultural areas where these systems are installed • Improving water pressure and bigger sub-interventions for drip irrigation systems • High quality seed and mechanization assistance to bring water in some areas where there is no water; improve the quality of pipelines • Affordable drip irrigation systems that cover larger areas EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 106 Exhibit 2.3: Analysis of Differences between Treatment and Comparison Group Results Of the six SBEP initiatives examined through this evaluation, only the project’s “Grow More Corn” Initiative was selected for attention through a formal “impact evaluation” as defined in the Agency’s Evaluation Policy. For the study’s “impact evaluation” of the corn initiative, a Difference-in-Differences (DID) model was used to analyze the change in outcomes between the treatment and comparison groups, before and after the intervention. The “first difference” in this method is the difference before the intervention (baseline) and after the intervention (endline). The “second difference” is the difference between the beneficiary group (treatment) and nonparticipant group (comparison). Thus, two rounds of data are required. The benefits of this model is that it removes biases in endline comparisons between the treatment and comparison group that could be the result from inherent differences between the two groups, as well as biases from comparisons over time that could be the result of trends. Given the non￾random treatment allocation, however, this model cannot account for time-variant characteristics that might affect the treatment and comparison group differently, and relies on the parallel trend assumption, that changes in outcomes between the treatment and comparison group would have been identical in the absence of the intervention. Within this framework, the following DID specification was used: 𝑌𝑖 = 𝛽𝑜 + 𝛽1𝐷𝑖 + 𝛽2𝐼𝑡 + 𝛽3𝐷𝑖 𝐼𝑡 + 𝛿𝑋𝑖 + 𝜀𝑖 Here, 𝑌𝑖 is the outcome of interest, 𝐷𝑖 is the treatment status dummy variable, 𝐼𝑡 is the endline period dummy variable, 𝐷𝑖 𝐼𝑡 is an interaction term of treatment status and endline period. 𝐷𝑖 captures possible differences between the treatment and comparison group prior to the intervention, 𝐼𝑡 captures aggregate factors that would cause changes in the outcome over time even in the absence of the intervention. Thus, the coefficient of interest, 𝛽3, is the treatment effect after controlling for time-trends and initial differences between the treatment and comparison groups. 𝑋𝑖 is a matrix of relevant covariates to increase the efficiency of 𝛽3. Specifically, 𝑋𝑖 contains the following farm-level variables: size of farm, farm registration, proportion of farm income from total income, and whether past investments have been made. The model comparing the treatment group to the larger comparison group also includes the following individual-level covariates: gender, age, and awareness of economic development plans. Control for regions and treatment waves were also included in all models. Standard errors were adjusted for clustering at the region level. FIGURE 2: DIFFERENCE-IN-DIFFERENCES METHOD Source: Khandker, Koolwal & Samad (2010) EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 107 Findings The DID model specified above was conducted for different outcome measures comparing the treatment group to the comparison group with farmers in the same regions, and separately comparing the treatment group to the larger comparison group with farmers in other regions. The findings are shown below. Job Creation Increase in jobs among SBEP beneficiary farms were modest and not statistically significant when compared to the nonparticipant farms in the same regions over time. When compared to nonparticipant farms in other regions, there is a decrease in jobs, but this effect is not statistically significant. The two figures below show the treatment effect (square marker) and the 95% confidence interval (bars) for the three different outcomes on job creation. For example, Figure 2 shows that SBEP beneficiary farms saw an increase of 1.47 total workers relative to nonparticipant farmers in the same regions. However, this effect is not statistically significant since the 95% confidence interval crosses 0. FIGURE 3: TREATMENT EFFECTS ON JOB CREATION, COMPARED TO NONPARTICIPANT FARMERS IN SAME REGIONS FIGURE 4: TREATMENT EFFECTS ON JOB CREATION, COMPARED TO NONPARTICIPANT FARMERS IN OTHER REGIONS 1.47 -1.61 2.05 -10 -8 -6 -4 -2 0 2 4 6 8 10 Total workers Total full-time workers Total paid employees -3.14 -2.42 -2.58 -10 -8 -6 -4 -2 0 2 4 6 8 10 Total workers Total full-time workers Total paid employees EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 108 Perceptions and Expectations With respect to the financial situation of their farms, there was a 26 percentage point increase in the proportion of SBEP beneficiaries reporting that their farm’s financial situation was better in 2016, relative to nonparticipant farmers in the same regions. This effect was statistically significant at the 0.1 level. This effect is even larger, a 30 percentage point increase, relative to farmers in other regions. SBEP’s “Grow More Corn” beneficiaries were also the most optimistic with respect to their farm’s financial prospects for the coming year, but this effect is not statistically significant. SBEP beneficiaries are no more likely to make new investments next year compared to nonparticipant farmers in the same regions or in other regions. The two figures below show the treatment effect (square marker) and the 95% confidence interval (bars) for the three different outcomes on perceptions and expectations. For example, Figure 4 shows a 14 percentage point increase in the proportion of SBEP beneficiary farms who believe their farm’s financial situation will improve in the next 12 months relative to nonparticipant farmers in the same regions. However, this effect is not statistically significant, since the 95% confidence interval crosses 0. FIGURE 5: TREATMENT EFFECTS ON PERCEPTION AND EXPECTATIONS, COMPARED TO NONPARTICIPANT FARMERS IN SAME REGIONS FIGURE 6: TREATMENT EFFECTS ON PERCEPTION AND EXPECTATIONS, COMPARED TO NONPARTICIPANT FARMERS IN OTHER REGIONS 0.26 0.14 -0.02 -0.60 -0.40 -0.20 0.00 0.20 0.40 0.60 Better financial situation (2016 vs last yr) Financial situation will improve next 12m Likely new investment next year 0.30 -0.02 -0.09 -0.60 -0.40 -0.20 0.00 0.20 0.40 0.60 Better financial situation (2016 vs last yr) Financial situation will improve next 12m Likely new investment next year EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 109 Yields The effect on yields is rather difficult to determine due to missing responses to these questions which results in a small sample size. Moreover, given the non-random assignment to treatment waves and the staggered years when each treatment wave started receiving drip irrigation, it is not possible to conduct the DID analysis for the entire treatment group. Instead each treatment wave was compared to the overall comparison groups. The results below show that there have been small but statistically significant increases in yields for the SBEP beneficiaries. Yields from the year after drip irrigation was started are statistically significant for all three treatment waves. SBEP beneficiaries from wave 1 saw an increase of 0.49 tons/hectare compared to nonparticipant farmers the year after they adopted drip irrigation and an increase of 0.85 tons/hectare after the second year. These results are similar when compared to nonparticipant farms in other regions. The 1-year effect is similar for SBEP beneficiaries from wave 2; however, this increase in yield is not sustained for a second harvest. The 1-year effect for SBEP beneficiaries from wave 3 is smaller, a 0.18 tons/hectare increase after the first year. This effect is not sustained for a second year. TABLE 1: TREATMENT EFFECT ON MAIZE YIELDS, ONE AND TWO YEARS POST-INTERVENTION Number of Hectares Planted The same limitations regarding the high non-response, small sample size, and staggered treatment waves are present here. However, the results below show that there is a statistically significant effect on hectares planted for waves 1 and 3. In particular, SBEP beneficiary farms from wave 1 have increased 1.3 hectares of maize planted the year after the intervention and increased 1.9 hectares of maize planted by the second year after the intervention. There is no statistically significant effect on hectares of maize planted for treatment wave 2. For treatment wave 3, we see an even larger increase after the first year, 1.8 more hectares relative to nonparticipant farms. However, this increase is not sustained for the second year. TABLE 2: TREATMENT EFFECT ON HECTARES OF MAIZE PLANTED, ONE AND TWO YEARS POST-INTERVENTION Wave 1 Wave 2 Wave 3 Wave 1 Wave 2 Wave 3 Post 1-year 0.487*** 0.500* 0.175** 0.468*** 0.418* 0.109 Post 2-years 0.850*** -0.281 0.0274 0.818*** -0.324 -0.0119 *** p<0.01, ** p<0.05, * p<0.1 Treatment vs Comp A Treatment vs Comp A+B Wave 1 Wave 2 Wave 3 Wave 1 Wave 2 Wave 3 Post 1-year 1.309*** 0.180 1.780*** 1.312*** 0.172 1.760*** Post 2-years 1.877*** 0.421 -0.0552** 1.877*** 0.423 -0.0623*** *** p<0.01, ** p<0.05, * p<0.1 Treatment vs Comp A Treatment vs Comp A+B EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 110 Difference-in-Differences Regression Results Treatment vs Comparison A (1) (2) (3) (4) (5) (6) (7) (8) (9) VARIABLES DID DID with covariates DID with covariates, FE DID DID with covariates DID with covariates, FE DID DID with covariates DID with covariates, FE Round x Treatment 0.568 0.415 1.472 -1.304 -1.244 -1.612 0.765 0.603 2.049 (0.804) (0.441) (2.138) (2.217) (2.174) (2.058) (0.498) (0.282) (2.256) Treatment 1.839 -0.447 -1.595 1.439 0.581 1.979 2.398 0.500 -1.257 (2.361) (1.484) (2.520) (2.351) (2.184) (2.139) (1.479) (1.065) (2.998) Round -1.335* -1.296** -1.349** -0.534 -0.650 -0.605 -0.832 -0.750 -0.868 (0.543) (0.343) (0.411) (0.388) (0.398) (0.417) (0.383) (0.535) (0.620) Size of farm 0.888 0.813 0.0136 -0.0342 0.870** 0.778 (0.384) (0.485) (0.184) (0.153) (0.241) (0.398) Farm registered 0.473 0.179 0.553 0.427 0.679 0.445 (1.082) (0.772) (0.738) (0.701) (0.914) (0.754) Farm income / total HH income 0.473 0.472** 0.428* 0.487** 0.227 0.201 (0.222) (0.0874) (0.171) (0.120) (0.268) (0.119) Recent past investments 1.140 1.498 0.543* 0.559** 0.523 1.173 (0.776) (1.306) (0.188) (0.146) (0.691) (1.358) Polog region 1.731 0.695 3.034* (1.219) (0.373) (1.062) Northeast region 1.144 0.729 1.325 (1.054) (0.356) (1.026) Vardar region 0.342 -0.732* 0.994 (0.559) (0.272) (0.673) Treatment group wave 1 6.749 0.756 7.949 (11.11) (2.708) (10.84) Treatment group wave 2 -0.243 -1.637 0.312 (1.676) (0.836) (1.081) Treatment group wave 3, omitted - - - Constant 4.495*** 0.664 -0.120 2.457*** 0.990 0.535 1.707** -1.366 -2.706 (0.455) (1.819) (3.010) (0.320) (0.421) (0.589) (0.434) (1.167) (2.680) Observations 514 513 513 490 489 489 522 521 521 R-squared 0.015 0.043 0.075 0.015 0.035 0.056 0.022 0.041 0.091 *** p<0.01, ** p<0.05, * p<0.1 Total workers Total full-time workers Total paid employees Robust standard errors in parentheses. Standard errors adjusted for clustering at the region level. First model is a standard DID; second model includes baseline covariates; third model accounts for fixed effects of regions and treatment waves. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 111 Treatment vs Comparison A (16) (17) (18) (19) (20) (21) (22) (23) (24) VARIABLES DID DID with covariates DID with covariates, FE DID DID with covariates DID with covariates, FE DID DID with covariates DID with covariates, FE Round x Treatment 0.196** 0.207** 0.261* 0.213* 0.211* 0.144 0.00950 0.0277 -0.0155 (0.0526) (0.0515) (0.0895) (0.0876) (0.0785) (0.127) (0.102) (0.0751) (0.0866) Treatment 0.0762** 0.0679 -0.0812 0.189 0.197 0.237 0.248 0.0574 0.164 (0.0234) (0.0305) (0.0775) (0.118) (0.151) (0.233) (0.130) (0.131) (0.100) Round -0.0498 -0.0639 -0.0572 -0.207* -0.208* -0.208 0.00543 -0.0801 -0.0490 (0.102) (0.100) (0.0952) (0.0872) (0.0839) (0.0895) (0.0703) (0.0440) (0.0488) Size of farm -0.00310 -0.000212 0.0178 0.0229 0.00350 -0.00480 (0.0254) (0.0244) (0.0507) (0.0501) (0.0160) (0.0149) Farm registered -0.0777 -0.0912 -0.102 -0.0920 0.191 0.102 (0.0546) (0.0597) (0.0453) (0.0428) (0.107) (0.0649) Farm income / total HH income 0.00798 0.00682 -0.0147 -0.0184 -0.0245 0.0109 (0.0142) (0.0123) (0.0402) (0.0418) (0.0328) (0.00857) Recent past investments 0.0989* 0.0843* 0.0681 0.0525 0.361** 0.251** (0.0311) (0.0315) (0.0442) (0.0389) (0.0870) (0.0460) Polog region -0.0946*** -0.0438 -0.0173* (0.0144) (0.0310) (0.00562) Northeast region -0.000241 -0.0154 0.463*** (0.0183) (0.0741) (0.0418) Vardar region 0.0597** 0.193** 0.239*** (0.0152) (0.0463) (0.0153) Treatment group wave 1 0.123 -0.289** -0.132* (0.114) (0.0887) (0.0458) Treatment group wave 2 0.123 -0.0438 0.0504 (0.0870) (0.177) (0.121) Treatment group wave 3, omitted - - - Constant 0.197** 0.190** 0.228** 0.378*** 0.394** 0.411** 0.293 0.209 0.0735*** (0.0375) (0.0343) (0.0502) (0.0433) (0.0755) (0.110) (0.146) (0.111) (0.00902) Observations 506 505 505 463 462 462 522 521 521 R-squared 0.059 0.074 0.089 0.115 0.125 0.158 0.058 0.248 0.384 *** p<0.01, ** p<0.05, * p<0.1 Better financial situation (2016 vs last yr) Financial situation will improve next 12m Likely new investment next year Robust standard errors in parentheses. Standard errors adjusted for clustering at the region level. First model is a standard DID; second model includes baseline covariates; third model accounts for fixed effects of regions and treatment waves. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 112 Treatment vs Comparison A+B (1) (2) (3) (4) (5) (6) (7) (8) (9) VARIABLES DID DID with covariates DID with covariates, FE DID DID with covariates DID with covariates, FE DID DID with covariates DID with covariates, FE Round x Treatment -0.107 -2.183 -3.141 -1.491 -1.679 -2.415 -0.113 -1.996 -2.580 (0.645) (2.005) (1.852) (2.046) (2.427) (2.433) (0.593) (1.732) (1.702) Treatment 2.548 1.344 3.057 1.916 1.194 2.777 2.987* 2.032 3.282 (2.089) (2.048) (2.159) (2.129) (2.453) (2.437) (1.508) (1.563) (1.985) Round -0.660 -0.753** -0.668* -0.346 -0.521 -0.379 0.0470 -0.127 -0.0903 (0.493) (0.297) (0.328) (0.232) (0.310) (0.329) (0.590) (0.481) (0.522) Male respondent -0.217 -0.00481 0.0325 0.178 -0.104 -0.278 (0.177) (0.295) (0.177) (0.233) (0.195) (0.341) Age of respondent -0.0163** -0.0149 -0.0129 -0.0127 -0.00409 -0.0142 (0.00490) (0.00993) (0.00786) (0.00873) (0.00660) (0.0122) Aware of economic development plans -0.741 -1.052** -0.678 -1.107* -0.473 -0.480 (0.387) (0.382) (0.632) (0.555) (0.454) (0.552) Size of farm 0.751** 0.654** 0.148 0.0143 0.607*** 0.549*** (0.214) (0.203) (0.141) (0.0913) (0.0921) (0.102) Farm registered 0.398 0.158 0.599 0.323 0.591 0.458 (0.658) (0.621) (0.575) (0.483) (0.609) (0.648) Farm income / total HH income 0.361 0.450** 0.436** 0.495*** 0.182 0.218 (0.202) (0.153) (0.168) (0.113) (0.166) (0.135) Past recent investments 0.686 0.777 0.427** 0.582** 0.426 0.593 (0.372) (0.543) (0.168) (0.160) (0.399) (0.611) Southeast region 0.739 1.763*** 0.288 (0.408) (0.226) (0.372) Southwest region 1.236*** 0.962*** 0.593 (0.277) (0.177) (0.378) Pelagonia region 1.406 1.618** -0.250 (0.731) (0.463) (0.401) Polog region 1.736*** 2.142*** 1.522*** (0.337) (0.275) (0.139) Northeast region 2.169*** 2.384*** 0.529 (0.523) (0.248) (0.615) Vardar region 0.801 0.235 0.150 (1.086) (0.408) (0.718) Treatment group wave 1 1.494 1.902 2.454 (3.207) (3.397) (3.012) Treatment group wave 2 -2.111* -1.776* -1.616 (0.962) (0.853) (0.994) Treatment group wave 3, omitted - - - Constant 3.785*** 2.154*** 0.622 1.980*** 1.084 -0.346 1.117** -0.481 -0.402 (0.476) (0.572) (0.590) (0.357) (0.669) (0.601) (0.438) (0.407) (0.890) Observations 740 679 679 711 656 656 750 687 687 R-squared 0.018 0.076 0.097 0.020 0.058 0.109 0.026 0.061 0.087 *** p<0.01, ** p<0.05, * p<0.1 Robust standard errors in parentheses. Standard errors adjusted for clustering at the region level. First model is a standard DID; second model inclused covariates that were not balanced at baseline; third model accounts for fixed effects of treatment waves. Total workers Total full-time workers Total paid employees EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 113 Treatment vs Comparison A+B (16) (17) (18) (19) (20) (21) (22) (23) (24) VARIABLES DID DID with covariates DID with covariates, FE DID DID with covariates DID with covariates, FE DID DID with covariates DID with covariates, FE Round x Treatment 0.117 0.236* 0.302* 0.0105 0.00757 -0.0160 -0.0790 -0.0477 -0.0926 (0.103) (0.0981) (0.127) (0.158) (0.116) (0.132) (0.114) (0.0656) (0.0718) Treatment 0.0890** -0.0150 -0.166 0.260* 0.274* 0.261 0.283** 0.0869 0.185 (0.0261) (0.0400) (0.0896) (0.125) (0.138) (0.237) (0.110) (0.101) (0.105) Round 0.0293 0.0127 0.0185 -0.00382 -0.0448 -0.0485 0.0940 -0.0431 -0.0239 (0.117) (0.0877) (0.0900) (0.154) (0.112) (0.117) (0.0902) (0.0542) (0.0551) Male respondent -0.0156 -0.00466 -0.0889 -0.0913** -0.116 -0.0374 (0.0332) (0.0415) (0.0506) (0.0361) (0.0645) (0.0421) Age of respondent -0.00423*** -0.00308** -0.000630 -0.000612 -0.00519*** -0.00387** (0.000872) (0.00112) (0.00122) (0.00127) (0.00132) (0.00114) Aware of economic development plans -0.0200 -0.0196 -0.0468 0.0135 0.0570 0.0818 (0.0454) (0.0469) (0.0646) (0.0631) (0.0771) (0.0529) Size of farm -0.0148 -0.000200 0.00478 0.0153 0.00168 -0.00630 (0.0194) (0.0231) (0.0456) (0.0358) (0.0212) (0.0124) Farm registered -0.0681 -0.0849* -0.0573 -0.0312 0.173** 0.115* (0.0503) (0.0419) (0.0552) (0.0526) (0.0674) (0.0533) Farm income / total HH income 0.0254 0.0294 0.000622 0.00276 -0.00340 0.0172 (0.0187) (0.0215) (0.0353) (0.0390) (0.0196) (0.0150) Past recent investments 0.105* 0.0940* 0.136 0.129 0.383*** 0.317*** (0.0478) (0.0415) (0.0813) (0.0741) (0.0618) (0.0674) Southeast region 0.150*** -0.249*** 0.123*** (0.0210) (0.0364) (0.0225) Southwest region 0.222*** 0.0422 0.121** (0.0345) (0.0232) (0.0381) Pelagonia region 0.124** -0.0994** -0.0385 (0.0386) (0.0403) (0.0292) Polog region 0.0809*** -0.0760** -0.0115 (0.0137) (0.0207) (0.0234) Northeast region 0.114** -0.162*** 0.347*** (0.0404) (0.0419) (0.0620) Vardar region 0.181*** 0.0994 0.221*** (0.0448) (0.0783) (0.0327) Treatment group wave 1 0.0294 -0.252* -0.143** (0.0457) (0.110) (0.0426) Treatment group wave 2 0.146 -0.0434 -0.00656 (0.0806) (0.187) (0.141) Treatment group wave 3, omitted - - - Constant 0.184*** 0.366*** 0.147 0.308*** 0.405*** 0.432** 0.258** 0.469** 0.253* (0.0310) (0.0881) (0.164) (0.0514) (0.0646) (0.139) (0.0857) (0.142) (0.120) Observations 723 663 663 651 606 606 750 687 687 R-squared 0.035 0.074 0.094 0.051 0.077 0.112 0.051 0.310 0.376 *** p<0.01, ** p<0.05, * p<0.1 Financial situation will improve next 12m Likely new investment next year Robust standard errors in parentheses. Standard errors adjusted for clustering at the region level. First model is a standard DID; second model inclused covariates that were not balanced at baseline; third model accounts for fixed effects of treatment waves. Better financial situation (2016 vs last yr) EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 114 Baseline Balance of Means Variable Treatment Group Comparison Group A CompA+B T = CompA T = CompA+B Male respondent 0.979 0.788 0.799 0.187 0.040 (0.144) (0.410) (0.402) Age 41.42 44.99 45.66 0.312 0.059 (9.446) (12.88) (13.17) Completed secondary education 0.604 0.620 0.704 0.899 0.332 (0.494) (0.487) (0.457) Member of farmer association/coop 0.104 0.0272 0.0168 0.319 0.184 (0.309) (0.163) (0.129) Aware of econ. development plans 0.646 0.315 0.208 0.141 0.024 (0.483) (0.466) (0.407) Size of farm (Hectares) 3.833 2.451 2.124 0.036 0.004 (1.038) (1.049) (1.012) Owns at least some part of farm 0.958 0.880 0.919 0.411 0.497 (0.202) (0.325) (0.273) Farm is registered 0.771 0.337 0.285 0.079 0.009 (0.425) (0.474) (0.452) Intercrop 0.896 0.946 0.933 0.636 0.645 (0.309) (0.227) (0.251) Farm income / total HH income (categories) 3.604 2.647 2.466 0.028 0.008 (0.818) (1.087) (1.143) Made investments in last year 0.563 0.212 0.171 0.092 0.024 (0.501) (0.410) (0.377) Financial situation better than last year 0.273 0.197 0.184 0.047 0.014 (0.451) (0.399) (0.388) Financial situation will improve 0.568 0.378 0.308 0.207 0.082 (0.502) (0.487) (0.463) Future investment likely 0.542 0.293 0.258 0.151 0.041 (0.504) (0.457) (0.438) Baseline Characteristics P-Values EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 115 Variable Treatment Group Comparison Group A CompA+B T = CompA T = CompA+B Maize- Irrigation used 0.969 0.871 0.838 0.151 0.008 (0.177) (0.337) (0.369) Maize - Fertilizer used 1 0.823 0.779 0.034 0.018 (0) (0.383) (0.416) Maize - Mechanization used 1 0.769 0.745 0.082 0.012 (0) (0.423) (0.437) Forage - Irrigation used 1 0.815 0.809 0.075 0.008 (0) (0.390) (0.394) Forage - Fertilizer used 0.750 0.761 0.688 0.855 0.547 (0.442) (0.429) (0.465) Forage - Mechanization used 0.833 0.630 0.599 0.025 0.021 (0.381) (0.485) (0.492) Total workers 6.333 4.495 3.785 0.493 0.268 (11.32) (7.553) (6.065) Total full-time workers 3.896 2.457 1.980 0.584 0.403 (11.30) (4.166) (3.446) Total paid-workers 4.104 1.707 1.117 0.203 0.095 (11.57) (7.482) (5.940) % of total arable land used for maize 0.272 0.204 0.193 0.515 0.403 (0.287) (0.208) (0.220) % of total arable land used for forage 0.139 0.0914 0.119 0.289 0.641 (0.246) (0.144) (0.202) Baseline Characteristics P-Values EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 116 Exhibit 2.4: Yield and Production Data and Supplementary Graphs Corn grain yield (kg/Ha) production date (tons/Ha) as well as these data for corn silage are provided in this exhibit as are additional graphs that show trends for other farmers in the regions in which SBEP worked as well as national statistics were those are available from Macedonia’s State Statistical Office. 0.00 5.00 10.00 15.00 20.00 25.00 2012 2013 2014 2015 2016 Corn Silage Yields for Exhibit 2.4 Treatment Farmers SBEP Regions Comparison A Comparison B National Data 0.00 2.00 4.00 6.00 8.00 10.00 12.00 2012 2013 2014 2015 2016 Corn Grain Yields for Exhibit 2.4 Treatment Farmers SBEP Regions Comparison A Comparison B National Data EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 117 Corn Grain Corn Grain Yield (kg/ha) Treatment Farmers SBEP Regions Comparison A Compariso n B 2012 8.80 9.02 4.83 2013 9.56 8.83 5.02 2014 9.35 8.29 5.75 2015 8.77 7.22 5.70 2016 11.03 10.12 6.38 Corn Grain Production (tons/hectare) Treatment Farmers SBEP Regions Comparison A Compariso n B 2012 6.25 11.1 1.96 2013 12.00 10.85 4.31 2014 13.13 10.51 4.44 2015 13.01 10.17 4.50 2016 14.21 14.66 5.68 0 2 4 6 8 10 12 14 16 2012 2013 2014 2015 2016 Corn Grain Production Exhibit 2.4 Treatment Farmers SBEP Regions Comparison A Comparison B EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 118 Corn Silage Corn Silage Yield (kg/ha) Treatment Farmers SBEP Regions Comparison A Comparison B National Data 2012 5.06 16.21 7.43 13.04 2013 12.23 13.26 5.12 14.82 2014 11.90 12.70 5.37 18.68 2015 11.43 12.57 6.02 17.26 2016 17.65 16.82 6.33 21.56 Corn Silage Production (tons/hectare) Treatment Farmers SBEP Regions Comparison A Comparison B 2012 15.75 11.11 4.19 2013 19.95 11.37 3.54 2014 16.95 11.30 4.48 2015 16.39 13.02 5.16 2016 22.91 14.67 5.69 0.00 5.00 10.00 15.00 20.00 25.00 2012 2013 2014 2015 2016 Corn Silage Production for Exhibit 2.4 Treatment Farmers SBEP Regions Comparison A Comparison B EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 119 Sub-Study 3: Women in Business Initiative Photo Credit: SBEP (USAID/Macedonia). This SBEP Women in Business initiative’s performance review is based on secondary sources, including project quarterly reports and primary data collected by the evaluation team through a survey conducted by the MSI evaluation team. For the survey, 12 of the 23 women-owned or managed firms that received assistance from this program between its inception and the end of FY16 responded. The team also consulted comparable survey data from 34 out of 56 women-owned firms not assisted by SBEP, which the evaluation team contacted. Other data sources the evaluation team utilized included interviews with SBEP’s partner for the Women in Business initiative, the Association for Local and Rural Development (ALRD); the Women’s Advisory Board, in the Polog region, which helped to steer this effort since its beginning, and was expected to continue in that role, as an Advisory Board for the Egalite Foundation — an NGO formed to mobilize and manage funds donated to support this initiative beyond the termination of USAID’s assistance. MSI also interviewed Ecolog, an international private-sector firm operating in Polog that contributed funding to this initiative and talked with members of the SBEP staff. Introduction Unemployment and low workforce participation rates in parts of the country are a chronic problem for Macedonia. In 2016, the country ranked 25th highest among 141 countries on unemployment, despite declining from a high of 38.70 percent in the first quarter of 2005 and a record low of 22.10 percent in the third quarter of 2017. Unemployment was at 31 percent in 2012 when SBEP started, and that same year the overall workforce participation rate was 56.5 percent, with an important gap between men, with a 62.2 percent activity rate, and women at 43.8 percent nationally, and notable differences between regions. Polog, a statistical region on the western side of the country had the lowest rate of labor force participation in 2012, and that ranking persists today, with an activity rate of 47.2 percent in 2016. Polog, where SBEP sited its Women in Business initiative is the only majority INTRODUCTION LABOR FORCE ACTIVITY RATES BY REGION OF MACEDONIA. Source: State Statistical Office of Macedonia (2012) EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 120 ethnical Albanian region in the country, and cultural practices play a role in women’s low level of workforce participation. As of 2006, the last year in which national labor force statistics were reported on an ethnic basis, female participation in the labor force among ethnic Albanian women was very low compared to their ethnic Macedonian counterparts. Published research on these differences, as well as USAID’s gender assessment, indicated that cultural/religious norms and women’s educational attainment level both factor into women’s level of entrepreneurship and employment in Polog. Nationally, the proportion of women reported by the State Statistical Office to be employers runs at about 25 percent of all employed persons with that designation. In Polog, this share is believed to be significantly lower, but as interviewees in the region explained, there is no data source on this question, official or otherwise. Lower levels of educational attainment among ethnic Albanian women is viewed as being another factor that contributes to a lower level of participation in the workforce among ethnic Albanian women than among ethnic Macedonian women, as illustrated in the figure below. Level of education may also likely to play into women’s decisions about whether to start a business, though research shows that while this is not necessarily a determining factor it may affect their rate of success.7 7 Peters, Ricardo M. “The Relationship between Levels of Education of Entrepreneurs and Their Business Success”, 2011. Source: World Bank, Labor Market Inequalities in FYR Macedonia: New Evidence on Gender and Ethnicity, 2015. Source: Pounova, Marija. Gender Mainstreaming in the Republic of Macedonia. Beyond the EU Lenses, Centre for Formation Europeenne, 2013. Re-analysis of survey data from the Center for Research and Policymaking, Skopje. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 121 Evolution of the SBEP Women in Business Initiative SBEP’s Women in Business Initiative started early in the life of the project and emerged during SBEP’s launch even in Polog in 2012 attended by local mayors and other interested parties including “the Economic Chamber of North-West Macedonia, the Polog Regional Development Center (RDC), and the German Society for International Cooperation (GIZ - RED). In addition, the Municipal and Local Economic Development offices, financial institutions and certainly private sector entities complete the spectrum of collaborators”, according to SBEP quarterly reporting.8 Both GIZ and the Northwest Chamber had activities that could contribute to a cluster of efforts in this arena. This focus was also consistent with the memorandum of understanding SBEP had signed earlier with the Polog Regional Development Center. Including a focus on women in SBEP’s portfolio was also aligned with the thrust of USAID/Macedonia’s 2010 Gender Assessment, which “highlighted women and entrepreneurship, noting that “Macedonia has the third highest gender pay gap in the region, few women are managers or owners of businesses, and [there are] low levels of female entrepreneurs.” USAID’s assessment further recommended: • Across the board and especially in programming seeking to involve women from more traditional backgrounds, build features into projects that will address constraints on women’s ability to participate due to their family role and responsibilities. • Design entrepreneurship programs specifically to address barriers faced by women who wish to start up or expand their businesses. Building on GIZ’s existing plan to establish a women’s entrepreneurs cluster, and the chamber’s suggestion that such a cluster could be housed within the chamber, SBEP took on the role of mapping the scope, size and potential of existing women-owned or managed businesses in Polog. To execute this early task, SBEP called upon ALRD.9 Among other things, ALRD found that rural women in Polog primarily work from home and that the majority of women’s businesses are linked to producing goods and services for weddings, religious celebrations and various other festivities.10 Planning for establishing a women’s entrepreneurship cluster continued over several months through meetings with female entrepreneurs in attendance. SBEP, GIZ and the chamber encouraged them to participate. Their numbers grew from an initial 10 to roughly 20 by 2014. They also received technical assistance from SBEP, GIZ and the chamber through Konet, a local business support organization, in their efforts to develop a “vision, mission and goals” for a women entrepreneur’s cluster or other organizational structure they would create. In retrospect the evolution of this initiative had several important developmental aspects that received attention both sequentially and concurrently, in that once each of these stages began it proceeded in parallel with the others until SBEP phase itself out of the initiative during the project’s final year, as outlined in the figure below. In addition, Exhibit 3.1 provides a chronology of this initiative. 8 SBEP Quarterly Report: October-December 2012. 9 SBEP Quarterly Report: July-September 2013. 10 SBEP Quarterly Report: April-June 2016. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 122 Scoping and Operationalizing the Initiative -- Deciding to focus (mainly) on existing businesses and equipment needs entrepreneurs, at a relatively constant dollar level -- Women's Advisory Board role in selection process -- Focus with ALRED on existing businesses and equipment needs -- Assistance provided to 25 women-owned firms Mobilizing continuous financial flows -- Building awareness of the initiateve and spreading "success stories" about assisted firms -- Created a Fund to accept and manage financial contributions -- Secured initial funding flow contributions from Ecolog Internationalv ($15,000) and two Polog municipalities ($10,000 each) Institutionalization -- Developed relationships with multiple possible future initiative mentors, including other donors -- Helped establish the Egalite Foundation as a future home. -- Help SBEP staff launch AIRE as a legacy organization with skills and a continuing committment to this initiative. Assistance to women-owned businesses started very quickly under this model. • The first business supported was Naser-nara in Kamenjane owned by Ms. Rabije Sinani, whose business produces cakes and sweets for weddings. As Ms. Sinani looked to expand her business, SBEP supported her with a refrigerator and a machine for rolling dough. The expansion of her business had the benefit of increasing opportunities for her suppliers. Ms. Sinani also trained 20 women from Kamenjane and neighboring villages on how to make wedding cakes and sweets with a further aim to employ at least 10 of them on a long-term basis in her workshop. • The second business supported by SBEP was Princesha, an artisanal tailoring shop owned by Ms. Mjaftime Fejzi, which produces wedding gowns and traditional dresses in Gostivar. SBEP provided two sewing machines and professional ironing equipment to the business. In turn, Ms. Fejzi trained 15 women, of which five were employed during the high season. Notably, as a direct result, the business doubled its production of wedding dresses from 8 to 16 a day.11 In all, 25 firms were provided with assistance in the $3,000 to $5,000 range through the initiative. Additional stories can be found in SBEP quarterly reports which are available on line and can be located with a simple search that highlights USAID/Macedonia and SBEP, and specifies quarterly report. A listing of firms the Women in Business initiative assisted is provided in Exhibit 3.3. For each firm listed the evaluation team includes all of the information to found in quarterly reports. As the table indicates, accessible reporting on the specifics of SBEP’s assistance under this initiative was most extensive at the start of the program. On this point, one interview respondent said much the same thing to the evaluation team: “We could have used some data and metrics to monitor, but only for certain documented activities. I can say that continual monitoring and evaluation of the program implementation has not been executed according to available documentation.” SBEP Partner Interview 11 SBEP Quarterly Report: July-September 2013. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 123 Women in Business Initiative Results Key program results of interest to USAID/Macedonia for this evaluation include the effects of SBEP’s initiatives on the perceptions and expectations of project beneficiaries, changes in their production and product quality and jobs created as a result of SBEP assistance. Findings on these anticipated outcomes presented in this section come from evaluation team interviews and surveys and from SBEP’s own performance reports. Perceptions and Expectations Through its survey of 12 of the 25 women-owned businesses SBEP supported, and individual interviews with other program stakeholders, the evaluation team sought information on the effects or impact of this initiative. Exhibit 3.2 provides a summary of survey responses as well as a profile of the subset of beneficiaries the team interviewed. In summary their answers indicate that as a result of SBEP assistance their sales of goods/services increased (58%) and that their customer base had increased (50%) as a result of new investments in the firm, increased marketing and better packaging. Women business owners felt they now had the best possible equipment for their firms (83%) and some said that they were likely to invest further during the coming year (33%). The same percentage characterized their firms as being in a better financial situation this year than last, and that they expected thing to e still better the following year. Overall, 83% of the women SBEP had helped said that they were satisfied with their firm’s performance and 74% said they were optimistic about their firm’s future prospects and they outlook for their family or household situation. With respect to beneficiary expectations, one issue was raised by an initiative partner during one of the final evaluation interviews in early 2017 that may still warrant attention: “We do not have relevant documentation or evidence needed to legally transfer the equipment we purchased for women’s businesses to them because there was no official contract signed with the women businesses on the support from the Fund. We have ended up with no papers in our hand that this equipment was handed over to the businesses. This is a current problem for us and we assume we will have problems with the accounting for this.” - SBEP Partner Interview Additional perspectives on the underlying question “what difference did the SBEP Women in Business initiative make?” were provided through interviews with initiative partners and donors. Illustrative views on that question are provided below. Consistently, these types of respondents focused on the motivational and confidence building effects of the program, more so than the equipment or other inputs it provided. When asked what the top three things are that women-own businesses supported by SBEP valued about the program, one interviewee responded: 1. “The fact that the support was obtained by the community, towards women businesses and this changed the mentality 2. Promotion of their products/services, which brought to extension of the market and number of clients, and 3. New knowledge, cooperation and experience that was necessary for future actions and next steps.” - SBEP Partner Interview “In my opinion, the SBEP Women-in￾Business initiative made remarkable change of the mentality of beneficiaries (women and young people), bringing a “hope” that their businesses can survive and further develop.” - Women’s Advisory Board Member EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 124 Another interview respondent who was asked about “the most significant change” the SBEP women in business initiative brought about, told the evaluation team that: “In my opinion, this initiative made a remarkable change of the mentality of beneficiaries (women and young people), bringing a “hope” that their businesses can survive and further develop. For instance, a hair dressing studio, and one that makes hand-made soaps. The most significant change according to me is non-material and this is an opportunity they have to grow their businesses.” - Initiative Fund Donor Interview Job Creation The close out and final performance report delivered to USAID on SBEP states that 119 jobs were created as a result of the Women in Business initiative. Through its review of descriptions of businesses SBEP supported found in quarterly reports, including performance indicator tracking sheets, and the project’s final report, the evaluation team tried to associate job creation with specific types of firms SBEP had helped. As the table of assisted women-owned firms that participated in this initiative shows, data on job creation was available for only a few firms, and in most cases what was found were projections about how many jobs might be created, rather than numbers of jobs actually crated. Similarly, indirect job creation through supply chains was present for some firms, but these too were mainly projections. USAID may have received additional information on job creation under this initiative, but the evaluation team did not see that data. With the data available, a rough calculation can be made that indicates that, on average SBEP investments of between $3,00 to $5,00 yielded a return of 4.76 jobs a year or two hence, but nothing more specific can be said about what kinds of firms generated higher numbers of jobs than others, or why that might have been the case. Women in Business Initiative Sustainability SBEP made a concerted effort to add resources to the budget for this initiative while it was being implemented and to establish the kinds of public and private sector relationships that might have produced annual funding allotments to sustain the initiative’s support to women-owned businesses. Despite the project’s best efforts, the type of locally-based, long term funding commitments flowing to the Fund SBEP helped create did not materialize, as interview responses explained to the evaluation team: “No additional donors beyond Ecology and two municipalities has been received. However the initiative has been recognized as an interesting one, by the Swiss organization SDC, and already had some visits and meetings with their IME (Increasing Market Employability) project . The British Embassy representatives also showed their interest to learn more about our experience with the Fund.” So far no interest from other women’s groups. We have been approached by one local NGO that had a British Embassy grant for doing a similar program but no funding for this initiative was discussed. They came to learn our practices and lessons learned.” This has left the initiative dependent for funds on whatever mix of donors can be identified. As one interview respondent conveyed, at least one such source had been found before the project terminated: “Earlier this year, the Egalite Foundation applied for and received an institutional support grant through the Swiss-funded Civica Mobilitas program. This type of support will be directed towards strengthening the organization, improving its internal structure and procedures, enhancing promotion and building the staff’s skills and knowledge in networking, advocating and EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 125 effective work in local communities. The financial support for the first year is US$ 22,000, while the extension into a second year will depend on its success in the first.”12 - Women’s Advisory Board Representative One factor the evaluation team observed that may limit the effectiveness of this sustainability strategy is the presence of other projects and initiatives that have emerged in Macedonia which have a similar focus including the Swiss SDC IME project mentioned above, the World Bank’s Local and Regional Competitiveness project, and USAID’s Regional Economic Growth (REG) Project, USAID/Macedonia’s current Development Credit Authority and microenterprise initiatives, and possibly others. To the degree that these activities exist, even in part because the SBEP Women in Business initiative stimulated interest and provided aspects of what may be a model, should be understood as evidence of the impact of what SBEP undertook. 12 SBEP Quarterly Report: April-June 2013. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 126 Exhibit 3.1 Women in Business Initiative Chronology January-March 2013: Business leaders and aspiring businesswomen were brought together for support and action. The Women’s Entrepreneurs Cluster was established and promoted as a cluster within the Economic Chamber of Northwest Macedonia. ALRD was contracted to map out opportunity areas for women’s entrepreneurship in Polog. April-June 2013: First business, a cake workshop in Kamenjane, was supported. July-September 2013: Two businesses were supported: An artisanal tailoring shop for wedding gowns and traditional dresses, Princesha in Gostivar, owned by Ms. Mjaftime Fejzi, was supported with two sewing machines and professional ironing equipment. The Children’s Education Center in Rea was supported with beds, chairs, tables, educational and outdoor toys, and furniture. October-December 2013: SBEP supported a tailor shop in Gostivar, owned by Ms. Azbije Jonuzi, which produces traditional dresses and wedding gowns, labels and uniforms for a transport company and draperies for buses. SBEP provided the business with five different sewing machines and one industrial iron, which would double her production and lead to 3-4 new employees January-March 2014: USAID SBEP signed an MoU with Ecolog, a major company in the Polog region, which agreed to donate $15,000 to match USAID’s funding for supporting women-owned or women￾operated business, a first of its kind in the region. Another business, Jehona’s Event Agency in Tetovo, was supported with sewing machines and professional ironing equipment. April-June 2014: The Municipality of Bogovinje allocated $10,000 for the fund to be used to co-finance Polog-based women and youth businesses identified in cooperation with SBEP, ARLD and the Women Advisory Board. Eight businesses were identified for support. July-September 2014: Three new businesses were approved for support. ALRD began to implement the SBEP, Ecolog and Bogovinje Municipality-supported fund. October-December 2014: A consulting and translating firm located in Tetovo was selected for support. January-March 2015: The Municipality of Tetovo officially joined the Fund for Supporting Women and Youth Businesses, allocating $ 6,000. Three local businesses from the Municipality of Bogovinje also joined the fund, with a total contribution of $10,500 in funds and equipment. One new small business, a photo studio owned by 26-year old Hanife Amzai from Gostivar, received support from the fund. April-June 2015: The Municipality of Brvenica contributed $5,000 to the fund. Two new businesses were selected to receive support from the fund: A beauty salon owner from Zhelino and an organic soap producer from Tetovo. July-September 2015: Egalite, the Foundation for the Support of Women and Youth Entrepreneurship, was established, with the expectation that it would become the future home of the Women in Business initiative SBEP terminates. October-December 2015: The Municipality of Gostivar joined the initiative and contributed $10,000 to the foundation. Two new businesses were supported: Aya Farm, a cow farm from the village of Pirok, received a milking machine, a milk-processing machine and a hay-collector, which helped them improve operations and expand production. Sassa Pack, a cardboard packaging producer from the village of Zherovjane, was also supported. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 127 January-March 2016: The Women’s Advisory Board approved the support of four newly proposed small businesses. Egalite received a grant from the Swiss funded Civica Mobilitas program for institutional development. April-June 2016: During this quarter, SBEP summarized and wrapped up its involvement in the Women in Business initiative, turning leadership of this effort over to the Egalite Foundation. The Women’s Advisory Board, it is expected, will “continue to steer the Foundation and engage with it leaders…to spread the word about the Foundation and track down potential investors.” EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 128 Exhibit 3.2: Women in Business Survey Summary The first part of this survey asked about the respondent’s businesses. The second part asked about the respondent’s awareness, experience and perceptions of SBEP. Business Trends in Respondent’s Organizations over the past 2-5 Years (since 2012)13 Business Focus of respondent firms: • Half of the firms (6) make products, such as cheese, organic soap, skin cream, and wedding decorations • Half sell services, including sewing wedding dresses, cutting hair, and classes for children See Exhibit 3.3 for a full listing of assisted firms. Sales of the organization’s services/goods: • Increased: 58% • Decreased: 33% Customer base: • New customers started buying our services/goods: 50% • One respondent said her firm had lost customers (12%) • On an open-ended question, respondents identified several reasons why business sales had increased: o Increased marketing o Better educated clients o Packaging o New investments by the firm Business Equipment Adequacy of equipment and automation: • Best possible mix given the size of their business (83%) • Cost and lack of sources constraints further improvements of this sort Business Financing Original financing for the respondent’s firms: • Personal savings: 91% • Government subsidy involved: 8% Source of normal operating costs: • Earnings from the firm: 92% 13 Answers to questions asked of this survey’s 12 respondents are shown in percentages based on the number of respondents that actually answered each question, ignoring missing responses (valid percent). Profile of the Evaluation’s Women in Business Survey Respondents (12) Over the course of its Women in Business initiative, SBEP reportedly provided assistance to 25 female entrepreneurs in the Polog Region. Of these, the evaluation team was able to arrange to conduct in-person interviews with 12 of these beneficiaries. Of the 12 women interviewed, 11 reported that they had started the businesses they run, and 9 are the owners of those businesses, seven of which were registered with the government in their own names. Nine of the respondents were between 29 and 49 years of age, while two were younger (youth). Half of the women had university degrees. All of the businesses the respondents represented were established after 2000 and operate from a single location. None were members of any business association, nor were any of the respondents aware of any economic development plans for their region or the country as a whole. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 129 • One respondent receives some financial help from her family Major investments (new equipment, additional space), since start-up: • Yes (42%) – four with business profits; one with a loan • No (58%) Difficulties encountered running the financial side of the business, including obtaining loans, because they are women: • No: 83% • Yes: 17% If the firm has or were to get a line of credit, in whose name would that be set up: • Mine -- Only one respondent answered this question, and this was her answer Employment in Respondent Firms Number of people who work at the organization? • 0-9: 92% • 10-19: 8% Number of people working for the organization over the past 2-3 years: • Increased: 8% • Stated the same: 75% Increased market demand was cited as the reason for bringing on additional employees according to the one respondent who said employment had increased. Does this type of work have good prospects for young people to get started earning a living after finishing school? • Yes: 42% • About the same as other types of work: 50% Respondent Organization’s Current Situation and Future Prospects Overall, considering the financial situation and business development efforts made by the organization where you work over the past 2-5 years, how satisfied are you with its performance? Very satisfied Somewhat satisfied Somewhat unsatisfied 8% 75% -- • On an open-ended question about why they were satisfied or very satisfied with their organization’s performance, 10 respondents (48%) cited the increased number of tourists coming to Macedonia as the primary reason. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 130 How would you characterize the current financial situation of the organization where you work? Better than last year About the same as last year Worse than last year 33% 50% -- What are your expectations for the organization’s financial situation over the next 12 months? Will improve Will stay the same Will get worse 33% 58%% -- Did the organization’s owners/leaders will make investments in it, out of their own or the organization’s resources, (beyond normal operating costs) over the past year (2016)? • Yes: 25% • No: 58% How likely is it that the owners/leaders of this organization will make investments in it, out of their own resources, (beyond normal operating costs) next year? Very likely Somewhat likely Not sure Somewhat unlikely Very unlikely 8% 25% 49% -- -- If respondent’s business does not account for 100% of the family’s income, from what other sources does the household obtain money? • Private sector employment in a firm we don’t own: 58% • Public sector employment: 25% • Pensions: 8% Considering your firm’s future prospects and other sources of funds, what is the outlook for your family or household situation? Very optimistic Somewhat optimistic Not sure Somewhat pessimistic Very pessimistic 8% 66% 16% -- -- Respondent’s Awareness and Experience with SBEP Respondent was aware of SBEP before this survey • Yes: 58% Source of initial awareness? Television or newspaper Another firm or person SBEP approached us Some other way 57% 29% 14% -- EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 131 Types of interaction with SBEP and how respondent rated their value for themselves and their organization Type of Interaction with SBEP Percent Responding “Yes” of Eight (8) Responses Participated in a SBEP training program 38% Received equipment or inputs for this business from SBEP 38% Received technical advice from SBEP specific to this business 13% Attended a SBEP conference or workshop 13% Business received publicity as a function of its interactions with SBEP -- As to the impact of SBEP on respondents from this group of women owned/managed firms, the interview process did not elicit long narrative descriptions, possibly because of respondent lack of experience with interviews or the time involved. Examples the evaluation team found in story texts in SBEP quarterly reports provide a more qualitative sense of the difference that SBEP assistance was viewed as making in the businesses assisted and in the lives of the women involved. See Exhibit 3.3 for data on specific firms the evaluation team found in SBEP reports. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 132 Exhibit 3.3 The table below consolidates information the evaluation team extracted from SBEP quarterly reports, in person interviews it conducted and its survey of 12 of the 25 women-owned firms SBEP assisted. A total of 26 firms were identified. SBEP’s close out and final performance report speaks of 25 firms, suggesting that one of those listed below as not yet assisted was dropped. Firms Assisted under the SBEP Women in Business Initiative and Registration Status (List of firms from SBEP elaborated based on MSI Quarterly Report reviews and evaluation interviews) Year Name of company14 Location SBEP Financial Support15 Registration status16 Jobs Created in Assisted Firms) (projected or actual) Sales and indirect/ supply chain effects 2013 1 Naser-Nara Bakery, wedding cakes and other goods Kamenjane (Bogovinje) Large refrigerator; dough laminator Registered before support Owner plans to employ at least 10 individuals she trains to make cake (projected) “Without the laminator I could produce 22 packs of pasta a day, now I can do much more.” (actual) Owner plans to train 20 women in the area on making cakes.(projected) Owner plans to obtain HACCP certification and supply local shops and restaurants (projected) Owner currently buys milk, butter and eggs from village women and plans to increase her business with these local suppliers (projected) 2 Princesha Sowing Gostivar 2 sewing machines; professional ironing equipment Registered before support Owner plans to engage 5 individuals she has trained during high season (projected) Owner reported to SBEP that production doubled since obtaining equipment SBEP provided; she can now produce 16 dresses/day, instead of 8 (actual). Owner committed to training 10 women (projected) Owner plans further development of supply chain of embroidery and handicrafts locally. 14 List of firm names and locations provided by SBEP staff 15 Data on SBEP financial contributions, project numbers of jobs and projected supply chain effects located in SBEP quarterly reports. 16 Provided in a separate communication from SBEP Staff EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 133 Year Name of company14 Location SBEP Financial Support15 Registration status16 Jobs Created in Assisted Firms) (projected or actual) Sales and indirect/ supply chain effects 2014 3 Textile Design Gostivar 2 sewing machines; industrial iron Registered before support 4 REA child care center Marshal Tito bb, Tetovo Beds, chairs, tables, educational toys and outdoor toys/furniture Registered before support Owner anticipates hiring two additional employees (projected) Owner is considering looking for a larger location (projected) “I started with 4 employees; now I have 13. All of them are women, 22 to 45 years old.” Owner’s evaluation interview response 5 Jehona Events Agency Ilindenska bb, lokal 20, Tetovo Registered before support 6 De Graf Studio graphic design and print Sedlarce (Bogovinje) Registered after support 7 Women’s FM Club Kamenjane (Bobovinje) Not functional 8 Fashion Studio Shine Kuzman Josifovski 104, Tetovo Equipment Registered before support 9 Hairdressing and cosmetics Zhelino Registered after support 10 Gama Accounting Ilindenska br6- 1/6, Tetovo Registered after support 11 3D printing ART Design Bogovinje Registered before support 12 Flower shop and events decorations Gorno Sedlarce (Bogovinje) Registered after support 13 Cosmetics and creams Kamenjane ul. 101 bb (Bogovinje) Has an artisan license 14 Akize Rizvance, farming and cheese production Ul. Leninagrdska 29, Gostivar Equipment Registered before support EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 134 Year Name of company14 Location SBEP Financial Support15 Registration status16 Jobs Created in Assisted Firms) (projected or actual) Sales and indirect/ supply chain effects 15 Butik Trend, textile shop/ design and sowing Kamenjane Equipment Registered before support 16 Glamour Tetovo, sowing and sale of wedding dresses Tetovo Registered after support 2015 17 Photo studio AKON, photography, video production Gostivar Registered before support 18 Phi school, educational services - tutoring, mentoring Kompleksi Saranda, Tetovo Registered after support 19 AZUR soaps and cosmetics, handmade organic soaps Tetovo Registered before support 20 Hairdressing cosmetic studio Zhelino Zhelino Registered after support 2016 21 AYA FARM, dairy products Pirok, Bogovinje Milk making machines; milk processing machines; haymaker Registered before support 22 Sassa Pack, production of paper packaging Zherovjane, Bogovinje Registered before support 23 Sara Sinani, video production Aleksandar Jovanovski Zhurcin bb, Tetovo Equipment Not implemented yet 24 Futura still Ul. 101, br. 166, Jegunovce Equipment Not implemented yet 25 Happy Club Tearce Ul. 101/1, 1224 Tearce Equipment Not implemented yet 26 Vida Filigri, handmade filigree Zhelino Equipment Registered after support EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 135 Sub-Study 4: Adventure Travel Initiative Photo credits: left: SBEP (USAID/Macedonia); center and right: Adventure Travel News This performance review of the SBEP Adventure Travel initiative is based on multiple sources, including project quarterly reports and a 2016 survey that reached 21 of 37 travel organization that worked with SBEP to introduce this initiative in Macedonia. The evaluation team also conducted interviews for this section with SBEP’s partner, the Adventure Travel Trade Association (ATTA), the government Agency for Tourism, and SBEP staff. Introduction In 2013, the year SBEP initiated its work on the Adventure Travel initiative, Macedonia received approximately 702,000 tourists, about 57 percent of which were from foreign countries.17 Although tourist arrivals have risen since then, as the chart on the right indicates, Macedonia continues to rank lower than most other countries in Western and Southern Europe, and well below countries with vibrant tourism sectors, such as Italy, Greece and Turkey. On the World Economic Forum’s Travel and Tourism Competitiveness Index for 2017, Macedonia ranks 89 out of 141 countries.18 Nevertheless, according to the World Travel and Tourism Council, “The total contribution of travel and tourism to GDP [gross domestic product] was MKD 36.5 billion (6.7 percent of GDP) in 2016” compared with 1.4 percent of GDP in 2014. 17 National Tourism Strategy Republic of Macedonia, Kohl & Partner 18 Available at: http://www3.weforum.org/docs/WEF_TTCR_2017_web_0401.pdf Source: State Statistical Office of Macedonia EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 136 Evolution of the SBEP Adventure Travel Initiative SBEP’s Adventure Travel initiative grew out of discussions about a rural tourism initiative in rural municipalities in the Polog region in 2012, and focused from the beginning on hiking, horseback riding and other activities that tourists in this region enjoyed. By September 2013, SBEP’s vision had expanded to include a larger range of activities and geographic areas and to target the fast-growing niche market for adventure travel. This led to a partnership with Seattle-based ATTA, a global membership organization made up of tour operators, tourism boards, specialty travel agents, guides, and accommodations, media, and service providers that seek to promote adventure businesses and destinations. The market ATTA represents encompasses “hard” adventure travel, such as caving, mountain climbing, paragliding and kite surfing; “soft” adventures, including backpacking, cycling, fishing, hiking, hunting, rafting and sailing, and “non-adventure” travel ranging from attending festivals and fairs to educational programs, including language learning. Globally, adventure travel was already growing fast in 2013 and was estimated to be worth $263 billion worldwide that year. An ATTA market study report said at the time that it was growing at a rate of 65 percent per year. The Adventure Travel component of SBEP partnered with ATTA and involved collaboration with the USAID Europe and Eurasia Bureau’s new Regional Economic Growth project and with Macedonia’s Ministry for the Promotion and Support of Tourism. “Yesterday, a group of 20 international media and tour operators crossed the border between Albania and Macedonia. During the next four days we'll … get to know the local touristic offers and get in touch with the tourism sector. After Kosovo and Albania, Macedonia is the last country on the list, as part of the two-week tour "Adventure week – Western Balkan." The aim of the tour is to connect the local markets for active tourism in these countries with the tour operators from North America and Western Europe in order to attract more foreign tourists. Some of the participants are: the chief editor of National Geographic Traveler, Keith Bellows; the editor of Yahoo Travel, Jo Piazza; the editor for Southeast and East Europe of Lonely Planet, Brana Vladisavljevic, as well as other influential media from this sector. karas4training.com, October 2014 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 137 Under the Adventure Travel initiative, SBEP quarterly reports show that the project played a broad role in introducing adventure travel to Macedonia, included helping with planning and publicizing its offerings abroad and in the international media. SBEP supported an Adventure EDU training, which ATTA experts delivered to 60 tourism industry professionals from Macedonia to build the capacity of key industry members to deliver adventure travel services. SBEP also supported Adventure Week, an event that helped establish cross-border partnerships and cooperation in the region between ATTA, the National Tourism Agency, and local tourism stakeholders. Notably, Adventure Week showcased local adventure travel stakeholders to international buyers and ATTA’s network, facilitating opportunities for local businesses to establish international business relations. Additionally, SBEP reported that it held an event, Discover Krushevo, that highlighted Krushevo’s tourist attractions. As the initiative evolved, SBEP helped establish new sporting events and draw tourists to locations that offered multiple active tourism opportunities. By supporting these events, SBEP sought to increase demand from domestic and foreign tourists, and motivate the organizers to build their capacities and adopt international standards to earn more business. The initiative also promoted the publishing of an article in Cross Country magazine on paragliding in Macedonia, providing information about paragliding sites and tourism opportunities in the country. Aside from supporting events and publicizing adventure travel, SBEP also provided direct support to local businesses. Partnering with the Aviation Federation of Macedonia, SBEP provided essential equipment for the take-off and landing locations of four paragliding sites, including a solar￾powered weather station, windsocks, a frame for checking the paragliding gear, information boards, and installation of a repeater system for amplifying radio waves to facilitate communication among paragliders at greater distances. Exhibit 4.1 provides a chronology for this initiative that highlights additional contributions to the development of this market. Adventure Travel Initiative Results Economic gains from adventure travel flow directly from the number of tourist visits, the duration of their stay, the money they spend, and how they spend it. For Macedonia (and globally), adventure travel broke new ground and totally redefined Macedonia as understood by travelers and the industries on which the traveler’s experience depends. The effects of SBEP’s efforts are evident in the number of activities, events, locations, partner organizations, and unpaid advertising the initiative mobilized and documented in quarterly reports. Quantitative data on tourist gains attributable to the initiative and to adventure travel worldwide are more difficult to pinpoint because national statistics, such as those shown above for Macedonia, capture national improvements on standard measures, but they lack the precision needed to monitor niche markets. Notably, other organizations have begun to notice Macedonia’s adventure travel potential and are working toward quantitative approaches for tracking progress. These include industry surveys that try to gauge gains based on information businesses share. In this vein, the evaluation team conducted a survey of travel and sports-related firms and associations with which SBEP worked on this initiative, which is discussed below and in Exhibit 4.2. Activities that break down national statistics to track changes in microsites closely linked to adventure travel attractions are complementing these kinds of efforts. A 2016 Market Analysis of Tourism along Western and Eastern circuits and the Central Corridor in Macedonia, produced under an EU and World Bank–funded Local and Regional Competitiveness Project with the government, is advancing this strategy through a circuit-based microsite approach to segmenting national data to track segments of the tourism market. Using a less granular regional approach, which includes tourists who visit cultural and historic sites and those who engage in sports, the evaluation team graphed tourist arrivals during SBEP period for the four regions in which it worked most extensively. As the Figure below shows, trends varied across these regions, performing best in Vardar during the period. Given how new the Adventure Travel initiative is EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 138 for Macedonia, it would not be surprising if tourists in that niche market grew in other SBEP focus regions, but could not be detected, making alternative tracking approaches even more crucial. Perceptions and Expectations The evaluation team used a combination of individual interviews and an industry-focused phone survey to gather perceptions of SBEP’s key partners in the Adventure Travel initiative. Describing its impressions of SBEP’s contributions, Macedonia’s Tourism Agency told the evaluation team: “Adventure Tourism is one the of the fastest growing subsectors of tourism in the country. The SBEP project earned very high respect not only from us as an agency, but also from the private sector. We were very satisfied with the Adventure Next Balkan conference. It was the first event of this type in the region, and it achieved its objective. We collaborated with them on other activities as well and are planning more for the future.” ATTA had similarly high praise for SBEP, telling the evaluation team: “It was the professionalism and understanding of the SBEP team that helped achieve results and early outcomes. When we had first started work in Macedonia, there had been no “team spirit” in the industry, but with every next event, after SBEP became involved, we could see how the industry was more open for collaboration and partnership, not just in Macedonia, but around the region. We hope that momentum will not be lost with the closing of the project.” In interviews with tour operators, travel agencies and associations, and sports-related firms, the evaluation’s survey research on adventure travel (see Exhibit 4.2) found this same sense of enthusiasm and optimism. The vast majority of respondents (91 percent) said their sales have risen since 2012, and their customer base has grown (43 percent), increased primarily by tourists who are looking for challenges and adventure (67 percent) and who tend to spend more when they visit (76 percent). These service organizations participate in regional and seasonal events (86 percent), say their financial situation has improved (67 percent), and have recently invested in their businesses (62 percent) and have plans to invest further in the future. 0 10,000 20,000 30,000 40,000 50,000 60,000 70,000 80,000 90,000 2011 2012 2013 2014 2015 2016 Tourist Arrivals by Year in Regions where SBEP Was Active Pelagonia Polog Vardar Region Northeast Region EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 139 Job Creation Among the tourism-related firms surveyed, 47 percent employ less than 10 people. Another 24 percent have 10 – 20 employees, and a small number of firms are larger. In response to an MSI survey question, 26 percent of these firms indicated that they had increased the size of their staff in the past year or two. The remainder indicated that the size of their firm remained unchanged during this period. SBEP, in its performance management tracking, estimated that about 165 jobs could be linked to its Adventure Travel initiative, but this number was reportedly extrapolated from the projected revenues resulting from the media value of the promotional and advertising efforts within the initiative and is not an actual count of jobs added. Adventure Travel Initiative Sustainability Evidence of the likely sustainability of SBEP’s Adventure Travel initiative emerges from the evaluation, including: • Strong growth in this subsector worldwide • Early returns in this market-driven industry in the form of increased sales and new customers to Macedonian firms that have been involved • Firm backing from government agencies and high-level offices • Continued involvement of ATTA, SBEP’s primary partner and an industry leader in this niche market • Successor projects that are advancing the initiative’s agenda further, including USAID’s Regional Economic Growth (a regional development project), and the EU and World Bank’s local and regional competitiveness project with the government • The continued involvement of SBEP’s legacy organization Association for Innovations in Rural Economies in this sector. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 140 Exhibit 4.1: Adventure Travel Initiative Chronology July – September 2013: SBEP established a partnership with ATTA. October – December 2013: The Macedonia Adventure Travel initiative was launched. A Memorandum of Understanding was signed with the National Agency for Promotion of Tourism. January – March 2014: Adventure EDU training was held for 60 local tourism industry representatives in Skopje in partnership with ATTA. A four-day tour throughout Macedonia was organized for ATTA’s Executive Director. April – June 2014: AdventureWeek, a specialized nine-day familiarization trip, was announced. July – September 2014: SBEP organized Discover Krushevo, a one-day event held on July 27 to promote Krushevo’s tourist offerings. Equipment for paragliding sites in Krushevo and Treskavec was installed. AdventureWeek Western Balkans took place at the end of August and the beginning of September. October – December 2014: Several inbound tour operators and SBEP/Tourism Agency representatives attended the annual ATTA World Summit in Killarney, Ireland, from October 6 – 9. At the summit, Macedonia was declared the host of the first ever Western Balkans Adventure Summit, the Adventure Connect Regional Event planned for the spring of 2016. In preparation for this event and to promote Macedonian adventure sports, a paragliding cup competition in precision landing took place in Treskavec, near Prilep. January – March 2015: About 50 members of the tourism industry in Macedonia attended the EDU workshop that took place in March, covering the topics of risk management and operational excellence. SBEP assisted in the development of a hiking trail in Western Macedonia, a new adventure travel product that would be a continuation of the Via Dinarica. Coordination activities made for Adventure NEXT – Balkans, the regional Western Balkans Adventure Travel Conference in May 2016, continued. April – June 2015: Several SBEP-supported events took place, including the Krali Marko Bouldering Event, Demir Kapija Rock Climbing Outdoor Festival, Mariovo Off-Road Biking Event, Azot Biking Event, Mavrovo Mountain Bike Race, and the Gostivar Open Paragliding Cup. July – September 2015: Additional SBEP-supported adventure sport events took place, including Shara Mountain Sky Race, Popova Shapka Mountain Festival, Krali Marko Trails, and Discover Pelagonia events. Macedonia was promoted as a biking destination at Eurobike in Germany. October – December 2015: Macedonia was announced as the next host of Adventure NEXT in Ohrid during the annual ATTA World Summit in Chile. January – March 2016: ATTA held a two-day workshop with the tour-operators and destination representatives participating in the Adventure NEXT – Balkans conference. SBEP organized a study tour to a paragliding area in Italy for the Mayor of Krushevo and local paragliding pilots so that they could get better acquainted with the local economic development practices of the hosts. April – June 2016: Adventure NEXT - Balkans, the first regional conference in adventure travel in Europe, took place in May in Ohrid, hosting nearly 300 delegates from 30 destinations worldwide. SBEP helped provide a meteorological station for one more paragliding take-off site and artificial grass for a paragliding take-off site in Krushevo. SBEP supported four additional adventure sports events. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 141 Exhibit 4.2: Adventure Tourism Survey Summary The first part of this survey asked about the respondent’s organization. The second part asked about the respondent’s awareness, experience and perceptions of SBEP. Business Trends Affecting Respondent’s Organizations19 Perceived importance of travel and tourism in Macedonia’s economic development plans: • Very important: 84 percent Sixty-two percent of respondents were aware of national or local economic development plans in Macedonia, and 38 percent had participated in their development through task forces or public meetings. Changes in respondent’s organization during the past 2 – 5 years (since 2012) Sales of the organization’s services/goods: • Increased: 91 percent • Decreased: 9 percent Customer base: • New customers started buying our services/goods: 43 percent • We increased the number/types of services/goods we offer: 23 percent • On an open-ended question, respondents identified some of the new services they have added, which included a rock climbing space and sponsorship of a new competitive sports event. Organization’s customers or clients age: • Becoming older as a group: 33 percent • Becoming younger as a group: 14 percent • No visible change: 52 percent Organization’s customers or clients’ interests: • Looking to rest and relax: 14 percent 19 Answers to questions asked of this survey’s 21 respondents are shown in percentages based on the number of respondents who actually answered each question, ignoring missing responses (valid percent). Profile of the Evaluation’s 21 Adventure Tourism Phone Survey Respondents From an SBEP list of 40 individuals with whom the project had worked on Adventure Tourism (including government agency representatives), the evaluation team identified 37 who were directly involved in this sector as tour operators, travel firm or association representatives, or sports club or business venture owners/managers. Of these 37, the team reached and interviewed representatives of 21 (57 percent) of these organizations. Initial survey questions showed that 67 percent of respondents were ages 30 – 49, and another three individuals (14 percent) were 29 years old or under (i.e., youth). 95 percent were men and 91 percent held university degrees; and 76 percent had worked in this field for six years or more. Eight (38 percent) indicated that they were owners or owners/managers of tourism￾related businesses; another eight (38 percent) said they worked for associations involved in tourism, and nine worked for sports or wellness organizations. Regardless of whether they worked in for-profit or non-profit establishments, 57 percent of the survey’s respondents indicated that they had founded the organizations they represented. As to the aspects of travel or tourism with which those organizations focused, most respondents identified 2 – 3 features, with sports and other recreation activities (including equipment) cited most frequently (19 percent of respondents), followed by cultural activities, including historic sites and festivals (17 percent), accommodations for visitors (15 percent), and travel and transportation services (13 percent). Close to 80 percent are registered organizations, and 91 percent have only one location from which they operate; 85 percent of the organizations respondents represented were founded after 2000. 71 percent of the organizations that belong to an association said they benefit most from its training and advice (43 percent) and its facilitation of sales of their services (33 percent). EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 142 • Looking for challenges, activities: 67 percent • No visible change: 19 percent Customer or client’s spending: • Increased: 76 percent • No visible change: 24 percent Cost of doing business: • Increased: 76 percent • No visible change: 24 percent Respondent Organization’s Employment Opportunities Number of people who work at the organization? • 0 – 9: 48 percent • 10 – 19: 24 percent • 20 – 49: 14 percent • 50+: 14 percent Number of people working for the organization during the past 2 – 3 years: • Increased: 29 percent • Stated the same: 67 percent Does this type of work have good prospects for young people to get started earning a living after finishing school? • Yes: 38 percent • About the same as other types of work: 24 percent • Fewer opportunities for youth: 33 percent Respondent Organization’s Business Development Activities during the Past 2 – 5 Years Organization participated in regional, seasonal or other tourism promotion events? • Yes: 86 percent • Answers to the survey’s open-ended question showed that firms surveyed participated in events in a number of European locations as well as events in Macedonia, citing business promotion and their interest in attracting more of Macedonia’s growing number of tourists to their location. Organization received technical advice, training or additional equipment: • Yes: 81 percent • No: 19 percent Value/importance of technical advice received: • High: 83 percent Value/importance of training I or other employees received: • High: 89 percent EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 143 Value/importance of additional equipment investments: • High: 100 percent Respondent Organization’s Current Situation and Future Prospects Overall, considering the financial situation and business development efforts made by the organization where you work over the past 2 – 5 years, how satisfied are you with its performance? Very satisfied Somewhat satisfied Somewhat unsatisfied 43 percent 48 percent 5 percent • On an open-ended question about why they were satisfied or very satisfied with their organization’s performance, 10 respondents (48 percent) cited the increased number of tourists coming to Macedonia as the primary reason. How would you characterize the current financial situation of the organization where you work? Better than last year About the same as last year Worse than last year 67 percent 19 percent 5 percent What other factors are most likely to have an impact on the future prospects of the organization where you work? Factors Affecting Future Business Prospects Percentage of Respondents saying “Very Important” Business production/service levels 94% Availability of subsidies for businesses 93% Access to finance 88% Business access to international markets 84% Business cost of production/service delivery 84% Economic conditions in the EU and other large economies 73% Foreign investment in Macedonian business 69% Business access to markets throughout Macedonia 69% General economic conditions in Macedonia 68% Price of business inputs 67% Macedonian government policies 65% Macedonian government leadership 63% International product standards 63% What are your expectations for the organization’s financial situation over the next 12 months? Will improve Will stay the same Will get worse 52% 38% -- Did the organization’s owners/leaders make investments in it out of their own or the organization’s resources (beyond normal operating costs) over the past year (2016)? • Yes: 67 percent • No: 33 percent How likely is it that the owners/leaders of this organization will make investments in it out of their own resources (beyond normal operating costs) next year? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 144 Very likely Somewhat likely Not sure Somewhat unlikely Very unlikely 52% 19% 29% -- -- Considering these future prospects in your professional situation, what is the outlook for your family or household situation? Very optimistic Somewhat optimistic Not sure Somewhat pessimistic Very pessimistic 62% 33% 6% -- -- Respondent’s Awareness, Experience and Perceptions of SBEP Respondent was aware of SBEP before this survey. • Yes: 88 percent Source of initial awareness? Television or newspaper Another firm or person SBEP approached us Some other way 25% 28% 23% 23% Types of interaction with SBEP and how respondent rated their value for themselves and their organization Type of Interaction with SBEP Perceived Value High Moderate Low Participated in an SBEP training program 86% 14% -- Received equipment or inputs for this business from SBEP 86% 14% -- Participated in an SBEP-organized tourism promotion activity 86% 14% Business received publicity as a function of its interactions with SBEP 85% 8% 8% Received technical advice from SBEP specific to this business 80% 20% -- Attended an SBEP conference or workshop 60% 40% -- Regarding the impact of SBEP on respondents from this group of tourism, travel and sports-related organizations, the evaluation’s “most significant change” question brought forth the following beneficiary observations: • I got the idea to increase the business by buying another old building that I intend in the coming period to equip. • We got the support and they helped us to realize “Discover Krushevo.” • It took our organization to a higher level. • Great financial support • Promotion of our business • We got more clients thanks to partners we’ve met because of SBEP. • Beneficial cooperation in education, technical assistance, contacts and promotion of rural development • Project enabled us to return the old touristic offer of Prespa Lake and employ people from Resen. • No significant change • Increasing performance standards EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 145 • We offered value added services to clients and we paid to volunteers for marketing and promotion. • Direct promotion of the company and more sales • Learning about other associations that are in the same industry • Familiarization with foreign companies that are in the same sector, exchange views and opinions with them and signing partnerships EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 146 Sub-Study 5: Wild-Gathered Products Initiative Photo credits (left and right): Dragan of Stip; center: USAID-funded Western Balkans Geotourism Network Leave No Trace training program. This performance review of the SBEP Wild-Gathered Products initiative is based on multiple sources, including project quarterly reports, primary data collected by the MSI evaluation team through interviews, the United States Agency for International Development’s (USAID) 2008 Profile of the Macedonian Wild-Gathered Products Value Chain, and updated export data from the United Nations (UN) database, Comtrade, through 2016. Interviews for this review were carried out with the following: • A representative of the Macedonian public enterprise Makedonski Shumi, which is responsible for the management of forest resources • A representative of the Mavrovo National Park-Mavrovi Anovi on the western side of the country, where SBEP provided technical assistance and equipment for three wild product collection or purchasing centers • The master trainer on non-wood forest product collection from ProBio, an SBEP local partner organization • SBEP staff. Introduction Macedonia has a long history in gathering non-wood wild forest products both for personal consumption and as a source of income for some people living in otherwise underdeveloped areas where such products grow freely. In 2008, USAID/Macedonia’s Abribusiness Program (2007 – 2013) identified this subsector as one of potential interest for its export potential and family income generation prospects for those involved in wild product collection. USAID’s 2008 market profile for the wild products sector, cited above, identified mushrooms, wild berries and medicinal plants as three of the main forest products for which markets exist, the largest being the market for mushrooms in European Union (EU) countries. Within the EU, the study showed that Italy was consistently the strongest individual country market, accounting for 65 percent of EU sales in 2007. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 147 Source: USAID 2008 Profile, op cit. When this AgBiz profile of wild-gathered products was written, the number of wild product gatherers was estimated to be about 12,000. Gatherers were not formally trained and were not linked together in an association or similar network. Gatherers sold to traders who came around during growing season and then sold wild products to domestic processors and export firms. Evolution of the SBEP Wild-Gathered Products Initiative SBEP’s Wild-Gathered Products Initiatives grew out of discussions with local private and public-sector stakeholders in the Polog region, and wild product gathering was sometimes the only source of income for poor families living in or near forests. The initiative, which began in 2012, had the following three main dimensions: • At the policy level, SBEP collaborated with national authorities, including Makedonski Shumi, a public enterprise responsible for the management of forest resources, including non-wood forest products; the Polog Regional Development Center; and representatives of the Mavrovo National Park to strengthen legal frameworks for wild product gathering and rationalizing value chain mechanisms for their collection, sale and export. • At the collection and purchasing center level, SBEP worked with private sector entrepreneurs and Mavrovo Park authorities to establish centers where wild product gatherers could sell the forest products they collected and commercial buyers would purchase these supplies. At the start of this effort, SBEP noted in a quarterly report that “the Park currently lacks a mechanism for tracking the quantities of wild-gathered products collected in the National Park, who collects it, or who the final buyers are.” EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 148 • At the level of individual wild product gatherers, SBEP worked with various stakeholders to develop and provide one-day training courses on wild product gathering for which participants earned certificates that helped them obtained gathering licenses (after a licensing requirement was introduced for the government in 2013 by Makedonski Shumi, the public enterprise that oversees non-wood forest products). SBEP’s vision for this initiative included creating 300 stable seasonal jobs and to generate improved incomes of up to $7,800 annually for skilled wild product gatherers (a figure more than 60 percent above the average national per capita income). SBEP also envisioned the creation of links between the collection centers it would help establish and private sector firms — within and beyond Polog — that could engage in the processing or export of non-wood forest products. By the time SBEP began to examine the wild-gathered products value chain, Makedonski Shumi had also begun to strengthen its involvement in this area, issuing regulations and setting standards for the collection of non-wood forest products and for the training, certification and licensing of wild product gatherers. Wild-Gathered Products Initiative Results Business Framework and Practices Adopted SBEP, along with its partner, PointPro, organized and facilitated meetings between Mavrovo National Park representatives about the adoption and implementation of a new, sustainable business model for wild product collection and sales. This included meeting with major buyers of wild-gathered products to explore opportunities for a collaborative model between the park and the respective companies. To this end, the park authorities explained to the evaluation team that they began formally recruiting firms with which they would establish arrangements for purchasing wild products through an open call, under which firms would pay 30,000 MKD (Macedonian denars) to operate as a registered collection company. At the end of each contract period, firms would also pay a fee to the park based on the number of kilograms of wild products purchased. In 2016, five firms — all of which were export oriented — held contracts with the park. In addition to helping link Mavrovo Park authorities with firms that would buy products through the collection centers in the park, SBEP also sought to build links between gatherers in Polog and outside the park, and with firms in other regions. In Terace, one of nine Polog municipalities, SBEP introduced to municipal representatives a Pelagonija-based company, Vigan, a processor of wild mushrooms, herbs, and fruits. Vigan followed up by initiating purchases of wild berries from regional gatherers in Polog. “We are familiar with the SBEP project, being directly or indirectly involved in the implementation of non-timber product (NTP) initiatives in several regions. We have collaborated with SBEP on the development of some rulebooks for regulating wild-gathered products and one guidebook for collectors which is on our website. From our involvement with SBEP there was an institutional benefit gained in the area of very sensitive social, economic and environmental challenges involved in the exploration and management of NTP initiatives. SBEP enhanced the whole process and system, enabling better public-private dialogue, creation of quality documents and at the same time the dissemination of knowledge and experience on all levels of stakeholders.” - Makedonski Shumi Representative EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 149 Wild Product Collection and Purchasing Centers Established In 2013, SBEP began working with local stakeholders to identify arrangements and sites for wild product collection or purchasing centers, in parallel with a market analysis and business model framework it commissioned from a local partner (see text box). Mebi Prima, a private sector firm, was the first center to open in the village of Pirok, a rural municipality of Bogovinje in Prolog, where SBEP’s dialogue with the mayor had helped identify an entrepreneur interested in entering this market. Subsequent discussions with Mavrovo National Park authorities identified three sites within the park where additional centers could be established. According to park authorities, SBEP’s help with the establishment of Mavrovo Park collection centers included the purchase of measuring and storage devices and dryers for herbs. Billboards with educational information about wild-gathered products were also provided. According to park authorities, the three centers that received assistance were Vrbeni, in a park neighborhood unit; Pilana, on the livestock cooperative premises; and Jance, on the premises of a primary school. Recordkeeping at each site — based on purchase bills provided to companies that buy wild products from individual gatherers — are provided to park authorities annually. Other sources provided the evaluation team with conflicting information on these centers, suggesting that although SBEP had equipped all the park sites, only one of them — the Vrbeni location — was operational as of December 2016. One of the other two centers was reportedly still not registered, and the other was not yet operational. MSI was unable to determine the exact status of these collection centers; when the study data were being collected, weather conditions prevented the team from visiting the sites. According to park authorities, mushrooms account for 90 percent of wild products gathered in Mavrovo Park. Berries are also collected and sold, though much of that product is retained for domestic consumption, one respondent indicated. There are, however, some exceptions to this observation. For example, outside the Mavrovo Park, SBEP worked with Mebi Prima to contract with gatherers to collect blueberries one year in August. As explained in SBEP’s quarterly report, “The month-long blueberry season yielded purchases of over 10 tons of fruit. Collector incomes per month from this short season saw a drastic improvement of up to 115 percent — from 8.935 MKD (approximately USD$186) to 19.252 MKD (approximately USD$428) on average — extremely promising results. Now into wild apples and juniper season, the Center has already purchased over 500 tons of juniper berries. SBEP’s research and experience to date shows that collectors can work 8 – 10 months during the year. They can earn a regular income of up to USD$950 per month, more than twice the average Macedonian salary.” In 2013, USAID’s new SBEP project asked one of its partners, PointPro Consulting, to take another look at the wild-gathered products market and help develop a sustainable business model for the sector. Its report echoed the earlier AgBiz study in finding the market for wild-gathered products to be fragmented, with less than a dozen firms involved in wild mushroom and berries purchasing and exporting, and about half that many trading in medicinal and aromatic plants. PointPro’s survey of 30 gatherers in the Mavrovo region, which provided a baseline for SBEP’s work, found that 81 percent were men, and that almost half said that this work was their family’s only source of income. We can say that getting the support from the SBEP project, we achieved to accomplish most crucial segments of the technical process and administrative procedure [for the new business model]: equipping the collection centers, providing training and education for collectors, preparation of guidelines for collectors … and at the same time strengthening the capacity of our institution by involving us in all steps and activities. - Mavrovo Park Representative EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 150 Another observation made during interviews was that despite Mavrovo Park authority’s efforts to encourage the collection of other wild products, the collection of medicinal plants was limited. One interview respondent suggested that this might be because none of the firms with which the park has collection center contracts buys the plants. This is notable given that written reports on wild product markets in Europe indicate that the beauty product industry demand is rising. Notably, SBEP’s first quarter report for FY2016 indicated that it was working with Mavrovo Park authorities to review park policies and the performance of several of the firms currently holding two-year collection center purchasing contracts with the park, with one issue being the range of wild products those firms purchase. Upgrading Wild Product Gatherers’ Knowledge and Qualifications During its Wild-Gathered Products initiative, SBEP, through its local partner, ProBio, trained nearly 2,000 gatherers on best practices for this sector. Although SBEP’s final report indicated that 1,770 collectors were trained (among them 235 women), the initiative’s master trainer provided the evaluation team with a somewhat higher total in December 2016, and its regional distribution (see text box). SBEP documents describe these workshops as helping to establish ground rules for sustainable collection in areas of intense exploitation, such as Mavrovo National Park, and promoting the formalization of the sector. The workshops also introduced the necessary minimum level of traceability and quality assurance needed to guarantee the sector’s continued competitiveness, which helps supply the cosmetic, pharmaceutical and food processing industries in Macedonia. Participants were awarded a completion certificate at the end of one-day training sessions provided to wild product gatherers who participated under SBEP. To legally gather and sell wild products commercially, Makedonski Shumi has required citizens to obtain licenses since 2013. In Mavrovo Park, authorities described two types of licenses they issue, one for 400 denars each year to individuals who live within the grounds and gather on a noncommercial basis, and a commercial license for gathering that nonresidents may purchase for 1,500 denars per year. Park authorities declined to provide the exact number of licensed gathers operating in the park, saying that it was believed to be 570 – 600. For gatherers operating commercially and hoping to sell what they collect to export-oriented firms or in the domestic organic food market, the park has also arranged for the provision of certificates of origin since 2015 through PEOSERT, an accredited international certificate company. SBEP reporting indicates that it engaged in outreach and promotional activities, such as television advertisements and educational methods, to increase the reach and effectiveness of the Wild-Gathered Products initiative. A television advertisement was created and broadcast widely to inform collectors about opportunities to attend trainings and become certified through SBEP. Additionally, study tours to existing collection centers (in Makedonski Brod, Probistip and Veles) were organized for the new collection centers’ representatives to promote information sharing and help learn best practices. In 2013, SBEP, in consultation with Makedonski Shumi, produced a manual on sustainable Wild Product Gatherers Trained (per ProBio, December 2016) Polog Region: 365 Mavrovo National Park: 942 Pelagonia Region: 213 Northeast Region: 450 Vardar Region: 325 Total: 2,295 “We are issuing only approximately 3,000 [commercial gathering] licenses for collection per year. We estimate that annually there are 10,000 gatherers nationally.” - Makedonski Shumi Representative EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 151 collection of wild herbs and mushrooms. The manual contains information on about 40 of the most common herbs and mushrooms found in the forests across Macedonia, along with details on how and when they are collected. The manual is designed to be portable to help aid in collection, so collectors can bring it along with them when going out to collect. The manuals are available in Macedonian and Albanian languages, and each collector that was trained by SBEP received a copy together with their certificates. Makedonski Shumi representatives indicate that this manual is also available on its website. Wild Product Initiative Impact SBEP reporting and USAID’s overall vision of the project suggest that the Wild-Gathered Product initiative was selected as an element of the program for two distinct reasons. From a regional development perspective (the guiding rationale for this project), the wild product sector represented an important opportunity for improving links along a value chain that holds potential for expanding business opportunities, as well as creating jobs and improving worker income in otherwise isolated and low￾income, high-unemployment sub-regions in the country. At the individual level, these same prospects, along with rigorous requirements in the organic product and export markets, justified investments in wild product–gatherer workforce development efforts that, in turn, could help individuals working in this sector to increase the market value of the products they gather, and therefore their incomes. Wild Product Gatherer Jobs and Income Job Creation Job creation is a core objective of all the initiatives supported under SBEP. Performance Indicator #1 for the project tracks new jobs created as a result of SBEP-linked activities. Targets were identified for performance on this indicator annually, but not for specific initiatives under the project. Thus, the project had no specific job creation target to meet for its Wild-Gathered Products initiative. Nevertheless, in its project closeout and final report, SBEP attributed 616 new jobs created over the life of the project to the Wild-Gathered Products initiative, of which 41 were reportedly jobs taken by women. In the wild grown products subsector, the meaning of jobs differs from how it is used in other sectors because collectors are actually self-employed individuals who sign purchasing contracts with firms that buy non-wood forest products, including mushrooms, berries and medicinal plants. To operate commercially and sell what they gather legally, wild product collectors have been required to have a government license since 2013. The public enterprise, Makedonski Shumi, told the evaluation team it is currently issuing 3,000 collection licenses per year, knowing that another estimated 7,000 people collect forest products without licenses, probably for their personal use. Under its Wild-Gathered Products initiative, SBEP trained gatherers in small groups in Mavrovo National Park and elsewhere in the country. Participants received a completion certificate at the end of this one-day training. SBEP’s closeout performance and financial report states that 1,770 collectors — of which 235 were women — were trained in 74 one-day sessions. In an interview with SBEP’s master trainer from ProBio, the evaluation team was given slightly higher figures, disaggregated by region (see text box). Participants in SBEP trainings did not receive a commercial gathering license from SBEP because the project was not a licensing agent. Commercial gathering licenses for collecting forest products in the Mavrovo National EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 152 “The baseline information which SBEP collected in 2012 showed that the average monthly income was US$198. The data from 2016 showed that it had risen to US$281, i.e., that there had been an increase of 42 percent. Even though wild products collection is regarded as seasonal employment, the collectors can still work some 10 months during the year, collecting various products and earn a regular income. As the majority of the population in these rural areas is unemployed, this is an excellent opportunity to secure a livelihood.” SBEP Closeout Performance and Financial Report, April 2017 The evaluation team did not independently collect income data from wild product gatherers. However, if it is assumed that SBEP partner, PointPro, replicated the survey it conducted and reported on it to SBEP in November 2013, these data are likely to be reliable regarding the change in incomes reported. It may be worth noting that PointPro’s original survey reported a monthly income range of 6,000 MKD to 18,000 MKD for gatherers, rather than what may be an average of 9,900 MKD, which converts to US$198 today, five years later. Thus, the gains recorded may be higher or lower for individuals depending on where they fell in this range and the range that prevailed in 2016. Export Growth and Regional Development As observed by SBEP early in it work on wild products, data that would help USAID to test the hypothesis that development of the wild product sector in the regions in which the project worked has contributed to regional economic development by expanding regional earnings from exports of wild Park cost 1500 MKD, according to park authorities. SBEP’s master trainer for this initiative was unable to tell the evaluation team how many of the SBEP trained collectors had obtained commercial licenses that would allow them to sell what they gather and make their work consistent with generally accepted definitions of seasonal labor — the evaluation team was told that many gatherers reportedly work about 10 months a year. In its review of SBEP quarterly reports, the evaluation team did not find information on the number of SBEP-trained gatherers who obtained licenses either on their own or as part of an arrangement with a licensed purchasing company (Makedonski Shumi reported that there are 3,000 of these, five of which are licensed to purchase in Mavrovo National Park.) What the evaluation team found instead were projections near the start of the project Source: SBEP Quarterly Report: January – March 2013 period of how many jobs it was hoped the project’s investment in the Wild-Gathered Products initiative would yield. Among the documents the evaluation team reviewed on this topic after SBEP had closed, this estimate is the closest it could find as a basis for the jobs created number attributed to the Wild-Products Initiative on that SBEP performance indicator. At the individual income level, SBEP’s closeout and final report indicates that the initiative had “quite an impact.” EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 153 products, or increased domestic consumption. SBEP itself did not monitor and report to USAID either the volume or the value of wild product sales in the collection centers it helped establish in Mavrovo Park or those of the private sector collector it assisted, Mebi Prima. Discussions with Mavrovo Park authorities suggested that records are being kept in the operational centers, using measuring devices provided by SBEP and as part of the purchasing bill protocols the park requires and uses to establish fees owed by the firms it contracts to buy from gatherers at those sites. However, the evaluation team did not try to obtain or analyze those records. An interview response from a representative of Mavrovo Park indicated that the volume of wild products sold from park collection sites had been increasing each year. Additionally, this interviewee said: “We can provide you unofficial and only approximate data regarding the registered quantity of non-timber products exported per year (for the period of the last few years, excluding 2016): 20,000 to 25,000 tons. The real and exact data [on volume gathered and sold] cannot be found in any statistical sources, but the registered number of exports is definitely in the Customs Office of Macedonia.” Mavrovo Park Representative Similarly, no documentation of this sort was provided during the team’s interview with a Makedonski Shumi Representative. This interview, however, yielded estimates in this regard. “My estimation of quantity of collected fresh NTP is 50,000 tons per year, against 5,000 tons of declared exported quantities; the ratio of fresh/dried NTP is 1:4. Overall 70 percent of the collected quantity is mushrooms; the remainder involves berries and herbs. Part of the quantities of collected fruits like berries, is simply “disappearing” because it is transformed in the recordkeeping into general fruits quantities. On the collection side, we are issuing approximately 3,000 licenses for non-timber forest products per year. We know the actual number is larger. We estimate that annually we have 10,000 active gatherers. At the next stage, we have 3,000 collection companies that are purchasing with licenses.” Makedonski Shumi Representative MSI researched Macedonia State Statistical Office export reports to better ground these estimates and provide USAID with clearer information on national-level results to which the SBEP Wild-Gathered Products initiative may have contributed, but found that the office did not provide sufficiently disaggregated commodity data. To move forward, the evaluation team shifted to the UN’s Comtrade database, which provides up-to-date export information for most countries, including Macedonia. Using USAID’s 2008 profile as a starting point, MSI matched product descriptions used for mushrooms, berries and medicinal plants in that report to current Harmonized System (HS) codes. Data were then drawn for the five-year period from 2012 to 2016 for each product. Markets on which data were pulled included the world for all products and Italy for mushrooms, based on the large market for Macedonian mushrooms reported in 2008. Exact comparisons to data in USAID’s 2008 report were not possible, however, because data could not be pulled from the no-cost, online database for the EU, which is the unit of analysis the 2008 report used. Nevertheless, the data extracted were useful. In particular, they showed no important change in exports over SBEP period for fresh or dried mushrooms, or for berries of the types examined in 2008 or medicinal plants. However, exports of processed mushrooms (cans or jars) increased dramatically, suggesting that SBEP’s efforts to connect wild product gathering in Polog with mushroom processing in Pelagonia may well have been on track with market trends. The Tables below present export data on wild-gathered products from Macedonia. Regarding these data, the distance is far between what the evaluation learned in interviews at project sites and export EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 154 statistics, and no assumptions should be made about causality without additional data that verify intermediate stage transitions. However, these data may generate questions and hypotheses, and USAID and SBEP’s various stakeholders may wish to follow up on those.20 Mushroom Exports from Macedonia to the World (https://comtrade.un.org/data) HS 070959: Vegetables; truffles and mushrooms (other than of the genus Agaricus), fresh or chilled HS 071239: Vegetables; mushrooms, other than the genus Agaricus and truffles, whole, cut, sliced, broken or in powder but not further prepared, dried HS 200390: Vegetable preparations; mushrooms other than of the genus Agaricus, prepared or preserved otherwise than by vinegar or acetic acid 20 USAID/Macedonia may already have improved data on the country’s wild products production and markets. While completing this sub-study report, MSI noted a recent reference to a report by EPI Centar on this topic through ARD, which the Mission may have commissioned: http://www.epicentar.mk/index.php/component/tlpportfolio/portfolio/value-chain-analysis-of￾the-wild-gathered-products-in-macedonia. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 155 Wild Berry Exports from Macedonia to the World (https://comtrade.un.org/data) HS 081120: Fruit, edible; raspberries, blackberries, mulberries, loganberries, black, white or red currants and gooseberries, uncooked or cooked, whether containing added sugar or other sweetening matter Medicinal Plant Exports from Macedonia to the World (https://comtrade.un.org/data) HS 1211: Plants and parts of plants (including seeds and fruits), used primarily in perfumery, pharmacy; for insecticidal, fungicidal or similar purposes, fresh or dried, whether crushed or powdered Wild-Gathered Products Initiative Sustainability The Wild-Gathered Products initiative ended in April – June 2016, according to SBEP quarterly reports. SBEP’s last quarterly report that discussed this initiative did not report any plans for sustaining the initiative in another institution. Furthermore, in SBEP’s closeout and final report, a section on sustainability brought closure to some of the initiatives that were discontinued as this learning-by-doing project moved forward, but little was said about the sustainability of initiatives that were supported for most of the project’s life span. Regarding the Wild-Gathered Products initiative, much of what SBEP and the PointPro business plan recommended at the start of SBEP’s involvement with Mavrovo National Park have been institutionalized by that entity. Similarly, aspects of its work with Makedonski Shumi appear to have been fully absorbed into that organization. Sustainability for wild product collectors and the firms SBEP helped identify as product buyers either inside the park or in the private sector is apparently expected to be market based. The evaluation team found that the reporting did not account for the trainings provided by ProBio during the SBEP period, for which a continuing audience is likely given that training and certificates appear to be a precondition for commercial gathering licensing. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 156 Exhibit 5.1 Wild-Gathered Products Initiative Chronology July – September 2012: SBEP’s Wild-Gathered Products initiative is launched. October – December 2012: SBEP began establishing Collection Centers for wild medicinal herbs, mushrooms and forest fruits in the Polog region. January – March 2013: The first collection centers were established, and the first round of trainings for collectors began. April – June 2013: The first collection center for wild herbs and mushrooms opened in rural Polog. Four trainings for collectors of wild herbs and products were conducted in the municipalities of Bogovinje, Vrapchishte and Tetovo, resulting in up to 150 trained collectors. July – September 2013: The first batches of blueberries and juniper were purchased by the newly established Collection Center for Wild Herbs and Mushrooms in Bogovinje, demonstrating potential for major impact on rural job creation and income improvement. October – December 2013: SBEP trained 227 new collectors. PointPro developed a study to determine the most appropriate business model for Mavrovo National Park. January – March 2014: Three trainings were organized for wild herb and mushroom collectors in Polog — 95 new collectors were trained, 21 of which were women. Television advertisements informing the local population about the opportunity to engage in wild herb and mushroom collection and encouraging them to apply for training and certification aired for a month in February – March on regional TV stations in Polog. The USAID Mission Director visited Mavrovo National Park and attended a gathering of wild herb and mushroom collectors in the village of Vrben in January. April – June 2014: Nine trainings in sustainable collection of wild-gathered products took place, six of which were in the Northeast region and were the first trainings in that region. July – September 2014: Three trainings in sustainable collection of wild-gathered products took place in Polog and the Northeast region. Mavrovo National Park licensed three companies to purchase wild￾gathered products on its territory and opened one of three planned collection centers. October – December 2014: In the Northeast and Polog regions, 153 new collectors participated in seven trainings on the sustainable collection of wild-gathered products. January – March 2015: Eighty-two new collectors, including 18 women, participated in three training sessions: two in the Northeast and one in the Vardar region — the first training held in Vardar. April – June 2015: In the Northeast and Vardar regions and in Mavrovo National Park, 242 new collectors — 37 of which were women — participated in 11 trainings. July – September 2015: SBEP trained 103 new collectors, including 13 women, in five training sessions in Mavrovo National Park and in the Northeast and Vardar regions. Three companies signed two-year agreements with the park to purchase wild-gathered products from its territory. October – December 2015: Seventy-five new collectors participated in four trainings in Vardar. January – March 2016: SBEP assisted Mavrovo National Park in revising its policy regarding the licensing of companies interested in purchasing wild-gathered products from its territory because the previous policy did not show the expected results. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 157 April – June 2016: SBEP’s Wild-Gathered Products initiative concluded. The initiative trained 1,770 collectors in sustainable collection in 74 training sessions. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 158 Sub-Study 6: Light Manufacturing Initiative Photo Credits: SBEP (USAID/Macedonia) (left and right) and European Bank for Reconstruction (center). This performance review of the SBEP Light Manufacturing initiative is based on secondary sources, including project quarterly reports and primary data collected by the MSI evaluation team through interviews with pairs of firms that SBEP staff matched as buyers and potential suppliers. Interviews and discussions for this review were also carried out with a representative of the government agency Invest Macedonia; staff of the Macedonian Association of Metal and Electrical Industry (MAMEI), an association that is expected to be the permanent home for this effort; and SBEP staff members. Introduction In 2016, manufacturing in Macedonia accounted for 11 percent of gross domestic product. This sector, with 7,967 registered firms, represented 12.2 percent of all firms in 2016, even though the manufacturing sector lost 188 firms between 2011 and 2016. Employment in this sector was reported at 136,899 for the final quarter of 2016. The vast majority of manufacturing enterprises are small — 6,198 firms employed 1 – 19 people in 2015, accounting for 81 percent of employment in the sector. Much of Macedonia’s manufacturing is export oriented, with automotive components, apparel and high technology exports all contributing to these flows. Historically, heavy industry has dominated this sector, which today is more flexible and market oriented. Although there is no standard definition of “light manufacturing,” the term is generally used to refer to products for which all steps in the production process are carried out under a single roof. However, no national data series exists on the number of firms or proportion of the labor force in Macedonia that is engaged in light manufacturing. Reviews of Macedonia’s positive economic growth rates in recent years refer to businesses of this sort and Macedonia’s business focus on customer service and product quality. They also stress the relevance of light manufacturing capacity and the relatively low cost of labor in Macedonia for international firms that are considering investing or establishing their own manufacturing plants there. USAID/Macedonia’s involvement in light manufacturing in the country dates to at least 2007. This subsector was identified then as one of three target sectors (apparel, light manufacturing, and information and communications technology) for attention and assistance from a vertical supply chain perspective in the Mission’s new five-year Macedonia Competitiveness Project (MCP), implemented by the Carana Corporation. As of the midterm evaluation for this project, USAID’s team was working with a small number of light manufacturers primarily in the automotive components field, and with a much larger set of firms outside of the three target subsectors identified in the initial MCP plan.21 From FY2010 to FY2014, USAID supported another project, Investment Development and Export 21 To download the evaluation, visit http://usaidprojectstarter.org/sites/default/files/resources/pdfs/Macedonia%20Competitiveness%20Evaluation.pdf. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 159 Advancement Support (IDEAS), which was implemented by Booz Allen Hamilton. IDEAS interacted with firms in this subsector to some degree because it provided support to the government on improving the enabling environment for business, attracting foreign direct investment, promoting exports, and providing hands-on assistance to a dozen firms to “build their capacity and conduct product and market development activities.” Light manufacturing emerged again as an initiative under SBEP, also implemented by Carana, which began at about the time that MCP was expected to close, possibly reflecting the continuity of corporate management between these two USAID-contracted activities. Evolution of the SBEP Light Manufacturing Initiative Light Manufacturing was one of the first initiatives activated under SBEP. From the start, the initiative focused on import substitution to achieve Macedonia’s goals under its 2007 Law on Balanced Regional Development. The SBEP Second Quarterly Report for FY2013 paraphrases the project’s theory of change for this initiative as follows: “Connections between local small manufacturers and larger firms have been broken or are difficult to establish. Low quality, high prices, or the inability to deliver sufficient quantities on time are among the obstacles for establishing critical business relationships. Larger companies, whether Macedonian or foreign, resort to importing supplies. SBEP outreach has shown that many firms are willing to substitute imports with local supply, but do not have the time or resources to identify potential contractors.” SBEP’s approach to addressing an opportunity for business expansion in regions outside of Skopje aimed to integrate small- and medium-size manufacturing firms into vertical supply chains by identifying opportunities for them to supply products to larger Macedonian firms and international firms operating in Macedonia. In the initial stage of this effort, SBEP commissioned a consultant team to identify large firms with supply chain requirements currently being filled with imports. These efforts focused first on the two regions in which the project initially worked, Pelagonia and Polog. Once firms with supply chain needs were identified, the consultant team began to locate Macedonian firms that had the capacity and interest to fill those supply chain needs locally. This customized “matchmaking” service, which characterized a portion of the Light Manufacturing Initiative throughout SBEP’s project lifetime, was labor intensive and often involved a long gestation period before discussions between the larger and smaller firms involved resulted in trial orders and contracts. Within a relatively short time, SBEP’s light manufacturing consultant team had met and discussed supply opportunities with a large number of Macedonian firms located across the country. By mid-2013, the consultant team had met with more than 40 companies and identified import substitution needs and possible suppliers. To grow its list of possible supplier firms further and to complement another SBEP initiative, the project collaborated with USAID’s IDEAS project in late 2013 to conduct a survey covering 50 companies in the Polog region. During 2013, SBEP’s consultants also researched issues affecting the light manufacturing sector and reached an initial set of conclusions that helped guide their work (see text box). EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 160 Building on these early experiences, SBEP’s consultants moved into what became a second aspect of this initiative, one that formalized the project’s efforts to identify local suppliers using value chain mapping exercises through which it screened and catalogued more than 150 companies from the subsector to develop value chain maps with information about the stakeholders in the chains. The catalogue that resulted from these efforts evolved further, becoming a searchable online resource. As the existence of SBEP’s inventory of light manufacturing firms became known, requests for referrals followed, including from the Agency for Foreign Investments and Export Promotion of the Republic of Macedonia (the government agency that promotes direct foreign investment and exports, and operates the website Invest Macedonia), and the Office of the Deputy Prime Minister for Economic Affairs. Requests from these entities grew over time to include sourcing requests that would involve exporting to other countries as well as import substitution opportunities with foreign firms operating or investing in Macedonia. In part, SBEP’s work on an inventory and database of Macedonian light manufacturing firms paralleled work already under way at MAMEI, which focused on export opportunities and had begun under USAID’s IDEAS project (2010 – 2014). The range of experiences and expertise developed through these aspects of SBEP’s Light Manufacturing initiative coalesced in the fall of 2016 with the first International Suppliers’ Conference and business-to-business (B2B) matching event: The Macedonia Manufacturing Expo, which took place on September 15 in Skopje. SBEP and USAID collaborated to organize this conference with MAMEI, Invest Macedonia, the American Chamber of Commerce in Macedonia, the embassies of both Switzerland and the Netherlands, and other parties. During the conference, SBEP reported in its fourth quarter FY2016 Quarterly Report, “A total of 170 B2B meetings took place, and around a dozen “SBEP succeeded in inviting a large number of participants to the Expo, including 15 foreign companies (already settled in the economic zones) and about 60 Macedonian companies which had direct (rotating) meetings with those 15. Overall, more than 300 B2B meetings took place among participating company representatives.” - Invest Macedonia Representative Results of SBEP’s 2013 Analysis of Light Manufacturing in Macedonia • The light manufacturing/subcontracting sector (metal processing, mechanical engineering and electrical engineering) is one of the most complex and diversified sectors in the country. The sector includes production of various types of products for different applications, such as processing equipment, metal enclosure and furniture, automotive parts and assemblies, household appliances, safes and metal doors, spare parts and equipment maintenance, metal constructions, and so on. • This is an important sector for the country. There are more than 2,000 companies involved in this sector with about 25,000 employees, which is about 25 percent of the total number of employees in the manufacturing sector, or about 6 percent of the total number of employees in the country. • The total exports of the subcontracting sector amount to 1.1 billion euros, of which 72 percent are base metals (iron alloys, iron and steel, and fabricated steel products), 10 percent are machinery and mechanical appliances, 14.5 percent are electrical machinery and equipment, and 3 percent are transport vehicles and parts. • The companies in the sector continuously invest in new equipment and processes to improve efficiency of operations and meet customer demand. However, there is very little cooperation within the sector. • The main barriers to sector growth identified through the interviews are access to new customers and markets, access to finance, and practical experience of the labor. SBEP First Quarterly Report, FY2014 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 161 deals are already being negotiated.” Notably, SBEP also reported that “around 90 percent of the [B2B] meetings were prescheduled through the online matching system that had been developed by the project.” Light Manufacturing Initiative Results This section focuses on the results and outcomes of project activities under this initiative. Evaluation team findings regarding these are presented in three clusters. The first focuses on matchmaking efforts in line with the Light Manufacturing initiative’s original intent, which was to capitalize on import substitution opportunities in the supply chains of large Macedonia-based firms to help smaller Macedonian manufacturing firms expand their business by producing local substitutes acceptable to these larger firms. The second subsection examines results associated with SBEP’s investment in developing an inventory of light manufacturing firms, and the third focuses on the Macedonian Manufacturing Expo that the project organized in 2016. Customized Matchmaking: Identifying Supply Chain Opportunities for Macedonian Firms To understand the results of SBEP’s matchmaking activities, the evaluation team planned to interview as sample of pairs of buyer and seller firms to learn about their perceptions of these efforts and what became of them. Rather than searching for success stories only, the evaluation team hoped to learn why some matches led to trial orders and long-term business arrangements, but others did not. To that end, the team sought information from the SBEP staff, which then requested information from the pair of consultants who worked with SBEP on the Light Manufacturing initiative. This request was made at the same time that requests were sent to SBEP for lists of participants in other initiatives: Grow More Corn, Women in Business, Wild-Grown Product, and Adventure Tourism. In response to these requests except for Light Manufacturing, SBEP provided MSI with detailed lists of initiative participants, including contact information. For Light Manufacturing, however, the response received after several additional exchanges consisted of the names of six firms with which SBEP had worked, and a list of 54 firms that participated as either buyers or sellers at the September 2016 Manufacturing Expo in which SBEP played a leading role. Regarding the six firms with which SBEP had consulted, the response did not indicate whether the firms had been buyers or sellers, and thus the team could not identify buyer-seller pairs to interview. MSI asked SBEP for more and better information that it could use to set up pairs interviews, and SBEP staff tried to help, but were unable to obtain the needed data. Although some additional data were eventually delivered, it included only (a) contact information for export opportunities with foreign firms brought to SBEP by Invest Macedonia, which were different from the Macedonia-based buyer-seller matchmaking efforts that were central to SBEP’s characterization of the Light Manufacturing initiative; (b) contact information for government entities with which SBEP worked in this sector; and (c) potential buyer firms that were later invited to the 2016 Macedonia Manufacturing Expo that SBEP organized. To proceed, based on the names of six firms SBEP provided, MSI’s survey partner, GfK Skopje, attempted to conduct pretests of the interview instrument for this initiative, reporting back to MSI that firms it contacted had declined to participate. As it had contact information for only a few firms and pretest refusals occurred late in the study’s data collection period, MSI ended up abandoning planned interviews with “pairs” of buyer and supplier on this sector. At that point it opted to more carefully examine secondary source data for light manufacturing initiative results information. This exercise produced the following table, based on SBEP Quarterly Reports. The table identifies potential buyer and seller firms by name, geographic area in Macedonia, type of business, and the type of product one might produce for the other. Columns D (demand) and S (supply) indicate whether the focus of a EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 162 matchmaking effort was identified by the potential buyer or the interested seller. The table’s final column includes any documentation found in quarterly reports that indicates whether SBEP-facilitated interactions among pairs resulted in trial orders or contracts. As the table on the following page shows, the team’s review of quarterly reports identified 24 instances in which SBEP’s consultant team worked with Macedonia-based firms to try to identify imported supply chain products that could potentially be supplied instead by Macedonian firms. As the table shows, the team found only some of the information it sought in the reports examined. Notably, positive outcome information on these matchmaking attempts — meaning orders placed or contracts signed — was in quarterly reports for only three of these 24 efforts (12 percent). The number of matchmaking attempts on which the evaluation team found the information summarized in the table differs from totals reported in the closeout and financial report on SBEP. In that document, the project team counts the following as an accomplishment of the Light Manufacturing initiative: “[the project] supported over 130 MSMEs [micro, small and medium enterprises] with BDS [business development] services, of which 21 established buyer-seller relationships, i.e., entered the supply chain of lead firms.” Regarding the first of these figures, it may refer to light manufacturing firms that SBEP identified and entered into its inventory of firms rather than those for which it tried to establish a seller relationship with a larger firm in Macedonia using an import substitution approach to gaining access to the larger firm’s supply chain. The second figure, 21 firms, is closer to the 24 attempts to make matches that the evaluation team could confirm using SBEP quarterly reports. However, the confirmation provided by the evaluation came closer to validating that some sort of buyer-seller conversation had been developed than it did to demonstrating that commitments resulting in local firm sales into larger firm supply chains had actually been made. What is missing from this document-based review of matchmaking results under the Light Manufacturing initiative is information the team had hoped to obtain from interviews about the degree to which these experiences built confidence among small manufacturing firms and inspired both large and small firms to seek new, local partners to help them build and service their businesses. To some extent, these types of qualitative findings can be found in stories included in SBEP quarterly reports. Searchable Light Manufacturing Sector Inventory Based on reporting from SBEP and government sources, the inventory of light manufacturing firms the project developed appears to be a valuable resource that will continue to be used. The information developed by SBEP, reportedly on 250 firms, is now on the MAMEI website, which already had a similar inventory it developed and automated under USAID’s IDEAS project. MAMEI staff say that Invest Macedonia and the Cabinet Office of the Vice Prime Minister for foreign investments regularly request information on local firms and matching assistance to respond to requests they receive from foreign sources. Despite this usage pattern, MAMEI staff told the evaluation team that the status of the database ownership had not been fully determined as of December 2016. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 163 Import Substitution Matches Attempted Between Large Macedonia-based Firms and Smaller Macedonian Light Manufacturers Year Buyer Region/ Municipality Large Firm/ Type of Products D Light Manufacturer/ Type of Supply S Proposed Match Outcome 1 2013 Pelagonia - Prelip Comfy Angel – Bedding (D) ✓ Jovanovski Holding Polyethylene and polypropylene bags Order placed for 6,500 pieces 2 2013 Johnson Controls – Electronics (auto) ✓ 3 2013 Pelagonia - Prelip Vitaminka Local artisans ✓ Promotional items with firm’s logo 4 2013 Pelagonia - Bitola Cermat – ice cream & frozen foods ✓ Plamteks (Skopje) – possibly; or Mak Tojs; or Demasil in Kriva Planka Vitaminka (biscuits) – possibly IML packaging; biscuits for ice cream sandwiches 5 2013 Skopje Swisslion LLC - confectionary ✓ Plamteks (Skopje) – possibly; or Mak Tojs; or Demasil in Kriva Planka IML packaging 6 2013 Pelagonia - Prelip Milina – ice cream and frozen foods ✓ Plamteks (Skopje) – possibly; or Mak Tojs; or Demasil in Kriva Planka IML packaging 7 2013 Pelagonia - Prelip Miekara Miki - dairy ✓ 8 2013 Pelagonia – Bitola Ideal Shipka - dairy ✓ Plamteks (Skopje) – possibly; or Mak Tojs; or Demasil in Kriva Planka IML packaging 9 2013 Pelagonia - Bitola BiMilk - dairy ✓ Plamteks (Skopje) – possibly; or Mak Tojs; or Demasil in Kriva Planka IML packaging 10 2013 Pelagonia - Prelip Mikrontech – electric AC motors ✓ 11 2013 Pelagonia - Prelip Pro Vista – printing house ✓ Printing on various materials 12 2013 Pelagonia - Bitola Kromberg and Schubert – cable and wiring (auto) ✓ 13 2013 Pelagonia - Prelip Pip Orion – agro mechanical parts ✓ EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 164 Year Buyer Region/ Municipality Large Firm/ Type of Products D Light Manufacturer/ Type of Supply S Proposed Match Outcome 14 2014 Zdravje Radovo ✓ Plamteks (Skopje) – possibly; or Mak Tojs; or Demasil in Kriva Planka IML packaging 15 2014 Southwestern - Ohrid Five (5) firms ✓ SMD (Skopje), possibly; or Man-Dra Controlling systems for wood pellet stoves 16 2014 Company that develops and implements controlled atmosphere storages ✓ Food producers Partnership approach between these parties 17 2014 Pelagonia - Prelip Izo Staklo ✓ Rubber products 18 2014 Polog - Gostivar RBI ✓ 19 2015 Polog Tetovo Ecolog ✓ Jabolcello (Resen) - apple juice producer from Resen – possibly; BiMilk – dairy, possibly List of products to plug into firm’s supply chain 20 2015 Vadar - Veles Brako ✓ Filand (Kumanovo) - possibly Polyurethane products; parts for possible medical line Signed agreement 21 2015 Northeastern – Kriva Palanka Demastil – toy producer ✓ Finalmeplast (Skopje) or O.S.A. (Skopje) – possibly, or Alatnica HB (Ohrid) Tool production Orders placed with two of three tool shops SBEP identified 22 2015 Southwestern - Kichevo FDI company, Key Safety Systems (KSS) ✓ Inputs for airbag cushions 23 2015 Eastern - Kocani FDI Amphenol – interconnect products ✓ Procompo (Italian FDI) – possibly 24 2015 Polog – Tetovo TIZD Indbus Procompo (Italian FDI) – possibly Two directional: composite materials from Procompo to Indbus; metal construction from Indbus to Procompo EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 165 Responses to interview questions the evaluation team asked Invest Macedonia respondents indicated that they were quite aware of the database, had used it through MAMEI and thought that it was a positive contribution to the business environment, and felt that MAMEI should be recognized for that. These statements indicate continuing demand and a smooth transition from SBEP to MAMEI in this arena. Large-Scale, Event-Based B2B Matchmaking for Light Industries in Macedonia From an efficiency standpoint, the 2016 Macedonian Manufacturing Expo may represent a turning point for local firms that have the capacity and drive needed to win a role in the supply chains of larger domestic and foreign firms. Trade fairs worldwide serve this purpose in many industries. Returns on this investment will be a function of how many of the 300 reported B2B meetings resulted in supply contracts for Macedonia’s approximately 60 seller participants, and whether test orders turned into follow-up contracts and recurrent sales. They also depend upon whether the Expo becomes a permanent feature of the Macedonian business landscape instead of being remembered as a one-time experiment. Regarding orders emerging from the 2016 Expo, the SBEP quarterly report dated January 2017 stated that by the end of the conference, about a half dozen “deals” were under discussion. With SBEP closed, follow-up to determine how many and what types of transactions resulted from the Expo would seem to fall to MAEMI and Invest Macedonia. However, as the evaluation team’s interview with Invest Macedonia demonstrated, that organization (like SBEP itself) has problems learning about transactions that follow from B2B introductions it makes, regardless of the forum in which that occurs. Regarding whether the Expo will become a recurrent opportunity for local and foreign buyers to meet and begin working with Macedonian sellers, a poster from the 2016 Expo suggests that this was the intention, and it forecast that the next such event would be held in October 2017. An online search by the evaluation team failed to locate information confirming that this even occurred or that a future Expo is being planned. However, MSI found information describing online B2B experiences that Invest Macedonia arranged for local firms with firms from another country in 2017. MAMEI explicitly recognized the value of making the Expo or something like it permanent when it responded to an evaluation question on “next steps” by mentioning future expos as one of its priority recommendations for USAID, which included: • “Support to MAMEI, institutional and financial, as MAMEI is the only organization working the area of ‘matchmaking’ (B2B) and doing relevant research on the field. One of the key roles of Invest Macedonia is connecting Macedonian companies with foreign companies and vice versa. However, there is no relevant exact indicator (or data) regarding successful and established partnerships [that result]. Usually the companies are not willing very much to answer questionnaires or provide feedback after the matching. Very often we are informed about the results of company connections by the foreign company, not the Macedonian firm. - Invest Macedonia Representative EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 166 • More events as we have organized with Invest Macedonia in September this year, to be included in the future, or even establish such meetings/fairs annually. • More initiatives are needed to explore the potential of the light manufacturing sector, as it is a kind of neglected sector in the existing scope of governmental promotional interventions.” MAMEI Representative SBEP’s Light Manufacturing Initiative and National Trends Although USAID’s investment in the SBEP Light Manufacturing initiative was relatively small, the effort’s focus was on broad national goals. Significant unemployment, averaging 30 percent or more, is a chronic problem. It is higher among youth and in less economically developed regions than it is in Skopje, the capital. Imports are consistently higher than exports. Under the Light Manufacturing initiative, SBEP set out to reverse these patterns by identifying opportunities for creating jobs in the less-developed regions in which it worked by encouraging firms working in those regions to “buy locally” wherever an imported product in their supply chain could be replaced with one that was manufactured locally. Through this initiative, SBEP set out to plant an idea and show that it could work by demonstrating how to both importing firms and small manufacturers that had the capacity to produce the same or comparable products locally. The effort matches up well with other USAID projects that have focused on export expansion. Together, over time, this combination of forces could potentially impact the country’s trade imbalance and its jobs deficit in favorable ways. From this perspective, monitoring macro-level trends makes sense, even when USAID investments themselves will not change those trends. However, they may change people’s knowledge, beliefs and actions. To that end, the evaluation team captured data from the U.N. Comtrade Database on imports and exports between Macedonia and the world for the SBEP period. The data show that Macedonia’s trade gap declined from USD$2.5 billion in 2012 to USD$1.9 billion because exports grew faster than imports, which spiked upward in 2014, returning to about their 2012 level by 2016. The figure below shows this modest narrowing of the country’s trade gap. Relationship of Macedonian Exports to Imports, 2012 – 2016 (USD, billions) SBEP was not the cause of this narrowing, but organizations like Invest Macedonia and MAMEI may become contributors to further change if they continue along the path SBEP encouraged. Job Creation Job creation is a core objective of all the initiatives supported under SBEP. Performance Indicator # 1 for the project tracks new jobs created as a result of SBEP-linked activities. For the Light Manufacturing 0 2,000 4,000 6,000 8,000 2012 2013 2014 2015 2016 Imports Exports EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 167 initiative, SBEP’s closeout performance and financial report attributes 53 of the 1,673 jobs the project created to this initiative. These new jobs would most likely have been created in the firms that won supply contracts with larger Macedonia-based firms. MSI’s review of SBEP quarterly reports for supporting information found one example of job creation among the light manufacturers with which the project worked: “The toy producing company — Demastil from Kriva Palanka — hired 25 new employees (15 women and 10 men) as a result of the investment it made earlier in additional IML production equipment.” Demastil was identified in quarterly reports in connection with six firms that expressed an interest in IML packaging, as the table indicates. This example is the only instance the evaluation team found in quarterly reports specifically attributing new jobs created to a named Macedonian company. Light Manufacturing Initiative Sustainability For the Light Manufacturing initiative, SBEP made an early and deliberate effort to identify mechanisms through which this initiative might continue after USAID’s project funding ended. By 2015, SBEP had identified MAMEI as a potential hub for continuing matchmaking and light manufacturing inventory building after SBEP terminated (see text box). This option has since been developed, and MAMEI became the designated follow-on location for the SBEP light manufacturing sector inventory and database and potentially for other services the initiative provided under SBEP. Although this appears to have been an obvious choice, even though MAMEI is a relatively new organization (2014) with a small staff, MAMEI itself raised cautions about the sustainability of SBEP’s Light Manufacturing initiative, saying: “When we talk about the sustainability of this initiative, we cannot say a lot of positive things. SBEP is ending, and we have to allocate another funding. We constantly update the database for our own needs in order to be as accurate as possible. We do maintenance by the other users mentioned earlier [Invest Macedonia, TIRZ, the Cabinet Office of the Vice Prime Minister, the Ministry of Finance, and the Minister for FD].” MAMEI Representative Like MAMEI, Invest Macedonia expressed uncertainty about the future of the Light Manufacturing initiative: “[MAMEI] should find some sustainable funding source or model of funding.” Invest Macedonia Representative “SBEP’s supply chain initiative and the services that the consultants are providing to both the local companies and the foreign direct investment in the country have proven to fill a gap that has existed for quite some time. Therefore, the consultants have been considering ways to ensure sustainability and further development of this type of service. One of the alternatives is to nest it within MAMEI. Established in 2014, MAMEI has been supported from the beginning by the Swiss Import Promotion Programme within the Swiss Global Enterprise (SGE). SBEP also plans to support MAMEI with some activities, so the consultants organized a meeting this quarter between representatives of SBEP and SGE Switzerland to discuss potential joint activities that could contribute to MAMEI’s growth and development. The association has also received financial support from the Ministry of Economy this past quarter to develop supply chain software and a database that would include and serve local suppliers and other participants in the light manufacturing supply chains.” SBEP Third Quarterly Report, FY2015 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 168 Exhibit 6.1 Light Manufacturing Initiative Chronology January – March 2013: Consultants were hired under the success-fee model to connect local light manufacturing suppliers to anchor firm supply chains. April – June 2013: Consultants identified four local light manufacturing suppliers for Johnson Controls and four for Comfy Angel in this pilot phase. July – September 2013: The first match was made under the initiative. Comfy Angel placed an order for approximately $14,400 with local supplier Jovanovski Holding, with the possibility of a long-term partnership worth $130,000 annually. October – December 2013: Supply chain agents identified a business case for IML food packaging within the initiative. The agents conducted a survey of 50 companies founded or supported by diaspora investment in Polog to identify the profile of investors and discover the issues they face when establishing a business. January – March 2014: Supply chain consultants continued working on developing the business case for IML plastic containers used in food packaging. April – June 2014: The consultants, in cooperation with USAID IDEAS project, developed a database of 160 light manufacturing companies from Macedonia. July – September 2014: Consultants identified a plastic producing company in Kriva Palanka that was likely to invest in equipment for IML packaging production. October – December 2014: Plastic producing companies Demastil from Kriva Palanka and Plamteks from Skopje invested in equipment for IML packaging production with support from SBEP supply chain consultants. January – March 2015: Demastil, a plastic toy producer from Kriva Palanka, diversified its production and increased its workforce by 35 percent after investing in additional IML production equipment. Government institutions recognized the importance and necessity of the consultants’ work and continued to approach them for information and support regarding inquiries from potential foreign investors. April – June 2015: Government institutions in charge of attracting foreign investment contacted the consultants and asked for assistance in identifying local suppliers that could satisfy the requirements of foreign companies. July – September 2015: An online software solution — a database of companies from the light manufacturing sector in the country — began development. It will feature automated company profile creation, an easy and fast search option, and an overview of the industry’s supply chain. October – December 2015: Plans began for a suppliers’ conference in September 2016 that would provide a forum for B2B meetings between a dozen local and foreign buyers and 50 – 70 local suppliers. Invest Macedonia and the Cabinet of the Deputy Prime Minister for Economic Affairs expressed initial support for the idea. January – March 2016: SBEP continued organizing the suppliers’ conference and B2B event. Invest Macedonia and the American Chamber of Commerce in Macedonia agreed to participate and help organize the conference. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 169 April – June 2016: Preparations continued for the September 15 International Suppliers’ Conference and B2B event. The U.S. Chamber of Commerce, the Netherlands Embassy, and the German Chamber of Commerce AHK confirmed their support in organizing the event. The firm Brako from Veles signed an agreement with the SPS Group from the Netherlands. July – December 2016: Continued preparations resulted in a well-received Macedonian Manufacturing Expo on September 15 attended by 15 buyer firms and 60 smaller Macedonian manufacturing firms. During the event, more than 300 B2B meetings took place between representatives of these firms, many of which had been arranged electronically in advance using MAMEI software. Also during this period, several opportunities involving foreign firms interested in obtaining products from Macedonian firms emerged through contacts established with government agencies, MAEMI and other donors. These differ from SBEP’s portfolio of matchmaking efforts focused on import substitution because exports are the focus, and the lead firms involved are based outside of Macedonia. However, these are attractive opportunities for Macedonian light manufacturers, and SBEP is working with local firms to respond to the following: • “German companies interested in serial production of pressure vessels for fire extinguishers and for polypropylene and high-density polypropylene for railways; • Dutch companies looking for stainless steel testing and lab equipment, piping, and equipment for power stations, with a potential to expand to Germany; and • Danish companies in search of suppliers of composite materials for special use, such as composite tapes (fiberglass with other components) for special usage in the chemical, petro and oil industry.” SBEP Fourth Quarterly Report, FY2015 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 170 ANNEX D: DATA COLLECTION INSTRUMENTS Survey Instrument – Farms (main survey) Interview Log Data: Date Day Month Year GfK Interviewer(s) (1) (2) Region Municipality Village GPS coordinates Interview Language 1. Macedonia 2. Albanian Respondent’s native language Interview Venue 2. On the farm 2. Other location If other location, identify that location Introduction Read or recite the full introduction to the respondent before starting the interview. Ask the respondent if he/she agrees to be interviewed for this survey. Provide the respondent with a card that has GfK’s contact information on it (even if it does not include the name of the interviewer). Gender Screening Question Is the person you are about to interview a female? 1. Yes 2. No END THE INTERVIEW AND ASK FOR THE FEMALE OWNER/DECISION MAKER [Do not proceed if the only person available to interview is a male. It is intended that all respondents for this particular set of interviews will be women. Be prepared to use a female interviewer if there is any My name is __________. I am from GfK, a research firm in Skopje. We are here today conducting a survey of farms in this area. Our questions are about what farms in this areas produce, the farming methods that are used, and farmers' views about their current situation and prospects for the future. The research we are conducting will help organizations that are supporting regional development in Macedonia decide what kinds of assistance to provide. This includes both Government of Macedonia agencies and assistance programs funded externally, such as the assistance program for Macedonia funded by the United States, through its’ assistance agency, USAID. We are asking you and other farmers to help us understand economic conditions and opportunities, and efforts to improve them in this municipality. We will speak with many people like you as we gather answers. We will be writing what we learn, but we will not give out the names of specific individuals we speak with during this survey in our report. Before we begin, do you have any questions about what I just explained to you about our purpose in being here? We expect that our conversation with you today will take about 25 minutes of your time. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 171 cultural sensitivity about men interviewing women in the particular municipality or village or home-based business environment where this interview is taking place. Note that the response to this question need not be recorded as a gender of respondent question is included in the interview below. This screening question is simply intended to ensure that interviewers do not proceed with interviews with males.] Respondent Screening Question The respondents for these interviews should be the primary farmer/farm family head, i.e., the person who is most likely to make decisions about planting and about technology choices for the farm. In some instances, this person may be the primary farmer but not the farm family head, e.g., a son of an elderly farm owner, or a paid farm manager for an owner who is not the primary farmer. 1. How would you describe your relationship to this farm and its owner? 2. I am the owner of this farm 3. I rent this farm from the owner 4. I am a member of the owner’s family 5. I am a member of the renter’s family 6. I am the paid manager (not a family member) 7. Other [If the answer is (3), (4) or (6) for this question, the GfK interviewer for this survey should ask who is the principal decision-maker and determine whether this interview should be conducted with that person instead, if he/she is available.] If no appropriate individual is at home/the farm, reschedule if possible or END INTERVIEW. Corn Farm Screening Question 2. Is corn one of the crops this farm currently grows? 1. Yes 2. No – if this farm was not interviewed in 2014, END THE INTERVIEW If this farmer or someone else from this farm was interviewed in 2014 and responded “no” to the previous question about growing corn currently, ask: 3. Did you grow corn in the past? 1. Yes, but not now 2. No 3. DK If the answer to the question above is “yes, but not now”, ask: 4. Why did you stop growing corn? Open-ended answer is expected – try to get as full a story as possible from any farm interviewed in 2014 that is not growing corn commercially in 2016 - noting any mention of drip irrigation or SBEP in the story of why the farm quit growing corn. After learning why the farm has stopped growing corn (since 2014), END THE INTERVIEW. Demographics – Respondent EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 172 “As we begin this interview, I would like to ask you a few questions about yourself” 5. Gender of Respondent 1) Male 2) Female 6. How old are you? Enter reported age GfK interviewer to mark which of the following groups this respondent belongs to. 1) >29 (youth) 2) 30-34 3) 35-39 4) 40-44 5) 45-49 6) 50-54 7) 55-59 8) >60 7. What is the highest level of schooling you completed? 1) Primary school or uncompleted secondary school 2) Completed secondary school or uncompleted university 3) University degree or post graduate degree Respondent Membership in Business Associations “I’d like to ask you a few questions about farm-related organizations you might be a part of.” 8. Are you a member of a farmer’s association, cooperative or other farm related business group? 1) Yes 2) No – GO TO NEXT SECTION  Awareness of Economic Development Planning 9. What are the names of the farm related membership organizations to which you belong? Name of Farm-Related Membership Organizations to which this farm or you personally belong Type of Organization 1 = Farmer’s Association; 2 = Cooperative; 3 = Other type of business group 1 2 3 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 173 10. How does membership in this (these) organization(s) benefit you and this farm? (Choose as many as are appropriate) 1) Facilitates purchases of inputs – Y/N. 2) Provides advice and training to the farm – Y/N 3) Facilitates sales of products- Y/N 4) Represents firms with government – Y/N 5) DK/NA 11. At what level(s) of government does the organization you are a member of represent your situation/views and those of other firms to the government? 1) Municipal level 2) Regional level 3) National level 4) All of the above 5) DK Respondent Awareness of/Participation in Economic Development Planning – Public-Private Sector Dialogue “Now I’d like to ask you a few questions about your awareness and involvement in economic development planning for this area – the municipality or region.” 12. Are you aware of any plan(s) for the economic development of this area? 1) Yes 2) No - go to next section  Farm Demographics 13. At what level(s) do the economic development plans with which you are personally the most familiar address economic development issues -- local/municipal level, regional level or national level? 1) Local/municipal 2) Regional 3) National 14. What are the name(s) of the economic development plan(s) at these levels with which you are personally familiar? Please provide the name(s) if you recall them: Names or descriptions of economic improvement plans the respondent is personally familiar with a) National Plans ___________________________________________________________________________ b) Regional Plan __________________________________________________________________________ c) Municipal Plan ___________________________________________________________________________ EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 174 15. Did you personally participate in the development of any of these economic development plan? 1) Yes 2) No, skip to the next question Q17 16. In what ways did you participate in the creation or implementation of economic development plans at these levels? (Select all that apply) Type of personal involvement in the development/ implementation of economic development plans Local/ municipal level plan (X) Regional level plan (X) National level plan (X) Active, regular participant in a planning group or committee (e.g., made suggestions, helped organize the process) Attended public meetings/briefings on the plan or reviewed a written version Worked to implement the plan once it was approved/adopted Other type of involvement No personal involvement 17. In what ways have the specific economic development plans you are most aware of been of benefit to you or your business? Specific Benefits of Economic Development Plans to this Farm or Me Personally Local/municipal plan(s) Regional plan(s) National plan(s) Demographics – Farm 18. What is the size of this farm – total number of hectares? 1. Less than 1 hectare 2. 1-5 hectares 3. 6-10 hectares 4. 11-20 hectares 5. More than 20 hectares 6. I don’t have arable land – END INTERVIEW 7. Don’t know – END INTERVIEW 19. How many hectares (out of the total number for this farm) are used for commercial purposes (crops, livestock, nurseries), as opposed to personal use, e.g., kitchen gardens, house, etc.? Enter number of hectares GfK to then enter the correct code below for the number given in answer to the question above: 1. Less than 1 hectare 2. 1-5 hectares 3. 6-10 hectares 4. 11-20 hectares 5. More than 20 hectares EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 175 20. Of the hectares used for commercial purposes, what percentage is allocated to crops, livestock, orchards and vineyards, raising seeds or trees, or other commercial purposes? Commercial Uses of Land Hectares in Commercial Use Allocated for This Purpose GfK to calculate percentage in SPSS 1. Farming crops on arable land 2. Keeping livestock 3. Raising fruit in orchards/vineyards 4. Raising seeds/trees in nurseries 5. Other commercial uses TOTAL 100% 21. Does the family that farms this land own it or is the farm land rented from someone else? 1. Owned by the family that farms it 2. Rented from someone else 3. Part of it is owned; part of it is rented 22. Is this farm registered as business entity or in the Registry of Ministry of Agriculture? (Check one response). 1) Yes 2) No 3) DK 23. If the farm is registered, how is it registered – as what type of entity? 1. Yes, as a business enterprise (sole proprietorship or limited liability company) 2. Yes, as an individual holding (individual farmer) 3. No, it is not registered either way 4. I don’t know Farm Workers/Jobs Linked to Farm “Now I would like to understand how many people work on this farm and whether they are men or women; whether they work full-time, part-time, or only seasonally, and whether they are paid or unpaid farm workers.” (Ask specific follow-up questions to complete the table below.) 24. Overall, how many people does this farm currently employ? Enter absolute number GFK interviewer to indicate correct cluster into which the respondent’s absolute number (above) fits: 1) 0-9 2) 10-19 3) 20-49 4) 50-249 5) 250+ 6) DK EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 176 25. How many of this farm’s current employees are full-time, part-time, or seasonal employees? (Total for this question should be the same as the absolute number of employees the respondent provided above) Type Number Percentage (GfK to later calculate %) Full-time Part-time, year round Seasonal Total employees 100% 26. Approximately what percentage of current employees are female? Gender Number Percentage (GfK to later calculate %) Female Male Total employees 100% 26A. Approximately what percentage of current employees are youth? Gender Number Percentage (GfK to later calculate %) Age 18-29 30 or Older Total employees 100% 26B. Does this type of work have good opportunities for young people to get started earning a living after they finish school? Better opportunities than in other types of work About the same as other types of work Fewer opportunities than in other types of work 27. How many current employees are family members and what percentage are hired workers? Type Number Percentage (GfK to later calculate %) Family Members Hired workers Total employees 100% 28. How many of the family members who currently work for the farm are paid versus working without pay? Family Member Farm Workers Number Percentage (GfK to later calculate %) Paid a salary Unpaid Total Family Member Worker/Employees 100% EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 177 29. Has the number of people working on this farm increased, stayed the same, or declined over the past few years (2-5 years)? 1) Increased 2) Stayed the same 3) Declined 4) DK 30. If the number of people employed by this farm changed (increased/decreased) over the past 3-5 years (since 2012), please describe how the number of farm employees has changed over this period. 2012 2013 2014 2015 2016 Total Farm Employees By Year 31. If the number of people employed by this farm changed (by more than just 1-2 people) over the past 2-5 years (increased or declined), what explains that change – what explains why the number of employees increased or declined? Open-ended answer expected. Hectares Used to Grow Corn 32. What percentage of the arable land on which this farm cultivates crops is used to grow corn? Crops Grown Number of hectares on which corn or other crops were planted in 2016 GfK to calculate in SPSS percentage of Hectares in Commercial Crop Production Maize (Eating Corn) Forage Corn All Other Crops Total hectares used for farming crops on arable land 100% Total here should be the same as for “farming crops on arable land” in the earlier question regarding all commercial uses of the farm’s land. If “all other crops” is greater than 10% of hectares on which commercial crops are grown, ask: 33. What are the other main crops you grow on this farm? Open-ended answer expected – write in a list of other major crops grown besides corn EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 178 Investments in Production Aids “Now I’m going to ask a few questions about this farm’s use of production aids for corn and other crops.” Fertilizer 34. Is fertilizer used to grow corn on this farm and when was its use introduced? Maize (Eating Corn) Forage (Fodder Corn) Fertilizer Used (mark X if yes) Initial Year Used: Select one: Pre-2012, 2012, 2013, 2014, 2015, 2016 Fertilizer Used (mark X if yes) Initial Year Used: Select one: Pre-2012, 2012, 2013, 2014, 2015, 2016 35. Is mechanization used to grow corn on this farm, for which corn crops and when was it introduced? Maize (Eating Corn) Forage (Fodder Corn) Mechanization Used (mark X if yes) Initial Year Used: Select one: Pre-2012, 2012, 2013, 2014, 2015, 2016 Mechanization Used (mark X if yes) Initial Year Used: Select one: Pre-2012, 2012, 2013, 2014, 2015, 2016 36. What types of mechanization does this farm use with the corn crop? (Choose all that apply) 1. Mechanized planting 2. Mechanized harvesting 3. Other mechanization 37. If other mechanization is used, explain what type: Seed 38. Are high-yield, hybrid, or other specialized corn seed varieties used in growing corn on this farm? Maize (Eating Corn) Forage (Fodder Corn) High-yield, hybrid or other specialized corn seed are used (mark X if yes) Initial Year Used: Select one: Pre-2012, 2012, 2013, 2014, 2015, 2016 High-yield, hybrid or other specialized corn seed are used (mark X if yes) Initial Year Used: Selection one: Pre-2012, 2012, 2013, 2014, 2015, 2016 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 179 Irrigation 39. Of the hectares this farm planted in corn in 2016, how many of those hectares were irrigated and by what method? Type of Irrigation System Number of hectares of corn covered by this type of irrigation system in 2016 Maize (Eating Corn) hectares GfK to calculate percentage Forage (Fodder Corn) hectares GfK to calculate percentage Surface Sprinkler Drip irrigation Sub-surface Other irrigation None (hectares of corn not irrigated in 2016) Total number of hectares planted by type of corn 100% 100% Note: Totals in this table should match totals (in hectares) for these two kinds of corn in previous answer. 40. In what year did you first use drip irrigation for growing corn? Enter initial year 41. For what other crops do you use drip irrigation, and in what year did you start using drip irrigation for this crop? Other crops for which this farm uses drip irrigation Initial Year Drip Irrigation Was Used (X) Before 2012 2012 2013 2014 2015 2016 42. Between 2012 and 2016 how has the percentage of the farm’s corn crop changed in terms of the percentage of the crop using fertilizer, improved seeds, mechanization or irrigation? (answers in this table should be related to the total number of hectares the respondent said he used in these years to grow (a) maize (eating corn) and (b) forage). Type of Crop Hectares Planted Using Fertilizer Hectares Planted Using Mechanization Hectares Planted Using Improved Seed (e.g., hybrid) Hectares Planted Using any type of Irrigation Hectares Planted Using Specifically “Drip” Irrigation (subset of previous column’s answers) 2012 2016 2012 2016 2012 2016 2012 2016 2012 2016 Maize (Eating Corn) Forage (Fodder Corn) EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 180 43. Since you began using drip irrigation and other modern techniques (fertilizer, new seed varieties, or mechanization) to grow corn, have other farmers come to see what you are doing and followed your lead by adopting some of these practices? 1. Yes 2. No If the previous answer was yes, ask: 44. Approximately how many farmers have adopted a new/modern technology or approach to corn farming based on your example or what they learned from you? New/modern technology or approach for corn farming Number of farmers I know who have adopted this since I did or by watching what I did or learning from me Drip irrigation for growing corn Enter number Improved seed for growing corn Fertilizer Mechanization Corn Productivity and Sales In the questions I’m going to ask now some will be about before you started using drip irrigation in XXXX, and some will be about after (Q39). “I would now like to ask a few questions about productivity and sales for your corn crops over the past several years (since 2012)” 45. Before we turn to these questions let me ask you whether you keep written (annual) records of the number of hectares you plant by type of crop planted, and other information such as the size of your harvest, and how much you sell and earn from each crop? 1. Yes, I keep annual written records by crop for what I plant, harvest, sell and earn 2. No, I keep overall records for the farm, but not by crop, i.e., I have written records on total acres planted, total volume sold, but not crop by crop 3. No, I do not have written annual records concerning hectares planted, tons sold, by crop or overall. Based on the respondent’s answer, GfK’s interviewer should tell the respondent that he/she should feel free to look at their written records as they answer this next set of questions. Give the respondent time to open those records if they choose to look at them. GfK interviewers should complete the box below without asking the respondent a verbal question: 46. Respondent used written records to answer questions on corn planting, harvesting and sales. (GfK to mark X under yes or no) Yes No EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 181 47. Maize (Eating Corn) Production, Sales and Costs Column 1 Column 2 Column 3 Column 4 Column 5 Column 6 Hectares Planted in Maize (Eating Corn) Tons of Maize Harvested Tons of Maize Sold Commercially Total Value of Maize Sales (in denars) Maize Production Costs (total, in denars) 2016 – total 2016 – planted with drip irrigation 2015 – total 2015 – planted with drip irrigation 2014 – total 2014 – planted with drip irrigation 2013 - total 2013 – planted with drip irrigation 2012 - total 2012 – planted with drip irrigation Notes: (1) During Analysis GfK to calculate yield as kilos per hectare harvested, earnings per ton/maize, cost of production per ton/maize, and profits (sales les costs); (2) we collected data for 2012 and 2013 for some but not all of the farmers who will be interviewed in 2016. During analysis we will compare responses in 2014 to those given in 2016 as an accuracy check on responses, but we plan to use the answers given in 2014 on the table above for the “impact” portion of our analysis, as recall in 2016 could be quite poor if written records do not exist/were not consulted in providing answers in either 2014 or 2016. 48. Forage (Fodder Corn) Column 1 Column 2 Column 3 Column 4 Column 5 Column 6 Hectares Planted in Forage (Fodder Corn) Tons of Forage Harvested Tons of Forage Sold Commercially Total Value of Forage Sales (in denars) Forage Production Costs (total, in denars) 2016 – total 2016 – planted with drip irrigation 2015 – total 2015 – planted with drip irrigation 2014 – total 2014 – planted with drip irrigation 2013 - total 2013 – planted with drip irrigation 2012 - total 2012 – planted with drip irrigation Notes – same as for maize above EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 182 Farm Livestock and Use of Farm Crops for Feeding “Now I would like to ask a couple of questions about livestock you may keep and the degree to which you feed those livestock with corn grown on this farm.” 49. Which of the following types of livestock, and how many of each does this farm raise/keep for commercial purposes? 50. What portion of their feed comes from corn grown by the farm today (2016) compared to 2012? Column 1 Column 2 Sources of Food for Livestock in 2016 Type of Livestock Number of Animals Number fed with corn from this farm Number fed with other farm products Number feed purchased food 2012 2016 2012 2016 2012 2016 2012 2016 Cattle Sheep Goats Poultry Other In data analysis GfK will calculate the percentage of animals fed corn from this farm and the percentage fed purchased food. 51. Overall, has the percentage of animal feed that comes from this farm increased, decreased or stayed about the same over the past 2-5 years (since 2012)? 1. Increased 2. Stayed the same 3. Decreased 4. DK 52. If this farm reduced the amount of livestock feed (per animal) that it purchased (between 2012 and 2016), what explains that shift in livestock feed sources? Open-ended answer expected 53. If you purchase feed for your livestock, where does it come from? 1. Local sources in this region of Macedonia 2. Other parts of Macedonia 3. Imported from other countries 4. DK Crop Buyers/Customers “Now I’d like to learn about the types of buyers or customers who purchase your farm crops – the same crops we have discussed under other questions above” 54. In percentage terms, what portion of this farm’s corn sales are bought by each type of buyer, i.e., individuals, a cooperative the farmer belongs to, farms or firms in this region of Macedonia, farms/firms in other parts of Macedonia, or as exports? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 183 Corn Products Product Sold by Type of Buyer, in Percentage of Sales (2016) Total Individuals Cooperative Domestic Firms/ Farms* Exports Product is sold directly to individual buyers/ consumers Product is sold to the cooperative to which I belong Sold to Firms or farms in this municipality or region Sold to Firms or farms in other regions in Macedonia Product is sold for export to other countries Maize (Eating Corn) 100% Forage (Fodder Corn) 100% 55. Did the way this farm sells its corn products – or the types of buyers who purchase the farm's corn change in any important ways between 2012 and 2016? 1. Yes 2. No, if “no” skip to the next section If the answer to the previous question was yes, ask: 56. How is the distribution of corn buyers today different from the distribution of buyers in 2012? Open-ended response is expected. Draw out useful information about how buyers changed and why that happened. Subsidies 57. Does your farm receive a subsidy for hectares planted in corn? 1. Yes 2. No If yes on the preceding question, 58. What portion (percentage) of the cost of producing corn is offset by this subsidy? Enter percentage If “yes” on subsidy for corn received in the preceding questions, 59. How hard or easy would you say it is to obtain a monetary or tax subsidy for growing corn on your farm? 1. Very easy 2. Somewhat easy 3. Somewhat difficult 4. Very difficult 60. Are there any other types of incentives your farm has received from government to grown corn? 1) Yes 2) No 61. If yes, please describe these incentives for growing corn Open ended response expected. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 184 Assistance Received on Farm Production, Quality and/or Sales 62. Over the past several years (since 2012), has your farm received technical advice or assistance from other organizations or programs, including government or foreign donors/organization on improving your volume or quality of farm production and sales? 1) Yes 2) No 63. If yes, which of types of assistance did you receive and did you pay for this assistance or was it provided without charge by some other organization? Type of Assistance Received We paid for this assistance (X) We received this assistance without charge from another Organization (X) Name of Program or Organization that Provided this assistance Value-Importance of this assistance or improvement to the development and status of this business 1 = High 2 = Moderate 3 = Low Technical advice (off-site) on crop production, quality or sales – I/we visited their location On-site consultation/assistance at our location on crop production, quality or sales Training for myself or other farm employees on some aspect of our business Equipment or other inputs to improve our crop production, quality, post￾harvest handling or other aspect of sales If SBEP was named in the previous question as a provider of assistance, skip the next question 64. 64. Are you aware of an assistance program funded by the U.S. government called the Small Business Expansion Project (SBEP) that is working with Macedonian farms to improve their production, product quality or sales? (Some people may know of it as the USAID project or by some other name. Try to determine whether the respondent actually knows the program even if he/she doesn’t call it SBEP) 1) Yes (by some name)___________________ Go Q65 2) No – if no proceed to next question below Q64A 64A. If the answer above was “no” to being familiar with SBEP, ask: While you may not be familiar with that organization, I’d like to ask if you are familiar with a national campaign in Macedonia called “Grow More Corn”? 1) Yes – proceed to SECTION on Q75 [SUSTAINABILITY (skipping over SBEP specific questions] 2) No – if no proceed to Q79 [SECTION on Farm Sales and Household Earnings] 65. When, in what year, did you become aware of SBEP? Enter year EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 185 66. How did you become aware of SBEP? (or whatever name they call this program.) Type of Contact or Exposure to SBEP Yes Saw a story on SBEP on TV or in the newspaper Learned about SBEP from another farm or business assisted by SBEP or another contact who knew about SBEP SBEP representatives sought out my farm; they called or came to visit us? Other 67. What types of interactions have you had with the SBEP program? Type of Interactions with SBEP Yes (X) Value-Importance of this interaction with SBEP to the development and status of this farm 1 = High 2 = Moderate 3 = Low Attended a SBEP conference or workshop Participated in a SBEP training program Received technical advice from SBEP specific to this specific business Received equipment or inputs for this business from SBEP Participated in a SBEP organized tourism promotion activity Business received publicity as a function of its interactions with SBEP Other types of interactions 68. If “other types of interactions” is selected above, please describe the other types of interactions this business had with SBEP. Other types of Interactions with SBEP (not listed above) Value-Importance of this interaction with SBEP to the development and status of this business 1 = High 2 = Moderate 3 = Low 1 2 3 For corn farming specifically: Which of the following did the assistance you received from SBEP? Specific Type of Corn Production Assistance Yes, if ever received (X) Initial Year Received Received in 2016 Modern corn planting and cultivation protocols or guidelines Corn Seed – for maize (eating corn) Corn seed – for forage (fodder corn) Drip irrigation system equipment Other Farming equipment (mechanization) Other corn specific assistance EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 186 If “yes” on drip irrigation was received from SBEP, in the table above, was received from SBEP: 69. How large a drip irrigation system did SBEP originally give you? (Answer in hectares covered by the original drip irrigation system SBEP provided) Enter number of hectares covered by original drip irrigation system from SBEP If “yes” on drip irrigation was received from SBEP: 70. Has the original system from SBEP been enlarged beyond the initial size to cover more hectares of corn or other crops? 1) Yes, the farm’s drip irrigation system has been enlarged since SBEP’s original system was given to us. 2) No If “yes” on enlarged: 71. By how much has it been expanded? (Answer in hectares – how many hectares are covered by enlargements to the drip irrigation system SBEP provided) Number of additional hectares covered by enlargement GfK to calculate this table Original SBEP drip irrigation system size (in hectares covered) Drip irrigation systems enlargement since SBEP’s system was received (in hectares covered) Total hectares covered by SBEP drip irrigation system for this farm and any enlargements 72. If “yes, the drip irrigation system SBEP provided has since been enlarged, how was that enlargement financed? 1) Purchased from the farm’s savings 2) Purchased with a bank loan 3) Purchase was partially subsidized by government 1. If purchase of additional drip irrigation equipment was partially subsidized by government, what government program provided this help? Enter response 4) Other method of obtaining additional drip irrigation equipment. 1. If “other” method is selected, what was that other method? Enter response Most Significant Change Question 73. For you personally or for this farm, what was the “most significant change” that emerged from all of your interactions with SBEP? This could be anything that happened that in your mind has long-term importance for how you and this farm will go forward. An open-ended response in the form of a story is expected – whatever the respondent answers, draw them into a discussion where they explain the change they think is significant and how it came about. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 187 Sustainability “Now I want to talk for a minute about the “Grow More Corn” initiative, and more specifically the use of drip irrigation to increase corn yields – will continue and be provided to other farmers. “ “The “Grow More Corn” idea, we are told, was that with the right start up support, many farmers would become exposed to drip irrigation for corn production and other modern corn farming practices, they would adopt them and that would, fairly quickly, stimulate economic growth and job creation in the farm sector in regions beyond Skopje, the capital. That, in turn, would contribute to balanced economic development.” 74. Is your understanding of the of the idea behind “Grow More Corn” the same as ours? 1) Yes 2) No 3) Other understanding: Open ended explanation of other understanding 75. Based on what you know and what other people in this sector – on farms, in associations and farm supply firms, or in the government – say to you, how likely is it that drip-irrigation for corn production and other elements of the “Grow More Corn” initiative will be adopted by increasing numbers of farmers in Macedonia after external donor assistance for this program ends? 1. Very likely 2. Somewhat likely 3. I can’t tell 4. Somewhat unlikely 5. Very unlikely 76. Is it your sense that more would need to be done to create a long-term home and support for the “Grown More Corn” initiative in Macedonia - to help more farmers adopt drip irrigation and other aspects of this initiative? 1. Yes – further assistance and support will be needed to make drip irrigation for corn widely popular 2. No, it will continue to grow on its own 3. DK If the answer was yes to the previous question, 77. How do you think continued Macedonian efforts to foster the Grow More Corn initiative might be organized? Or is something already happening that you know about or are part of that will help ensure that this initiative expands to helps more farmers and stimulates economic growth on a balanced regional basis? Open- ended answer expected EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 188 Farm Sales and Household Earnings “These next few questions are about how the farm earnings contribute from corn and other farm products contribute to your income and how you see the outlook for earnings in the near future” 78. How much did you earn from this farm last year (2015) in comparison to 2012? Income earned from the farm Macedonian Denars U.S. Dollars 2012 (whole year) GfK to calculate during analysis 2015 (whole year) GfK to calculate during analysis 79. What percentage or proportion of your total household income comes from this farm and its earnings? (By household we mean your immediate family and any relatives who live with you in your home.) 1. 100%, only source of family income 2. 75% 3. 50% 4. 25% or less If the answer to the sources of household income question above is less than 100%, ask: 80. What other sources of family income, including salaries earned by other household members does your family have? and check the appropriate boxes. (More than one answer is allowed. Fill in blank rows if an unlisted income source is described.) Other sources of household income Yes (✓) a) Private sector 1. Salary from employment in a privately owned business 2. Self-employment (own a business other than the farm) 3. Remittances from others 4. Pension from a private company b) Public sector 1. Salary from government employment 2. Pension from the government Income and Investment Perceptions and Expectations 81. Overall, how would you characterize the financial situation of this farm business in 2013 compared to the previous year? 1. Better than last year 2. About the same 3. Worse than last year 4. Don’t know/can’t tell 82. What are your expectations for this the farm’s financial situation over the next 12 months? 1. Will improve 2. Will stay the same 3. Will be worse 4. Don’t know/can’t tell EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 189 83. What factors, in your view, exert the greatest influence on whether the farm’s financial situation will improve or get worse? And how important is each factor? Factors (GfK to complete this table based on open-ended answer above) Mark yes if this factor was mentioned by the respondent (X) How important is this factor? 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know General economic conditions in Macedonia Economic conditions in the EU and other large economies Macedonian government leadership Macedonian government policies Availability of subsidies – for businesses Access to finance Foreign investment in Macedonian businesses Business access to domestic markets throughout Macedonia Business access to international markets Business production/service levels Business cost of production/service delivery Price of business inputs International product standards Other DK/NA 84. Have you (or the owner if you are not the owner) made any new investments in this farm business in the past few years (2012-2016), beyond normal business inputs)? 1. Yes 2. No 85. How likely it is that new investment (beyond normal inputs) will be made in this farm business next year (2017)? 1. Very likely 2. Somewhat likely 3. Not sure 4. Somewhat unlikely 5. Very unlikely For quality supervision purposes, please write down respondent’s contact information Name of respondent_____________________________________ Address_________________________________________ Village/Municipality_______________________ Telephone number___________________________________________ Closing Statement Thank you for being so kind as to talk with us today. What you have told us will help others come to know how farmers like you are doing and how you see the prospects for future development of your farm, municipality and region. In about two years we will be coming back to this area to learn whether and how things have changed for you and others who live here. We very much hope to be able to speak with you again at that time. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 190 Semi-Structured Interview Instrument – Federation of Farmers of Macedonia (FFM) My name is __________. I am from Management Systems International, an international development firm in the United States. We are here today conducting interviews with organizations working in this region and playing an active role in its economic development. The research we are conducting will help organizations that are supporting regional development in Macedonia decide what kinds of assistance might be provided in the future. This includes both Government of Macedonia agencies and assistance programs funded externally, such as the assistance program for Macedonia funded by the United States, through its assistance agency, USAID. We are asking you to help us understand economic conditions and opportunities, and efforts to improve them in this region. We will speak with many people like you as we gather answers. We will be writing what we learn, but we will not give out the names of specific individuals we speak with during this survey in our report. Before we begin, do you have any questions about what I just explained to you about our purpose in being here? We expect that our conversation with you today will take about one hour. Interviewer Name Interview Date Basic Information 1. Interview Location Name of Region Name of Municipality 2. Interview Participants FFM Title Name In this interview, our questions focus on initiatives that a U.S. Government assistance program to Macedonia called the Small Business Expansion Program (SBEP)? Our particular interest is their “Grow more Corn” Initiative. We have learned from them that organization was SBEP’s partner in organizing and implementing this initiative. When did FFM become SBEP’s partner for the “Grow more Corn” Initiative? How long this cooperation lasted? What were your responsibilities in this partner role? Can you please describe the various things you did to implement the initiative, and also tell us what SBEP itself did to implement the “Grow more Corn” Initiative? How did your roles differ? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 191 How constant was the “Grow more Corn” Initiative over the life of the SBEP project – or if it changed a lot over the years, how did it change? From what you know, how familiar are farmers that SBEP did not deal with directly with the Grow More Corn Initiative? Other farmers in the four program regions? Other farmers in other regoins? From what you learned from the businesses in the SBEP “Grow more Corn” initiative, what were the top three things about the program that business owners valued most? And why did they think each one was important? 1 2 3 From your national view, what effect on farmers did the SBEP “Grow More Corn” Initiative seem to have beyond the farmers that SBEP provided with drip irrigation systems and protocols? Have you seen any information or evidence that would suggest that the Grow More Corn Initiative inspired other farmers to go out and buy their own drip irrigation systems? If you’ve seen evidence of other farmers buying the kind of drip irrigation systems that SBEP gave the farmers in its program, how do they pay for this equipment? In your judgement, will the “Grow more Corn” Initiative continue to exist even after the SBEP project closes down? If yes, how will that happen? What is the 50/50 program and is it helping farmers buy drip irrigation? How many farmers have gotten drip irrigation with the 50/50 program? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 192 Whose idea was the 50/50 program and how did it get started? Most Significant Change Question How has participating in this “Grow more Corn” Initiative with SBEP changed your own organization? What has been the “most significant change” in FFM as a result of your involvement? Is there anything that cooperation with SBEP’s project has done for you in terms of capacity building of your organization, publicity, promotion with implementing “Grow more Corn” Initiative? Looking back, how might the SBEP project done a better job on the “Grow more Corn” Initiative? What recommendations would you have for those who might try to sustain this Initiative after SBEP closes? Is there anything else that you want to tell us , and we did not ask you? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 193 Survey Instrument – Women in Business Interview Log Data ID No. _________ Date A1. Day A2. Month A3. Year A4. GfK Interviewer(s) A5. Region A6. Municipality A7. Village A8. GPS coordinates A9. Interview Language 1) Macedonia 2) Albanian A10. Respondent’s native language 1) Macedonian 2) Albanian 3) Romani 4) Turkish 5) Other, which_______________ A11. Interview Venue 1) Business location 2) Other location (go to A12.) A12. If other location, identify that location _______________________________________ ____ Introduction Read or recite the full introduction to the respondent before starting the interview. Ask the respondent if she agrees to be interviewed for this survey. Provide the respondent with a card that has GfK’s contact information on it (even if it does not include the name of the interviewer). Gender Screening Question My name is __________. I am from GfK, a research firm in Skopje. We are here today conducting a survey of businesses in this area. Our questions are about what firms in this area produce and businesses’ views about their current situation and prospects for the future. The research we are conducting will help organizations that are supporting regional development in Macedonia decide what kinds of assistance to provide. This includes both Government of Macedonia agencies and assistance programs funded externally, such as the assistance program for Macedonia funded by the United States, through its assistance agency, USAID. We are asking you and other business owners and managers to help us understand economic conditions and opportunities, and efforts to improve them in this municipality. We will speak with many people like you as we gather answers. We will be writing what we learn, but we will not give out the names of specific individuals we speak with during this survey in our report. Before we begin, do you have any questions about what I just explained to you about our purpose in being here? We expect that our conversation with you today will take about 20-25 minutes of your time. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 194 1. For GfK staff: Is the person you are about to interview a female? 3. Yes – INT. Go to next question 4. No END THE INTERVIEW AND ASK FOR THE FEMALE OWNER/DECISION MAKER [INT. Do not proceed if the only person available to interview is a male. It is intended that all respondents for this particular set of interviews will be women owner/managers of businesses.] [For GfK: Be prepared to use a female interviewer in municipalities where there is a cultural sensitivity about men interviewing women in the municipality, village or home-based business environment where this interview will take place. Note that the response to this question need not be recorded as a gender of respondent question is included in the interview below. This screening question is simply intended to ensure that interviewers do not proceed with interviews with males.] Respondent Relationship to the Business 2. What is your relationship to this business? 9) I am the owner 10) I am a member of the owner’s family 11) I am the paid manager (not a family member) 12) Other __________________________________ 3. Are you the principal decision-maker for this business? 1) Yes, I am the principle decision-maker for this business, or I am actively involved in the decision-making process 2) No, I am a not a principal decision-maker- INT. END THE INTERVIEW AND ASK FOR THE DECISION MAKER. CONTINUE ONLY IF THE DECISION MAKER (OWNER/MANAGER) IS FEMALE. [INT. If the answer is No, the GfK interviewer for this survey should ask who is the principal decision-maker and determine whether this interview should be conducted with that person instead, if she is available.] Demographics – Respondent INT. Read: “As we begin this interview, I would like to ask you a few questions about yourself.” 4. Gender of Respondent 1) Male 2) Female – INT. Go to next question 5. How old are you? INT. Enter reported age 5.a INT. GfK interviewer to mark which of the following groups this respondent belongs to. 1) >29 (youth) 2) 30-34 3) 35-39 4) 40-44 5) 45-49 6) 50-54 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 195 7) 55-59 8) >60 6. What is the highest level of schooling you completed? 1) Primary school or uncompleted secondary school 2) Completed secondary school or uncompleted university 3) University degree or post graduate degree 4) Refuses to answer Respondent Membership in Business Associations INT. Read: “I’d like to ask you a few questions about business organizations you might be a part of.” 7. Are you a member of a business association/or chamber of commerce? 1) Yes 2) No – INT. GO TO NEXT SECTION  Respondent Awareness of/Participation in Economic Development Planning – Public-Private Sector Dialogue 8. What are the names of the business organizations or associations to which you belong? Name of Business Organizations or Associations to which I belong 1) 2) 3) 9. How does membership in this (these) organization(s) benefit you and this firm? (INT. Choose as many as are appropriate) 1) Facilitates purchases of inputs 2) Provides advice and training to the businesses 3) Facilitates sales of products 4) Represents firms with government 5) DK/NA 10. At what level(s) of government does the organization you are a member of represent your situation/views and those of other firms to the government? 1) Municipal level 2) Regional level 3) National level 4) All of the above 5) DK/NA EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 196 Respondent Awareness of/Participation in Economic Development Planning – Public-Private Sector Dialogue INT. Read: “Now I’d like to ask you a few questions about your awareness and involvement in economic development planning for this area – the municipality or region.” 11. Are you aware of any plan(s) for the economic development of this area? 1) Yes 2) No – INT. go to next section -> Demographics – Business 12. At what level(s) do the economic development plans with which you are personally the most familiar address economic development issues -- local/municipal level, regional level or national level? 1) Local/municipal 2) Regional 3) National 13. What are the name(s) of the economic development plan(s) at these levels with which you are personally familiar? Please provide the name(s) if you recall them Names or descriptions of economic improvement plans the respondent is personally familiar with d) National Plans ____________________________________________________________________________ _______ ____________________________________________________________________________ _______ e) Regional Plan ____________________________________________________________________________ _______ ____________________________________________________________________________ _______ f) Municipal Plan ____________________________________________________________________________ _______ ____________________________________________________________________________ _______ 14. Did you personally participate in the development of any of these economic development plan? 1) Yes 2) No – INT. skip the next question 15. In what ways did you participate in the creation or implementation of economic development plans at these levels? (Select all that apply) EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 197 Type of personal involvement in the development/ implementation of economic development plans Local/ municipal level plan (1) Regional level plan (2) National level plan (3) 1) Active, regular participant in a planning group or committee (e.g., made suggestions, helped organize the process) 1) 2) 3) 2) Attended public meetings/briefings on the plan or reviewed a written version 1) 2) 3) 3) Worked to implement the plan once it was approved/adopted 1) 2) 3) 4) Other type of involvement 1) 2) 3) 5) No personal involvement 1) 2) 3) 16. In what ways have the specific economic development plans you are most aware of been of benefit to you or your business? Specific Benefits of Economic Development Plans to this Business or Me Personally 1) Local/municipal plan(s) 2) Regional plan(s) 3) National plan(s) Demographics – Business INT. Read: “Now I would like to ask you a number of questions about this business.” 17. When (in what year) was this business started? 1) Year business was started INT. Enter Year 2) DK/NA 18. Who founded the business? 1) I started this business 2) My parents or another relative started the business 3) Other people, not relatives, started this business 4) DK/NA 19. Is this business registered as a business enterprise? (INT. Check one response). 1) Yes 2) No – INT. Skip next question 3) DK/NA - INT. Skip next question 20. If the business is registered, is it registered in your name? 1) Yes 2) No, it is registered in the name of one of my relatives 3) No, it is registered in the name of a non-relative 4) DK/NA 21. If you know or remember, how was the start-up of this business originally financed? 1) From my personal savings EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 198 2) With the savings of other family members 3) With a loan from family or friends 4) With a loan from a bank 5) Government subsidies (please name which type below if you choose this response) 6) Loan or incentive from some other institution (please name which institution below if you choose this response) 7) By other people, not me or my family 8) DK/NA 22. How are the on-going operations of this business financed today? 1) Exclusively through earnings of the firm 2) Earnings of the firm plus help from family members (savings) 3) Earnings plus a loan or line of credit from a bank or other institution 4) Other. - If other is selected, please explain below: 23. After the business’ start-up year, were further substantial investments made in this business, e.g., new equipment, expanded physical space? 1) Yes 2) No – INT. if “no”, skip the next question 24. If further investments were made in this business after its start-up year, how have they been financed? 1) Business profits – 1) Yes/2) No 2) My personal savings, or the savings of other family members – 1) Yes/2) No 3) Loans from banking or similar institutions – 1) Yes/2) No 4) Government funds or subsidies - 1) Yes/2) No (If 1) Yes, please name it which!) 5) Loan or incentive from some other institution - 1) Yes/2) No (If 1) Yes, please name which institution below if you choose this response) 6) Donation or gift from another organization or program to benefit my business – 1) Yes/2) No 7) By other people, not me or my family – 1) Yes/2) No 8) DK/NA 25. Have you, as a female business owner or manager, encountered any particular difficulties when registering or filing documents with government or banking institutions on behalf of this firm? 1) Those activities are not my responsibilities 2) Those activities are my responsibility, but I have not encountered any particular difficulties because I am a woman. The situation is the same for me as it would be for a male business owner. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 199 3) The activities are my responsibility and yes, I have encountered difficulties. INT. Ask: Please describe the difficulties or issues you faced: 26. If this firm has loans or a line of credit from a bank or other institution, in whose name is that loan or credit? 1) In my name as the owner/manager of the business 2) In the name of another family member. 3) In the name of another person or organization, not a family member 4) DK/NA 27. Have you, as a female business owner or manager, encountered any particular difficulties when attempting to organize start-up funding or on-going credit or investment financing for this firm? 1) Those activities are/were not my responsibilities 2) Those activities are my responsibility, but I have not encountered any particular difficulties because I am a woman. The situation is the same for me as it would be for a male business owner. 3) The activities are my responsibility and yes, I have encountered difficulties. INT. Ask: Please describe the difficulties or issues you faced: 28. Are there any other aspects of owning or running this business that you have found to be particularly difficult because you are a woman? Please explain: Business Type and Scale 29. Does this business primarily make products (things) or does it primarily provide services? 1) This business primarily makes products – INT. Go to next question (question 30.) 2) This business primarily provides services - INT. Skip question (go to question 31.) EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 200 30. If the business makes products, what are the specific types of products your business makes? Open ended answer expected. 31. If the business provides services, what are the specific services it provides? Open ended answer expected. 32. Based on the products or services described, into which broad business cluster does this firm best fit – in which category is it registered? (INT. The list below is from the State Statistical Office’s report on “business enterprises”. (It might be helpful if GfK interviewers had a copy of this list that they could show the respondent instead of reading off all of these choices) 1) Agriculture, forestry and fishing 2) Mining and quarrying 3) Manufacturing 4) Electricity, gas, steam and air conditioning supply 5) Water supply, sewerage, waste management and remediation activities 6) Construction 7) Wholesale and retail trade; repair of motor vehicles and motorcycles 8) Transportation and storage 9) Accommodation and food service activities 10) Information and communication 11) Financial and insurance activities 12) Real estate activities 13) Professional, scientific and technical activities 14) Administrative and support service activities 15) Public administration and defense; compulsory social security 16) Education 17) Human health and social work activities 18) Arts, entertainment and recreation 19) Other service activities 20) Activities of households as employers; undifferentiated goods- and services-producing activities of households for own use 21) Activities of extraterritorial organizations and bodies 33. How extensive, or what is the scale, of this business – total number of locations? 1) Only one main plant or business location – INT. Skip next question 2) Multiple business locations. – INT. Go to next question 3) DK/NA - INT. Skip next question EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 201 34. If the business operates in multiple locations, how many locations does it operate in? INT. Enter number Workers/Jobs Linked to the Business INT. Read: “Now, I’m going to ask you a few questions about the people who work for this business.” 35. Overall, how many people does this business currently employ? INT. Enter absolute number 36. GfK interviewer to indicate correct cluster into which the respondent’s absolute number (above) fits: 1) 0-9 2) 10-19 3) 20-49 4) 50-249 5) 250+ 6) DK/NA 37. How many of this firms’s current employees are full-time, part-time, or seasonal employees? (Total for this question should be the same as the absolute number of employees the respondent provided above – Question 35.) Type Number Percentage (GfK to later calculate %) 1) Full-time 2) Part-time, year round 3) Seasonal Total employees 100% 38. Approximately what number of current employees are female? Gender Number Percentage (GfK to later calculate %) 1) Female 2) Male Total employees 100% 38.a Approximately what percentage of current employees are youth? Gender Number Percentage (GfK to later calculate %) 1) Age 18-29 2) 30 or older Total employees 100% EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 202 38.b Does this type of work have good opportunities for young people to get started earning a living after they finish school? 1) Better opportunities than in other types of work 2) About the same as other types of work 3) Fewer opportunities than in other types of work 39. How many current employees are family members and what number are hired workers? Type Number Percentage (GfK to later calculate %) 1) Family Members 2) Hired workers Total employees 100% 40. How many of the family members who currently work for the farm are paid versus working without pay? Family Member Firm Workers Number Percentage (GfK to later calculate %) 1) Paid a salary 2) Unpaid Total Family Member Worker/Employees 100% 41. Has the number of people employed by this business increased, stayed the same, or declined over the past few years (2-5 years)? 1) Increased – INT. Go to next question (question 42.) 2) Stayed the same - INT. Skip next 2 questions, 42. And 43. (go to question 44.) 3) Declined - INT. Go to next question (question 42.) 4) DK/NA - INT. Skip next 2 questions, 42. And 43. (go to question 44.) 42. If the number of people employed by this business increased or decreased, how has the number of employees of this business changed over the past 2-5 years? (If the business was not started until after 2012 just leave any years in which it did not exist blank on this table) 2012 (1) 2013 (2) 2014 (3) 2015 (4) 2016 (5) Total Employees 43. INT. If the number of people employed by this business changed (by more than just a few people) over the past 2-5 years (increased or declined), ask: What explains that change – what explains why the number of employees increased or declined? Open-ended answer expected. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 203 Machinery and Automation 44. How would you characterize the extent and types of machinery and automation your business uses? 1) Best possible mix for this size business – INT. Go to next section - Business Volume, Sales and Costs 2) Adequate but possibly not the best mix for this size business – INT. Go to next question 3) Minimal, more or different machinery or automation is needed - INT. Go to next question 4) DK/NA 45. If you are not using what you consider to be the best possible mix of machinery and automation, why are you not using what you feel is the best mix? 1) Cost 2) Lack of knowledge about how to use other types of machinery or automation 3) Lack of available sources of other types of machinery or automation 4) Machinery and/or automation is not appropriate in our type of business 5) DK/NA Business Volume, Sales and Costs INT. Read: “Now I would like to ask a few questions about the volume of goods/services this firm produces and your sales and costs over the last few years.” 46. Over the past 2-5 years (since 2012, or more recently if this business started after 2012) has the volume of goods/services this firm produces stayed about the same, increased, or decreased? 1) Increased 2) Stayed the same 3) Decreased 4) DK/NA 47. Over this same period have your unit cost for producing your products or services (cost per unit) stayed about the same, increased or decreased? 1) Increased 2) Stayed about the same 3) Decreased 5) DK/NA 48. What has been the (approximate) value (earnings) on the volume of products or services this firm produced over the last several years (2012-2015)? (“Value of the volume of products/services produced” should approximately equal sales) (If the business was not in existence in the early years on this table, leave those years when it didn’t exist blank) EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 204 1) Value (earnings) on the Volume of Goods/Servic es Produced – 2012 (Total Sales In Denars) 2) Value (earnings) on the Volume of Goods/Servic es Produced – 2013 (Total Sales In Denars) 3) Value (earnings) on the Volume of Goods/Servic es Produced – 2014 (Total Sales In Denars) 4) Value (earnings) on the Volume of Goods/Servic es Produced – 2015 (Total Sales In Denars) 5) Value (earnings) on the Volume of Goods/Servic es Produced – 2016 (Total Sales In Denars) Value (sales) of all goods/servic es produced 49. If sales declined or increased over this period, which if any of the following occurred? 1) Some previous customers bought fewer units than before 2) Some previous customers stopped buying our goods (we lost customers) 3) Some previous customers bought more units than before 4) New customers started buying our goods (we gained customers) 5) Other important changes occurred that explain why our sales increased/decreased. If yes on “other change”, please describe these other changes: Open ended answer expected 50. Where are the customers for your business’ products located? 1) This region of Macedonia - Y/N 2) Other parts of Macedonia - Y/N 3) Other countries – some of our products are exported - Y/N 51. If the respondent indicates that some of the business’ products/services are exported, ask: What percentage of your sales comes from exports? INT. Enter percentage 52. Are the products/services your business produces expected to meet any of the following types of product quality standards? 1) Macedonian national product quality standards 2) EU standards 3) Other standards 4) DK/NA 53. How well equipped do you feel this business is to meet the product quality standards it is expected to meet? 1) Very qualified – it is easy for this business to meet all relevant standards 2) Mostly qualified 3) Not well qualified EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 205 54. If the answer was mostly or not well qualified, ask: what would it take to improve this business’ ability to meet the various product quality standards that would benefit the business to be able to meet? Open-ended response is expected 55. Overall – considering your level of production, product sales, investments in equipment and training, and production costs, are you generally satisfied with the performance of this business between 2012 (or start-up if that occurred later) and the present time? 1) Very satisfied 2) Somewhat satisfied 3) Somewhat unsatisfied 4) Very unsatisfied 56. If the respondent answered that she is generally satisfied with the performance of this business over the past several years (since 2012 or later, if it was established after 2012), what best explains why performance has been satisfactory or better? Open-ended answer expected 57. If the respondent answered that she is generally unsatisfied with the performance of this business over the past several years (since 2012), what best explains why performance has been unsatisfactory or worse? Open-ended answer expected Assistance Received on Goods/Services Delivery, Quality and/or Sales INT. Read: “Now I’d like to understand what kinds of assistance (advice, training, or equipment) this business – and you personally – may have invested in or received at no cost over the past few years.” EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 206 58. Has your business purchased/paid for technical advice, training, on-site consultation/assistance or additional equipment for your business (beyond your business start-up investments) in the past 2-5 years? 1) Yes 2) No 59. If yes, which of these kinds of investments in your business did you purchase over the past 2-5 years? How would you describe the value of the assistance you purchased for the development and current status of this business? Type of Assistance Received 1) Yes, purchased/paid for this type of assistance or improvement 2) Value-Importance of this assistance or improvement to the development and status of this business 1 = High 2 = Moderate 3 = Low 1) Technical advice (off site) – I/we visited their location 1) 1 = High 2 = Moderate 3 = Low 2) On-site consultation/assistance 1) 1 = High 2 = Moderate 3 = Low 3) Training for myself or the business’ employees 1) 1 = High 2 = Moderate 3 = Low 4) Additional equipment for the business, beyond start-up investments 1) 1 = High 2 = Moderate 3 = Low 60. Has your business received technical advice, training, on-site assistance or equipment at no cost from any external organization in the past 2-5 years? 1) Yes 2) No 61. If yes, which of these kinds of investments in your business at no cost over the past 2-5 years and from what specific organizations or programs were these types of assistance received? How would you describe the value of the assistance you received at no cost to your business? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 207 Type of Assistance Received 1) Yes, received this type of assistance or improvement at no cost (X) 2) Value￾Importance of this assistance or improvement to the development and status of this business 1 = High 2 = Moderate 3 = Low 3) Name of Organization(s) or Program that Provided Assistance at no cost 4) Type of Provider Organization or Program 1 = Gov’t of Macedonia 2 = IPARD (Instrument for Pre-Accession Assistance in Rural Development) 3= International Agency or Foreign Gov’t donor 4 = Macedonian private sector or diaspora 5 = Other foreign owned business 1) Technical advice (off site) – I/we visited their location) 1) 1 = High 2 = Moderate 3 = Low If there are multiple providers, identify each one 1,2,3 Identify type of provider for each provider of each type of assistance listed 2) On-site consultation/ assistance 1) 1 = High 2 = Moderate 3 = Low If there are multiple providers, identify each one 1,2,3 Identify type of provider for each provider of each type of assistance listed 3) Training for myself or the businesses’ employees 1) 1 = High 2 = Moderate 3 = Low If there are multiple providers, identify each one 1,2,3 Identify type of provider for each provider of each type of assistance listed 4) Additional equipment for the business, beyond start-up investments 1) 1 = High 2 = Moderate 3 = Low If there are multiple providers, identify each one 1,2,3 Identify type of provider for each provider of each type of assistance listed INT. If SBEP was named in the previous question as a provider of assistance, skip the next question. 62. Are you aware of an assistance program funded by the U.S. government called the “Small Business Expansion Project” (SBEP) that is working with Macedonian businesses on improving their quality and/or the sales of their goods or services? (Some people may know of it as the USAID project or by some other name. Try to determine whether the respondent actually knows the program even if he/she doesn’t call it SBEP) 1) Yes (by some name) 2) No – INT. if no proceed to section Income and Investment Perceptions and Expectations 63. When, in what year, did you become aware of SBEP? Enter year 64. How did you become aware of SBEP? (or whatever name they call this program.) Type of Contact or Exposure to SBEP Yes 1) Saw a story on SBEP on TV or in the newspaper 1 2) Learned about SBEP from another business assisted by SBEP or another contact who knew about SBEP 1 3) SBEP representatives sought out my business; they called or came to visit us 1 4) Other, what ________________________________ 1 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 208 65. What types of interactions have you had with SBEP? Type of Interactions with SBEP 1) Yes 2) Value-Importance of this interaction with SBEP to the development and status of this business 1 = High 2 = Moderate 3 = Low 1) Attended a SBEP conference or workshop 1) 1 = High 2 = Moderate 3 = Low 2) Participated in a SBEP training program 1) 1 = High 2 = Moderate 3 = Low 3) Received advice from SBEP specific to this specific business 1) 1 = High 2 = Moderate 3 = Low 4) Received equipment or inputs from SBEP 1) 1 = High 2 = Moderate 3 = Low 5) Business received publicity as a function of its interactions with SBEP 1) 1 = High 2 = Moderate 3 = Low 6) Other types of interactions 1) 66. If other “types of interactions” is selected above, please describe the other types of interactions this business had with SBEP. Other types of Interactions with SBEP (not listed above) Value-Importance of this interaction with SBEP to the development and status of this business 1 = High 2 = Moderate 3 = Low 1) 1 = High 2 = Moderate 3 = Low 2) 1 = High 2 = Moderate 3 = Low 3) 1 = High 2 = Moderate 3 = Low 67. As a female business owner/manager, did you feel that the types of assistance you received from SBEP was well-suited to any special needs you have or issues you face as a woman in business? 1) Very well-suited to my needs – Go to Question 68 2) Mostly well-suited to my needs – Go to Question 70 – Most Significant Change 3) Not that well-suited to my needs or the issues I face as a woman in business? – Go to Question 69 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 209 68. If the respondent answered “very well-suited”, ask: What made SBEP’s assistance “very well-suited for women in business”? Open-ended answer expected 69. If the respondent answered “not that well-suited”, ask: What made SBEP’s assistance “not that well-suited for women in business” – or how could it have been better suited? Open-ended answer expected Most Significant Change Question 70. Considering all of the ways you may have interacted with SBEP over the years since you became aware of this U.S. funded project, what has been the “most significant change” or effect on you and this business from those interactions? Income and Investment Perceptions and Expectations INT. Read: “These next few questions are about your views on the prospects for this business in the near term and your overall family welfare.” 71. Overall, how would you characterize the financial situation of the business we have been talking about today in 2016 compared to previous years? Better than last year 1 About the same 2 Worse than last year 3 DK/NA 4 72. What are your expectations for this business’ financial situation over the next 12 months? Will improve 1 Will stay the same 2 Will be worse 3 DK/NA 4 73. Did you (or the owner if you are not the owner) make any new investments in this business in 2016 (beyond normal inputs)? 1) Yes 2) No 3) DK/NA EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 210 74. How likely is it that new investment (beyond normal inputs) will be made in this business next year? 1) Very likely 2) Somewhat likely 3) Not sure 4) Somewhat unlikely 5) Very unlikely 6) DK/NA 75. What percentage of your household income comes from the business we have discussed today? (By household I mean your immediate family and relatives who live with you in your home? INT. Enter percentage 76. If this business does not account for 100% of your household’s income, from what other sources does the household also obtain money? 1) Private sector employment in a business we don’t own 2) Public sector (government) employment 3) Other businesses we own/self-employment 4) Pensions 5) Remittances 6) Other 77. What factors, in your view, exert the greatest influence on whether this business financial situation will improve or get worse over the next few years? Say that you will read through a list and then ask which ones are the most important in determining the business’s financial condition and, in turn, your family/household welfare. It might help if GfK interviewers had a copy of this list for the respondent to look at. Factors How important is this factor? 1 = Very Important 2 = Moderate importance 3 Low importance 4 = Can’t say, don’t know 1) General economic conditions in Macedonia 1 2 3 4 2) Economic conditions in the EU and other large economies 1 2 3 4 3) Macedonian government leadership 1 2 3 4 4) Macedonian government policies 1 2 3 4 5) Access to finances ( banks, loans) 1 2 3 4 6) Availability of subsidies – for businesses 1 2 3 4 7) Foreign investment in Macedonian businesses 1 2 3 4 8) Business access to markets throughout Macedonia 1 2 3 4 9) Business access to international markets 1 2 3 4 10) Business production/service levels 1 2 3 4 11) Business cost of production/service delivery 1 2 3 4 12) Price of business inputs 1 2 3 4 13) International product standards 1 2 3 4 14) Other INT. Go to Question 78 15) DK/NA EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 211 78. If “other” was selected, ask the respondent to explain “other”: INT. Open-ended response expected 79. Overall, and considering how the prospects for the business we have been discussing impacts your family/household welfare, what is your outlook on the future for your family/household? 1) Very optimistic 2) Somewhat optimistic 3) Not sure 4) Somewhat pessimistic 5) Very pessimistic A13. Name of company _______________________________ A14. Address _________________________________________ A15. City _______________________ A16. Name of respondent ___________________________________________ A17. Position in the company _______________________________________ A18. Telephone number ___________________________________________ Closing Statement Thank you for being so kind as to talk with us today. What you have told us will help others come to know how businesses like yours are doing and how you see the prospects for future development of your firm, municipality and region. We thank you again for your cooperation with this study. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 212 Survey Instrument – Tourism Interview Log Data Date A1. Day A2. Month A3. Year A4. GfK Interviewer(s) A6. Region A7. Municipality A8. Village A9. GPS coordinates A10. Interview Language 1.Macedonia 2.Albanian A11. Respondent’s native language 2) Macedonian 2) Albanian 3) Romani 4) Turkish 5 ) Other, which___ A12. Interview Venue 3) Business location 4) Other location A13. If other location, identify that location Introduction Read or recite the full introduction to the respondent before starting the interview. Ask the respondent if she agrees to be interviewed for this survey. Provide the respondent with a card that has GfK’s contact information on it (even if it does not include the name of the interviewer). My name is __________. I am from GfK, a research firm in Skopje. We are here today conducting a survey of businesses that are involved in various aspects of the travel and tourism industry in Macedonia. Our questions are about what role that you or your firm/institution plays in this industry and how you view the current status of this industry and its prospects for the future. The research we are conducting will help organizations that are supporting regional development in Macedonia decide what kinds of assistance to provide. This includes both Government of Macedonia agencies and assistance programs funded externally, such as the assistance program for Macedonia funded by the United States, through its’ assistance agency, USAID. We are asking you and other travel and tourism sector business owners and managers to help us understand tourism in Macedonia and efforts to expand travel and tourism in this area. We will speak with many people like you as we gather answers. We will be writing what we learn, but we will not give out the names of specific individuals we speak with during this survey in our report. Before we begin, do you have any questions about what I just explained to you about our purpose in being here? We expect that our conversation with you today will take about 20-25 minutes of your time. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 213 Respondent Relationship to the Travel/Tourism Sector 1. In what capacity are you involved in travel and tourism in Macedonia? 1. As the owner or manager of a business that is involved, at least to some degree, in the travel and tourism sector 2. As a representative of an association or other non-profit group that is involved in the travel/tourism sector 3. As a local, regional or national government representative. 4. Other. If other is selected, please explain: If no relationship between the respondent and the travel/tourism industry is established through the answer to this question, verify that you are talking to the right person as you may need to shift to a different respondent, e.g., the business owner or manager, or possibly discontinue the interview. Demographics – Respondent “As we begin this interview, I would like to ask you a few questions about yourself” 2. For how many years have you been involved in aspects of the travel and tourism business sector in Macedonia? 1. Less than 2 years 2. 2-5 years 3. 6-10 years 4. More than 10 years 3. Gender of Respondent 1) Male 2) Female 4. How old are you? ____years (or into which of the following groups does your age fall)? 1) >29 (youth) 2) 30-34 3) 35-39 4) 40-44 5) 45-49 6) 50-54 7) 55-59 8) >60 5. What is the highest level of schooling you completed? 1) Primary school or uncompleted secondary school 2) Completed secondary school or uncompleted university 3) University degree or post graduate degree EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 214 Respondent Awareness of/Participation in Economic Development Planning – Public-Private Sector Dialogue 6. Are you aware of any plan for the economic development of this area? 1) Yes 2) No - go to next section 7. At what level did this plan address economic development -- local/municipal, regional or national? 1) Local/municipal 2) Regional 3) National 8. Do you know the name of this economic development plan? Please provide the name if you recall it. Enter name or DK 9. How important is travel and tourism in Macedonia as an element of this economic development plan? 1) Very important – a strong focus of the plan 2) Somewhat important 3) Not very important 4) DK 10. Did you personally participate in the development of this particular economic development plan? 1) Yes 2) No, skip to the next question 11. In what ways did you participate in the creation or implementation of this plan? (Select all that apply) 1. Active, regular participant in a planning group or committee (e.g., made suggestions, helped organize the process) – Y/N 2. Attended public meetings/briefings on the plan or reviewed a written version – Y/N 3. Worked to implement the plan once it was approved/adopted – Y/N 4. Other, please describe: 12. In what ways has this specific economic development plan been of benefit to you or the organization you represent? Open-ended answer – answers should include some specific aspect of how the plan or economic development result of the plan has affected the respondent or the business, e.g., provided specific incentives, provided specific benefits. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 215 Demographics – Businesses “Now I would like to ask you a number of questions about the business you own or manage.” 13. When (in what year) was this business started? 1) Year business was started Enter year 2) DK 14. Who founded the business? 1) I started this business 2) Other members of my family started the business 3) Other people, not relatives, started this business 4) DK 15. Is this business registered as a business enterprise? (Check one response). 1) Yes 2) No 3) DK 16. In what aspects of the travel and tourism sector in Macedonia is this business involved. (Select all that apply) Aspects of Travel/Tourism Sector Business Includes Yes (X) Accommodation for visitors. 1.yes 2.no Food and beverage serving activities. 1.yes 2.no Road, rail, water or air transport services for tourists 1.yes 2.no Transport equipment rental 1.yes 2.no Travel agencies and other reservation services. 1.yes 2.no Cultural activities, e.g., historic sites, music/art festivals 1.yes 2.no Sports and other recreation activities, including sports equipment rental or instruction 1.yes 2.no Other 1.yes 2.no If “other” is marked above, please briefly explain below: 17. How extensive, or what is the scale, of this business – total number of locations? 1) Only one main business location Y/N 2) Multiple business locations. Y/N 5) DK 18. If the business operates in multiple locations, how many locations does it operate in? Enter number EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 216 Workers/Jobs Linked to the Business “Now, I’m going to ask you a few questions about the people who work on this business” 19. Overall, how many people does this business currently employ? 1) 0-9 2) 10-19 3) 20-49 4) 50-249 5) 250+ 99) DK/NA 20. Approximately what percentage of this business’ current employees are full-time, part-time or seasonal employees? Type Number GfK to calculate Percentage Full-time Part-time Seasonal 100% 21. Approximately what percentage of current employees are female? Gender Number GfK to calculate Percentage Female Male 100% 22. Approximately what percentage of current employees are youth? Gender Number Percentage (GfK to later calculate %) Age 18-29 30 or Older Total employees 100% 23. Does this type of work have good opportunities for young people to get started earning a living after they finish school? 1) Better opportunities than in other types of work 2) About the same as other types of work 3) Fewer opportunities than in other types of work EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 217 24. Has the number of people employed by this business increased, stayed the same, or declined over the past few years (2-5 years)? 1) Increased 2) Stayed the same – int. skip the next two questions and go to next section 3) Declined 4) DK – int. skip the next two questions and go to next section 25. If the number of people employed by this business increased or decreased, how has the number of employees of this business changed over the past 3-5 years? (If the business was not started until after 2012 just leave any years in which it did not exist blank in this table) 2012 2013 2014 2015 2016 Total Employees 26. If the number of people employed by this business changed (by more than just a few people) over the past 2-5 years (increased or declined), what explains that change – what explains why the number of employees increased or declined? Open-ended answer expected. Respondent Business’ Membership in Business Associations 27. Are you or is this business a member of a business association/chamber? 1) Yes 2) No – int. go to next section I’d like to understand/learn a bit about the business organizations you belong to: 28. What are the names of the business organizations or associations to which you or this business belongs? Name of Business Organizations or Associations to which I belong 1 2 3 29. How does membership in this (these) organization(s) benefit you and this business? (Choose as many as are appropriate) 6) Facilitates purchases of inputs or supplies 7) Provides advice and training to the business 8) Facilitates sales of services 9) Represents firms with government 10) DK/NA 30. At what level(s) of government does the organization you are a member of represent your situation/views and those of other firms to the government? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 218 6) Municipal level 7) Regional level 8) National level 9) All of the above 10) DK Business Volume, Sales and Costs If the business provides services, ask the following questions: 31. Over the past 2-5 years (since 2012, or more recently if this business started after 2012) has this business’ volume of services provided stayed about the same, increased, or decreased? 1) Increased 2) Stayed the same 3) Decreased 4) DK or N/A 32. Approximately how many total customers has your business served each year, and of those, how many do you estimate were tourists? 2012 2013 2014 2015 2016 Total Customers Tourists Total Tourists From Macedonia Other Countries In which of the following ways have the tourists your business served changed since 2012? 33. Age of tourist customers: 1. Becoming older as a group 2. Becoming younger as a group 3. About the same, no visible change 4. DK 34. Activity Preferences of tourist customers: 1. Becoming less physically active – looking to rest, take “look and listen tours” 2. Becoming more physically active – looking for challenges, active role in activities, lessons 3. About the same, no visible change 4. DK 35. Spending by tourist customers: 1. Increasing, per capita 2. Declining, per capita 3. About the same, no visible change 4. DK EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 219 36. Over the same period have your unit costs for providing services stayed about the same, increased or decreased? 1) Increased 2) Stayed about the same 3) Decreased 4) DK or N/A 37. What has been the (approximate) value (earnings on) the volume of services this firm provided over the last several years (2012-2015)? (“Value of the volume of services provided” should approximately equal sales) (If the business was not in existence in the early years on this table, leave those years when it didn’t exist blank) Value of the Volume of Services Provided – 2012 Total Sales In Denars Value of the Volume of Services Provided – 2013 Total Sales In Denars Value of the Volume of Services Provided – 2014 Total Sales In Denars Value of the Volume of Services Provided – 2015 Total Sales In Denars Value of the Volume of Services Provided – 2016 Total Sales In Denars Value (sales) of all good produced 38. If sales declined or increased over this period, which, if any, of the following occurred? 4) Some previous customers bought fewer services than before – Y/N 5) Some previous customers stopped buying our services (we lost customers) – Y/N 6) We reduced the number and types of services we offer – Y/N 7) Some previous customers bought more services than before – Y/N 8) New customers started buying our services (we gained customers) Y/N 9) We increased the number and types of services we offer – Y/N 10) Other important changes occurred that explain why our sales increased/decreased – Y/N. If yes on “other change” , please describe these other changes: Open-ended answer expected 39. If the business increased the number and types of services it offers over the past several years (since 2012) what new services were added and why? New Tourism Related Service Added Year Reason for Adding this Service 40. Overall – considering this businesses’ volume of services, sales, investments in equipment and training, and service delivery costs, are you generally satisfied with the performance of this business between 2012 (or start-up if that occurred later) and the present? 1) Very satisfied 2) Somewhat Satisfied EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 220 3) Somewhat Unsatisfied 4) Very unsatisfied 41. If the respondent answered that she/he is generally satisfied with the performance of this business over the past several years (since 2012 or later, if it was established after 2012), what best explains why performance has been satisfactory or better? Open-ended answer expected 42. If the respondent answered that she/he is generally unsatisfied with the performance of this business over the past several years (since 2012), what best explains why performance has been unsatisfactory or worse? Open-ended answer expected 43. Over the past several years, have you or the organization you represent participated with others in any regional, seasonal or other tourism promotion events? 1) Yes – go to question 44 2) No– go to next section 3) DK – go to next section 44. If the respondent answers yes in the previous question: Which tourism promotion events (up to 3) in which you have participated along with others had the greatest impact in terms of stimulating tourism – meaning it led to increased numbers of tourists after the event, increased tourism income for businesses in that sector, or increased tourism related jobs in the municipalities or regions where the event was held? Name of Tourism Promotion Event Year What specifically it promoted Who Organized the Event What specifically has been its impact 1. 2. 3. Assistance Received on Services Delivery, Quality and/or Sales “Now I’d like to understand what kinds of assistance (advice, training, or equipment) this business – and you personally – may have invested in or received at no cost over the past few years.” 45. Has your business received technical advice, training, on-site consultation/assistance or additional equipment for your business (beyond your business start-up investments) in the past several years (since 2012)? 1) Yes 2) No – int. go to next section EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 221 46. If yes, which types of assistance did you receive, and did you pay for this assistance or was it provided without charge by some other organization? Type of Assistance Received We paid for this assistance (X) We received this assistance without charge from another organization (X) Name of Organization that Provided this assistance Value-Importance of this assistance or improvement to the development and status of this business 1 = High 2 = Moderate 3 = Low Technical advice (off site) – I/we visited their location 1,yes 2,no 1,yes 2,no 1 = High 2 = Moderate 3 = Low On-site consultation/assistance 1,yes 2,no 1,yes 2,no 1 = High 2 = Moderate 3 = Low Training for myself or the business’ employees 1,yes 2,no 1,yes 2,no 1 = High 2 = Moderate 3 = Low Additional equipment for the business, beyond our start-up investments 1,yes 2,no 1,yes 2,no 1 = High 2 = Moderate 3 = Low If SBEP was named in the previous question as a provider of assistance, skip the next question. 47. Are you aware of an assistance program funded by the U.S. government called the Small Business Expansion Project (SBEP) that is working with Macedonian businesses on improving their quality and/or the sales of their tourism business or related services? (Some people may know of it as the USAID project or by some other name. Try to determine whether the respondent actually knows the program even if he/she doesn’t call it SBEP) 1) Yes (by some name) 2) No – if no proceed to the following section 48. When, in what year, did you become aware of SBEP? Enter year 49. How did you become aware of SBEP? (or whatever name they call this program) Type of Contact or Exposure to SBEP Yes Saw a story on SBEP on TV or in the newspaper 1,yes 2,no Learned about SBEP from another business assisted by SBEP or another contact who knew about SBEP 1,yes 2,no SBEP representatives sought out my business; they called or came to visit us 1,yes 2,no Other 1,yes EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 222 2,no 50. What types of interactions have you had with the SBEP program? Type of Interactions with SBEP Yes (X) Value-Importance of this interaction with SBEP to the development and status of this business 1 = High 2 = Moderate 3 = Low Attended a SBEP conference or workshop 1,yes 2,no 1 = High 2 = Moderate 3 = Low Participated in a SBEP training program 1,yes 2,no 1 = High 2 = Moderate 3 = Low Received technical advice from SBEP specific to this specific business 1,yes 2,no 1 = High 2 = Moderate 3 = Low Received equipment or inputs for this business from SBEP 1,yes 2,no 1 = High 2 = Moderate 3 = Low Participated in a SBEP organized tourism promotion activity 1,yes 2,no 1 = High 2 = Moderate 3 = Low Business received publicity as a function of its interactions with SBEP 1,yes 2,no 1 = High 2 = Moderate 3 = Low Other types of interactions 1,yes 2,no 51. If “other types of interactions” is selected above, please describe the other types of interactions this business had with SBEP. Other types of Interactions with SBEP (not listed above) Value-Importance of this interaction with SBEP to the development and status of this business 1 = High 2 = Moderate 3 = Low 1 1 = High 2 = Moderate 3 = Low 2 1 = High 2 = Moderate 3 = Low 3 1 = High 2 = Moderate 3 = Low EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 223 Most Significant Change Question 52. Considering all of your interactions with SBEP what has been the “most significant change” that occurred following, or as a result of those interactions, for this business, or for you personally? Income and Investment Perceptions and Expectations “These next few questions are about your views on the prospects for this business in the near-term and your overall family welfare.” 53. Overall, how would you characterize the financial situation of the business we have been talking about today in 2016 compared to previous years? Better than last year 1 About the same 2 Worse than last year 3 DK/NA 99 54. What are your expectations for the business’ financial situation over the next 12 months? Will improve 1 Will stay the same 2 Will be worse 3 DK/NA 99 55. Did you (or the owner if you are not the owner) make any new investments in this business in 2016 (beyond normal inputs)? 1) Yes 2) No 3) DK/NA 56. How likely is it that new investment (beyond normal inputs) will be made in this business next year? 5) Very likely 4) Somewhat likely 3) Not sure 2) Somewhat unlikely 1) Very unlikely 99) DK/NA 57. What factors, in your view, will exert the greatest influence on whether this business’ financial situation will improve or get worse over the next few years? Say that you will read through a list and then ask which ones are the most important in determining the EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 224 business’ financial condition and, in turn, your family/household welfare. It might help if GfK interviewers had a copy of this list the respondent could look at. Factors How important is this factor? 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know General economic conditions in Macedonia 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know Economic conditions in the EU and other large economies 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know Macedonian government leadership 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know Macedonian government policies 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know Availability of subsidies – for businesses 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know Access to finance 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know Foreign investment in Macedonian businesses 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know Business access to markets throughout Macedonia 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know Business access to international markets 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know Business production/service levels 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know Business cost of production/service delivery 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know Price of business inputs 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know International product standards 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 225 Other DK/NA 58. If “other” was selected, ask the respondent to explain “other”: Open-ended response expected 59. Overall, and considering how the prospects for the business we have been discussing impacts your family/household welfare, what is your outlook on the future for your family/household? 1) Very optimistic 2) Somewhat optimistic 3) Not Sure 4) Somewhat pessimistic 5) Very pessimistic A14. Name of company _______________________________ A15. Address_________________________________________ A16. City_______________________ A17. Name of respondent___________________________________________ A18. Position in the company_______________________________________ A19. Telephone number___________________________________________ Closing Statement Thank you for being so kind as to talk with us today. What you have told us will help others come to know how businesses like yours are doing and how you see the prospects for future development of your firm, municipality and region. We thank you again for your cooperation with this study. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 226 Survey Instrument – Light Manufacturing (Supplier Version) Interview Log Data Date A1. Day A2. Month A3. Year A4. GfK Interviewer(s) A6. Region A7. Municipality A8. Village A9. GPS coordinates A10. Interview Language Macedonia Albanian A11. Respondent’s native language 1) Macedonian 2) Albanian 3) Romani 4) Turkish 5) other, which___ A12. Interview Venue 5) Business location 6) Other location A13. If other location, identify that location Introduction Read or recite the full introduction to the respondent before starting the interview. Ask the respondent if she agrees to be interviewed for this survey. Provide the respondent with a card that has GfK’s contact information on it (even if it does not include the name of the interviewer). Respondent Relationship to this Business My name is __________. I am from GfK, a research firm in Skopje. We are here today conducting a survey of businesses in Macedonia and their involvement in regional economic development initiatives. Our questions are about what role that you or your firm has played in such efforts. The research we are conducting will help organizations that are supporting regional development in Macedonia decide what kinds of assistance to provide. This includes both Government of Macedonia agencies and assistance programs funded externally, such as the assistance program for Macedonia funded by the United States, through its assistance agency, USAID. We are asking you and other business owners and managers to help us understand business involvement in regional development initiatives in Macedonia. Among these we include efforts to make firms in Macedonia aware of each other’s interests in buying inputs locally as well as supplying products for local firms and markets. We will a number of firms doing business in Macedonia as we gather answers. We will be summarizing what we learn, but we will not give out the names of specific individuals we speak with during this survey in our report. Before we begin, do you have any questions about what I just explained to you about our purpose in being here? We expect that our conversation with you today will take about 20-25 minutes of your time. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 227 In what capacity are you involved in this business? 1. I own this business. 2. I am a member of the owner’s family and am a principal decision-maker in the business. 3. I am the paid manager of this business, not a family member. 4. Other. If other is selected, please explain: Use this question to determine whether you are speaking with the owner/manager and if not try to shift and interview a respondent who is in a top decision-making position. Demographics – Respondent “As we begin this interview, I would like to ask you a few questions about yourself” For how many years have you been involved in this business? 1. Less than 2 years 2. 2-5 years 3. 6-10 years 4. More than 10 years Gender of Respondent 1) Male 2) Female How old are you? ____years (or into which of the following groups does your age fall)? 1) >29 (youth) 2) 30-34 3) 35-39 4) 40-44 5) 45-49 6) 50-54 7) 55-59 8) >60 What is the highest level of schooling you completed? 1) Primary school or uncompleted secondary school 2) Completed secondary school or uncompleted university 3) University degree or post graduate degree Demographics – Businesses “Now I would like to ask you a number of questions about the business you own or manage.” When (in what year) was this business started? 3) Year business was started Enter year EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 228 4) DK Who founded the business? 5) I started this business 6) Other members of my family started the business 7) Other people, not relatives, started this business 8) DK Is this firm registered as a business enterprise in Macedonia? (Check one response). 1) Yes 2) No 3) DK What type of business is this -- in which sector does it operate? (Select only one) 1. Agriculture, forestry and fishing 2. Mining and quarrying 3. Manufacturing 4. Services, including wholesale/retail, construction, financial and transport services 5. Tourism, including accommodations 6. Other. If other is chosen, explain below If this is a manufacturing business, what are the main types of products does this firm make? Open ended answer GfK interviewer, please look at the SITC code list at the back of this interview and enter the SITC codes that best match the products the respondent has described. It is ok to ask the respondent to help you enter the right product category numbers below. How extensive, or what is the scale, of this business – total number of locations? 1) Only one main business location Y/N 2) Multiple business locations. Y/N 5) DK If the business operates in multiple locations, how many locations does it operate in? Enter number Workers/Jobs Linked to the Business EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 229 “Now, I’m going to ask you a few questions about the people who work on this business Overall, how many people does this business currently employ? 1) 0-9 2) 10-19 3) 20-49 4) 50-249 5) 250+ 99) DK/NA Approximately what percentage of this business’ current employees are full time, part time, or seasonal employees? Type Percentage Full time Part time Seasonal 100% Approximately what percentages of current employees are female? Gender Percentage Female Male 100% Approximately what percentages of current employees are youth (aged 18-29)? Age Group Number Percentage (GfK to later calculate %) Age 18-29 30 or Older Total employees 100% 1. Does this type of work have good opportunities for young people to get started earning a living after they finish school? 1) Better opportunities than in other types of work 2) About the same as other types of work 3) Fewer opportunities than in other types of work Has the number of people employed by this business increased, stayed the same, or declined over the past few years (2-5 years)? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 230 1) Increased 2) Stayed the same 3) Declined 4) DK If the number of people employed by this business changed (by more than just a few people) over the past 2-5 years (increased or declined), what explains that change – what explains why the number of employees increased or declined? Open ended answer expected. Business Volume, Sales and Costs Over the past few years (since 2012), has this businesses’ volume of goods/services provided stayed about the same, increased, or decreased? 5) Increased 6) Stayed the same 7) Decreased 8) DK or N/A What has been the (approximate) value (earnings on) the volume of goods or services this firm provided over the last several years (2012-2015)? (“Value of the volume of good or services provided” should approximately equal sales) (If the business was not in existence in the early years on this table, leave those years when it didn’t exist blank) Value of the Volume of Goods/Services Provided – 2012 Total Sales In Dinars Value of the Volume of Goods/Services Provided – 2013 Total Sales In Dinars Value of the Volume of Goods/Services Provided – 2014 Total Sales In Dinars Value of the Volume of Goods/Services Provided – 2015 Total Sales In Dinars Value (sales) of all goods produced If sales declined or increased over this period, which if any of the following occurred? 11) Some previous customers bought fewer goods/services than before – Y/N 12) Some previous customers stopped buying our goods/services (we lost customers) – Y/N 13) We reduced the number and types of goods/services we offer – Y/N 14) Some previous customers bought more goods/services than before – Y/N 15) New customers started buying our goods/services (we gained customers) Y/N 16) We increased the number and types of goods/services we offer – Y/N 17) Other important changes occurred that explain why our sales increased/decreased – Y/N. If yes on “other change” , please describe these other changes: Open ended answer expected EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 231 If the business increased the number and types of goods/services it offers over the past several years (since 2012) what new goods/services were added and why? New Goods/Services Added Year Reason for Adding this Good/Service Over the same period have your unit costs for providing goods/services stayed about the same, increased or decreased? 5) Increased 6) Stayed about the same 7) Decreased 8) DK or N/A Do you import some of the inputs for the goods and services you produce? 1) Yes 2) No – if no skip to first question on exports What portion of your inputs come from countries other than Macedonia? 1) Less than 25% 2) 26 to 50% 3) Over 50% Where do most of your imports originate? 1) Nearby/neighbor non-EU countries – Y/N 2) EU countries – Y/N 3) Other more distant countries – Y/N 4) DK To customers in what location do you sell most of the products your business sells? 1) This region of Macedonia (the one where the business is headquartered) - Y/N 2) Other parts of Macedonia - Y/N 3) Export sales to businesses in other countries - Y/N If the respondent answered export sales, ask: To what countries do you export at least some of your products? 1) Nearby/neighbor non-EU countries – Y/N 2) EU countries – Y/N 3) Other more distant countries – Y/N 4) DK EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 232 What percent of your sales come from exports? 1) Less than 25% 2) 26 to 50% 3) Over 50% SBEP Are you aware of an assistance program funded by the U.S. government called the Small Business Expansion Project (SBEP) that is working with Macedonian businesses on improving their quality and/or the sales of their tourism business or related services? (Some people may know of it as the USAID project or by some other name. Try to determine whether the respondent actually knows the program even if he/she doesn’t call it SBEP) 1) Yes (by some name) 2) No – if no proceed to Q XX When, in what year, did you become aware of SBEP? Enter year How did you become aware of SBEP? (or whatever name they call this program.) Type of Contact or Exposure to SBEP Yes Saw a story on SBEP on TV or in the newspaper Learned about SBEP from another business assisted by SBEP or another contact who knew about SBEP SBEP representatives sought out my business; they called or came to visit us? Other One of SBEP’s initiatives over the past five years has involved identifying businesses that operate in Macedonia and working with them to identify imported input they use that might be available from Macedonian companies, or might be things that Macedonian companies could start to produce against custom orders. After identifying products that firms operating in Macedonia would consider buying locally instead of importing, SBEP researched existing firms that might be potential suppliers of such products. Was your firm one of the firms that SBEP reached out to when it was looking for potential suppliers of specific types of products or services? And if yes, can you please describe how they approached you and in what year? After you met SBEP, did it introduce you to any firms that might be potential buyers for your existing products, or new products you had the capacity to create? Could you describe this process and and how many firms SBEP introduced you to and for what specific reasons? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 233 Could you share with us the names of the candidate buyer firms to which you were introduced? What were the results of this “match making” process between you and possible local buyer firms for one or more of your products or services? Did any of the introductions lead to contracts, if so can you describe their scale, duration, value or other characteristics? Are you supplying products to the firms to which SBEP introduced you? What is your impression of the kind of “match making” program in which you were involved with SBEP? Is it something that could be useful for a large number of firms that might be able to supply products or services to other firms operating in Macedonia that currently import some of their inputs? Or was your situation fairly unique and such a program might only be useful for a few firms? SBEP is closing soon, and those involved in its transition are thinking about how this business to business program might be sustained? One idea would place it inside a government agency. Another might try to leave it in the private sector to operate on a fee basis. What are your thoughts about whether the program should be continued and how that could be accomplished? Thinking about all of your interactions with SBEP, what has been the “most significant change” in your business to emerge as a result of your interactions with the SBEP program? Open ended answer – in the form of a story – is expected here. Respondent Awareness of/Participation in Economic Development Planning – Public-Private Sector Dialogue “Now I’d like to ask you a few questions about your awareness and involvement in economic development planning for this area – the municipality or region.” In particular we are interested in plans that include a focus on balanced regional growth and development, i.e., plans that promote grown and development in regions outside of Skopje. SBEP, the program we have just discussed, was loosely linked to the idea of strengthening growth and development in other regions. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 234 Are you aware of any plan for the economic development of this area – this municipality or region? 3) Yes 4) No - go to next section At what level did this plan address economic development -- local/municipal, regional or national? 4) Local/municipal 5) Regional 6) National Do you know the name of this economic development plan? Please provide the name of the plan to which you are referring, if you recall it. Enter name of plan or DK Is balanced economic growth in Macedonia a priority of this plan, i.e., promoting economic growth and development in regions outside of Skopje? 1) Yes 2) No 3) Can’t recall From your perspective, how important is balanced regional growth and development in Macedonia? 1) Very important 2) Somewhat important 3) Not very important 4) No opinion Did you personally participate in the development of the particular economic development plan we discussed? 1) Yes 2) No, skip to the next question In what ways did you participate in the creation or implementation of this plan? (Select all that apply) a. Active, regular participant in a planning group or committee (e.g., made suggestions, helped organize the process) – Y/N b. Attended public meetings/briefings on the plan or reviewed a written version – Y/N c. Worked to implement the plan once it was approved/adopted – Y/N d. Other, please describe: EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 235 In what ways has this specific economic development plan for been of benefit to you or the organization you represent? Open ended answer – answers should include some specific aspect of the plan or economic development result of the plan has affected the respondent or the business, e.g., provided specific incentives and/or provided specific benefits. Perceptions and Expectations for the Future These last few questions are about your views on the prospects for this business in Macedonia. Overall, how would you characterize the financial situation your business in 2016 compared to previous years? Better than past years About the same Worse than past years DK/NA What are your expectations for this the businesses’ financial situation over the next 12 months? Will improve Will stay the same Will be worse DK/NA Did you (or the owner if you are not the owner) make any new investments in this business in 2016 (beyond normal inputs)? 1) Yes 2) No 99) DK/NA How likely is it that new investment (beyond normal inputs) will be made in this farm business next year? 5) Very likely 4) Somewhat likely 3) Not sure 2) Somewhat unlikely 1) Very unlikely 99) DK/NA What factors, in your view, will exert the greatest influence on whether this businesses’ financial situation will improve or get worse over the next few years? Say that you will read through a list and then ask which ones are the most important in determining the firm’s financial condition and, in turn, your family/household welfare. It might help if GfK interviewers had a copy of this list the respondent could look at. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 236 Factors How important is this factor? 1 = Very Important 2 – Moderate importance 3- Low importance 4 – Can’t say, don’t know General economic conditions in Macedonia Economic conditions in the EU and other large economies Macedonian government leadership Macedonian government policies Availability of subsidies – for businesses Foreign investment in Macedonian businesses Business access to markets throughout Macedonia Business access to international markets Business production/service levels Business cost of production/service delivery Price of business inputs International product standards Other DK/NA A14. Name of company _______________________________ A15. Address_________________________________________ A16.City_______________________ A17.Name of respondent___________________________________________ A18. Position in the company_______________________________________ A19. Telephone number___________________________________________ Closing Statement Thank you for being so kind as to talk with us today. What you have told us will help others come to know how businesses like you are doing and how you see the prospects for future development of your firm, municipality and region. We thank you again for your cooperation with this study. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 237 Survey Instrument – Light Manufacturing (Buyer Version) Interview Log Data Date A1. Day A2. Month A3. Year A4. GfK Interviewer(s) A6. Region A7. Municipality A8. Village A9. GPS coordinates A10. Interview Language Macedonia Albanian A11. Respondent’s native language 3) Macedonian 2) Albanian 3) Romani 4) Turkish 5) other, which___ A12. Interview Venue 7) Business location 8) Other location A13. If other location, identify that location Introduction Read or recite the full introduction to the respondent before starting the interview. Ask the respondent if he/she agrees to be interviewed for this survey. Provide the respondent with a card that has GfK’s contact information on it (even if it does not include the name of the interviewer). My name is __________. I am from GfK, a research firm in Skopje. We are here today conducting a survey of businesses in Macedonia and their involvement in regional economic development initiatives. Our questions are about what role that you or your firm has played in such efforts. The research we are conducting will help organizations that are supporting regional development in Macedonia decide what kinds of assistance to provide. This includes both Government of Macedonia agencies and assistance programs funded externally, such as the assistance program for Macedonia funded by the United States, through its assistance agency, USAID. We are asking you and other business owners and managers to help us understand business involvement in regional development initiatives in Macedonia. Among these we include efforts to make firms in Macedonia aware of each other’s interests in buying inputs locally as well as supplying products for local firms and markets. We will a number of firms doing business in Macedonia as we gather answers. We will be summarizing what we learn, but we will not give out the names of specific individuals we speak with during this survey in our report. Before we begin, do you have any questions about what I just explained to you about our purpose in being here? We expect that our conversation with you today will take about 20-25 minutes of your time. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 238 Respondent Relationship to the Business In what capacity are you involved in this business? 5. I own this business. 6. I am a member of the owner’s family and am a principal decision-maker in the business. 7. I am the paid manager of this business, not a family member. 8. Other. If other is selected, please explain: Use this question to determine whether you are speaking with the owner/manager and if not try to shift and interview a respondent who is in a top decision-making position. Demographics – Respondent “As we begin this interview, I would like to ask you a few questions about yourself” For how many years have you been involved in this business? 5. Less than 2 years 6. 2-5 years 7. 6-10 years 8. More than 10 years Gender of Respondent 1) Male 2) Female How old are you? ____years (or into which of the following groups does your age fall)? 9) >29 (youth) 10) 30-34 11) 35-39 12) 40-44 13) 45-49 14) 50-54 15) 55-59 16) >60 What is the highest level of schooling you completed? 1) Primary school or uncompleted secondary school 2) Completed secondary school or uncompleted university 3) University degree or post graduate degree Demographics – Businesses “Now I would like to ask you a number of questions about the business you own or manage.” EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 239 When (in what year) was this business started? 5) Year business was started Enter year 6) DK Who founded the business? 9) I started this business 10) Other members of my family started the business 11) Other people, not relatives, started this business 12) DK Is this firm registered as a business enterprise in Macedonia? (Check one response). 1) Yes 2) No 3) DK What type of business is this -- in which sector does it operate? (Select only one) 7. Agriculture, forestry and fishing 8. Mining and quarrying 9. Manufacturing 10. Services, including wholesale/retail, construction, financial and transport services 11. Tourism, including accommodations 12. Other. If other is chosen, explain below If this is a manufacturing business, what are the main types of products does this firm make? Open ended answer GfK interviewer, please look at the SITC code list at the back of this interview and enter the SITC codes that best match the products the respondent has described. It is ok to ask the respondent to help you enter the right product category numbers below. How extensive, or what is the scale, of this business – total number of locations? 1) Only one main business location Y/N 2) Multiple business locations. Y/N 5) DK EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 240 If the business operates in multiple locations, how many locations does it operate in? Enter number Workers/Jobs Linked to the Business “Now, I’m going to ask you a few questions about the people who work on this business Overall, how many people does this business currently employ? 1) 0-9 2) 10-19 3) 20-49 4) 50-249 5) 250+ 6) DK/NA Approximately what percentage of this business’ current employees are full time, part time, or seasonal employees? Type Percentage Full time Part time Seasonal 100% Approximately what percentages of current employees are female? Gender Percentage Female Male 100% Approximately what percentages of current employees are youth (aged 18-29)? Age Group Number Percentage (GfK to later calculate %) Age 18-29 30 or Older Total employees 100% 2. Does this type of work have good opportunities for young people to get started earning a living after they finish school? 4) Better opportunities than in other types of work EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 241 5) About the same as other types of work 6) Fewer opportunities than in other types of work Has the number of people employed by this business increased, stayed the same, or declined over the past few years (since 2012)? 1) Increased 2) Stayed the same 3) Declined 4) DK If the number of people employed by this business changed (by more than just a few people) over the past 2-5 years (since 2012) -- (increased or declined), what explains that change – what explains why the number of employees increased or declined? Open ended answer expected. Business Volume, Sales and Costs Over the past few years (since 2012), has this businesses’ volume of goods/services provided stayed about the same, increased, or decreased? 9) Increased 10) Stayed the same 11) Decreased 12) DK or N/A What has been the (approximate) value (earnings on) the volume of goods or services this firm provided over the last several years (2012-2015)? (“Value of the volume of good or services provided” should approximately equal sales) (If the business was not in existence in the early years on this table, leave those years when it didn’t exist blank) Value of the Volume of Goods/Services Provided – 2012 Total Sales In Dinars Value of the Volume of Goods/Services Provided – 2013 Total Sales In Dinars Value of the Volume of Goods/Services Provided – 2014 Total Sales In Dinars Value of the Volume of Goods/Services Provided – 2015 Total Sales In Dinars Value (sales) of all goods produced If sales declined or increased over this period, which if any of the following occurred? 18) Some previous customers bought fewer goods/services than before – Y/N 19) Some previous customers stopped buying our goods/services (we lost customers) – Y/N 20) We reduced the number and types of goods/services we offer – Y/N 21) Some previous customers bought more goods/services than before – Y/N 22) New customers started buying our goods/services (we gained customers) Y/N EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 242 23) We increased the number and types of goods/services we offer – Y/N 24) Other important changes occurred that explain why our sales increased/decreased – Y/N. If yes on “other change” , please describe these other changes: Open ended answer expected If the business increased the number and types of goods/services it offers over the past several years (since 2012) what new goods/services were added and why? New Goods/Services Added Year Reason for Adding this Good/Service Over the same period have your unit costs for providing goods/services stayed about the same, increased or decreased? 9) Increased 10) Stayed about the same 11) Decreased 12) DK or N/A Do you import some of the inputs for the goods and services you produce? 3) Yes 4) No – if no skip to first question on exports What portion of your inputs come from countries other than Macedonia? 4) Less than 25% 5) 26 to 50% 6) Over 50% Where do most of your imports originate? 5) Nearby/neighbor non-EU countries – Y/N 6) EU countries – Y/N 7) Other more distant countries – Y/N 8) DK To customers in what location do you sell most of the products your business sells? 4) This region of Macedonia (the one where the business is headquartered) - Y/N 5) Other parts of Macedonia - Y/N 6) Export sales to businesses in other countries - Y/N If the respondent answered export sales, ask: To what countries do you export at least some of your products ? 5) Nearby/neighbor non-EU countries – Y/N 6) EU countries – Y/N 7) Other more distant countries – Y/N EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 243 8) DK What percent of your sales come from exports? 4) Less than 25% 5) 26 to 50% 6) Over 50% SBEP Are you aware of an assistance program funded by the U.S. government called the Small Business Expansion Project (SBEP) that is working with Macedonian businesses on improving their quality and/or the sales of their tourism business or related services? (Some people may know of it as the USAID project or by some other name. Try to determine whether the respondent actually knows the program even if he/she doesn’t call it SBEP) 1) Yes (by some name) 2) No – if no proceed to Q XX When, in what year, did you become aware of SBEP? Enter year How did you become aware of SBEP? (or whatever name they call this program.) Type of Contact or Exposure to SBEP Yes Saw a story on SBEP on TV or in the newspaper Learned about SBEP from another business assisted by SBEP or another contact who knew about SBEP SBEP representatives sought out my business; they called or came to visit us? Other One of SBEP’s initiatives over the past five years has involved identifying businesses that operate in Macedonia and working with them to identify imported input they use that might be available from Macedonian companies, or might be things that Macedonian companies could start to produce against custom orders. If you had an interaction like this with the SBEP team, can you descried for us how it began and what kinds of products you identified for which SBEP could try to help you identify local suppliers, including the year in which this interaction occurred? Once you and SBEP identified imported inputs for which you would consider using local sources, what transpired next? Did SBEP identify possible suppliers to you? Did they identify one or more than one option and were you then introduced to candidate supplier firms? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 244 How long after your original discussions with SBEP about local suppliers for some of your imported inputs did SBEP introduce you to candidate local suppliers for these products? Could you share with us the names of the candidate supplier firms to which you were introduced? What were the results of this “match making” process between you and possible local supply firms for one or more of the inputs you import? Did you find a good match, or were the candidate firms SBEP identified not a good fit for your needs? Did any of the introductions lead to contracts, if so can you describe their scale, duration, value or other characteristics? Are you still using suppliers to whom SBEP introduced you? What is your impression of the kind of “match making” program in which you were involved with SBEP? Is it something that could be useful for a large number of firms that import some of their inputs from outside Macedonia? Or was your situation fairly unique and such a program might only be useful for a few firms? SBEP is closing soon, and those involved in its transition are thinking about how this business to business program might be sustained? One idea would place it inside a government agency. Another might try to leave it in the private sector to operate on a fee basis. What are your thoughts about whether the program should be continued and how that could be accomplished? Thinking about all of your interactions with SBEP, what has been the “most significant change” in your business to emerge as a result of your interactions with the SBEP program? Open ended answer – in the form of a story – is expected here. Respondent Awareness of/Participation in Economic Development Planning – Public-Private Sector Dialogue “Now I’d like to ask you a few questions about your awareness and involvement in economic development planning for this area – the municipality or region.” In particular we are interested in plans that include a focus on balanced regional growth and development, i.e., plans that promote grown and EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 245 development in regions outside of Skopje. SBEP, the program we have just discussed, was loosely linked to the idea of strengthening growth and development in other regions. Are you aware of any plan for the economic development of this area – this municipality or region? 5) Yes 6) No - go to next section At what level did this plan address economic development -- local/municipal, regional or national? 7) Local/municipal 8) Regional 9) National Do you know the name of this economic development plan? Please provide the name of the plan to which you are referring, if you recall it. Enter name of plan or DK Is balanced economic growth in Macedonia a priority of this plan, i.e., promoting economic growth and development in regions outside of Skopje? 4) Yes 5) No 6) Can’t recall From your perspective, how important is balanced regional growth and development in Macedonia? 5) Very important 6) Somewhat important 7) Not very important 8) No opinion Did you personally participate in the development of the particular economic development plan we discussed? 3) Yes 4) No, skip to the next question In what ways did you participate in the creation or implementation of this plan? (Select all that apply) e. Active, regular participant in a planning group or committee (e.g., made suggestions, helped organize the process) – Y/N f. Attended public meetings/briefings on the plan or reviewed a written version – Y/N g. Worked to implement the plan once it was approved/adopted – Y/N h. Other, please describe: EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 246 In what ways has this specific economic development plan for been of benefit to you or the organization you represent? Open ended answer – answers should include some specific aspect of the plan or economic development result of the plan has affected the respondent or the business, e.g., provided specific incentives and/or provided specific benefits. Perceptions and Expectations for the Future These last few questions are about your views on the prospects for this business in Macedonia. Overall, how would you characterize the financial situation your business in 2016 compared to previous years? Better than past years About the same Worse than past years DK/NA What are your expectations for this the businesses’ financial situation over the next 12 months? Will improve Will stay the same Will be worse DK/NA Did you (or the owner if you are not the owner) make any new investments in this business in 2016 (beyond normal inputs)? 1) Yes 2) No 3) DK/NA How likely is it that new investment (beyond normal inputs) will be made in this farm business next year? 5) Very likely 4) Somewhat likely 3) Not sure 2) Somewhat unlikely 1) Very unlikely 0) DK/NA What factors, in your view, will exert the greatest influence on whether this businesses’ financial situation will improve or get worse over the next few years? Say that you will read through a list and then ask which ones are the most important in determining the firm’s financial condition and, EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 247 in turn, your family/household welfare. It might help if GfK interviewers had a copy of this list the respondent could look at. Factors How important is this factor? 1 = Very Important 2 – Moderate importance 3- Low importance 4 – Can’t say, don’t know General economic conditions in Macedonia Economic conditions in the EU and other large economies Macedonian government leadership Macedonian government policies Availability of subsidies – for businesses Foreign investment in Macedonian businesses Business access to markets throughout Macedonia Business access to international markets Business production/service levels Business cost of production/service delivery Price of business inputs International product standards Other DK/NA A14. Name of company _______________________________ A15. Address_________________________________________ A16.City_______________________ A17.Name of respondent___________________________________________ A18. Position in the company_______________________________________ A19. Telephone number___________________________________________ Closing Statement Thank you for being so kind as to talk with us today. What you have told us will help others come to know how businesses like you are doing and how you see the prospects for future development of your firm, municipality and region. We thank you again for your cooperation with this study. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 248 Survey Instrument – Agribusiness Interview Log Data Date A1. Day A2. Month A3. Year A4. GfK Interviewer(s) A6. Region A7. Municipality A8. Village A9. GPS coordinates A10. Interview Language Macedonia Albanian A11. Respondent’s native language 4) Macedonian2) Albanian3) Romani 4) Turkish 5 )other, which___ A12. Interview Venue 9) Business location 10) Other location A13. If other location, identify that location The respondents for these interviews should be the owners or paid managers for registered agribusinesses in the Macedonian government’s business structure categories 10 (agribusinesses) and 11 (beverages) who are the primary decision-makers for those businesses. Introduction Read or recite the full introduction to the respondent before starting the interview. Ask the respondent if he/she has agrees to be interviewed for this survey. Provide the respondent with a card that has GfK’s contact information on it (even if it does not include the name of the interviewer). My name is __________. I am from GfK, a research firm in Skopje. We are here today conducting a survey of agribusinesses in this area. Our questions are about what firms in this areas produce and businesses’ vies about their current situation and prospects for the future. The research we are conducting will help organizations that are supporting regional development in Macedonia decide what kinds of assistance to provide. This includes both Government of Macedonia agencies and assistance programs funded externally, such as the assistance program for Macedonia funded by the United States, through its assistance agency, USAID. We are asking you and other businesses to help us understand economic conditions and opportunities, and efforts to improve them in this municipality. We will speak with many people who like you as we gather answers. We will be writing what we learn, but we will not give out the names of specific individuals we speak with during this survey in our report. Before we begin, do you have any questions about what I just explained to you about our purpose in being here? We expect that our conversation with you today will take about 20-25 minutes of your time. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 249 Respondent Relationship to the Business 1. Does your family own this business or does it belong to someone else? 1. I/we own the business 2. It belongs to someone else 3. Other If other, explain “other”_______________ 2. What is your relationship to this business? 1)I am the owner 2) I am a member of the owner’s family 3) I am the paid manager (not a family member) 4) Other (explain below)__________________ 3. Are you the principal decision-maker for this business? 1) Yes, I am the principle decision-maker for this business 2) No, I am a not the principal decision-maker- END THE INTERVIEW AND ASK FOR DECISION MAKER [If the answer is no the GfK interviewer for this survey should ask who is the principal decision-maker and determine whether this interview should be conducted with that person instead, if he/she is available.] Demographics – Respondent “As we begin this interview, I would like to ask you a few questions about yourself” 4. Gender of Respondent 1)Male 2) Female 5. How old are you? ____years (or into which of the following groups does your age fall)? 1) 15-19 2) 20-24 3) 25-29 4) 30-34 5) 35-39 6) 40-44 7) 45-49 8) 50-54 9) 55-59 10) 60-64 11) 65 + 99) DK/NA 6. What is the highest level of schooling you completed? 1) Primary school 2) Uncompleted secondary school 3) Completed secondary school EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 250 4) Uncompleted university 5) university degree 6) Postgraduate degree 99)DK/NA Respondent Membership in Business Associations 7. Are you a member of a business association/chamber? 1) Yes 2) No-go to next section I’d like to understand learn a bit about the business organizations you belong to: 8. Can you tell me which organizations they are – the name and type of organization (Column 1) 9. For each of the organizations the manager belongs to, ask: in which of the following ways does this organization help you? (Read the headings for Columns 2-5) and mark each one the respondent says applies) [Note – multiple answers are ok on this question . 8. Name of Associations 9. How Does Membership Benefit You and this Business? Facilitates purchase of supplies Provides advice/training to the business Facilitates sales of products Represents businesses with government 1.__________________ 1 2 3 4 2. _________________ 1 2 3 4 3. _________________ 1 2 3 4 Respondent Awareness of/Participation in Economic Development Planning – Public-Private Sector Dialogue 10. Are you aware of any economic improvement plans for this area – municipality, region? 3) Yes 4) No- go to next section 11. What sort of economic improvement plans are you aware of? Can you identify them by name or tell me if they are national, regional, municipal or plans that your business association developed? Names or descriptions of economic improvement plans the respondent is aware of 1 National Plans 2 Regional Plan 3 Municipal Plan 4 Business Association Plan EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 251 12. Have you personally participated in the development of any plans for improving economic prospects around here? 1) Yes 2) No-go to next section 13. If yes, what type of plan did you participate in developing and how did you participate? Enter names of plans under the type/level of plan shown in Column 2. Add rows if needed. In Column 3 describe the way in which you participated. (Columns 1 and 2). 13. Plans you played a role in developing 14. In what way did you participate? 1) attended meetings, 2) made suggestions 3) reviewed a plan 4) Other, what______________ National Plans 1 1234 Regional Plan 2 1234 Municipal Plan 3 1234 Association Plan 4 1234 Demographics – Business “Now I would like to ask you a number of questions about this business.” 15. When the business was created or started operations? 16. How extensive, or what is the scale, of this business – total number of locations? 1) Only one main plant/business location 2) Headquarters and 1-5 production and/or sales locations 3) Headquarters and 6-10 production and/or sales locations 4) Headquarters and more than 10 production and/or sales locations 5) Other -if “other” explain below____________________________________ 17. Is this business registered as a business enterprise? (Check one response). 1) Yes, as a business enterprise (sole proprietorship or limited liability company). 2) No, it is not registered 3) I don’t know 18. If the answer is “yes”, as what year it was registered and enter that year in Column 2 __________________year EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 252 Workers/Jobs Linked to the Business “Now, I’m going to ask you a few questions about the people who work on this business 19. Overall, how many people does this business employ?__________employees 1) 0-9 2) 10-19 3) 20-49 4) 50-249 5) 250+ 99) DK/NA 20. How many of these employees are full time versus part time or seasonal , and what portion are women? (Column 3 separately for men and women) 1 Full time ________Total 2 Part Time ________ Total 3 Seasonal ________ Total 4 Full time _______ Female 5 Part Time ________Female 6 Seasonal _______Female 21. How many, if any, of these employees are members of the business owner’s family (not including the owner) ________ Total ________Female 22. How many, if any, of these family members who are employees are unpaid (do not receive a salary like other employees)? ________ Total ________Female Production This question focuses on the product lines the agribusiness produces. Columns 1 and 2 on the left show all recognized product lines in the global Harmonized System (HS) at the four digit level. You should be able to find the general group(s) -- at the two digit level – that fits the business you are interviewing. Ask about all of the product lines listed for those categories and put a check mark in Column 3 for each product line the respondent says they make, e.g., sausages (1601) and vinegar (2209). Note – GfK interviewers should not expect that respondents will be familiar with the HS codes. Most will not be likely to know them. Conduct the interview asking about product lines and specific products in words not numbers. The number are here to help simplify the analysis and to help you look up how to code particular types of products if you need to (using the HS product sheet from GfK) Note – you do not have to read through all of the product lines shown on the table. Start this section by establishing in words what kinds of products the business manufactures, and then ask about specific product lines only for the broad types of product you were told they make. When you finish entering EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 253 check marks for Column 3 ask: “Is that all? Did we miss any types of products you produce? That will help you fill in any gaps. For each product line where you place a check mark in Column 3 – ask all of the questions in the remaining columns. For Column 4 ask for a more detailed description, e.g, pork sausages and apple cider vinegar may be the kind of answers you receive and write down. [After the interview you may be able to check the full HS code list and find a six digit HS code that matches what you heard. If you do, add that to the description in Column 4] For column 5 first ask how the business counts volume in terms of the product you are asking about, e.g., sausages may be counted in tons, vinegar may be counted in cases (of 12 bottles). When you write in the answer on Column 5 enter both the number of units and the unit used, e.g., tons, cases. For column 6 enter “increased”, “decreased or same”. HS Code (four digits) Basic product line description 23. 1)Yes 2) No 24. More detailed product description or HS code (six digits), multiple codes 25.Unit 26. Number of Units Produced (2013 estimate)- for category from question23 (4 digit) 27. Production Increased, decreased, or stated the same during past 3 years, since 2011? for category from question23 (4 digit) 1601 Sausages 1602 Prepared/ Preserved meat, other 1603 Extracts of meat/fish 1604 Prepared/ preserved fish 1605 Crustaceans 1701 Cane or beet sugar 1702 Sugar, caramel, , chemical 1703 Molasses 1704 Sugar confection 1801 Cocoa beans 1802 Cocoa shells 1803 Cocoa paste 1804 Cocoa butter 1805 Cocoa powder 1806 Chocolate, other cocoa EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 254 HS Code (four digits) Basic product line description 23. 1)Yes 2) No 24. More detailed product description or HS code (six digits), multiple codes 25.Unit 26. Number of Units Produced (2013 estimate)- for category from question23 (4 digit) 27. Production Increased, decreased, or stated the same during past 3 years, since 2011? for category from question23 (4 digit) 1901 Malt 1902 Pasta 1903 Tapioca 1904 Foods prep by swelling 1905 Bread, pastry 2001 Veg, nuts prep with vinegar 2002 Tomatoes prep 2003 Mushrooms prep 2004 Vegetables, frozen 2005 Vegetables prep not frozen 2006 Fruit, nuts prep with sugar 2007 Jams, jellies 2008 Fruit, nuts, prep other 2009 Fruit juices 2101 Extracts of coffee, tea 2102 Yeast, baking powder 2103 Sauces, mixed, mustard 2104 Soups, broth 2105 Ice cream 2106 Food prep protein 2201 Water, natural 2202 Water, sweetened 2203 Beer EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 255 HS Code (four digits) Basic product line description 23. 1)Yes 2) No 24. More detailed product description or HS code (six digits), multiple codes 25.Unit 26. Number of Units Produced (2013 estimate)- for category from question23 (4 digit) 27. Production Increased, decreased, or stated the same during past 3 years, since 2011? for category from question23 (4 digit) 2204 Wine 2205 Vermouth, similar 2206 Fermented, cider 2207 Ethyl alcohol, 80% + 2208 Ethyl alcohol, gin, whiskey 2209 Vinegar & substitutes 2301 Meal, residue, not for human 2302 Bran, sharps, residue-maize 2303 Starch residue 2304 Soybean oil residue 2305 Peanut oil residue 2306 Oilcakes other residue 2307 Wine lees 2308 Veg waste, for animals 2309 Other prep food for animals 2401 Tobacco, refuse 2402 Cigars, cigarettes 2403 Other tobacco products Investments in Production Aids -- Machinery and Automation Start by reviewing answers to the question above that indicated on which products the firm manufactures. Focus on the top five products the firm produces, using information about the number of units produced and the scale of the business. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 256 Bring down the top five products identified in the table above on all products manufactured. In some cases, businesses will manufacture fewer than five products. That is ok – you do not need to provide information in the question below on five products if the business you are interviewing makes only three or four (less than 5). Use the true number of products produced when answering this question. For each of up to five top products a business produces, ask what type of machinery or automation is used and for how long it has been used. 28. Products (from check marks above) 29. Type of machinery or automation used 1) individual artisan tools 2) mass production machinery 3) automation 30. For how many years has this type of machinery or automation been used for this product? 1 123 2 123 3 123 4 123 5 123 31. How would you characterize the extent and types of machinery and automation your business uses? Best possible mix for this size business and product lines is being used 1-go to next section Adequate, but possibly not the best mix for this size business and product lines 2--go to next question Minimal, more machinery or automated is needed or different types 3-go to next question DK/NA 99 32. If you are not using what you consider the best possible mix of machinery and automation for your products, why are you not using what you feel is the best mix. (Respondent can choose more than one answer) Cost 1 Lack of knowledge of how to use other types of machinery 2 Lack of availability/source of other types of equipment or automation 3 Machinery and automation is not appropriate for our businesses products 4 DK/NA 99 Manufacturing Productivity and Sales “Considering the products you manufacture and the types of production aids your business uses, I would now like to asк a few questions about productivity and sales for your main products.” Indicate that you will ask about units produced and sales for 2012 and 2013 but that you only need what they recall for 2012 or can estimate for 2013, and to understand if production, sales and prices are rising or EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 257 falling -- you do not need absolutely precise answers, good estimates for each year for these values will suffice. For the top five products, you will ask about the highest recent production level (number of units manufactured) and the amount sold and price as well as the cost of production. In total only ask this set of questions about five products or product lines Use the same list of high volume products as for the prior question. [Note that HS codes can be used to identify these products if you have the correct codes.] 33. High Volume Product or Product Line 34. Units (e.g., tons, cases) Number of units produced Amount Sold (units) Total value (sales)-in denars Total production Cost in denars 35. 2012 36. 2013 37. 2012 38. 2013 39. 2012 40. 2013 41. 2012 42. 2013 1 2 3 4 5 DK/NA- the respondent refuses to give information 99 – go to next section Production Inputs - Sources In addition to asking about production, I have a few questions to ask about the agricultural inputs you use to produce your products. 43. For the products you manufacture, what are the 1-5 main agricultural inputs? Enter in Column 1 44. For each of these inputs, I am going to ask you about the source – whether you buy it in this municipality or region; whether it comes from another part of Macedonia or whether it is imported. Put an (✓) in Column 2, 3 or 4. 45. For the imported items, ask why they import each agricultural product they import – not available locally, lower cost, higher quality or more reliable supply than locally. Multiple reasons and (✓) are allowed on this question. 43. Main Agricultural Inputs for Your Firm’s High Volume Products 44. Source of Agricultural Input 45. If Imported, Reasons Why Macedonia – this Municipality or Region Macedonia – Other Regions Imported from Another Country (Name of Country) Which country Not available locally Lower cost than local sources Higher quality than local sources More reliable supply than local sources 1 1 2 3 1 2 3 4 2 1 2 3 1 2 3 4 3 1 2 3 1 2 3 4 4 1 2 3 1 2 3 4 5 1 2 3 1 2 3 4 The company doesn’t buy agricultural inputs 98 – go to next section DK/NA- the respondent refuses to give information 99– go to next section Manufactured Product Quality -- Standards EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 258 46. For the agribusiness products you produce are there any national or international product quality standards you try or are expected to meet? Ask about national standards first. For every standard that applies to one or more of the firm’s products, write in the name of the specific standard. Then for each national standard as what products it applies to, whether the firm’s products already meet the standard and when they expect to meet it (if they haven’t already). 46. Type/Source of Standard 47 Manufactured products to which Standard applies (insert 4 digit HS code) Manufactured Product Status in Relation to Standards 48 Firm’s Product (s) already meet the standard 49 If not, when standard is likely to be achieved (year) National Standards 1 1) Yes 2)No 2 1) Yes 2)No 3 1) Yes 2)No EU Standards 4 1) Yes 2)No 5 1) Yes 2)No 6 1) Yes 2)No Other Standards 7 1) Yes 2)No 8 1) Yes 2)No 9 1) Yes 2)No 63а_10 Respondents is not aware of any standard 99– go to next section Assistance Received on Production, Quality and/or Sales Now I’d like to understand what kinds of assistance (advice, training, in-company direct assistance) this business – and you personally – may have received from the past few years. 50. From which, if any, external groups or organizations have you received advice, training or direct on￾site assistance in the past 1-3 years? 1) Yes, we have received help 2) No, we haven’t received help-go to q55 • Read the list in Column 1 to the respondent to suggest some types of groups or organizations that might have provided assistance and enter. For any of these types that the respondent says has provided assistance, obtain the name of the group or organization. Blank rows are at the bottom of this table to write in groups or organizations not listed in Column 1. • Once Column 2 is filled in, go back and ask for each organization that provided assistance what the focus of the assistance was (Column 4) and how it was delivered (Columns 5-10). 51.Type of Group or Organization 53. Types of Assistance received during past 1-3 years 54. Focus of Assistance EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 259 52. Name or other Description Of the group or organization Advice on￾site at another location Training or demonst￾ration at another location Direct assistance (services, goods, funds) (for example, input selection, production, technology use, packaging, sales 1)Input supply company 1 2 3 2)Equipment supply company 1 2 3 3)Business Association 1 2 3 4)Municipal LED office 1 2 3 5)Commerce or other ministry program 1 2 3 6)EU funded program, e.g., Leader/LAG 1 2 3 7)Other government program – U.S., other country 1 2 3 8)United Nations or other international agency program 1 2 3 Other, what? 1 2 3 [If the USAID Small Business Expansion Project (SBEP) was not identified by the respondent in answers to questions above, use the following questions to ask about this program directly. Note that the respondents may recognize this program by a different name that SBEP, e.g., the USAID Small Business program or the Carana program, or even just the USAID project for assistance to business…we do not know exactly how it is may be known at the business level in various parts of Macedonia 55. “Are you aware of any assistance programs funded by the U.S. government that are helping Macedonian businesses improve their use of farming technology and their crop production and sales?” 1) Yes 2) No 56. If yes, what is that program called, or how would you describe it? [If the respondent describes a program in response to the question, try to determine whether it is SBEP or another USAID funded program.] ___________________________________________________________________________ ________ ___________________________________________________________________________ ________ 99) The respondents is not aware for SBEP- go to q60 if the company has received assistance, if they haven’t received assistance go to next section 57. If the respondent knows of the SBEP program (by any name), ask: What contact have you had with this project called SBEP, or how do you know about it? Mark answers that match Column 1 options EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 260 below or use the blank rows provided to describe other types of contact mentioned. If the respondents is not aware of SBEP skip to question 60. 57. Type of Contact or Exposure to SBEP Yes 58. Focus or Topic Covered, e.g., crops, assistance to light industry, etc. Saw a story on TV or in the newspaper 1 Visited another firm or farm assisted by SBEP 2 Attended a SBEP conference or workshop 3 Participated in a SBEP training program 4 Received advice from SBEP to this business 5 Received equipment or inputs from SBEP 6 Other, what? If respondents have had contact with the SBEP program, ask 59. Where SBEP effects are important? Blank lines are provided to add items to the list of important effects based on what respondents say. 59. Aspects of Manufacturing Where SBEP Has Important Effects Sources of inputs 1 Quality of inputs 2 Production technology 3 Quality of production 4 Level of production 5 Helping find buyers 6 More effective interactions between government and farmer groups 7 Creation of new businesses 8 Other, what? 9 None of the above 99 60. Considering all of the sources from which you/your farm has received training, advice, materials, funds or other assistance over the past five years – including business organizations, your local or national government, assistance programs funded by other governments and foreign donors – what are the 1-3 examples of the types of assistance that made the biggest difference in the way the farm operates? 60. Examples of “High Impact” Type of Assistance received (advice, training , materials, funds) 61. Name of Provider Organization 62. How This Assistance Made a Difference in Business Operations (what changed because you/ your business received assistance?) 1.___________________ ___________________ ___________________ 2.___________________ ___________________ ___________________ EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 261 Production Buyers/Customers “Now I’d like to learn about the types of buyers or customers who purchase your high volume products – the same 1-5 products we have discussed under other questions above. 64. Enter a list of the top volume products in Column 1 – use the same list of high volume products as for questions above. [The HS codes can be used to identify them specifically of the correct codes are known. 63. Main Products the Firm Sells (Complete for up to five products, starting with the highest volume product sold) Type of buyers, % Total 64 Individuals 65 Domestic Businesses* 66 Export Product is sold directly to individual buyers/ consumers Sold to businesses in this municipality or region Sold to businesses in other regions in Macedonia Product is sold for export to other countries 100% 100% 100% 100% 100% DK/NA- the respondent refuses to give information 99– go to next section Household sources of income and perspectives These next few questions are about other sources of income of your family and perspectives in the near future. 67. Besides income from the business, does your household have other sources of income (by household I mean your immediate family and relatives who live with you in your home? 1) Yes 2) No- go to question 70 99) DK/NA- go to question 70 68. Which are they? Mark table below 69. How they contribute to the total income in percentages? Include the percentage contribution from the agribusiness first. More than one answer is allowed. Fill in blank rows if an unlisted income source is described. Once a source is identified, include the percentage in parenthesis sources of household income 68 69 write % Agribusiness 1 Private sector 2 Salary from employment in a privately owned business 3 Self-employment (own a business other than the farm) 4 Remittances from others 5 Pension from a private company 6 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 262 Public sector 7 Salary from government employment 8 Pension from the government 9 DK/NA 99 Income and Investment Perceptions and Expectations 70. Overall, how would you characterize the financial situation of this business in 2013 compared to the previous year? Better than last year 1 About the same 2 Worse than last year 3 DK/NA 99 71.What are your expectations for this the firm’s financial situation over the next 12 months? Will improve 1 Will stay the same 2 Will be worse 3 DK/NA 99 72. What factors, in your view, exert the greatest influenced on whether the firm’s financial situation will improve or get worse? Say that you will read through a list and then ask which ones are the most important in determining the firm’s financial conditions. Also ask – are there other factors besides the ones I read to you that are important. [Add the other factors the respondent mentions in the blank rows below. Factors General economic conditions in Macedonia 1 Economic conditions in the EU and other large economies 2 Macedonian government policies 3 Availability of subsidies 4 Foreign investment 5 Access to markets/buyers in Macedonia 6 Access to international markets 7 Production levels 8 Cost of production 9 Price of inputs 10 International product standards 11 Other, what_________________ 12 DK/NA 99 73. Did you (or the owner if you are not the owner) make any new investments in this business in 2013 (beyond normal inputs)? 1) Yes 2) No EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 263 99) DK/NA 74. How likely is it that new investment (beyond normal inputs) will be made in this farm business next year? 5) Very likely 4) Somewhat likely 3) Not sure 2) Somewhat unlikely 1) Very unlikely 99) DK/NA A14. Name of company A15. Address_________________________________________ A16.City_______________________ A17.Name of respondent___________________________________________ A18. Position in the company_______________________________________ A19. Telephone number___________________________________________ Closing Statement Thank you for being so kind as to talk with us today. What you have told us will help others come to know how businesses like you are doing and how you see the prospects for future development of your firm, municipality and region. In about two years we will be coming back to this area to learn whether and how things have changed for you and others who live here. We very much hope to be able to speak with you again at that time. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 264 Semi Structured Interview – Business and Farmer’s Association or Group Representatives Introduction We appreciate your willingness to be interviewed today. We are here today gathering information about how your municipality and other municipalities in this region view the progress they are making on local economic development and prospects for further development of farms and businesses in this area. This is be our second visit with your municipality. We had the pleasure of meeting and talking with you two years ago (2014), and this will give us the chance to learn about changes and progress you have made since that time. The research we are conducting will help others who are interested in regional development in Macedonia, including assistance programs for Macedonia funded by the U.S. Government’s economic assistance group called USAID, make decisions about how best to support regional development in this country. This is why we are asking you and other municipal officials to help us understand economic challenges and opportunities your local economic development plans and programs focus on and efforts that are underway to improve growth and jobs in this municipality. We will be writing about what we learn from municipalities across Macedonia, but we will not specify names of persons or municipalities in our report. Before we start this interview we would like to encourage you to ask your staff to contribute their views and ideas. Since this is a group interview, we hope you will help them feel free to contribute your thoughts as well as listen to others. There are no right or wrong answers, we are interested in learning about your opinions, perceptions and attitudes. Is our purpose clear and acceptable and can we begin? Mark this space to indicate agreement _____ Interviewer: _________________________________ Interview date: __________ 1. Name of Region: Name of Region Code Name of Region Code Eastern 1 Southeastern 5 Northeastern 2 Southwestern 6 Pelagonia 3 Vardar 7 Polog 4 Skopje 2. Name of the Municipality and Code22 (INT. Please enter name/number of municipality and code number from list provided by GfK Skopje) Enter name/number of municipality Municipality code number from GfK list 3. List of Interview Participants (INT. Should include individuals in leadership positions in local business and farm associations in the municipality, e.g., the local Chamber of Commerce. Fill in all the required information in each column for each person. Three is the limit to the size of this interview 22 Code can be entered before doing the interview or during transcription and codification of open ended questions. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 265 group and at least one of these three should represent farmers in the area and one should represent other types of enterprises.) Name (first, last) Name of Enterprise or Farmer’s Association or Group Title Length of time in position Contact Gender (M/F) Enter first name and last name Enter title (President, Director) Enter number of years individual has been in this position Enter contact phone or email Enter gender of the respondent 1) 2) 3) Municipal Type 4. Is this municipality considered to be primarily urban or primarily rural? Urban 1 Rural 2 5. Which of the following five sources of income are the #1, #2 and #3 most important sources of citizen income in this municipality? 1) Agriculture, forestry and fishing 2) Mining and quarrying 3) Manufacturing 4) Services, including wholesale/retail, construction, financial and transport services 5) Tourism, including accommodations Three top income sources for this municipality # 1 # 2 # 3 6. According to national data from the State Statistical Office, your municipality had about XX registered businesses last year, does that seem about right? [GFk use the table at the back of this interview to supply the correct figure for this municipality] 1) Yes 2) No 3) DK/NA 7. How are these businesses in your municipality divided up by sector? Approximately what percent fall into each group? Sector Rough Percentage of Businesses In this Sector 1) Agriculture, forestry and fishing (excluding independent agricultural holdings not registered as businesses) 2) Mining and quarrying 3) Manufacturing 4) Services, including wholesale/retail, construction, financial and transport services EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 266 5) Tourism, including accommodations 6) Other Total 100% 8. Approximately how many independent agricultural holdings are registered in this municipality? 9. Are exports an important source of revenue for businesses and farms in this municipality? 1) Yes (INT. Go to Question 10.) 2) No (INT. Go to Question 11.) 3) Don’t know/No answer (INT. Go to Question 11.) 10. Exports by businesses and farms in the municipality (INT. Write down the answer) DK/NA 1) Approximately what number of businesses and farms in the municipality export their products? 99 2) What are the main types of products exported from this municipality? 99 3) To what regions/countries are these exports generally sold? 99 11. Are imported inputs important for the success of businesses and farms in this municipality? 1) Yes (INT. Go to Question 12.) 2) No (INT. Go to Question 13.) 3) Don’t know/No answer (INT. Go to Question 13.) 12. Imports by businesses and farms in the municipality (INT. Write down the answer) DK/NA 1) Approximately what number of businesses in the municipality import inputs to their businesses? 99 2) What types of inputs for businesses are commonly imported by businesses and farms in this municipality? 99 3) From what regions/countries are these imports generally purchased? 99 Economic Status of Your Municipality 13. How do you estimate your municipality ranks compared to other municipalities in your region on overall GDP (the market value of all goods the municipality produces) and GDP per capita (total GDP for the municipality divided by the number of citizens)? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 267 13A. GDP overall (market value of all goods produced) 13B. GDP per Capita Much above the region’s average 1 Much above the region’s average 1 Somewhat above the region’s average 2 Somewhat above the region’s average 2 Same as the region’s average 3 Same as the region’s average 3 Somewhat below the region’s average 4 Somewhat below the region’s average 4 Much below the region’s average 5 Much below the region’s average 5 Don’t know/No answer 99 Don’t know/No answer 99 14. How would you describe overall unemployment in this municipality compared to 2014? 1) Getting worse 2) About the same 3) Declining 4) DK/NA 15. How would you describe overall youth (18-29) unemployment in this municipality compared to 2014? 1) Getting worse 2) About the same 3) Declining 4) DK/NA 16. How would you describe outmigration from this municipality compared to 2014? 1) Getting worse 2) About the same 3) Declining 4) DK/NA 17. Nationally, government statistics suggest that roughly 25% of businesses are owned and operated by women. In your municipality would you estimate that the percentage of women owned/operated businesses is: 1) Higher than that 2) About that level 3) Somewhat lower than that level 4) Much lower than that level 5) DK/NA 18. In this municipality, how supportive is the culture and business environment when women indicate that they are interested in becoming entrepreneurs? 1) More supportive than most – our community actively encourages and provides help and support to women who want to start businesses. 2) About average 3) Less encouraging than other municipalities – there are cultural factors as well as characteristics of the business environment here that discourage or make it more difficult for women to start businesses than it might be in a bigger or different municipality. 19. How would you compare your municipality’s overall economic situation in today to your municipality situation roughly two years ago (2014), when we last visited? 1) Greatly improved EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 268 2) Somewhat improved 3) Same as in 2014 (INT. Go to next section Municipal Development Plan, Question 21.) 4) Somewhat worse 5) Much worse 6) DK/NA (INT. Go to next section Municipal Development Plan, Question 21.) 20. (If the respondent’s said that the municipality’s economic situation had changed since 2014, please ask) What best explains why your municipality improved/did less well since 2014? Open ended response ______________________________________________________________________________________ ______________________________________________________________________________________ ______________________________________________________________________________________ ______________________________ Municipal Development Plan 21. Does your municipality have Municipal Economic Development Plan? 1) Yes (INT. Go to Question 22.) 2) No (INT. Go to next section LEADER/LAG Process, Question 31.) 22. INT. If Yes at previous question, ask: 22A. Do you know in what year was it prepared? 22B. What period of time does it cover? INT. Write down the year ______________ INT. Write down the time period: _________________ 23. What is the formal name of this plan? INT. Write down the name of the Plan: ___________________________________________________________________________________ ___________________________________________________________________________________ 24. Who or what entities were the primary authors of this plan? (e.g., the LED office?, some special community committee set up for this purpose?) ___________________________________________________________________________________ ___________________________________________________________________________________ ___________________________________________________________________________________ ___________________________________________________________________________________ ___________________________________________________________________________________ 25. In which of the following ways did the private sector participate on the development of the Municipal Economic Development Plan? Types of Private Sector Involvement in the Development of the Municipal Economic Development Plan Yes 1) Active, regular participant in a planning group or committee (e.g., made suggestions, helped organize the process) 2) Attended public meetings/briefings on the plan or reviewed a written version 3) Worked to implement the plan once it was approved/adopted 4) Other types of involvement, what? _____________________________________ 5) Private Sector representatives had no direct involvement EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 269 26. What are the main sources of fund through which this municipality finances its Economic Development Plan? Type of Funding Sources Specific sources from which this municipality received funding for its Economic Development Plan 1) National Government Sources 2) Regional sources 3) Municipalities own resources, including fees, municipal bonds, etc. 4) Private sector investment - domestic 5) Foreign direct investments or remittances 6) Other sources, what? _____________________________________ 27. If you answered that private sector investment (4) at previous question) is one of the sources of funds that finances action on this municipalities’ Municipal Economic Development Plan, how specifically does the business community here help to co-finance these efforts? Do individual businesses invest with their own funds in specific types of projects the community wants to undertake? Or is there some other way in which private sector resources play a financial role in implementing the Municipal Economic Development Plan? ___________________________________________________________________________________ ___________________________________________________________________________________ ______________ 28. As you understand the Municipal Economic Development Plan – what are its three main goals? The biggest or most important things your municipality hopes to achieve with in the years the plan covered? Main goals for strategies of you Municipal Economic Development Plan Goal 1) Goal 2) Goal 3) 29. What can you say about the progress this municipality has made on these goals since the Plan was created? Or specific difficulties it has encountered. Progress or difficulties since implementation of the plan started – are there any achievements you can describe? Goal 1) __________________________________________________________________ ________________________ Goal 2) __________________________________________________________________ ________________________ Goal 3) __________________________________________________________________ ________________________ 30. In addition to the main Municipal Economic Development Plan goals you have described, has the municipality taken any steps under its Plan to (1) reduce unemployment, particularly youth unemployment, or (2) reduce outmigration, or EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 270 (2) improve the business climate for would-be women and youth entrepreneurs? Are there any recent programs or achievements in these areas that you can describe since 2014? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ LEADER/LAG Process 31. Are you aware of the LEADER/LAG program from the EU that is encouraging municipalities to work together as “mini-regions” on economic opportunities and challenges they share? 1) Yes 2) No – (INT. if no skip to the next section on Regional Development Center) 32. Do you know if this municipality is a member of a LAG? 1) Yes 2) No - (INT. if no skip to the last question in this section, question 38.) 33. If yes, do you know the name of the LAG and what other municipalities are also members and when it was founded? LAG Name Other LAG Member Municipalities Year Founded 1 2 3 4 5 6 7 8 9 34. What role, if any, do members of the private sector play in this LAG? Do they attend meetings; contribute to the development or implementation of a LAG plan? What specific roles do private sector representatives have on a LAG? Please describe how your municipalities’ private sector is involved. ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ 35. From your understanding, what is the common vision or set of goals that led the municipal members of this LAG to join together – what is it that your municipalities believe they can do together to promote the economic development of your combined area that would not be possible as individual municipalities? ___________________________________________________________________________ ___________________________________________________________________________ __________________________________________________ EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 271 36. Are there any achievements you can talk about that your LAG has made in terms of achieving its vision or goals? ________________________________________________ 37. How does being in a LAG help this municipality achieve its own Municipal Economic Development Plan goals? ___________________________________________________________________________ _______________________________________________ 38. If your municipality is not a member of a LAG, do you know why that is the case? ___________________________________________________________________________ ___________________________________________________________________________ _______________________________________________ Regional Development Center INT. Read: In addition to your municipalities own economic development plans, this municipality is part of a region, and all of the regions in Macedonia have Regional Development Centers. 39. In your view, how has this municipality benefitted from there being an RDC in your region? ___________________________________________________________________________ ___________________________________________________________________________ _______________________________________________ 40. How could you benefit more from the existence of this regional center? ___________________________________________________________________________ ___________________________________________________________________________ __________________________________________________ Assistance from Others in Building Your Municipalities Economic Development Capacity 41. What kinds of assistance has the private sector in your municipality received from outside sources that has helped build its capacity, skills and experience in participating in municipal economic development planning and implementing economic development plans over the past five years? And from where has assistance to your municipalities’ privates sector come? Type of Organization Name of Organization Types of Assistance Received: 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other Value of assistance from this organization: 1 = High, 2 = Moderate, 3 = Low 1) Government 1 = Training, 1 = High 2 = Moderate EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 272 Type of Organization Name of Organization Types of Assistance Received: 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other Value of assistance from this organization: 1 = High, 2 = Moderate, 3 = Low 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 2) Local Expert Organizations 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 1 = High 2 = Moderate EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 273 Type of Organization Name of Organization Types of Assistance Received: 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other Value of assistance from this organization: 1 = High, 2 = Moderate, 3 = Low 3) External Donors or their Projects 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low SBEP -- (These questions are to be asked only in municipalities in Polog, Pelagonia and Northeast and Vadar) 42. Are people in the business and farm community in your municipality aware of a project called the Small Enterprise Expansion Program (SBEP)? 1) Yes 2) No 43. Has your organization cooperated with the Small Enterprise Expansion Program (SBEP) 1) Yes 2) No – (INT. If No, skip to section on Municipal Income and Investment Perceptions and Expectations, question 48.) 44. In what year did you become aware of SBEP? (or whatever name they call this program.) EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 274 45. How did your organization become aware of SBEP? Type of Contact or Exposure to SBEP Yes 1) I/we Saw a story on SBEP on TV or in the newspaper 2) I/we learned about SBEP from another municipality or other type of organization assisted by SBEP 3) SBEP representatives sought out our municipality/organization; they called or came to visit us 4) I/we sought out SBEP for assistance with our municipality/organization 5) Other, what? ____________________________________________________ 46. On which of the following areas or initiatives has the private sector in your municipality worked directly with SBEP? Area or Initiative 1) Yes 2) Nо Types of Assistance Received by the municipality or its citizens: 1 = Training, 2 = Technical Assistance/Consultations, 3 = Equipment, 4 = Funds, 5 = Other Value of assistance from this organization on this topic or initiative: 1 = High, 2 = Moderate, 3 = Low 1 Municipal Economic Development Planning 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 = Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 2 Development of a LAG 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 = Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 3 Promoting “Women in Business” 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 = Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 4 Promoting Adventure Tourism 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 = Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 5 Promoting Wild Product Harvesting and Sales 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 = Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 275 Area or Initiative 1) Yes 2) Nо Types of Assistance Received by the municipality or its citizens: 1 = Training, 2 = Technical Assistance/Consultations, 3 = Equipment, 4 = Funds, 5 = Other Value of assistance from this organization on this topic or initiative: 1 = High, 2 = Moderate, 3 = Low 6 Introducing smaller light manufacturing businesses to larger firms working in your region that might use them as suppliers 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 = Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 7 The Grow More Corn Initiative that promotes using drip irrigation systems and protocols 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 = Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 8 Other initiative: Please describe ____________________________ 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 = Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 9 Other initiative: Please describe ____________________________ 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 = Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low Most Significant Change Question 47. Considering all of the ways the private sector in this municipality may have interacted with SBEP over the years, what has been the “most significant change” or effect on your private and farm sector from those interactions? ___________________________________________________________________________ ___________________________________________________________________________ _____________________________________________ Municipal Income and Investment Perceptions and Expectations 48. Looking to the future, how would you characterize the economic situation and prospects for the private sector in this municipality in the coming year? Better than last year 1) About the same 2) Worse than last year 3) EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 276 49. What are the prospects for the private sector in this municipality making new monetary investments in the economic development of this area in 2017? 1) More than the previous year 2) About the same as the previous year 3) Less than this year 4) DK/NA 50. Which factors, in your view, exert the greatest influence on whether the financial situation of the private sector this municipality will improve or get worse over the next few years? Say that you will read through a list and then ask which ones are the most important in determining the business’ financial condition and, in turn, your family/household welfare. Factors How important is this factor? 1 = Very Important; 2 = Moderate importance; 3 = Low importance; 4 = Can’t say, don’t know 1) General economic conditions in Macedonia 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 2) Economic conditions in the EU and other economies 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 3) Macedonian government leadership 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 4) Macedonian government policies 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 5) Access to finances to support LAG actions 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 6) Foreign investment in Macedonian businesses 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 7) Access to markets throughout Macedonia 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 8) Access to international markets 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 9) Imports 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 10) Exports 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 11) International product standards 1 = Very Important 2 = Moderate importance 3 = Low importance EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 277 Factors How important is this factor? 1 = Very Important; 2 = Moderate importance; 3 = Low importance; 4 = Can’t say, don’t know 4 = Can’t say, don’t know 12) Other, what? ___________________________ 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 51. What are the most challenging current needs that the private sector in your municipality is facing today? ________________________________________________________________________ ____________________________________________________________ 52. What do you see as being this municipalities best future prospects for economic growth? ________________________________________________________________________ ________________________________________________________________________ __________________________________________________________ 53. We thank you for all of your answers and wish to ask one more question. Is there anything important about the private sector in this municipality that we did not ask you about – that you think we should know? ________________________________________________________________________ ________________________________________________________________________ __________________________________________________________ EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 278 For Reference Only – Not part of the Questionnaire Active business entities by sectors of activity according to NKD Rev.2, by municipalities, by years, situation as at 31 December 2015 Total Republic of Macedonia 70 139 Skopje 24 877 Aerodrom 3 511 Butel 1 367 Gazi Baba 2 735 Gjorche Petrov 1 507 Karposh 3 810 Kisela Voda 2 707 Saraj 427 Centar 6 767 Chair 1 830 Shuto Orizari 216 Arachinovo 82 Berovo 425 Bitola 3 897 Bogdanci 273 Bogovinje 356 Bosilovo 258 Brvenica 293 Valandovo 376 Vasilevo 170 Vevchani 107 Veles 1 793 Vinica 539 Vraneshtica - Vrapchishte 387 Gevgelija 1 213 Gostivar 2 362 Gradsko 81 Debar 465 Debarca 106 Delchevo 531 Demir Kapija 105 Demir Hisar 236 Dojran 97 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 279 Active business entities by sectors of activity according to NKD Rev.2, by municipalities, by years, situation as at 31 December 2015 Total Dolneni 194 Drugovo - Zhelino 278 Zajas - Zelenikovo 62 Zrnovci 64 Ilinden 498 Jegunovce 178 Kavadarci 1 639 Karbinci 78 Kichevo 1 419 Konche 70 Kochani 1 354 Kratovo 234 Kriva Palanka 535 Krivogashtani 105 Krushevo 233 Kumanovo 3 050 Lipkovo 216 Lozovo 51 Mavrovo and Rostusha 155 Makedonska Kamenica 187 Makedonski Brod 161 Mogila 158 Negotino 733 Novaci 83 Novo Selo 235 Oslomej - Ohrid 2 663 Petrovec 181 Pehchevo 172 Plasnica 55 Prilep 2 706 Probishtip 411 Radovish 819 Rankovce 53 Resen 459 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 280 Active business entities by sectors of activity according to NKD Rev.2, by municipalities, by years, situation as at 31 December 2015 Total Rosoman 119 Sveti Nikole 853 Sopishte 110 Staro Nagorichane 51 Struga 2 073 Strumica 2 378 Studenichani 233 Tearce 311 Tetovo 3 234 Centar Zhupa 78 Chashka 96 Cheshinovo - Obleshevo 144 Chucher - Sandevo 154 Shtip 1 787 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 281 Semi Structured Interview – Mayor and LED Representatives Introduction We appreciate your willingness to be interviewed today. We are here today gathering information about how your municipality and other municipalities in this region view the progress they are making on local economic development and prospects for further development of farms and businesses in this area. This is our second visit with your municipality. We had the pleasure of meeting and talking with you a little more than 2 years ago, and this will give us the chance to learn about changes and progress you have made since that time. The research we are conducting will help others who are interested in regional development in Macedonia, including assistance programs for Macedonia funded by the U.S. Government’s economic assistance group called USAID, make decisions about how best to support regional development in this country. This is why we are asking you and other municipal officials to help us understand economic challenges and opportunities your local economic development plans and programs focus on and efforts that are underway to improve growth and jobs in this municipality. We will be writing about what we learn from municipalities across Macedonia, but we will not specify names of persons or municipalities in our report. Mr/Madam Mayor, before we start this interview we would like to encourage you to ask your staff to contribute their views and ideas. Sometimes, as you know, staff will only listen when the Mayor is speaking. Since this is a group interview, we hope you will help them feel free to contribute their thoughts as well as listen to yours. There are no right or wrong answers, we are interested in learning about your opinions, perceptions and attitudes. Is our purpose clear and acceptable and can we begin? Mark this space to indicate agreement _____ Interviewer: _________________________________ Interview date: _____________ 1. Name of Region: Name of Region Code Name of Region Code Eastern 1 Southeastern 5 Northeastern 2 Southwestern 6 Pelagonia 3 Vardar 7 Polog 4 Skopje 2. Name of the Municipality and Code23 (INT. Please enter name/number of municipality and code number from list provided by GfK Skopje): Enter name/number of municipality Municipality code number from GfK list 3. List of interview participants (INT. Should include the Mayor, the head or assistant head of the LED and one other person if they wish – someone with knowledge about municipal economic 23 Code can be entered before doing the interview or during transcription and codification of open-ended questions. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 282 development planning, finances, etc. Fill in all the required information in each column for each person. Three is the limit to the size of this interview group. Fill in the Mayor and Head of LED department/sector information only if they participate, otherwise write in the titles of individuals who do participate.) Title Name (first, last) Length of Time in Position Contact Gender (M/F) Title Name Enter number of years individual has been in this position Enter contact phone or email Enter gender of the respondent 1) Mayor 2) Head of LED department / sector 3) 4) Municipal Business and Farming Profile 4. Is this municipality considered to be primarily urban or primarily rural? Urban 1 Rural 2 5. Which of the following five sources of income are the #1, #2 and #3 most important sources of citizen income in this municipality? 1) Agriculture, forestry and fishing 2) Mining and quarrying 3) Manufacturing 4) Services, including wholesale/retail, construction, financial and transport services 5) Tourism, including accommodations Three top income sources for this municipality # 1 # 2 # 3 6. According to national data from the State Statistical Office, your municipality had about XX registered businesses last year, does that seem about right? [GfK use the table at the back of this interview to supply the correct figure for this municipality] 1) Yes 2) No 3) DK/NA 7. Nationally, government statistics suggest that roughly 25% of businesses are owned and operated by women. In your municipality would you estimate that the percentage of women owned/operated businesses is: 1) Higher than that 2) About that level 3) Somewhat lower than that level 4) Much lower than that level 5) DK/NA EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 283 8. Approximately how many independent agricultural holdings are registered in this municipality? (GfK should say that this figure is one we did not find among published sources, so we are asking them). 9. Are exports an important source of revenue for businesses and farms in this municipality? 1) Yes (INT. Go to Question 10.) 2) No (INT. Go to Question 11.) 3) DK/NA (INT. Go to Question 11.) 10. Exports by businesses and farms in the municipality (INT. Write down the answer) DK/NA 1) Approximately what number of businesses and farms in the municipality export their products? 99 2) What are the main types of products exported from this municipality? 99 3) To what regions/countries are these exports generally sold? 99 11. Are imported inputs important for the success of businesses and farms in this municipality? 1) Yes (INT. Go to Question 12.) 2) No (INT. Go to Question 13.) 3) DK/NA (INT. Go to Question 13.) 12. Imports by businesses and farms in the municipality (INT. Write down the answer) DK/NA 1) Approximately what number of businesses in the municipality import inputs to their businesses? 99 2) What types of inputs for businesses are commonly imported by businesses and farms in this municipality? 99 3) From what regions/countries are these imports generally purchased? 99 Economic Status of Your Municipality 13. How do you estimate your municipality ranks compared to other municipalities in your region on overall GDP (the market value of all goods the municipality produces) and GDP per capita (total GDP for the municipality divided by the number of citizens)? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 284 13A. GDP overall (market value of all goods produced) 13B. GDP per Capita Much above the region’s average 1 Much above the region’s average 1 Somewhat above the region’s average 2 Somewhat above the region’s average 2 Same as the region’s average 3 Same as the region’s average 3 Somewhat below the region’s average 4 Somewhat below the region’s average 4 Much below the region’s average 5 Much below the region’s average 5 Don’t know/No answer 99 Don’t know/No answer 99 14. How would you describe overall unemployment in this municipality compared to 2014? 1) Getting worse 2) About the same 3) Declining 4) DK/NA 15. How would you describe overall youth (18-29) unemployment in this municipality compared to 2014? 1) Getting worse 2) About the same 3) Declining 4) DK/NA 16. How would you describe outmigration from this municipality compared to 2014? 1) Getting worse 2) About the same 3) Declining 4) DK/NA 17. How would you compare your municipality’s overall economic situation in today to your municipality situation roughly two years ago (2014), when we last visited? 1) Greatly improved 2) Somewhat improved 3) Same as in 2014 (INT. Go to next section on Municipal Development Plan, Question 19.) 4) Somewhat worse 5) Much worse 6) DK/NA (INT. Go to next section on Municipal Development Plan, Question 19.) 18. (If the respondent’s said that the municipality’s economic situation had changed since 2014, please ask) What best explains why your municipality improved/did less well since 2014? Open ended response ______________________________________________________________________________________ ______________________________________________________________________________________ ______________________________________________________________________________________ ________ EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 285 Municipal Development Plan 19. Does your municipality have Municipal Economic Development Plan? 1) Yes (INT. Go to Question 20.) 2) No (INT. if No, skip to Section LEADER/LAG Process, Question 30.) 20. INT. If yes at previous question, ask: 20A. In what year was it prepared? 20B. What period of time does it cover? INT. Write down the year ______________ INT. Write down the time period: _________________ 21. What is the formal name of this plan? INT. Write down the name of the Plan: ___________________________________________________________________________________ ___________________________________________________________________________________ 22. Who or what entities were the primary authors of this plan? (e.g., the LED office, some special community committee set up for this purpose?) ___________________________________________________________________________________ ___________________________________________________________________________________ ___________________________________________________________________________________ __________________________________________________________________________________ 23. What role did the business community, including farmers, play in preparing this Municipal Economic Development Plan? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ _______________________________________ 24. What role did civil society organizations and ordinary citizens play in preparing this Municipal Economic Development Plan? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ______________________________________ 25. Are there any segments of the population or groups of citizens that you wish had played a more active role in preparing your Municipal Economic Development Plan than occurred – people or groups who you would like to see participate more actively next time? ___________________________________________________________________________ ___________________________________________________________________________ _____________________________________ EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 286 26. What are the main sources of fund through which this municipality finances its Economic Development Plan? Type of Funding Sources Specific sources from which this municipality received funding for its Economic Development Plan 1) National government sources 2) Regional sources 3) Municipalities own resources, including fees, municipal bonds, etc. 4) Private sector investment - domestic 5) Foreign direct investment or remittances 6) Other sources 27. What were the three main goals or strategies of the Municipal Economic Development Plan for the period it covered? The biggest things your municipality hoped to achieve with in the years the plan covered? Main goals for strategies of you Municipal Economic Development Plan Goal 1) Goal 2) Goal 3) 28. What can you say about the progress you have made – or the obstacles you have faced – on each of these major goals since the plan was completed and implementation began? Progress and difficulties since implementation of the plan started – are there any achievements you can describe? Goal 1) _______________________________________________________________ _______________________________________________________________ _______________________________________________________________ __________________________________ Goal 2) _______________________________________________________________ _______________________________________________________________ _______________________________________________________________ __________________________________ Goal 3) _______________________________________________________________ _______________________________________________________________ _______________________________________________________________ ___________________________________ 29. If there are specific aspects of your Municipal Economic Development Plan, that you have not yet described, that focus on unemployment, particularly youth unemployment, or outmigration from this municipality can you tell us what those are and what progress you have made on them, if any? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ______________________________________________________________ EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 287 LEADER/LAG Process 30. Is this municipality a member of a LAG? 1) Yes 2) No – (INT. if no skip to last question in this LEADER/LAG Process section, Question 37.) 31. If Yes, ask: What is the name of the LAG and what other municipalities are also members? LAG Name Other LAG Members Year Founded 1 2 3 4 5 6 7 8 9 32. What is the registration status of this LAG? Status Answer 1) Fully registered (status LAG; NGO or other) 2) Application submitted; registration in progress 3) Currently preparing application 4) Other, please explain 33. What is the common vision or set of goals that led the members of this LAG to join together – what is it that you believe you can do together to promote the economic development of your combined area that would not be possible as individual municipalities? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ____________________________________________________ 34. What is the status of your LAG’s plan for achieving this vision or set of goals? 1) LAG already has a formal written vision/plan (INT. if yes, enter single or multiple year, and specify which years if multiple) _____________________________ 2) LAG is currently preparing a written vision/plan 3) LAG is just starting to talk about developing a vision/plan EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 288 35. Are there any achievements you can talk about that your LAG has made in terms of achieving its vision or goals? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ 36. How does being in a LAG help this municipality achieve its own Municipal Economic Development Plan goals? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ _______________________________________________________________ 37. If your municipality is not a member of a LAG, why is that the case? ___________________________________________________________________________ ___________________________________________________________________________ ________________________________________________________ Regional Development Center INT. read: In addition to your municipalities own economic development plans, this municipality is part of a region, and all of the regions in Macedonia have Regional Development Centers. 38. How the RDC has been involved in municipal economic development planning for your municipality – what role does it play in your local economic development planning process, if any? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ________________________________________________________ 39. How the municipality has been involved in regional economic development planning – have you been involved in creating the RDC’s regional development plan? Are there needs of your municipality reflected in the RDC’s Regional plan and activities? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ _______________________________________________________ 40. How has this municipality benefitted from there being an RDC in your region? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ________________________________________________________ EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 289 41. How could you benefit more from the existence of this regional center? ___________________________________________________________________________ ___________________________________________________________________________ ________________________________________________________ Assistance from Others in Building Your Municipalities Economic Development Capacity 42. What kinds of assistance has your municipality received from outside sources that has helped build its capacity, skills and experience with economic development planning and implementing economic development plans over the past five years? And from where has that assistance come? Type of Organization Name of Organization Types of Assistance Received: 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other Value of assistance from this organization: 1 = High, 2 = Moderate, 3 = Low 1) Government 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 2) Local Expert Organizations 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 1 = High 2 = Moderate 3 = Low EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 290 Type of Organization Name of Organization Types of Assistance Received: 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other Value of assistance from this organization: 1 = High, 2 = Moderate, 3 = Low 4 = Funds, 5 = Other 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 3) External Donors or their Projects 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 291 SBEP -- (These questions are to be asked only in municipalities in Polog, Pelagonia and Northeast and Vardar) 43. Are people (citizens) in your municipality aware of a project called the Small Enterprise Expansion Program (SBEP)? 1) Yes 2) No 44. Has your municipality cooperated with project called the Small Enterprise Expansion Program (SBEP)? 1) Yes 2) No - (INT. if No, skip to section on Municipal Income and Investment Perceptions and Expectations, Question 49.) 45. In what year did you become aware of SBEP? (or whatever name they call this program.) 46. How did your municipality become aware of SBEP? Type of Contact or Exposure to SBEP Yes 1) I/we saw a story on SBEP on TV or in the newspaper 2) I/we learned about SBEP from another municipality or other type of organization assisted by SBEP 3) SBEP representatives sought out our municipality; they called or came to visit us 4) I/we sought out SBEP for assistance with our municipality 5) Other (specify) _______________________________________________________ 47. On which of the following areas or initiatives has your municipality worked directly with SBEP? Area or Initiative 1) Yes 2) Nо Types of Assistance Received by the municipality or its citizens: 1 = Training, 2 = Technical Assistance/Consultat ions, 3 Equipment, 4 = Funds, 5 = Other Value of assistance from this organization on this topic or initiative: 1 = High, 2 = Moderate, 3 = Low 1 Municipal Economic Development Planning 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 2 Development of a LAG 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 3 Promoting “Women in Business” 1) Yes 1 = Training, 1 = High EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 292 Area or Initiative 1) Yes 2) Nо Types of Assistance Received by the municipality or its citizens: 1 = Training, 2 = Technical Assistance/Consultat ions, 3 Equipment, 4 = Funds, 5 = Other Value of assistance from this organization on this topic or initiative: 1 = High, 2 = Moderate, 3 = Low 2) Nо 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 2 = Moderate 3 = Low 4 Promoting Adventure Tourism 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 5 Promoting Wild Product Harvesting and Sales 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 6 Introducing smaller light manufacturing businesses to larger firms working in your region that might use them as suppliers 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 7 The Grow More Corn Initiative that promotes using drip irrigation systems and protocols 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 8 Other initiative: Please describe _______________________ 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low 9 Other initiative: Please describe _______________________ 1) Yes 2) Nо 1 = Training, 2 = Technical Assistance/Consultations, 3 Equipment, 4 = Funds, 5 = Other 1 = High 2 = Moderate 3 = Low EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 293 Most Significant Change Question 48. Considering all of the ways this municipality may have interacted with SBEP over the years, what has been the “most significant change” or effect on you and this municipality from those interactions? ___________________________________________________________________________ _______________________________________________________ Municipal Income and Investment Perceptions and Expectations 49. Looking to the future, how would you characterize the economic situation and prospects for of this municipality in the coming year? Better than last year 1) About the same 2) Worse than last year 3) 50. What are the prospects for the municipality making new monetary investments in the economic development in 2017? 1) More than the previous year 2) About the same as the previous year 3) Less than this year 4) DK/NA 51. Which factors, in your view, exert the greatest influence on whether the financial situation of this municipality will improve or get worse over the next few years? INT. Say that you will read through a list and then ask which ones are the most important in determining the municipality’s financial condition and, in turn, citizens' welfare. Factors How important is this factor? 1 = Very Important; 2 = Moderate importance; 3 = Low importance; 4 = Can’t say, don’t know 1) General economic conditions in Macedonia 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 2) Economic conditions in the EU and other economies 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 3) Macedonian government leadership 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 4) Macedonian government policies 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 5) Access to finances to support LAG actions 1 = Very Important 2 = Moderate importance 3 = Low importance EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 294 4 = Can’t say, don’t know 6) Foreign investment in Macedonian businesses 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 7) Access to markets throughout Macedonia 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 8) Access to international markets 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 9) Imports 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 10) Exports 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 11) International product standards 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 12) Other, what _________________________________________________ 1 = Very Important 2 = Moderate importance 3 = Low importance 4 = Can’t say, don’t know 52. What are the most challenging current needs that your municipality is facing today? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________ 53. What do you see as being this municipalities best future prospects for economic growth? ________________________________________________________________________ ________________________________________________________________________ ______________________________________________________________________ 54. We thank you for all of your answers and wish to ask one more question. Is there anything important about this municipality that we did not ask you about – that you think we should know? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ______________________________________________________________ EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 295 For Reference Only – Not part of the Questionnaire Active business entities by sectors of activity according to NKD Rev.2, by municipalities, by years, situation as at 31 December 2015 Total Republic of Macedonia 70 139 Skopje 24 877 Aerodrom 3 511 Butel 1 367 Gazi Baba 2 735 Gjorche Petrov 1 507 Karposh 3 810 Kisela Voda 2 707 Saraj 427 Centar 6 767 Chair 1 830 Shuto Orizari 216 Arachinovo 82 Berovo 425 Bitola 3 897 Bogdanci 273 Bogovinje 356 Bosilovo 258 Brvenica 293 Valandovo 376 Vasilevo 170 Vevchani 107 Veles 1 793 Vinica 539 Vraneshtica - Vrapchishte 387 Gevgelija 1 213 Gostivar 2 362 Gradsko 81 Debar 465 Debarca 106 Delchevo 531 Demir Kapija 105 Demir Hisar 236 Dojran 97 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 296 Active business entities by sectors of activity according to NKD Rev.2, by municipalities, by years, situation as at 31 December 2015 Total Dolneni 194 Drugovo - Zhelino 278 Zajas - Zelenikovo 62 Zrnovci 64 Ilinden 498 Jegunovce 178 Kavadarci 1 639 Karbinci 78 Kichevo 1 419 Konche 70 Kochani 1 354 Kratovo 234 Kriva Palanka 535 Krivogashtani 105 Krushevo 233 Kumanovo 3 050 Lipkovo 216 Lozovo 51 Mavrovo and Rostusha 155 Makedonska Kamenica 187 Makedonski Brod 161 Mogila 158 Negotino 733 Novaci 83 Novo Selo 235 Oslomej - Ohrid 2 663 Petrovec 181 Pehchevo 172 Plasnica 55 Prilep 2 706 Probishtip 411 Radovish 819 Rankovce 53 Resen 459 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 297 Active business entities by sectors of activity according to NKD Rev.2, by municipalities, by years, situation as at 31 December 2015 Total Rosoman 119 Sveti Nikole 853 Sopishte 110 Staro Nagorichane 51 Struga 2 073 Strumica 2 378 Studenichani 233 Tearce 311 Tetovo 3 234 Centar Zhupa 78 Chashka 96 Cheshinovo - Obleshevo 144 Chucher - Sandevo 154 Shtip 1 787 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 298 Semi-Structured Interview Instrument – LAG Representatives Round 2 (2016) LAG President and Executive Director (minimum = Director and President. maximum: 3 participants, total) Introduction My name is __________. I am from Management Systems International, an international development firm in the United States. We are here today conducting a survey of businesses in this area. Our questions are about how sub-regional LAGS that have formed around Macedonia, under the LEADER/LAG program we have learned from the EU, view their current situation and prospects for the future. The research we are conducting will help organizations that are supporting regional development in Macedonia decide what kinds of assistance might be provided in the future. This includes both Government of Macedonia agencies and assistance programs funded externally, such as the assistance program for Macedonia funded by the United States, through its assistance agency, USAID. We are asking you and other LAG leaders to help us understand economic conditions and opportunities, and efforts to improve them in this region. We will speak with many people like you as we gather answers. We will be writing what we learn, but we will not give out the names of specific individuals we speak with during this survey in our report. Before we begin, do you have any questions about what I just explained to you about our purpose in being here? We expect that our conversation with you today will take about one hour. Interviewer Name Interview Date Basic Information 3. Name of Region? Name of Region Code Name of Region Code Eastern 1 Southeastern 5 Northeastern 2 Southwestern 6 Pelagonia 3 Vardar 7 Polog 4 Skopje Name of LAG? 4. Which Municipalities are Members of this LAG? Enter name of each municipality in this LAG EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 299 5. LAG Members represented in this Interview Title Name Length of time in Position Contact (Phone or email) Municipality Represented President Years in position: Director Years Appointed: Write in Write-in Write in Write-in Current LAG Status 6. Planning Region in which this LAG is registered? 7. Current status of this LAG? Status Answer Fully registered ( status LAG ; NGO or other) Application submitted; registration in progress Currently preparing application Other, explain LAG Formation 8. In what year was this LAG first formed? 9. Can you please tell the story of how your LAG was formed and the assistance you received? Did you do it alone? Did you receive help from others? 10. What types of People represent their municipalities – only public sector representatives, or also private sector individuals, e.g., businesses, educators, etc? Please describe for us the composition of the members of this LAG including a sense of how equally divided members might be between the public and private sectors. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 300 11. If the LAG’s story about its formation included receiving assistance from others, ask which organizations helped them form their LAG and get it started, and what kinds of assistance did those organizations provide? Names of Organizations/ Individuals That Assisted with Forming this LAG Types of Assistance they Provided Over what Period of Time (approximately) 1 2 3 LAG Operations 1. How frequently does this LAG meet? 2. How large are LAG meetings, how many and what types of people come to LAG meetings? Is just anyone allowed to attend – both public and private sector -- or do you have to be or represent a member municipality? Frequency Average Number Attending Special Credentials or Criteria to Participate in LAG meetings, if any 3. What is this LAG’s “vision” of its objectives or goals – what makes it a LAG under the LEADER/LAG Concept? 4. Does the LAG have a specific plan or agenda for achieving this “vision” a. LAG already has a formal written vision/plan ( if yes, enter single or multiple year, and specify which years if multiple) b. LAG is currently preparing a written vision/plan c. LAG is just starting to talk about developing a vision/plan 5. What process did the members of this LAG use to reach a consensus on the various projects, results and outcomes they wanted to pursue together – meaning how you are identify what are the people’s needs, priorities, and solutions? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 301 6. How long did it take the LAG members to agree on its first agenda of joint activities, projects and results? 7. Regardless of what stage of development this LAG’s Vision/Plan is, what are the main aims of this LAG going forward? What are the key results or processes or projects various municipalities in this LAG have come together to work on? Please describe for some of the specific projects and outcomes this LAG hopes to accomplish for this sub-region. LAG Results in Economic and Cross-Cutting Sectors If this LAG’s vision and action focuses on particular sectors of the sub-regional economy, which of the following receive the highest priority as focus sectors for this LAG? 1. Agriculture, forestry and fishing 2. Mining and quarrying 3. Manufacturing 4. Services, including wholesale/retail, construction, financial and transport services 5. Tourism, including accommodations Three top priority sectors for this LAG. # 1 Enter a number between 1-7 above # 2 Enter a number between 1-7 above # 3 Enter a number between 1-7 above In these economic areas, what have been this LAG’s most important 1-2 results or outcomes to date? Cross Cutting Issues National statistical tables indicate that across the country, roughly 25% of businesses in Macedonia are women-owned. Does any aspect of this LAG’s plans seek to increase women’s participation in the economy as business owners? If yes, could you describe that feature or programs and how it is working. Nationwide, youth unemployment has been a problem for a number of years. Is this something that is improving or getting worse in this particular sub-region? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 302 Does any aspect of this LAG’s plans seek to increase women’s participation in the economy as business owners? If yes, could you describe that feature or programs and how it is working? Interactions at other Administrative Levels How does this LAG interact with the Mayors and economic development offices in the municipalities that make up the LAG – if those municipalities have their own single or multi-year planning exercises and action plans? How does this LAG and its member municipalities coordinate? At the regional level, every Macedonian region has a Regional Development Center (RDC). Does this LAG have a relationship with this region’s RDC? Could you please describe this LAG’s relationship with the regional RDC, including how the LAG and the RDC differ, or complement each other and work together, including how frequently the organizations meet and for what purposes. Funding for this LAG’s Vision/Plan Given where this LAG is in its certification process, what kinds of funding has this LAG received so far and from what sources, e.g., donations from member municipalities, regional or national government assistance, donations from external organizations, or funds that came in because of a LAG application to some entity Name of Source of Assistance Application or Competition Process Involved (Y/N) Approximate Amount ($) Approximate Time Frame (years) Macedonian Government Sources EU Related Funds linked to LEADER/LAG program Other Donors EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 303 Other Assistance Received Earlier in this interview we talked about assistance this LAG may have received when it was just forming. Now I’d like to ask a question or two about the U.S. Small Business Expansion Project (SBEP) which has been working with LAGs, among other things since 2012. Are you aware of the U.S. government project called the “Small Business Expansion Project” (SBEP) that is working with Macedonian businesses on improving economic development here in several ways? 1) Yes (by some name) 2) No – if no, proceed to the Income and Investment Perceptions and Expectations section. When, in what year, did you become aware of SBEP? Enter year How did you become aware of SBEP? (or whatever name they call this program.) Type of Contact or Exposure to SBEP Yes I/we Saw a story on SBEP on TV or in the newspaper I/we Learned about SBEP from another LAG or other type of organization assisted by SBEP SBEP representatives sought out our LAG; they called or came to visit us I/we sought out SBEP for assistance with our LAG Other What types of interactions have you had with SBEP? Type of Interactions with SBEP Yes (X) Value-Importance of this interaction with SBEP to the development and status of this business: 1 = High: 2 = Moderate: 3 = Low Attended a SBEP conference or workshop Participated in a SBEP training program Received advice from SBEP specific to this particular LAG Participated in Public-Private Sector Dialogues on LAG opportunities/challenges that SBEP helped organize/facilitate Received inputs or money from SBEP Our LAG received publicity as a function of its interactions with SBEP Other types of interactions Add more rows if needed EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 304 Most Significant Change Question Considering all of the ways this LAG may have interacted with SBEP over the years since you became aware of this U.S. funded project, what has been the “most significant change” or effect on you and this LAG from those interactions? Income and Investment Perceptions and Expectations Overall, how would you characterize the economic situation of the sub-region this LAG covers in 2016 compared to before your LAG was formed? Better than last year About the same Worse than last year DK/NA What are your expectations about the economic status of this sub-region under the LAG over the next 12 months? Will improve Will stay the same Will be worse DK/NA Has the LAG made new monetary investments in the vision and plans of this LAG during 2016? 1) More than the previous year 2) About the same as the previous year How likely is it that new investments will be made by this LAG next year? 1) Very likely 2) Somewhat likely 3) Not sure 4) Somewhat unlikely 5) Very unlikely 6) DK/NA Which factors, in your view, exert the greatest influence on whether the financial situation of this LAG sub-region will improve or get worse over the next few years? Say that you will read through a list and then ask which ones are the most important in determining the business’ financial condition and, in turn, your family/household welfare. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 305 Factors How important is this factor? 1 = Very Important; 2 = Moderate importance; 3 = Low importance; 4 = Can’t say, don’t know General economic conditions in Macedonia Economic conditions in the EU and other economies Macedonian government leadership Macedonian government policies Access to finances to support LAG actions Foreign investment in Macedonian businesses Access to markets throughout Macedonia Access to international markets Imports Exports International product standards Other DK/NA 1. What are the most emerging needs that you have currently with your LAG operating? 2. Where do you see future opportunities available for this LAG? Please explain. 3. What might be your “top three recommendations” to others for future programming in the field of economic and regional development for this particular sub-region of Macedonia? Top Recommendations for Future Economic and Regional Development in Macedonia 1. 2. 3 We thank you for all of your answers and wish to ask one more question. Is there anything important about this LAG that we did not ask you about – that you think we should know? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 306 Semi-Structured Interview Instrument – RDC Representatives Round 2 (2016) Director and RDC Officer(s) (minimum = Director and one from the RDC, preferably Coordinator of the Business Center that is working under RDC umbrella. maximum: 3 participants, total) My name is __________. I am from Management Systems International, an international development firm in the United States. We are here today conducting a survey of Regional Development Centers. Our questions are about how regional RDCs that have formed around Macedonia, under the LEADER/LAG program we have learned from the EU, view their current situation and prospects for the future. The research we are conducting will help organizations that are supporting regional development in Macedonia decide what kinds of assistance might be provided in the future. This includes both Government of Macedonia agencies and assistance programs funded externally, such as the assistance program for Macedonia funded by the United States, through its assistance agency, USAID. We are asking you and other RDC leaders to help us understand economic conditions and opportunities, and efforts to improve them in this region. We will speak with many people like you as we gather answers. We will be writing what we learn, but we will not give out the names of specific individuals we speak with during this survey in our report. Before we begin, do you have any questions about what I just explained to you about our purpose in being here? We expect that our conversation with you today will take about one hour. Interviewer Name Interview Date Basic Information 12. Name of Region? Name of Region Code Name of Region Code Eastern 1 Southeastern 5 Northeastern 2 Southwestern 6 Pelagonia 3 Vardar 7 Polog 4 Skopje Interview Participants Title Name Length of time in Position Contact (Phone or email) Director Years in position: President Years Appointed: Write in . Years in position: EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 307 RDC Origins In what year was this Regional Development Center (RDC) first formed? In what year did it become fully operational in this location? In 2014, when we last visited this RDC, people in your positions across the country generally told us that they were in the process of developing, but in general, had not yet completed or issued their Regional Development Plans. Has there been progress towards the completion and issuance of a Development Plan for this Region? If it has been issued: What years does it cover? When was it issued? Year X to Year Y Month/year If it has been issued, ask where online or in the region you could obtain a copy (and bring it back with you please.) When we talked to the RDCs last, most told us that the process for identifying local impediments to economic development, and coming up with and discussing options that might improve the region’s prospects, was fairly informal, with most private sector stakeholders participating on an ad hoc basis. Did these processes continue on in an informal manner, or have you begun to formalize this region’s planning processes since then? Please give us a brief update on how you see the regional development planning processes having evolved and being likely to evolve going forward. As this region’s development planning process has evolved over the past two years, we were wondering if a greater local consensus exists now than back then about what are the greatest impediments to development in this region, and what are the greatest undeveloped opportunities. 13. What do you see today as being the top four (4) impediments to the region’s growth Number Top Impediments to Economic Improvement in this Region 1 2 3 4 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 308 14. Since we last visited in 2014, has the region made any progress in reducing or eliminating any of these impediments? Please tell us about the progress you have made or any of these impediments where you think there has been important progress. Progress on Priority Impediments in this Region Story Narrative (Put the story narrative on the correct line to match the number of the impediment in the previous question. Write up only the number of stories they provide easily – 1,2,3 or 4, don’t push)) Impediment 1 Impediment 2 Impediment 2 Impediment 4 Interviewers – try to get the essence of each story, i.e., “we saw this problem, we did this, we got help from these others – including how we got funding for the work or project, then we implemented our plan, and in the end, this is how it turned out (the results they achieved).” Study the elements in the previous story guide and direct these stories along a straight line to be sure that they can be written up in a useful way. Beginning, middle, end. Write up the stories in “their” voice, how they told it. Try to get all elements of each story you hear, and any put direct quotes in quote marks. 15. What do you see today as being the top four unrealized opportunities for improving economic growth in the region? Number Top Impediments to Economic Improvement in this Region 1 2 3 4 16. Since we last visited in 2014, has the region made any progress in acting on high priority unrealized opportunities for improving this region’s development performance? Please tell us about the progress you have made or any of the unrealized opportunities where you think there has been important progress. Progress on Priority Impediments in this Region Story Narrative (As above) Impediment 1 Impediment 2 Impediment 2 Impediment 4 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 309 See instructions on story writing above. Cross Cutting Issues National statistical tables indicate that across the country, roughly 25% of businesses in Macedonia are women-owned. Does any aspect of this region’s Development Plan seek to increase women’s participation in the economy as business owners? If yes, could you describe that feature or programs and how it is working. Nationwide, youth unemployment has been a problem for a number of years. Is this something that is improving or getting worse in this particular region? Does any aspect of this region’s Development Plan seek to increase women’s participation in the economy as business owners? If yes, could you describe that feature or programs and how it is working. Funding the RDC Development Plan and Specific Activities Over the past two years, how close has this RDC come to reaching its financial targets for investments in the region’s Development Plan? • 100% of the way to achieving those targets • 75% of the way • 50% of the way • Less than 50% What sources of funding are proving to be the most important in this region for financing the region’s Development Plan, including government, private sector partnerships, donors, diaspora (if relevant) and other sources? Number Names of Top Sources of Funds for Implementing the Region’s Development Plan 1 2 3 On our last visit in 2014, we talked with RDCs about their relationship with planning processes at the municipality level. Most RDCs told us that while they did a lot of the “big picture” work on identifying impediments and opportunities, region-wide, the details of their regional plans were very dependents on inputs from Mayors and others at the EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 310 municipal levels. We would appreciate from you a brief update on that earlier description. How is the relationship between the RDCs and municipalities working out today? Do the regional plans this RDC has developed have an influence at the municipal level as well as the other way around? Can you give us some examples of that? Since our last visit, the LEADER/LAG program has become more active in some “sub￾regions” in Macedonia. Is that the case here? How many LAGs, or pre-LAGs are there in this region at the present time? How does your RDC relate to and work with the LAGs that are emerging in this region? Can you briefly tell us how that story has evolved here and how it is working out? Assistance to the RDC from Donor Organizations Has this RDC received direct assistance from any donor organizations that work, e.g., the World Bank, the U.S. Government, the Swiss Government, the German Government, the EU or others. From which of these or other donors has this RDC received assistance, and what kinds of assistance have been received. (Organization’s Name) Types of Assistance Provided ( Mark each correct column) Technical Assistance Training Direct Funding Equipment 1 2 3 Are you aware of the U.S. government project called the “Small Business Expansion Project” (SBEP) that is working with Macedonian businesses on improving economic development here in several ways? 1) Yes (by some name) 2) No – if no proceed to the Income and Investment Perceptions and Expectations section. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 311 When, in what year, did you become aware of SBEP? Enter year How did you become aware of SBEP? (or whatever name they call this program.) Type of Contact or Exposure to SBEP Yes I/we Saw a story on SBEP on TV or in the newspaper I/we Learned about SBEP from another LAG/ RDC or other type of organization assisted by SBEP SBEP representatives sought out our RDC; they called or came to visit us I/we sought out SBEP for assistance with our RDC Other What types of interactions have you had with SBEP? Type of Interactions with SBEP Yes (X) Value-Importance of this interaction with SBEP to the development and status of this business: 1 = High: 2 = Moderate; 3 = Low Attended a SBEP conference or workshop Participated in a SBEP training program Received advice from SBEP specific to this particular RDC Participated in Public-Private Sector Dialogues on RDC opportunities/challenges that SBEP helped organize/facilitate Received inputs or money from SBEP Our RDC received publicity as a function of its interactions with SBEP Other types of interactions Add more rows if needed Most Significant Change Question Considering all of the ways this RDC may have interacted with SBEP over the years since you became aware of this U.S. funded project, what has been the “most significant change” or effect on you and this RDC from those interactions? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 312 Income and Investment Perceptions and Expectations Overall, how would this RDC characterize the economic situation in this region in 2016 compared to before your RDC was formed? Better than last year About the same Worse than last year DK/NA What are the RDC’s expectations about the economic status of this region over the next 12 months? Will improve Will stay the same Will be worse DK/NA Has the RDC made new monetary investments in support of its Development Plan during 2016? 3) More than the previous year 4) About the same as the previous year How likely is it that new investments will be made by the RDC next year? 1) Very likely 2) Somewhat likely 3) Not sure 4) Somewhat unlikely 5) Very unlikely 6) DK/NA What factors, in your view, exert the greatest influence on whether the financial situation of this region will improve or get worse over the next few years? Say that you will read through a list and then ask which ones are the most important in determining the business’s financial condition and, in turn, your family/household welfare. Factors How important is this factor? 1 = Very Important; 2 = Moderate importance; 3 = Low importance; 4 = Can’t say, don’t know General economic conditions in Macedonia Economic conditions in the EU and other economies Macedonian government leadership Macedonian government policies Access to finances to support our actions Foreign investment in Macedonian businesses Access to markets throughout Macedonia Access to international markets EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 313 Imports Exports International product standards Other DK/NA What might be your “top three recommendation” to others for future programming in the field of economic and regional development for this particular region of Macedonia? Top Recommendations for Future Economic and Regional Development in Macedonia 1. 2. 3. We thank you for all of your answers and wish to ask one more question. Is there anything important about the RDC, the Regional Development Plan and its implementation in this region that we did not ask you about – that you think we should know? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 314 Semi-Structured Interview Instrument – Association for Local and Rural Development (ALRD) My name is __________. I am from Management Systems International, an international development firm in the United States. We are here today conducting interviews with organizations working in this region and playing an active role in its economic development. The research we are conducting will help organizations that are supporting regional development in Macedonia decide what kinds of assistance might be provided in the future. This includes both Government of Macedonia agencies and assistance programs funded externally, such as the assistance program for Macedonia funded by the United States, through its assistance agency, USAID. We are asking you to help us understand economic conditions and opportunities, and efforts to improve them in this region. We will speak with many people like you as we gather answers. We will be writing what we learn, but we will not give out the names of specific individuals we speak with during this survey in our report. Before we begin, do you have any questions about what I just explained to you about our purpose in being here? We expect that our conversation with you today will take about one hour. Interviewer Name Interview Date Basic Information 1. Interview Location Name of Region Name of Municipality 2. Interview Participants ALRD Title Name In this interview, our questions focus on initiatives that a U.S. Government assistance program to Macedonia called the Small Business Expansion Program (SBEP)? Our particular interest is their “Women in Business” Initiative. We have learned from them that organization was SBEP’s partner in organizing and implementing this initiative. When did ALRD become SBEP’s partner for the “Women in Business” Initiative? How long this cooperation lasted? What were your responsibilities in this partner role? Can you please describe the various things you did to implement the initiative, and also tell us what SBEP itself did to implement the “Women in Business” Initiative? How did your roles differ? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 315 How constant was the “Women in Business” Initiative over the life of the SBEP project – or if it changed a lot over the years, how did it change? Did you and SBEP have metrics you used for monitoring how well the “Women in Business” Initiative was working – success measures, or customer satisfaction measures? If you did, can you please explain them and how you gathered the information on the status of these metrics over they years? Over the years on which you worked on the “Women in Business” Initiative, how well did it do on the measures you used – or what you saw from other evidence? Can you described any metrics that improved over the years, or stayed about the same as the first year, or seemed to be declining over time? From what you learned from the businesses the SBEP “Women in Business” initiative, what were the top three things about the program that business owners valued most? And why did they think each one was important? 1 2 3 In your judgement, will the “Women in Business” Initiative continue to exist even after the SBEP project closes down? If yes, how will that happen? Most Significant Change Question How has participating in this “Women in Business” Initiative with SBEP changed your own organization? What has been the “most significant change” in ALRD as a result of your involvement? Is there anything that cooperation with SBEP project done for you in terms of capacity building of your organization, publicity, promotion with implementing “Women in Business” Initiative? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 316 Looking back, how might the SBEP project done a better job on the “Women in Business” Initiative? What recommendations would you have for those who might try to sustain this Initiative after SBEP closes? Is there anything else that you want to tell us, and we did not ask you? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 317 Additional Semi-Structured Interview Instruments Ministry of Local Government/Bureau for Regional Development/Cabinet of Vice-Prime Minister for Economic Affairs (Regional Development) 1. Please explain us what is your jurisdiction in regional economic development and RDC? 2. How involved are you/your institution with the municipalities and the RDC in regional strategies/plans? 3. What roles are established for RDCs and the Mayor’s Councils – in the Ministry’s view? 4. Do you know the degree to which the private sector is involved in regional development planning and implementation? 5. Which regions are the furthest along in planning and allocating funds for regional development? Which factors contributed to that? 6. Which regions are least far along? Which factors contributed to that? 7. How are RDCs and regions being monitored or evaluated on their regional development efforts? 8. What roles are donors playing in fostering regional development? Do you know which ones are working with the RDCs and how? 9. Are the RDCs linked to the LEADER/LAG initiatives and how? 10. Are you familiar with USAID/SBEP project, and did you have any cooperation with the effort? 11. For you personally or for this institution, what was the “most significant change” that emerged from all of your interactions with SBEP? This could be anything that happened that in your mind has long-term importance. An open-ended response in the form of a story is expected – whatever the respondent answers, draw them into a discussion where they explain the change they think is significant and how it came about. 12. What will be your recommendation for future programming in field of economic and regional development? 13. Other broad questions that provide you with an overview for seeing individual RDCs and understanding the RDC role? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 318 Chambers of Commerce 1. Please tell us about your organizational structure, regional presence, number of companies that are your members, % of the total number of companies in MK, and most dominant sectors that your Chamber is presenting? 2. How involved is the Chamber in the regional Rural Development process? 3. How involved is it in the LED planning process at the municipal level? 4. Does the Chamber have its own regional development initiatives underway? 5. Is the Chamber involved in LEADER/LAG, and how? 6. Is it involved with other donor regional development activities and how? 7. Is the term public-private sector dialogue familiar to you – and if YES how advanced is used of this approach in Macedonia? 8. Is the Chamber involved with public-private sector dialogues – where, of what type? 9. Are you familiar with the USAID/SBEP project, and did you have any cooperation with them? 10. For you personally or for this institution, what was the “most significant change” that emerged from all of your interactions with SBEP? This could be anything that happened that in your mind has long-term importance. An open-ended response in the form of a story is expected – whatever the respondent answers, draw them into a discussion where they explain the change they think is significant and how it came about. 11. What will be your recommendations for future programming in field of economic and regional development? 12. Anything that we have not asked you that you wanted to share with us? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 319 Ministry of Tourism/ Agency for tourism/ Ministry of Environment / Makedonski Sumi / National Park Mavrovo 1. What is your institution jurisdiction in field of tourism/ wild gathered products? 2. Do you have any involvement with the regional or municipal level in implementation of your policies / activities? If yes, please describe it! 3. How involved is it in the LED planning process at the municipal level? 4. Does your institution have its own regional development initiatives underway? 5. Is your institution involved in LEADER/LAG? If so, how? 6. Is it involved with other donor regional development activities and how? 7. Is your institution involved with public-private sector dialogues – where, of what type? 8. Are you familiar with the USAID/SBEP project, and did you have any cooperation with them? 9. For you personally or for this institution, what was the “most significant change” that emerged from all of your interactions with SBEP? This could be anything that happened that in your mind has long-term importance. An open-ended response in the form of a story is expected – whatever the respondent answers, draw them into a discussion where they explain the change they think is significant and how it came about. 10. What will be your recommendation for future programming in field of economic and regional development? 11. Anything that we have not asked you that you wanted to share with us? EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 320 ANNEX E: SOURCES OF INFORMATION Persons Interviewed Local Action Groups (LAGs) • LAG: Prespa • LAG: Makedonski Shumi • LAG: Vardar • LAG: Aber • LAG: Osogovski Lisec • LAG: Tikvesh Wine Valley • LAG: Jegunovce • LAG: Bogovinje • LAG: Pelagonija • LAG: Agrolider • Liaison Entre Actions de Développement de l'Économie Rurale (LEADER) LAG: Probaton Regional Centers for Development • Regional Development Committee: Southeast • Regional Development Committee: Southwest • Regional Development Committee: Northeast • Regional Development Committee: East • Regional Development Committee: Vardar • Regional Development Committee: Skopje • Regional Development Committee: Polog • Regional Development Committee: Pelagonia • Rural Development Network • Association for Local and Rural Development (ALRD) Tourism • Agency for Promotion and Support of Tourism in Macedonia • Agency for Tourism of Macedonia Others • Northwest Chamber of Commerce • Probio Company • Swiss Contact IME Program • Ministry of Agriculture of Macedonia, Forestry and Water Treatment • Magan Mak Company – Drip Irrigation Supplier • Cabinet of Vice Prime Minister for Economic Affairs -- • Women’s Advisory Board • REA Child Care Center • Ecolog International • Invest in Macedonia • Mavrovo National Park • Makedonski Shumi EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 321 • EPI Center • National Agricultural Extension Agency • Ministry of Local Government – Skopje • Bureau for Regional Development • GIZ - RED Program • Farmers Federation – Skopje • Agency for Financial Support for Agriculture and Rural Development EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 322 Bibliography of Documents Reviewed Emerging Europe. 2016. Macedonia — Stepping Out Of the Shadow Of the Balkans. http://emerging￾europe.com/intelligence/macedonia-stepping-shadow-balkans/ European Commission. 2014. Leader Gateway. http://enrd.ec.europa.eu/enrd￾static/leader/en/leader_en.html Herzberg, Benjamin and Andrew Wright. The PPD Handbook: A Toolkit for Business Environment Reformers. Washington, D.C., The World Bank Group, 2006. Kohl & Partner. 2016. National Tourism Strategy for Macedonia (Draft). M. Peters, Ricardo & Brijlal, Pradeep. 2011. The Relationship between Levels of Education of Entrepreneurs and Their Business Success: A Study of the Province of Kwazulu-Natal, South Africa. Industry and Higher Education. 25. 265-275. 10.5367/ihe.2011.0048. https://www.researchgate.net/publication/233506745_The_Relationship_between_Levels_of_Education_ of_Entrepreneurs_and_Their_Business_Success_A_Study_of_the_Province_of_Kwazulu￾Natal_South_Africa Ramadani, Veland. The Economic Power of Small Businesses: The Case of Republic of Macedonia. Tetovo, Macedonia, South East European University at Tetovo, Faculty of Business and Economy, 2011. Selimi, Dr. Nasir. “Are the Enterprises of Albanians in Macedonia Marginalized in the Economy of the Country?” Academic Journal of Interdisciplinary Studies. 4.2. (2015). MCSER Publishing. Web. 29 Jan. 2017. The World Bank. 2017. Countries/FYR of Macedonia. http://www.worldbank.org/en/country/macedonia/overview USAID (United States Agency for International Development). 2013. Small Business Expansion Project: Performance Monitoring Plan Table FY 2013 USAID (United States Agency for International Development). 2014. Small Business Expansion Project: Performance Monitoring Plan Table 1 – FY 2014 USAID (United States Agency for International Development). 2014. Small Business Expansion Project: Performance Monitoring Plan Table 2 – FY 2014 USAID (United States Agency for International Development). 2014. Small Business Expansion Project: Performance Monitoring Plan Table 3 – FY 2014 USAID (United States Agency for International Development). 2014. Small Business Expansion Project: Performance Monitoring Plan Table 4 – FY 2014 USAID (United States Agency for International Development). 2015. Small Business Expansion Project: Performance Monitoring Plan Table 1 – FY 2015 USAID (United States Agency for International Development). 2015. Small Business Expansion Project: Performance Monitoring Plan Table 2 – FY 2015 USAID (United States Agency for International Development). 2015. Small Business Expansion Project: Performance Monitoring Plan Table 3 – FY 2015 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 323 USAID (United States Agency for International Development). 2015. Small Business Expansion Project: Performance Monitoring Plan Table 4 – FY 2015 USAID (United States Agency for International Development). 2016. Small Business Expansion Project: Performance Monitoring Plan Table 1 – FY 2016 USAID (United States Agency for International Development). 2016. Small Business Expansion Project: Performance Monitoring Plan Table 2 – FY 2016 USAID (United States Agency for International Development). 2016. Small Business Expansion Project: Performance Monitoring Plan Table 3 – FY 2016 USAID (United States Agency for International Development). 2016. Small Business Expansion Project: Performance Monitoring Plan Table 4 – FY 2016 USAID (United States Agency for International Development). 2012. Small Business Expansion Project: Quarterly Report 4 – FY 2012 USAID (United States Agency for International Development). 2013. Small Business Expansion Project: Quarterly Report 1 – FY 2013 USAID (United States Agency for International Development). 2013. Small Business Expansion Project: Quarterly Report 2 – FY 2013 USAID (United States Agency for International Development). 2013. Small Business Expansion Project: Quarterly Report 3 – FY 2013 USAID (United States Agency for International Development). 2013. Small Business Expansion Project: Quarterly Report 4 – FY 2013 USAID (United States Agency for International Development). 2014. Small Business Expansion Project: Quarterly Report 1 – FY 2014 USAID (United States Agency for International Development). 2014. Small Business Expansion Project: Quarterly Report 2 – FY 2014 USAID (United States Agency for International Development). 2014. Small Business Expansion Project: Quarterly Report 3 – FY 2014 USAID (United States Agency for International Development). 2014. Small Business Expansion Project: Quarterly Report 4 – FY 2014 USAID (United States Agency for International Development). 2015. Small Business Expansion Project: Quarterly Report 1 – FY 2015 USAID (United States Agency for International Development). 2015. Small Business Expansion Project: Quarterly Report 2 – FY 2015 USAID (United States Agency for International Development). 2015. Small Business Expansion Project: Quarterly Report 3 – FY 2015 USAID (United States Agency for International Development). 2015. Small Business Expansion Project: Quarterly Report 4 – FY 2015 EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 324 USAID (United States Agency for International Development). 2016. Small Business Expansion Project: Quarterly Report 1 – FY 2016 USAID (United States Agency for International Development). 2016. Small Business Expansion Project: Quarterly Report 2 – FY 2016 USAID (United States Agency for International Development). 2016. Small Business Expansion Project: Quarterly Report 3 – FY 2016 USAID (United States Agency for International Development). 2016. Small Business Expansion Project: Quarterly Report 4 – FY 2016 Databases For time series data the study team drew on information available from the State Statistial Office of the Republic of Macedonia as well as on the United Nations COMTRADE data base for information about Macedonian imports and exports. EVALUATION REPORT - IMPACT AND PERFORMANCE EVALUATION OF SBEP 325 ANNEX F: CONFLICT OF INTEREST DISCLOSURES All evaluation team members have completed and signed conflict of interest disclosure statements, which are on file at the MSI home office and can be provided upon request.