March 2018 (Revised May 8, 2018) This publication was produced at the request of the United States Agency for International Development for the E3 Analytics and Evaluation Project. It was prepared independently by Management Systems International, A Tetra Tech Company. Evaluation Impact Evaluation of the Women’s Leadership in Small and Medium Enterprises Activity in the Kyrgyz Republic ABSTRACT This report presents findings, conclusions, and lessons learned from an impact evaluation of the Kyrgyzstan Women’s Leadership in Small and Medium Enterprises (WLSME) activity. The evaluation used a randomized controlled trial design to examine the overall activity’s impact across four outcome measures: business growth, entrepreneurial leadership, networks, and business knowledge and practices. Female entrepreneurs with small or medium businesses in the tourism and garment sectors were eligible to participate in the activity. After a baseline survey was conducted on 843 eligible applicants, 568 women entrepreneurs were randomly assigned to the treatment group and 275 women entrepreneurs to the control group. The evaluation measured changes in outcomes at three follow-up points (at the end of the activity in 2015, at 12 months post-activity, and at 24 months post-activity) to examine the time trends of the effects. The evaluation found small positive impacts on business growth through short-term investments in capital inputs and longer-term investments in labor inputs, though these investments do not translate into higher sales or profits. Factors that may have affected the activity’s impact include loss in sample size due to attrition, persistent cultural and gender-related intra￾household dynamics, and lack of access to finance. Targeting this type of activity to the “most promising entrepreneurs” may have differential effects; when the “right” entrepreneurs are selected and provided with targeted technical assistance, the probability of significant positive outcomes may increase dramatically. IMPACT EVALUATION OF THE WOMEN’S LEADERSHIP IN SMALL AND MEDIUM ENTERPRISES ACTIVITY IN THE KYRGYZ REPUBLIC March 26, 2018 (Revised May 8, 2018) Contracted under AID-OAA-M-13-00017 E3 Analytics and Evaluation Project Prepared By: Alberto Chong, Principal Investigator Irene Velez, Technical Manager Photo Caption: WLSME participants share their business experiences during one of the training sessions. Credit: Gulmira Asanbaeva, WLSME Project Manager DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic iii CONTENTS Acknowledgments.........................................................................................................................vi Acronyms ......................................................................................................................................vii Executive Summary....................................................................................................................viii Activity Description .............................................................................................................................................. viii Evaluation Design................................................................................................................................................... viii Findings........................................................................................................................................................................ x Primary Question ..................................................................................................................................... x Secondary Question ..................................................................................................................................... x Conclusions............................................................................................................................................................... xi Lessons Learned...................................................................................................................................................... xii Introduction................................................................................................................................... 1 Activity Overview.......................................................................................................................... 1 WLSME Initiative....................................................................................................................................................... 1 Kyrgyzstan WLSME Activity................................................................................................................................... 1 Development Hypotheses ............................................................................................................ 2 Evaluation Questions .................................................................................................................... 3 Outcome Measures.................................................................................................................................................. 4 Gender Aspects of the Questions........................................................................................................................ 4 Evaluation Design.......................................................................................................................... 4 Data Analysis Methods............................................................................................................................................ 6 Balance Among Treatment and Control Groups ...................................................................... 7 Analysis of Non-Response at Follow-Up ..................................................................................... 8 Evaluation Strengths and Limitations ......................................................................................... 9 Primary Question Findings – Overall Impact........................................................................... 10 Summary of Findings..............................................................................................................................................10 Business Growth.....................................................................................................................................................13 Insights on Business Growth......................................................................................................................14 Entrepreneurial Leadership ..................................................................................................................................15 Networks..................................................................................................................................................................18 Business Knowledge and Practices.....................................................................................................................19 Potential Intra-Household Conflict ....................................................................................................................20 Secondary Questions Findings – Component Analysis............................................................ 23 Summary of Findings..............................................................................................................................................23 Business Growth.....................................................................................................................................................25 Entrepreneurial Leadership ..................................................................................................................................26 Networks..................................................................................................................................................................27 Business Knowledge and Practices.....................................................................................................................27 Highly Motivated Achievers .................................................................................................................................28 Conclusions .................................................................................................................................. 29 Lessons Learned .......................................................................................................................... 30 Annex A: Evaluation Statement of Work................................................................................. 32 Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic iv Annex B: Literature Review....................................................................................................... 43 Annex C: Qualitative Data Collection Sampling and Process ................................................ 45 Annex D: Survey Data Collection and Quality Assurance ...................................................... 47 Annex E: Empirical Methods...................................................................................................... 49 Annex F: Full Results .................................................................................................................. 51 Primary Question: Overall Impact (ANCOVA model).................................................................................51 Secondary Questions: Component Results (DID model) ............................................................................56 Annex G: Data Collection Instruments .................................................................................... 64 Annex H: Team Composition.................................................................................................... 88 Annex I: References .................................................................................................................... 90 Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic v FIGURES: Figure 1: Theory of Change ............................................................................................................................ 3 Figure 2: Kyrgyzstan Evaluation Design and Process ................................................................................................ 5 Figure 3: Prevalence of Access to Finance.................................................................................................................15 Figure 4: Hiring Treatment Effects Over Time.........................................................................................................21 Figure 5: Difference-in-Differences Method..............................................................................................................50 TABLES: Table 1: Sample Distribution by Survey Round.......................................................................................................... 6 Table 2: Baseline Balance Test ............................................................................................................................ 8 Table 3: Non-Response Rates With Respect to Baseline........................................................................................ 8 Table 4: Summary of Statistically Significant Results ...............................................................................................12 Table 5: Impact on Labor Inputs ..........................................................................................................................13 Table 6: Impact on Firm Size (SME Threshold)........................................................................................................14 Table 7: Impact on Decision-Making Role .................................................................................................................16 Table 8: Impact on Intra-Household Relationships and Gender Roles...............................................................17 Table 9: Impact on Networks ..........................................................................................................................18 Table 10: Impact on Business Knowledge and Practices........................................................................................20 Table 11: Decision-Making Categories Over Time .................................................................................................22 Table 12: Gender Roles Categories Over Time ......................................................................................................22 Table 13: Treatment Sample Distribution by Activity component......................................................................23 Table 14: Focus Group Sample Distribution.............................................................................................................45 Table 15: In-Depth Interviews Sample Distribution................................................................................................46 Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic vi ACKNOWLEDGMENTS The evaluation team thanks the Kyrgyz women entrepreneurs who shared their time to participate in this study, M-Vector Research and Consulting for conducting the surveys, and ACDI/VOCA for its collaboration throughout the study. They also thank Zura Mendikulova for facilitating focus groups discussions and conducting in-depth interviews; Altyn Kapalova for conducting in-depth interviews and providing data quality oversight during data collection; and Joan Martinez and Angelo Cozzubo for their statistical support. The evaluation team also benefited from the shared thoughts and experiences of the Women’s Leadership in Small and Medium Enterprises (WLSME) Community of Practice, including the WLSME Peru team (GRADE/Sector 3 Social Venture Group) and WLSME India team (CARE), as well as technical reviews from Gary Glass, Jacob Patterson-Stein, Jeremy Gans, and Andrew Green at Management Systems International. Finally, the evaluation team thanks Anastasia de Santos (the USAID activity manager for this evaluation), Wade Channell (in USAID’s Office of Gender Equality and Women’s Empowerment), and Bhavani Pathak (the USAID Contracting Officer’s Representative for this evaluation) for their continued support of this study, and FHI 360 for its work with the original evaluation design and baseline phase. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic vii ACRONYMS ANCOVA Analysis of Covariance APR Annual Percentage Rate BMT Business Management Trainings BT Fund Bai-Tushum Innovations Fund DID Difference-in-Differences E3 Bureau for Economic Growth, Education, and Environment (USAID) FGD Focus Group Discussions IDI In-Depth Interviews ML Market Linkages RCT Randomized Controlled Trial SME Small and Medium Enterprise TS/AF Technical Skills/Access to Finance USAID United States Agency for International Development USD United States Dollars WLSME Women’s Leadership in Small and Medium Enterprises Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic viii EXECUTIVE SUMMARY This is the final report of the impact evaluation of the Kyrgyzstan Women’s Leadership in Small and Medium Enterprises (WLSME) activity. The Office of Trade and Regulatory Reform in the United States Agency for International Development’s Bureau for Economic Growth, Education, and Environment (USAID/E3) commissioned the evaluation. The report summarizes the findings from three follow-up rounds conducted at the end of the activity in 2015, at 12 months post-activity, and at 24 months post￾activity, and provides conclusions and lessons learned. Activity Description USAID’s WLSME initiative aims to address women’s under-representation in the small and medium enterprise (SME) sector by funding and testing innovative interventions to increase the entry and growth of women-owned and women-managed SMEs in the developing world. The Kyrgyzstan WLSME activity directly addresses two critical barriers: (1) agency constraints, which impede adequate accumulation of human capital and managerial capital, and (2) relationship constraints, which limit women’s access to information and reduce opportunities to build and draw on social capital. ACDI/VOCA and its partner organization, Bai-Tushum Innovations Fund, implemented the Kyrgyzstan WLSME activity nationwide, between September 2013 and September 2015, with a modest budget of $2 million. The activity targeted 960 women in the garment, tourism, and agro-processing sectors with at least one non-family member employee and identified as potential high-growth entrepreneurs. The activity consisted of three components: (1) business management training (BMT), which covered topics such as negotiating skills, business planning, marketing, financial planning, productivity, and human resource management; (2) market linkages (ML), including stakeholder meetings, trade fairs, workshops on value chains and sub-sectors, semi-annual value chain stakeholder meetings, web resources, and an annual business plan competition; and (3) technical skills/access to finance (TS/AF), where only finalists and semi-finalists from the business plan competitions received customized assistance, including technical training, targeted technical assistance, mentorship, and exchange visits. The participants in this last component were also eligible to apply for small grants. Evaluation Design The Kyrgyzstan WLSME activity is based on the development hypothesis that if women business owners have greater human capital, social capital, and access to market information, then they will be more likely to grow their businesses and become entrepreneurial leaders. This impact evaluation tests this hypothesis with a randomized controlled trial, where eligible applicants to the overall WLSME activity are randomly assigned to a treatment and control group. Only women in the treatment group received access to the WLSME activity, but once enrolled, participation in the individual components of the activity was demand-driven. The evaluation answers the following questions: Primary Question (combined impact – treatment vs. control groups): 1. Compared to participants in the control group, do participants who are randomly assigned to receive the program have higher mean values on the following post-intervention outcomes: entrepreneurial leadership, business growth, business knowledge/practices, and social/business networks? Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic ix Secondary Questions (separate estimates across treatment components): 2. Compared to participants who only receive BMT, do participants also exposed to ML have higher mean values on the same set of outcomes listed under the primary question? 3. Compared to participants who only receive BMT, do participants also exposed to TS/AF have higher mean values on the same set of outcomes listed under the primary question? The baseline sample consisted of 843 women, of whom 568 were randomly assigned to the treatment group and 275 were randomly assigned to the control group. This excludes the 354 agro-processing sector participants who were not randomized into treatment.1 Of the 568 participants enrolled in the activity, 461 actually participated, 378 completed the BMT component, 251 participated in the ML component, and 126 participated in the TS/AF component. The evaluation team conducted the first follow-up survey at the end of the activity, between August and October 2015, with an 82 percent response rate. The team conducted the second follow-up survey between August and October 2016, with an 84 percent response rate. The team conducted the third follow-up survey between August and October 2017, with a 76 percent response rate. The third follow-up sample consisted of 644 women, of whom 434 are in the treatment group and 210 are in the control group. However, only 568 women responded to all three follow-up rounds (67 percent of the baseline sample). Balance tests2 were originally conducted at baseline but are reported again in this report to show that the remaining treatment group, after excluding the agro-processing participants, and the control group shared similar social and economic conditions before the WLSME activity started. Analysis of survey non-responses showed that non-response was random, as no correlation exists with treatment assignment, and that survey non-response does not depend on observable characteristics. To answer the primary question, the evaluation team reports the findings using the preferred estimation model, analysis of covariance (ANCOVA). To answer the secondary questions, the evaluation team used a difference-in-differences (DID) model. However, given that selection into the activity components was not randomized, it is not possible to look at a causal treatment effects within the activity (secondary questions), but only at correlations. Thus, the randomized controlled trial (RCT) design only corresponds to the primary question. This is because participants who self-selected into participating in the ML or TS/AF components are different in ways that correlate to the outcomes. For example, women who attended the additional WLSME components may have been the most motivated or had a stronger support network to be able to dedicate more time away from home compared to women who participated only in the BMT, so they would have done better with their business and leadership even in absence of the WLSME activity. The evaluation also included focus group discussions with participants at the end of the activity in 2015 and in-depth interviews with 40 of the positive outlier participants at 20 months post-activity (between the second and third follow-up rounds) to explore opinions and experiences about how and why the WLSME activity worked, as well as to gain insights into social and cultural dynamics. 1 All women in the agro-processing sector (n=354) were assigned to the treatment group due to contamination risk and ethical concerns raised by agro-processing participants typically living in small communities or working within cooperatives. 2 An important consideration in assessing baseline data for an impact evaluation is the balance in characteristics between the treatment and control groups. If substantial differences exist between treatment and control group characteristics, the control group may not be a valid representation of the counterfactual. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic x Findings Primary Question There are modest statistically significant effects from the overall WLSME activity; however, some effects are not sustained over time. With respect to business growth, there is a short-term increase of 11 percentage points on investments in capital inputs (such as raw materials, goods, or equipment for business), but this effect is not detected in the later rounds. Small investments in labor inputs are detected in the medium-term, with an increase of 0.32 paid household workers (23 percent increase over the control group) but no increase in the total number of workers, meaning that household members who were previously not paid started getting paid. Similarly, in the longer-term, there is an increase of 0.28 paid household workers (22 percent increase over the control group) and an increase of 0.18 workers (15 percent increase over the control group) due to hiring new employees. The evaluation found no impact on microenterprises crossing the SME threshold (5 – 250 workers), but when the analysis is constrained to paid workers only (rather than total workers), there is a statistically significant effect on the SME threshold during the third round. At 24 months post-activity, the WLSME activity results in a 20.2 percentage point increase in the likelihood of growing from a microenterprise to a SME. However, neither these investments nor firm size growth translate into higher sales or profits. As important context to these findings, Kyrgyzstan’s economy underwent significant regulatory changes in August 2015, right as the WLSME activity ended. Ascending to the regional Customs Union appears to have negatively impacted Kyrgyzstan’s competitive position in textile exports, which might have had some impact on the rate of growth in this sector. With respect to entrepreneurial leadership, in the short-term, the frequency of often/always managing sales and client relations without consulting anyone else increases 7.2 percentage points, but this effect is not sustained over time. In the longer-term, participants report having gained a decision-making role along different dimensions of the business, namely applying for a loan (8.2 percentage point increase) and marketing and advertising decisions (8.7 percentage point increase). However, they also report a decrease in deciding their own wage (9.5 percentage point decrease). This lack of consistency in the results could indicate underlying intra-household dynamics at play as the business grows to include more household members as employees. The effects on business knowledge and practices are even more inconsistent, varying along different dimensions for each follow-up round. This supports the argument that behavior is hard to change and that the WLSME intervention did not have a consistent or sustained effect in this regard. Finally, the sparse and small short-term effects on networks are not sustained. Secondary Question The component analysis revealed two main findings. The first is that the participants who were part of the ML component, but not part of the “most promising entrepreneurs” selected for the TS/AF component, show statistically significant effects on only a handful of variables by the third round. The largest effects that are unique to this component are a 16.4 percentage point decrease in the likelihood of agreeing with the statement “there are no gender inequality problems in my community” and a 22.8 percentage point increase in the likelihood that the respondent prefers to work as an employee in a business instead of managing or owning her own business. The second finding is that the participants who were selected for the TS/AF component show the most positive and statistically significant effects across the different outcome variables. For example, the TS/AF component is associated with statistically significant increases in average sales on average months, non￾household members who work in the business and who are paid, likelihood that people ask her for advice, likelihood of implementing professional advice, likelihood of having a written budget, and Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic xi likelihood that she prepares accountancy documents annually. However, the TS/AF component is also associated with negative effects on having a decision-making role in several aspects of the business. The findings from the component analysis and the in-depth interviews corroborate the activity’s usefulness for high achieving and motivated participants. In fact, the story that emerges for this particular group is quite different from the average effects detected for the overall treatment group (more information on this group can be found in the Secondary Question section of the report). This group shows a clear recognition of the benefits of the intervention: “It has given me a lot, now I don’t do everything, I know exactly what I want, I know the direction, I know exactly what I will achieve, I set a goal and go to it, in small steps, but I think I will succeed. The program helped me with that.” “Even if I don't feel myself a leader, this program has made a leader out of me. I am considered a leader in my community, in the village. In the village, I take part in various social events because people support me as a woman who can do something, who can build relations with women, who can make others follow her.” Conclusions Six general conclusions can be drawn from the WLSME intervention. 1. There are some small positive impacts on business growth through short-term investments in capital inputs and longer-term investments in labor inputs, though these investments do not translate into higher sale or profits. This may also be due to a combination of low statistical power due to decreases in the sample size over time and difficulty in measurement of these variables; the latter is frequently reported in related literature (McKenzie 2012). 2. Several observed short-term impacts were not sustained over time, implying that behavior is difficult to change due to the contextual realities of applying what one learned. It may be that a more intense intervention is required to overcome the contextual issues that push against the teachings of the intervention. Corroborating this idea is the observed ambiguity over cultural and gender-related intra-household dynamics, which have a prevalent role in women’s activities. 3. Access to finance remains a constraint for women entrepreneurs in the Kyrgyz Republic. The finding that investments in capital inputs in the first follow-up round were not sustained, coupled with the voiced need for loans during the in-depth interviews at 20 months post-activity and increased involvement in deciding whether to apply for a loan in the third follow-up round, points to access to finance as an important business aspect for these SMEs. However, there is also a supply side gap, where most of the financing available is expensive and suited for trade businesses that can offer immediate credit repayments. Despite the efforts from the implementing partner to develop new business loan products for SMEs within the WLSME activity, this was not accomplished and the financing landscape has still not changed as of 2018. 4. Effects on decision-making and business knowledge and practices vary along business dimensions for each follow-up round. This lack of consistency could mean underlying intra-household dynamics are at play as the businesses grow to include more household members as employees. 5. The compressed timeline and difficulty to recruit the targeted number of eligible participants to the WLSME activity could have resulted in the inclusion of entrepreneurs who were either not motivated or fully committed to the activity. In a way, the resulting sample may have been “negatively” biased at the origin. In addition, there was a loss of 31 percent of the sample between the baseline and the third follow-up round (at 24 months post-activity). The small net sample size could have resulted in insufficient power to detect some of the more outlying outcomes, such as sales and profits. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic xii 6. Policies or criteria that target the “most promising entrepreneurs” may be particularly important. As observed in the results, when the “right” entrepreneurs are selected and provided with targeted technical assistance, the probability of significant positive outcomes increases dramatically. Lessons Learned While it is important to design interventions based on existing literature, the design should also take into account contextual considerations to align with local needs and priorities. Two areas where the design did not fully integrate the context and could have more adequately taken into account the local constraints were in: (1) addressing intra-household dynamics and gender issues and (2) focusing more directly on access to finance and the supply and demand constraints associated with it. Given the cost of this type of intervention and the potential differential effect on the high achieving women, it is important to better target the activity to those who will benefit the most. While it is difficult to know ex ante who these people may be, trying to identify them might be worth it from the perspective of the success of the intervention. The obvious difficulty is that the typical characteristics that define these individuals, such as intrinsic drive and inherent motivation, are difficult to observe. But there are at least two ways to try to do this. One is to focus the intervention on women who have already participated in similar activities, as this may indicate inherent interest. Another is to make it costly to participants, either monetarily or in terms of effort. The latter was part of the WLSME Kyrgyzstan training model, requiring participants to pay a fee to engage in each business management training course. This was highly unusual within the Kyrgyz context, required a behavior shift by participants, and challenged recruitment efforts. Future designs should take into consideration the pros and cons of different mechanisms to optimize the type of participants sought, while maintaining accessibility. The time trends of activity impacts need to be considered when deciding on the design and evaluation of programs and when and how to measure outcomes. The WLSME activity faced a shortened implementation timeline due to an extended period for the impact evaluation design, participant recruitment, and baseline data collection. During this start-up phase, the implementer was not authorized to begin the intervention, shortening their implementation period. Thus, only two years were allotted for intervention implementation, which means that due to the graduating intensity program design, most women in the TS/AF activity participated for less than six months. A longer implementation period would have provided a more intense intervention dosage. However, one of the evaluation’s strengths was that, from its inception, the design envisioned measurement of outcomes through three follow-up rounds and up to 24 months post-activity. This enabled the evaluation team to understand what changes are made relatively quickly after the activity and which impacts take more time to materialize. Conducting multiple follow-up rounds and measuring effects more than a year after the activity ends are important to tracing the trajectories of impacts and measuring both short and longer￾term effects. It is important to recognize early the potential tensions between program implementation and impact evaluation, to address viable solutions during the design stage. This intervention highlights a frequently perceived tradeoff between program implementation and impact evaluation. While evaluators feel somewhat constrained by the leeway of implementers, implementers sometimes perceive limits to their ability to adapt ongoing interventions. This tension between implementation and evaluation is understandable, but not irreconcilable if addressed during the design stage. Future activities may explicitly consider this trade-off in the evaluation design by modestly increasing the sample size, so if mid-intervention changes are required the evaluation will not suffer, or by increasing the budget for additional contingencies. Another alternative is to pursue multi-phased programs that allow for a small Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic xiii pilot, outside of the formal RCT, thereby providing a period in which to collaborate, learn, and adapt before ‘locking down’ an intervention model that will necessarily remain consistent over the formal intervention period to maximize learning from the RCT. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 1 INTRODUCTION This is the final report of the impact evaluation of the Women’s Leadership in Small and Medium Enterprises (WLSME) activity in Kyrgyzstan. The Office of Trade and Regulatory Reform in the United States Agency for International Development’s Bureau for Economic Growth, Education, and Environment (USAID/E3) commissioned the evaluation. The E3 Analytics and Evaluation Project3 provided post-baseline support for the evaluation, including implementation monitoring, follow-up data collection, and final analysis and reporting. The impact evaluation used a randomized controlled trial (RCT) design to test how the overall WLSME activity affected women entrepreneurs and their businesses across four primary sets of indicators: business growth, entrepreneurial leadership, networks, and business knowledge and practices. Annex A provides USAID’s statement of work for the evaluation. The report summarizes the findings from three follow-up rounds conducted at activity end in 2015, at 12 months post-activity, and at 24 months post-activity, and provides conclusions and lessons learned. ACTIVITY OVERVIEW WLSME Initiative USAID’s WLSME initiative aims to address women’s relative absence in the small and medium enterprise (SME) sector by implementing specific measures to reduce critical barriers, so that women may benefit from labor market participation in both the short and medium terms. These barriers include: (1) agency constraints, which impede adequate accumulation of human capital and managerial capital, and thus limit women’s knowledge and business practices; (2) relationship constraints, which limit women’s access to information and, as a consequence, reduce the opportunities for women entrepreneurs to build and draw on social capital; and (3) external constraints, which result in gender-specific barriers that limit the presence and success of women entrepreneurs. The Kyrgyzstan WLSME activity directly addresses only the first two constraints. In September 2012, USAID awarded three WLSME activities in Kyrgyzstan, India, and Peru through a competitive process, each with a performance period of three years and a budget of around $1.5 million to $2 million. USAID also initiated impact evaluations for each of these three activities under the Financial Integration, Economic Leveraging, Broad-Based Dissemination (FIELD-Support) Leader with Associates Cooperative Agreement with FHI 360 that concluded September 30, 2014, following the completion of evaluation design and baseline data collection and analysis activities. Subsequently, USAID transferred implementation of two of the impact evaluations (Kyrgyzstan and India) to the E3 Analytics and Evaluation Project, while the Peru evaluation is being separately funded and completed through the Multilateral Investment Fund of the Inter-American Development Bank. Kyrgyzstan WLSME Activity ACDI/VOCA, in collaboration with its partner organization Bai-Tushum Innovations Fund (BT Fund), implemented the WLSME activity in Kyrgyzstan. The activity operated nationwide from September 2013 to September 2015 and targeted 960 women who operated enterprises in priority sectors (garment, 3 Team lead Management Systems International, a Tetra Tech company, implements USAID’s E3 Analytics and Evaluation Project in collaboration with partners Development and Training Services, a Palladium company; and NORC at the University of Chicago. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 2 tourism, and agro-processing), met minimum employee and loan size requirements, and were identified as potential high-growth entrepreneurs. Activity components were sequenced, with standard services and courses offered to all participants. As components became more tailored and specialized, they focused on progressively fewer women. The most intensive mentoring and skills development components were reserved for the most promising women entrepreneurs. The components of this activity were nested within each other and consisted of subsets of participants: • Component 1 – Human Capital Gap (Agency): Business management training (BMT) covered topics such as negotiating skills, business planning, marketing, financial planning, productivity, and human resource management. The BMT were demand-driven and market￾oriented, meaning that women had a choice to some extent on the topics they attended and when, based on their interest. However, they had to attend at least 24 hours of business training (four days of six-hour training sessions) to complete this component, plus a three-day business planning seminar. Women could continue with more training, up to 72 hours per participant. • Component 2 – Information and Social Capital Gap (Relationships): This market linkages (ML) component included stakeholder meetings, trade fairs, workshops on value chains and sub-sectors, semi-annual value chain stakeholder meetings, web page resources, and an annual business plan competition. Originally, only participants who completed their BMT requirements from Component 1 were invited to the activities under Component 2. However, starting in January 2015, this requirement was dropped to increase the take-up rate of the activity components to increase the power to detect a significant impact of the overall activity. • Component 3 – Technical Skills/Access to Finance: Finalists and semifinalists from the business plan competitions in Component 2 received this customized technical assistance and access to finance (TS/AF). Assistance included technical training, targeted technical assistance, mentorship, and exchange visits. Participants were also eligible to apply for small grants of between USD $200 and $2,000. Instead of the initially planned loan guarantee fund, BT&P Bank offered a new loan product with a lower interest rate. 4 All activity participants, including the control group, had general access to this loan and other loan products from the BT&P Bank. DEVELOPMENT HYPOTHESES USAID’s development hypotheses for the WLSME initiative are displayed graphically in Figure 1, highlighting each of the intended results of the components and the presumed causal linkages (arrows). While the diagram focuses on the three parallel constraints that are hypothesized to impede business growth and entrepreneurial leadership, which the WLSME initiative aims to address, the Kyrgyzstan activity directly addresses only the first two – agency and relationship constraints – shown in bolded boxes. The third constraint (external constraints), addressed through joint counseling with spouses and engagement with service providers, was not part of the Kyrgyzstan activity. The activity in Kyrgyzstan also included the possibility of increased investments in SMEs that may occur because of increased access and reduced cost of finance from the WLSME partnership with BT Fund. However, this final path of interest is beyond the WLSME initiative and is not being delivered exclusively to activity beneficiaries, so it is depicted below the dotted line. 4 Regulatory changes made the planned loan guarantee fund no longer feasible. BT&P Bank’s loan product had an annual percentage rate (APR) of 19 percent if paid on time (24 percent if late), lower than the market rate (28 percent) but higher than state bank loans (10 percent). State loans, however, are mostly oriented towards agro-businesses and come with challenges. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 3 FIGURE 1: THEORY OF CHANGE WLSME’s theory of change for this activity is that if agency constraints are reduced by increasing women entrepreneurs’ business knowledge and managerial capabilities, then this will improve their business and management practices, leading to increased business growth and profitability and increased entrepreneurial leadership. In addition, if relationship constraints are reduced by increasing women entrepreneurs’ flow of information and links to other value-chain actors, then this will increase their social capital and business networks, leading to increased business growth and profitability and increased entrepreneurial leadership. EVALUATION QUESTIONS Little empirical research exists that provides convincing evidence about which interventions have the greatest chance of success in terms of creating female-led SMEs and helping female business owners grow their businesses (see Annex B: Literature Review). The purpose of this evaluation is to provide a learning, accountability, and decision-making platform by clarifying the most important constraints to women’s business growth and leadership, and thereby the most effective means to unleash the potential of women’s entrepreneurship in the SME sector in Kyrgyzstan. This evidence is expected to be useful to USAID staff, other donors, host governments, and stakeholders to improve future programming to better address the barriers to women’s entrepreneurship at the SME level. USAID’s evaluation questions included here are taken directly from the evaluation protocol designed by FHI 360. Primary Question (combined impact – treatment vs. control groups): 1. Compared to participants in the control group, do participants who are randomly assigned to receive the program have higher mean values on the following post-intervention outcomes: entrepreneurial leadership, business growth, business knowledge/practices, and social/business networks? Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 4 Secondary Questions (separate estimates across treatment components): 2. Compared to participants who only receive BMT, do participants also exposed to ML have higher mean values on the same set of outcomes listed under the primary question? 3. Compared to participants who only receive BMT, do participants also exposed to TS/AF have higher mean values on the same set of outcomes listed under the primary question? Outcome Measures A number of outcomes (dependent variables) linked to the WLSME theory of change were defined to measure whether and how much change the activity caused for women entrepreneurs. These include: • Business Growth: This includes, but is not limited to, measures of sales, profits, number of employees, number and type of paid employees, hours worked, investments, and formality. • Entrepreneurial Leadership: This includes measures on decision-making in business, entrepreneurial vocation, level of independence, and women’s empowerment. • Social/Business Networks: This includes measures regarding participants’ involvement in professional networks, such as the number of other business owners with whom the woman discusses business matters, as well as commercial networks. • Business Knowledge and Practices: This includes measures on marketing, inventory management, costing and recordkeeping, and financial planning. Gender Aspects of the Questions USAID evaluation guidance calls upon Agency staff and evaluation teams to examine evaluation questions from a gender perspective and to incorporate gender issues into study designs. This WLSME activity is targeted at women only, and the evaluation does not intend to collect data from male stakeholders (neither spouses nor male value chain actors). Thus, it will not be possible to disaggregate data collected in this evaluation by gender or to look at the differential gender effects of the activity components. Nonetheless, the main objective of the WLSME initiative is to close the multiple existing gaps between women and men in SMEs. EVALUATION DESIGN This impact evaluation is based on a RCT design to answer the primary question. Figure 2 shows the evaluation design, timeline, and participation numbers. Eligible applicants to the Kyrgyzstan WLSME activity were interviewed for the baseline and then randomly assigned to participate in the activity on a rolling basis, starting in July 2013. Randomization into the treatment and control groups occurred at the batch level, with an average of 32 women, in a 2:1 ratio. The baseline sample consisted of 843 women, of whom 568 were randomly assigned to the treatment group and 275 women were randomly assigned to the control group. This excludes the 354 agro-processing sector participants who were not randomized into treatment but participated in the WLSME activity. 5 All numbers in the rest of this report exclude the agro-processing sector participants. The WLSME activity offered the BMT to the treatment group only, of which 461 participants attended at least one BMT session (81 percent take-up) and 378 participants completed the BMT component 5 ACDI/VOCA, USAID, and the evaluation team made the decision to exclude the agro-processing sector participants from the evaluation sample. See Briefing Note submitted to USAID on February 11, 2015. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 5 (67 percent take-up). This is consistent with the average take-up rate across studies for individuals offered business training, as McKenzie and Woodruff (2014) report. For most of the duration of the activity, participants had to complete the BMT component to access the ML activities. However, the activity changed this requirement in January 2015 to increase take-up of the activity components, increasing the power to detect a significant impact of the overall activity. Since beneficiaries self-select into participating in the ML component, it is not possible to attribute impact to the individual activity components. This change therefore does not affect the essence of the evaluation design, but it does increase the ability to detect impact of the overall activity. A total of 251 women participated in the ML component. The activity offered the TS/AF component to only 126 participants who were finalists and semifinalists of the business plan competition. FIGURE 2: KYRGYZSTAN EVALUATION DESIGN AND PROCESS The evaluation team conducted the first follow-up survey at the end of the WLSME activity, between August and October 2015, with an 82 percent response rate. The team conducted the second follow-up survey 12 months after the end of the activity, between August and October 2016, with an 84 percent response rate. The team conducted the third follow-up survey 24 months after the end of the activity, between August and October 2017, with a 76 percent response rate. The third follow-up sample consisted of 644 women, of whom 434 are in the treatment group and 210 are in the control group. However, only 568 women responded to all three follow-up rounds (67 percent of the baseline sample). During the last round of data collection, the main reasons for non-response in the survey included lack of contact due to a change in contact information, business closure, and refusal to participate. Table 1 shows the sample distribution between the baseline and three follow-up survey rounds. The next section further analyzes the non-response rate. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 6 TABLE 1: SAMPLE DISTRIBUTION BY SURVEY ROUND Baseline First Follow-Up Second Follow-Up Third Follow-Up All Rounds N= N= % of sample N= % of sample N= % of sample N= % of sample Treatment Group 568 459 81% 472 83% 434 76% 390 69% Control Group 275 228 83% 232 84% 210 76% 178 65% Total 843 687 82% 704 84% 644 76% 568 67% In addition to the survey, the evaluation team conducted six focus group discussions (FGDs) at the end of the activity and 40 in-depth interviews (IDIs) at 20 months post-activity (between the second and third follow-up rounds) with WLSME participants. The FGDs were intended to explore opinions and experiences about “how” and “why” the WLSME activity worked, as well as to gain insights into social and cultural dynamics. The IDIs had two main objectives: (1) explore potential links between the positive changes in outcomes and specific intervention components through a positive deviance selection approach, where positive outliers – those with good outcomes – were identified and interviewed; and (2) explore further the ambivalent results from the survey data, particularly with respect to business growth and empowerment. Annex C summarizes the selection process for the FGDs and IDIs. Annex D summarizes the survey data collection and quality assurance process. Annex H describes the composition of the evaluation team that carried out endline activities. Data Analysis Methods To answer the primary question about the impact of the overall activity, the RCT design provides the greatest potential for causal inference and minimizes potential bias from unmeasured confounding factors, since access to the activity is not based on any external measure or individual characteristics that may correlate with the outcomes or participation in the activity. Because eligible participants were randomly assigned to the treatment and comparison group, both groups have, on average, the same characteristics and thus would be statistically identical in the absence of the activity. If both groups are identical, differing only in exposure to the activity, then any difference in outcomes at the end of the evaluation can be attributed to the activity. An analysis of covariance (ANCOVA) is the preferred estimation method used to answer this question, given the nature of the intervention, the low autocorrelation of outcome measures, and the three data rounds available (McKenzie, 2016). Annex E provides more information on the ANCOVA model. The secondary questions refer to the marginal effect from the activity components, but cannot be objectively answered from an attribution perspective, since access to the activity components was not randomized. Instead, participants either self-selected or the “most promising entrepreneurs” were selected for participation. The evaluation team could argue that more motivated women, more ambitious entrepreneurs, or those who would have succeeded even without the activity are more likely to have participated in the activity components. Therefore, comparing women who participated across the sub-groups would systematically miscalculate the impact estimates of each activity component, since changes in the outcome variables would reflect not only their participation in the components, but also the set of characteristics (e.g., ambition, motivation) that led the group to participate in the activity components in the first place. Given this non-randomized selection into the activity components, the difference-in-differences (DID) estimation model is more useful than the ANCOVA model because it accounts for all observable baseline differences between the groups. However, there may still be unobserved time-varying differences that cannot be accounted for with a DID model. Therefore, these estimates cannot be concluded as impact or causality; rather, they are merely a sign of association between the activity components and the outcome variables. The results presented in the secondary Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 7 questions findings section correspond to the DID model, not ANCOVA. Annex E provides more information on the DID model. In addition to the quantitative models, qualitative analysis of the FGDs and IDIs was conducted using MaxQDA software. Transcripts, translated to English, were uploaded into the software and then reviewed in their entirety with special attention to participant opinions and feedback related to WSLME activity effectiveness, utility, and perceptions on business growth, networks, empowerment, and gender roles. Responses were sorted categorically and assigned descriptive “codes” to facilitate frequency and demographic cross-comparison. Common trends and themes were tracked, as were divergences and outliers from those trends and themes. Relevant informant quotes were also extracted from the transcripts as examples of participant opinion and sentiment on specific topics. BALANCE AMONG TREATMENT AND CONTROL GROUPS Balance between the treatment and control groups across key demographic and outcome variables is necessary to show that the groups are the same, on average, prior to the start of the intervention. If substantial differences exist between treatment and control group characteristics, the control group may not be a valid representation of the counterfactual. The baseline report showed balance across the two groups except in two instances: number of years the business operated and whether the respondent used credit (from any source) to borrow funding for her business in the past 12 months. Given changes to the sample due to exclusion of the agro-processing sector, the evaluation team conducted balance tests again on the remaining baseline sample. The team applied the student’s t-test for two independent samples with unequal variances. This test provided previewing evidence about the differences between the control and treatment groups before the intervention started. The evaluation team chose 12 characteristics or dimensions to establish whether significant differences existed between treatment and control groups at the baseline round. If the p-value associated with the t-test is small (p-value <0.05), evidence suggests that the average is different for both groups. Namely, the mean difference is significantly different from zero. On the contrary, when the p-value associated with the test is not small, it can be concluded that the means of both groups are not different. Table 2 shows the tests of balance between the treatment group and control at the baseline. It is expected that participants in these two groups have similar characteristics at the stage previous to the intervention, since the treatment group was randomly selected at an individual level, but this provides evidence that the remaining sample (after the exclusion of the agro sector) was also balanced at baseline. As shown, the sample proves to be balanced for all variables except used credit to borrow funding for her business in the past 12 months. At baseline, the control group was more likely to have had a loan in the previous 12 months. No other variable shows statistically significant differences in the presented model. Thus, the data show that the control and treatment groups shared similar social and economic conditions before the WLSME activity took place, and therefore may be compared validly by experimental methods. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 8 TABLE 2: BASELINE BALANCE TEST Characteristics Control Treatment Difference p-value Age 44.27 44.79 -0.52 0.53 Married (=1) 0.79 0.77 0.01 0.64 Higher education (=1) 0.48 0.51 -0.03 0.42 Owner (=1) 0.53 0.51 0.02 0.65 Years of operation 7.27 7.00 0.27 0.55 Full-time workers from the household 0.98 0.91 0.07 0.47 Full-time non-family workers 3.68 3.89 -0.21 0.73 Previous trainings/seminars (=1) 0.34 0.37 -0.03 0.39 Number of children under 18 1.69 1.82 -0.13 0.23 Sells in external markets 0.22 0.24 -0.01 0.72 Garment sector 0.64 0.59 -0.05 0.19 Used credit for business in past 12 months 0.59 0.51 0.08** 0.03 Number of observations 568 275 - - Note: Statistical significance is denoted by the following system: *** p<0.01, ** p<0.05, * p<0.1 ANALYSIS OF NON-RESPONSE AT FOLLOW￾UP Given the sample distribution between the baseline and three follow-up survey rounds, as shown in Table 1 above, the evaluation team examined whether non-response was random and not correlated with treatment assignment. Table 3 shows there is no statistically significant difference (p-values=0.46, 0.64, 0.99, and 0.37 for each round, respectively) in non-response between the treatment and control groups across each round. TABLE 3: NON-RESPONSE RATES WITH RESPECT TO BASELINE First Follow-Up Second Follow-Up Third Follow-Up All Rounds Treatment Group 19.2% 16.9% 23.6% 31.3% Control Group 17.1% 15.6% 23.6% 35.3% Total 18.5% 16.5% 23.6% 32.6% p-value 0.46 0.64 0.99 0.37 F-test 0.27 0.12 0.17 0.70 The evaluation team also examined whether non-response depends on observable characteristics. The team regressed follow-up survey completion on the same set of baseline variables, treatment status, and interaction terms of those baseline variables with the treatment variable across each round. Looking at the F-test on the interaction variable coefficients for the first follow-up round (p-value=0.27), second follow-up round (p-value=0.12), third follow-up round (p-value=0.17), and all rounds combined (p￾value=0.70), the evaluation team did not find differences in the observable composition of the treatment versus control groups despite differential non-response rates. Thus, although the non-response rate increases for each round, affecting the power of the study, it does not affect the validity of the findings presented below. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 9 EVALUATION STRENGTHS AND LIMITATIONS This impact evaluation reflects a rigorous approach to address the evaluation questions and contribute to the global knowledge on women’s entrepreneurship in the SME space. Two key strengths of this evaluation are: • The use of multiple follow-up rounds. This improves the efficiency of the data and available resources. Given the high variability in some of the outcome measures, collecting data at multiple rounds instead of at a single point in time increases power for the same sample size, or vice versa requires a smaller sample size for the same level of precision. • Measuring outcomes over the short- and longer-term after the activity ends. This takes into account the time trends of different variables, as some have short-term effects and others take longer to fully develop. For business training, one might expect firms to make some changes relatively quickly after training; however, firms could start some practices and then drop them. Furthermore, the full impact of training may take some time to materialize (McKenzie and Woodruff, 2014). Multiple follow-ups over two years helps to understand these dynamics and ensure that the evaluation does not miss potential impacts from the activity. Despite efforts to minimize potential threats to validity, some limitations remain: • Sample size loss due to exclusion of agro-processing participants and survey attrition may have affected the ability to detect changes. While excluding the agro-processing sector participants from the evaluation sample resulted in losing one-third of the baseline treatment group, this exclusion maintained the validity of an RCT design by keeping only the randomized participants in the evaluation. By the end of the evaluation, only 67 percent of the baseline sample had responded to all three follow-up rounds. While the evaluation design accounted for some attrition (5 percent loss each round), the actual attrition was larger than anticipated. Fortunately, attrition was random, as no correlation exists with treatment assignment, and survey non-response does not depend on observable characteristics. • Difficulty measuring profits and revenues. Owners of micro- and small enterprises typically do not keep written records of these items, and owners of larger firms who do keep records may be reluctant to share them because of concerns about theft or taxation issues. De Mel et al. (2009) studied several approaches to obtaining profits from microenterprises and concluded that, in their context at least, a simple direct question is more accurate and less “noisy” than calculating profits from revenues and expenses. However, collecting profits has proved difficult for many studies, with several studies not collecting profit data at all (McKenzie and Woodruff, 2014). Furthermore, since knowledge of how to manage a business, including how to calculate profit, is one of the desired outcomes, non-responses or inaccurate responses may be correlated with the treatment. • Response bias due to self-reported behaviors. It is possible that treated individuals may report certain behaviors because the activity told them this was important, rather than because they actually did them. There is also the possibility that answering the same survey three times could have affected how they responded to the survey; although this is less likely to have affected the treatment group differently than the control group. • Diffused, and shorter than anticipated, exposure to the activity components. The WLSME activity faced a shortened implementation timeline due to an extended period for the impact evaluation design, participant recruitment, and baseline data collection. Moreover, of the 568 female entrepreneurs enrolled in the WLSME activity, 81 percent participated in any BMT sessions, 66 percent completed the BMT component, and only 44 percent participated in any ML activities. Thus, most participants received incomplete versions of the WLSME activity, diluting the treatment dosage to which they were exposed. If changes in certain outcome variables are expected to be Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 10 driven by exposure to specific component activities, then the low take-up and short duration would make it harder to detect significant changes. This may have been the case for the network outcomes, which were the focus of the ML component, but less than half of the treatment group participated in this component. PRIMARY QUESTION FINDINGS – OVERALL IMPACT This section presents a summary of the short-term (first follow-up at the end of the activity), medium￾term (second follow-up at 12 months post-activity), and longer-term (third follow-up at 24 months post￾activity) effects of the overall Kyrgyzstan WLSME activity on the key outcome measures. These results correspond to the ANCOVA model, which takes into account the baseline and all three follow-up rounds. 6 The estimations below can be interpreted as intent to treat, which represents the average effect of having access to the activity (i.e., all of the women assigned to the treatment group remain part of the treatment group, regardless of actual participation).7 Annex F provides the full results. Summary of Findings There are modest statistically significant effects from the overall WLSME activity; however, some effects are not sustained over time. With respect to business growth, there is a short-term effect only on investments in capital inputs (such as raw materials, goods, or equipment for business) and a medium￾and longer-term effect on investments in labor inputs. However, these investments do not translate into higher sales or profits. Furthermore, there is no impact on microenterprises crossing the SME threshold (5 – 250 workers). However, when the analysis is constrained to paid workers only (rather than total workers), there is a statistically significant effect on the SME threshold during the third round. At 24 months post-activity, the WLSME activity results in a 20.2 percentage point increase in the likelihood of growing from a microenterprise to a SME. With respect to entrepreneurial leadership, in the short term there is an increase in the frequency of often/always managing sales and client relations without consulting anyone else, but this effect is not sustained. In the longer term, participants report having gained a decision-making role along different dimensions of the business, namely applying for a loan and marketing and advertising decisions. However, they also report a decrease in deciding their own wage. This lack of consistency supports other results that show that there are underlying intra-household dynamics at play as the business grows to include more household members as employees. 6 To test for robustness, the evaluation team computed Bonferroni-type corrections in all empirical results. The number of hypotheses (m) considered for this correction depends on the number of tests applied for each outcome variable within any specific category. In particular, the number of hypotheses considered are as follows: business growth outcomes (m=10), entrepreneurial leadership category (m=27), networks (m=7), and business knowledge and practices outcomes (m=22). The corrected p-values at 90 percent are 0.0010, 0.004, 0.014, and 0.005, respectively. The corrected p-values at 95 percent are 0.005, 0.002, 0.007, and 0.002, respectively. While the key findings in this report do not change, some outcome variables lose statistical significance in most categories. Because Bonferroni tends to give false negatives and requires high power – a structural weakness of the sample – the evaluation team would not want to put excessive emphasis on these additional results. 7 Intent to treat accounts for real-world implementation of interventions, which face imperfect take-up and attrition. While the average treatment effect on the treated (i.e., only the women assigned to the treatment group who actually participated in the intervention) could also be considered, it is not included in this report because treatment effects dependent on full compliance are not realistic in this context. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 11 The effects on business knowledge and practices are even more inconsistent, varying along different dimensions for each follow-up round. This supports the argument that behavior is hard to change and that the WLSME intervention did not have a consistent or sustained effect in this regard. Finally, the sparse short-term effects on networks are not sustained. Table 4 shows a summary of only the statistically significant results for each outcome category across the three follow-up rounds. Results reported in one round but not in the others, lost or gained statistical significance over time. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 12 TABLE 4: SUMMARY OF STATISTICALLY SIGNIFICANT RESULTS Outcome Measures Short-term effects (First follow-up) Medium-term effects (Second follow-up) Longer-term effects (Third follow-up) Business Growth • 11 percentage point (ppt.) increase in investments in capital inputs • No effect on sales or profits • No effect on SME threshold • Increased investment in labor inputs (0.32 more paid HH members; no effect on total number of workers) • No effect on sales or profits • No effect on SME threshold • Increased investment in labor inputs (0.28 more paid HH members and 0.18 more total workers) • Less time spent working in her business (0.27 months per year less) • No effect on sales or profits • 20.2 ppt. more likely to cross the SME threshold (paid workers only) Entrepreneurial Leadership Business Decision-making: • 7.2 ppt. more likely to often/always manage sales and client relationships without consulting anyone else Gender roles: • 6.5 ppt. less likely to agree with “it is OK if men chide women because they went out without any permission” • 6.5 ppt. less likely to agree with “role of women is to earn money and take care of her family” Business Decision-making: • No effect Gender roles: • 5.9 ppt. increased perception that there are no gender inequality problems in her community Business Decision-making: • 8.2 ppt. more likely to be part of deciding whether to apply for a loan • 8.7 ppt. more likely to be in charge of marketing and advertising decisions • 9.5 ppt. less likely to be part of deciding her own wage Gender roles: • No effect Networks • Increase of 1.0 in number of people that participant can ask for advice • 10.6 ppt. more likely to have implemented professional advice • 10.2 ppt. less likely to be an active member of any social group • No effect Business Knowledge & Practices • 5.0 ppt. less likely to compare price and quality of inputs with other suppliers’ products • 5.4 ppt. more likely to have written business goals for next 12 months • 11.2 ppt. more likely to keep written record of owner’s salary • 7.1 ppt. less likely that business has fixed salary for the owner • 4.6 ppt. less likely to agree that she develops work plans at regular intervals • 5.7 ppt. less likely to agree that she often anticipates future circumstances and plans how the business will deal with them • 6.9 ppt. less likely to agree that she constantly collects information about the business market • 7.0 ppt. less likely to agree that she is sometimes late for appointments/meetings Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 13 Business Growth Business growth outcomes include variables related to sales, profit, business cycle, time spent working in the business, number and type of paid employees, investments, and loans. Self-reported average sales and average profits do not show any statistically significant improvement, regardless of type of month (good, average, or bad) in any of the three rounds. The lack of statistical significance can be due to a combination of low statistical power and difficulty in measurement in the sales variable; the latter is frequently reported in related literature (McKenzie 2012). As important context to these findings, Kyrgyzstan’s economy underwent significant regulatory changes in August 2015, right as the WLSME activity ended. Ascending to the regional Customs Union appears to have negatively impacted Kyrgyzstan’s competitive position in textile exports, which might have had some impact on the rate of growth in this sector. The capital investments detected at the end of the WLSME activity, an 11 percentage point increase in purchasing raw materials, goods, or equipment for business, are no longer detected at 12 months or 24 months post-activity. These investments may have been done in the short-term as an immediate way to grow the business. It also points to a potential barrier of access to finance that was voiced during the IDIs, explained further below. The labor investments detected at 12 months post-activity continue to be statistically significant, albeit small in absolute terms, at 24 months post-activity. Table 5 shows in the second round, an increase of 0.32 paid household workers (23 percent increase over the control group) but no increase in the total number of workers, meaning that household members who were previously not getting paid started to get paid. In the third round, there is an increase of 0.28 paid household workers (22 percent increase over the control group) and an increase of 0.18 workers (15 percent increase over the control group) due to hiring of new employees. These results show a credible trend regarding increased labor inputs that reflect formal business growth, as more household members working in the business become paid employees rather than simply helping, and in the longer term, the SMEs start hiring more people. As participants hire more employees, they also report a small decrease in the time (months per year) they spend working in their business relative to the control group (0.27 months less). TABLE 5: IMPACT ON LABOR INPUTS Outcome Variable Follow-up 1 Treatment effect (std. error) Follow-up 2 Treatment effect (std. error) Follow-up 3 Treatment effect (std. error) Number of household members who have worked in business in the last 12 months 0.0342 0.010 0.176** (0.105) (0.040) (0.072) Number of paid household workers 0.0921 0.319** 0.275** (0.165) (0.122) (0.101) Number of non-household people who have worked in business in the last 12 months -0.297 0.006 0.130 (0.586) (0.153) (0.298) Number of paid non-household workers -0.102 -0.109 0.184 (0.588) (0.257) (0.500) Number of months per year spent working in the business owned or managed 0.033 -0.282 -0.271* (0.215) (0.233) (0.121) Note: Coefficients were obtained by ANCOVA regressions with region and sector fixed effects. Age, marital status, education level, business ownership, number of workers, participation in previous trainings, and number of children under 18 were included as control variables. Robust standard errors in parentheses. Statistical significance is denoted by the following system: *** p<0.01, ** p<0.05, * p<0.1. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 14 TABLE 6: IMPACT ON FIRM SIZE (SME THRESHOLD) Outcome Variable Follow-up 1 Treatment effect (std. error) Follow-up 2 Treatment effect (std. error) Follow-up 3 Treatment effect (std. error) Likelihood of crossing SME threshold (5 to 250 total workers) 0.007 0.000 0.046 (0.015) (0.022) (0.037) Likelihood of crossing SME threshold (5 to 250 paid workers) 0.019 0.013 0.202*** (0.037) (0.052) (0.045) Note: Coefficients were obtained by ANCOVA regressions with region and sector fixed effects. Age, marital status, education level, business ownership, number of workers, participation in previous trainings, and number of children under 18 were included as control variables. Robust standard errors in parentheses. Statistical significance is denoted by the following system: *** p<0.01, ** p<0.05, * p<0.1. There is no impact on microenterprises crossing the SME threshold (5 – 250 workers). However, when the analysis is constrained to paid workers only (rather than total workers), there is a statistically significant effect on the SME threshold during the third round. At 24 months post-activity, the WLSME activity results in a 20.2 percentage point increase in the likelihood of growing from a microenterprise to a SME. Insights on Business Growth During the IDIs, participants reflected on how the WLSME activity had helped their business and the remaining challenges. Three-fourths (n=30) of the positive cases described situations where their business had improved in some fashion due to what they learned from the BMT, and one-fourth (n=7) of those interviewees credited post-activity success with enabling the hiring of new employees or the acquisition of new equipment. Of those who saw no change in their business or saw their business decline, none blamed the activity, although two interviewees said they would have done better if the activity had given them a grant or helped them get a loan. “It was the same with me. I cannot say that after courses my sales improved, no. But this is not connected with the things that the program didn’t provide to us, this is the world recession now.” “It has given me a lot, now I don’t do everything, I know exactly what I want, I know the direction, I know exactly what I will achieve, I set a goal and go to it, in small steps, but I think I will succeed. The program helped me with that.” There was an expectation from the women entrepreneurs that the activity would provide access to capital. However, there were no effects in any of the three follow-up rounds on access to loans or business registration and formality. This lack of impact is reflected in the disappointment that came through during the IDIs: “I did not like that after we acquired knowledge and certificates, we started thinking that we could get a loan for business development. We tried to get a loan but failed. No one paid attention to us. We have small business. And they give loans only to big businesses and give no loans to small businesses. The certificate we acquired turned out to be unhelpful. If you gave us certificates, you should have given us the opportunity. This is my opinion. You just trained us. I did not like that we received no further support and help.” Or, as another participant put it, more succinctly: Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 15 “It would be better if assistance is provided in the form of capital.” This need for access to capital was also reflected in the survey, in which respondents (WLSME and non￾WLSME participants alike) reported during the baseline and at each follow-up round that “access to finance” is the biggest challenge they faced now in managing their business and the top challenge to growing their business in the future. The frequency of these responses ranged from 44 to 50 percent across the time rounds. It is also, by far, the most frequently reported challenge. The second challenge reported was “getting support from my family,” which ranged from 5 to 11 percent across the time rounds. While there was no difference in the likelihood of applying for a loan between treatment and control groups, the percentage of respondents who had applied for a loan or had borrowed cash in the last 12 months decreased consistently across each time round, as shown in Figure 3. The most frequently cited reasons for not applying for a loan were: (1) loan conditions are not attractive, and (2) worry that she would not be able to repay the loan. These findings point to a gap in access to finance, where women entrepreneurs voice the need for capital but have not been applying for loans or borrowing cash for their business. FIGURE 3: PREVALENCE OF ACCESS TO FINANCE Entrepreneurial Leadership The second key outcome category, entrepreneurial leadership, includes variables related to decision￾making in the business, entrepreneurial vocation, level of independence, and women’s empowerment. The findings on different dimensions of entrepreneurial leadership are centered on the participant’s role in decision-making in the business and the intra-household relationships that allow and empower women to thrive. The main takeaway regarding decision-making is that the particular aspects of the business where participants’ decision-making role changed vary across the three rounds, yielding inconsistent results. In the first round, the evaluation detected a 7.2 percentage point increase in the likelihood of often/always managing sales and client relationships. In the second and third rounds, this finding was no longer detected. While it is not possible to claim that the intervention resulted in better sales management or client relations as expected by the WLSME intervention, the trend from this outcome Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 16 variable is an encouraging sign of increased behavioral change for participants, on average, making these individual, non-consulted decisions more frequent. The third round generated several new findings. First, there is an 8.2 percentage point increase in the likelihood of having a decision-making role, alone or with spouse/partner, on whether to apply for a loan. The observed trend supports the argument that time is needed to affect behavioral change, but it also reinforces the idea that a key issue for women entrepreneurs in this intervention was access to capital likely in the form of loans or grants. In fact, the trend in this variable may be interpreted as one of increased proactiveness given their expressed need for capital and struggles faced to secure financing. Second, there is an 8.7 percentage point increase in the likelihood of having a decision-making role, alone or with spouse/partner, on marketing and advertising matters. This indicator captures a narrow, somewhat marginal and specific issue; however, it brings to the fore the idea that the WLSME intervention bore some positive impact with respect to entrepreneurial leadership. However, there is also a negative finding corresponding to a 9.5 percentage point decrease in the likelihood of having a decision-making role, alone or with spouse/partner, on deciding her own wage. Table 7 provides a summary of the findings in this section. TABLE 7: IMPACT ON DECISION-MAKING ROLE Outcome Variable Follow-up 1 Treatment effect (std. error) Follow-up 2 Treatment effect (std. error) Follow-up 3 Treatment effect (std. error) Likelihood that I (with my partner/spouse or another household member) decide if I should apply for a loan -0.019 -0.035 0.082** (0.038) (0.037) (0.031) Likelihood that I (with my partner/spouse or another household member) decide my own singular wage 0.005 0.050 -0.095** (0.034) (0.036) (0.036) Likelihood that I (with my partner/spouse or another household member) am in charge of marketing and advertising decisions 0.031 -0.039 0.087** (0.037) (0.060) (0.027) Likelihood that I often (or always) manage sales and client relations without consulting anyone else 0.072* 0.015 -0.008 (0.039) (0.030) (0.033) Note: Coefficients were obtained by ANCOVA regressions with region and sector fixed effects. Age, marital status, education level, business ownership, number of workers, participation in previous trainings, and number of children under 18 were included as control variables. Robust standard errors in parentheses. Statistical significance is denoted by the following system: *** p<0.01, ** p<0.05, * p<0.1. Outcomes variables stated as “likelihood” can be interpreted as percentage point change by multiplying the coefficient (treatment effect) by 100. Another important aspect of entrepreneurial leadership is related to intra-household relationships and gender roles. In this case, there are mixed findings across time (see Table 8). In the short term, there is conflicting evidence regarding women becoming more assertive in their relationships with their spouse or partner. For example, the first round revealed a 6.5 percentage point decrease in the likelihood of agreeing with the statement, “It is OK if men chide women because they go out without asking for permission from the spouse.” However, this positive effect is lost in the subsequent rounds. On the other hand, there is also a 6.5 percentage point decrease in the likelihood of agreeing with the statement, “The role of women is to earn money and take care of her family.” Similar to the previous case, this negative effect is lost in the subsequent rounds. Moreover, unexpected signs (positive/negative) in some of the coefficients, even if not statistically significant, depict a context in Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 17 which external constraints and gender roles may be at play and hard to change. However, these dynamics are not captured by the survey in the subsequent rounds; in the medium term, there is a 6.0 percentage point increase in the likelihood of agreeing with the statement, “There are no gender inequality problems in my community,” while in the longer term, there are no statistically significant effects with respect to gender roles. TABLE 8: IMPACT ON INTRA-HOUSEHOLD RELATIONSHIPS AND GENDER ROLES Outcome Variable Follow-up 1 Treatment effect (std. error) Follow-up 2 Treatment effect (std. error) Follow-up 3 Treatment effect (std. error) Likelihood that (strongly) agrees that “it is OK if men chide women because they went out without any permission” -0.065* -0.017 0.005 (0.036) (0.028) (0.049) Likelihood that (strongly) agrees that “the role of women is to earn money and take care of her family” -0.065* -0.019 -0.034 (0.038) (0.034) (0.038) Likelihood that (strongly) agrees that there are no gender inequality problems in my community -0.014 0.059** -0.038 (0.027) (0.020) (0.027) Note: Coefficients were obtained by ANCOVA regressions with region and sector fixed effects. Age, marital status, education level, business ownership, number of workers, participation in previous trainings, and number of children under 18 were included as control variables. Robust standard errors in parentheses. Statistical significance is denoted by the following system: *** p<0.01, ** p<0.05, * p<0.1. Outcomes variables stated as “likelihood” can be interpreted as percentage point change by multiplying the coefficient (treatment effect) by 100. This inconsistency in the quantitative results is also apparent in the qualitative findings, with women sometimes plainly stating there is no intra-household inequality, but when probed, they would describe undeniably unequal gender situations and practices. The issue is the implicit definition of equality assumed by women. Several women equate opportunity to equality. Interviewees claimed that ambitious, hard-working, and successful women can achieve equal status with men, therefore society is not unequal, as reflected in this respondent’s observation: “The society now provides all opportunities to women. Women should find their way themselves. If they try, they can achieve anything they want to. There are no restrictions. Anyone who wants to may become a member of council, a businessperson. Women have all opportunities in the society. There are no prohibitions. But everything depends on her and her family.” Social pressure may prevent some women from experiencing equality, but respondents said that is their fault or the fault of their families: “Women are responsible for all household chores. This is why they limit them so that they only engage on housework; this is why that cannot be leaders, cannot do business. Their rights are limited. They have been oriented to do one thing. This is why they cannot engage in business or other things; in addition, they do not have time.” This ambiguity about gender inequality is also blamed on education and urban-rural differences: “It is wrong to complain that we don’t have equality in Bishkek. If I lived in a rural area, there’s another situation there. Maybe these were the women from the regions? They have small businesses, and they expressed their opinion, they have gender inconsistency. I don’t feel any inequality in the capital city. Or maybe because this is the clothing industry with more women, Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 18 maybe that’s why I don’t feel it. If I ran some other business with more men, maybe I would feel it.” In addition, they point to cultural norms for propagating gender inequality: “For a boy, they build a house, create all conditions, marry, and arrange great parties. And for the girls, they do not build a house. Nobody thinks of building a house for a daughter, they think just to get her married. I think here you can see what different rights a boy and a girl have. This is a deep view on the issue. They build a house for a boy, and not for a girl. Interesting, right?” Finally, despite some women seeing improvements, others point out clear inequality between men and women, mainly driven by economic necessity. There was consistent agreement that women who could support themselves economically were more likely to enjoy a higher level of freedom from inequality and that this factor was intensified by high levels of male unemployment. “Yes, in the past, views were different and approaches were different. In the Soviet era, everyone worked equally and received equal salaries. Back then, men earned more than women. And today it’s vice versa. Men sit at home, earn little, or don’t work at all, and women have many opportunities to earn. And the views on these changes are being changed, or have already changed.” Networks For the networks outcomes, there are sparse effects that are not sustained over time (see Table 9). In the short-term, there was a 10.6 percentage point increase in the likelihood of implementing professional advice and an increase of 1.0 in the number of people the participant could ask for business advice, both consistent with the intended achievements of the WLSME intervention. However, these effects are lost in the subsequent rounds. It is disappointing to not detect any network effects, given the activity’s focus on market linkages. This points to both the short-term nature of the effect on networks or, perhaps more critically, to the clash between the intentions of the activity and the contextual reality faced by the women after the activity has ended. TABLE 9: IMPACT ON NETWORKS Outcome Variable Follow-up 1 Treatment effect (std. error) Follow-up 2 Treatment effect (std. error) Follow-up 3 Treatment effect (std. error) Likelihood of being an active member of any social group 0.031 -0.102* 0.006 (0.053) (0.045) (0.066) Likelihood of implementation of professional advice (business adviser, lawyer or accountant) during the past two years 0.106*** -0.003 0.020 (0.034) (0.041) (0.018) Number of people I can go to ask business advice 0.980** 0.604 -0.638 (0.393) (0.349) (0.571) Note: Coefficients were obtained by ANCOVA regressions with region and sector fixed effects. Age, marital status, education level, business ownership, number of workers, participation in previous trainings, number of children under 18 were included as control variables. Robust standard errors in parentheses. Statistical significance is denoted by the following system: *** p<0.01, ** p<0.05, * p<0.1. Outcomes variables stated as “likelihood” can be interpreted as percentage point change by multiplying the coefficient (treatment effect) by 100. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 19 Women may be able to acquire knowledge and the tools to apply it, but implementing such knowledge and expanding formal networks may be too daunting to overcome. This is reinforced by the IDIs, where only a couple of women report undertaking explicit formal functioning partnerships, even though many described having newly developed informal networks for cooperation, information sharing, and passing along what they had learned from the classes. The quantitative findings do not support that; on average, these loose, informal networks have had little bearing on relevant results. However, interviews with the positive outliers provide strong evidence that when networking has been seriously pursued by the most motivated women, it has been useful: “This program has united us. I have many acquaintances now; although we lived in the same city, we did not know each other before. Even if they come to me and ask to make a chapan, I immediately say that I do not sew them, it’s [other female entrepreneur] who sews chapans, so I give her contacts. It means that this program created a platform where we communicated, got acquainted, and even began to cooperate. For example, I do the embroidery, someone sews and asks me to do embroidery, not expensive ones, and I say OK. I do her a favor, and she can suggest me something. It’s has become much easier, I don’t grab every single order as before.” Similarly: “I can teach business planning to other people. For example, I share my experience with my employees. I don’t think only about myself. I advise them how to do something better. I acquired knowledge and got the certificate. I cannot give certificate to others, yet I can teach them and share my knowledge and experience with them, especially when it concerns women.” Business Knowledge and Practices Business knowledge and practices outcomes include implementation of marketing, operations, and accounting practices; negotiation skills; recording of budget; and future expectations, among other messages provided by the training delivered. The results related to this category are rather limited and the most inconsistent. In the first round, there was a positive effect of a 5.4 percentage point increase in the likelihood of having written business goals for the next 12 months and a negative effect of a 5.0 percentage point decrease in the likelihood of comparing price and quality of inputs with other suppliers’ products during the last three months. In the second round, these effects were no longer detected, but there was a positive effect of an 11.2 percentage point increase in the likelihood of recording the owner’s salary in a registry or computer. In the third round, the evaluation no longer finds an effect on the likelihood of recording the owner’s salary and even finds a negative effect of a 7.1 percentage point decrease in the likelihood that the owner has a fixed salary. Also in the third round, the evaluation detects new findings with respect to business management practices. On the positive side, there is a 7.0 percentage point decrease in participants agreeing that they are sometimes late for appointments or meetings. The WLSME intervention improved the understanding of women participants that being late for appointments should not be acceptable. On the negative side, there is a 4.6 percentage point decrease in participants agreeing that they develop work plans at regular intervals, a 5.7 percentage point decrease in participants agreeing that they attempt to anticipate future circumstances and plan how they will deal with them, and a 6.9 percentage point decrease in participants agreeing that they are constantly collecting information about the market in which their business operates. The lack of statistical evidence in most of the variables in this category and the varying aspects that are statistically significant in each survey round point to the difficulty of Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 20 changing entrepreneurs’ behavior to modify their business practices, but also that the knowledge of best practices does not stay with them over time. TABLE 10: IMPACT ON BUSINESS KNOWLEDGE AND PRACTICES Outcome Variable Follow-up 1 Treatment effect (std. error) Follow-up 2 Treatment effect (std. error) Follow-up 3 Treatment effect (std. error) Likelihood that compared price and quality of inputs with other suppliers’ products during the last three months -0.050* 0.017 -0.039 (0.029) (0.043) (0.023) Likelihood that records salary of the owner in a notebook, registry or computer 0.018 0.112* -0.003 (0.056) (0.052) (0.006) Likelihood that has no written goals for next 12 months -0.054** 0.034 0.051 (0.026) (0.025) (0.035) Likelihood that (strongly) agrees that “I develop work plans at regular intervals” 0.018 0.032 -0.046* (0.025) (0.036) (0.022) Likelihood that (strongly) agrees that “I am sometimes late for appointments or meetings” -0.016 -0.028 -0.070* (0.038) (0.050) (0.035) Likelihood that (strongly) agrees that “I often attempt to anticipate future circumstances and plan how I/my company will deal with them” -0.009 -0.021 -0.057* (0.017) (0.012) (0.028) Likelihood that (strongly) agrees that “I am constantly collecting information about the market in which my company operates” 0.002 -0.018 -0.069** (0.019) (0.017) (0.027) Note: Coefficients were obtained by ANCOVA regressions with region and sector fixed effects. Age, marital status, education level, business ownership, number of workers, participation in previous trainings, and number of children under 18 were included as control variables. Robust standard errors in parentheses. Statistical significance is denoted by the following system: *** p<0.01, ** p<0.05, * p<0.1. Outcomes variables stated as “likelihood” can be interpreted as percentage point change by multiplying the coefficient (treatment effect) by 100. Potential Intra-Household Conflict As shown in previous sections, the effects on decision-making and business knowledge and practices vary along different business dimensions for each follow-up round. Moreover, there is a lack of consensus on gender equality from the IDIs. The question is whether this lack of consistency may be signaling issues that to some extent limit the potential of women-owned enterprises. The evaluation found suggestive evidence that indicates underlying intra-household conflict when businesses grow to include more employees. While this cannot be affirmed with certainty, indicative results from the evaluation appear to confirm this hypothesis, in particular when comparing the patterns of hiring those who are workers and household members vis-à-vis workers who are not household members. The evaluation found that while initially more household members were hired to work at the firm, this increase stayed flat in the later period. On the other hand, the decrease in hiring non-household members remained flat initially, but then hiring increased considerably during the third follow-up survey. While this latter effect is not statistically significant, it reflects a pattern that is not consistent with a “smooth empowerment transition” implicitly assumed when designing these programs. Instead, this Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 21 shows a scenario in which household members may initially feel deserving of any success of the firm, and only later give way to more professional management in the form of hiring non-household workers. While not actual proof, this pattern is consistent with a period of turmoil within households due to the intervention. These findings are illustrated in Figure 4. FIGURE 4: HIRING TREATMENT EFFECTS OVER TIME Note: Coefficients were obtained by ANCOVA regressions with region and sector fixed effects. Age, marital status, education level, business ownership, number of workers, participation in previous trainings, and number of children under 18 were included as control variables. The only statistical significant effects are on household members who worked in business on the third round and paid household workers on the second and third round. Certain entrepreneurial leadership measures also appear to show some intra-household conflict patterns that support the finding described above. A preliminary read of key empowerment variables yields contradictory findings. On one hand, WLSME participants report they are less likely to be in charge of general business planning and this response becomes more negative, albeit not statistically significant, each subsequent survey round. On the other hand, there is an inconsistent pattern (positive/negative effects across rounds) regarding decision-making with respect to the type of work they do, setting their wage, and marketing and advertising. However, a closer look at the decision￾making categories suggests a pattern that may reflect some intra-household conflict, where women lose their decision-making power in crucial business categories but gain decision-making power in “softer” categories such as marketing and advertising. This is illustrated in Table 11. -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 Follow-up 1 Follow-up 2 Follow-up 3 HH members who worked in business (last year) Non-HH members who worked in business (last year) Number of paid household workers Number of paid non-household workers Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 22 TABLE 11: DECISION-MAKING CATEGORIES OVER TIME Outcome Variable Follow-up 1 Treatment effect (std. error) Follow-up 2 Treatment effect (std. error) Follow-up 3 Treatment effect (std. error) Likelihood that I (with my partner/spouse or another household member) am in charge of general business planning decisions -0.007 -0.006 -0.020 (0.032) (0.047) (0.021) Likelihood that I (with my partner/spouse or another household member) decide my own singular wage 0.005 0.050 -0.095** (0.034) (0.036) (0.036) Likelihood that I (with my partner/spouse or another household member) decide what type of work I will do 0.021 0.006 -0.029 (0.032) (0.035) (0.021) Likelihood that I (with my partner/spouse or another household member) am in charge of marketing and advertising decisions 0.031 -0.039 0.087** (0.037) (0.060) (0.027) Note: Coefficients were obtained by ANCOVA regressions with region and sector fixed effects. Age, marital status, education level, business ownership, number of workers, participation in previous trainings, and number of children under 18 were included as control variables. The only statistical significant effects were on the likelihood of deciding own singular wage and on likelihood of being in charge of marketing decision, only in the third round. Regarding gender roles, the inconsistent patterns (positive/negative effects across rounds) may also reflect some intra-household conflict. For instance, WLSME participants are more likely to agree, albeit not statistically significantly different, with statements that “it is OK if men chide women because they went out without permission” and “it is OK if men chide women if they do not take care of the children”. Furthermore, WLSME participants are also less likely to agree, albeit not statistically significant after the first follow-up round, with the statement “the role of women is to earn money and take care of her family”. These findings are illustrated in Table 12. TABLE 12: GENDER ROLES CATEGORIES OVER TIME Outcome Variable Follow-up 1 Treatment effect (std. error) Follow-up 2 Treatment effect (std. error) Follow-up 3 Treatment effect (std. error) Likelihood that (strongly) agrees that “it is OK if men chide women because they went out without any permission” -0.065* -0.017 0.005 (0.036) (0.028) (0.049) Likelihood that (strongly) agrees that “it is OK if men chide women if they do not take care of children” 0.038 -0.015 0.025 (0.028) (0.024) (0.031) Likelihood that (strongly) agrees that “the role of women is to earn money and take care of her family” -0.065* -0.019 -0.034 (0.038) (0.034) (0.038) Note: Coefficients were obtained by ANCOVA regressions with region and sector fixed effects. Age, marital status, education level, business ownership, number of workers, participation in previous trainings, and number of children under 18 were included as control variables. The only statistical significant effects were on the likelihood of deciding own singular wage and on likelihood of being in charge of marketing decision, only in the third round. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 23 SECONDARY QUESTIONS FINDINGS – COMPONENT ANALYSIS In the previous sections, this report addressed the impact of the overall WLSME activity. This section presents an analogous analysis that compares the correlation of two activity components (ML and TS/AF) relative to the basic training component (BMT). The group of analysis consists of women who engaged in BMT only, compared to Component A (defined as BMT + ML) and Component B (defined as BMT + ML + TS/AF). Table 13 lists the number of participants in each of these three groups. TABLE 13: TREATMENT SAMPLE DISTRIBUTION BY ACTIVITY COMPONENT Treatment Group Component Comparison Group Component A Component B Only BMT BMT + ML BMT + ML+TS/AF N % N % N % Participant 255 65% 165 65% 93 36% Non-participant 135 35% 90 35% 162 64% Total 390 100% 255 100% 255 100% Note: The evaluation sample for this analysis excludes the following individuals: (1) activity participants in the agriculture sector and (2) activity participants who were assigned to the treatment group but rejected to participate in any of the follow-up surveys, in spite of being registered in the baseline survey. Since selection into the activity components was not randomized, it is not possible to look at a causal treatment effect within the intervention, but only at correlations. This is because participants who self￾selected into participating in ML or TS/AF are different in ways that are correlated to the outcomes. Given the non-randomized selection into the activity components, the results reported below correspond to the DID model, not ANCOVA.8 In the discussion that follows, the estimates for Component A correspond to the added value of the ML component relative to the BMT only. The estimates for Component B correspond to the added value of the ML+TS/AF components relative to the BMT only. Summary of Findings The component analysis yields two main findings. The first is that the participants who were part of the ML component, but not part of the “most promising entrepreneurs” selected for the TS/AF component, show statistically significant effects on only a handful of variables by the third round. However, the largest effects that are unique to this component are a 16.4 percentage point decrease in the likelihood of agreeing with the statement, “There are no gender inequality problems in my community,” and a 22.8 percentage point increase in the likelihood that the respondent prefers to work as an employee in a business instead of managing or owning her own business. The second finding is that the participants who were selected for the TS/AF component show the most positive and statistically significant effects across the outcome variables. For example, the TS/AF component is associated with statistically significant increases in average sales on average months, non￾household members who work in the business and who are paid, likelihood that people ask her for 8 An additional statistical method proposed in the Evaluation Design Proposal, propensity score matching, was not conducted due to the small sample size. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 24 advice, likelihood of implementing professional advice, likelihood of having a written budget, and likelihood that she prepares accountancy documents annually. However, the TS/AF component is associated with negative effects on having a decision-making role on several aspects of the business. The tables that follow summarize only the statistically significant results for each component across the three follow-up rounds. Results reported in one round but not in the others, lost or gained statistical significance over time. Annex F provides the full set of results. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 25 Business Growth Activity Component Short-Term Associations (First follow-up) Medium-Term Associations (Second follow-up) Longer-Term Associations (Third follow-up) Component A (BMT + ML) • Increased likelihood of having a bank account for the business • Increased likelihood of loan approval • No effect on sales or profits • No effect on SME threshold • Increased likelihood of having a bank account for the business • Less number of days per week spent working in the business • No effect on sales or profits • No effect on SME threshold • Increase in number of HH members working in the business • No effect on sales or profits • No effect on SME threshold Component B (BMT+ML+TS/AF) • Decreased likelihood of having a bank account for the business • No effect on sales or profits • No effect on SME threshold • Increased likelihood of having a bank account for the business • Less number of days per week spent working in the business • Increase in total number of paid non-HH workers • Increase in total number of non-HH workers • No effect on sales or profits • No effect on SME threshold • Increased sales on an average month • Increased likelihood of having a bank account for the business • Less number of days per week spent working in the business • Increase in total number of paid non-HH workers • Increase in total number of non-HH workers • More bad months in the last year (business cycle) • No effect on SME threshold Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 26 Entrepreneurial Leadership Activity Component Short-Term Associations (First follow-up) Medium-Term Associations (Second follow-up) Longer-Term Associations (Third follow-up) Component A (BMT + ML) • Prefers to work as an employee instead of managing or owning a business Gender roles: • Decreased likelihood in agreeing with “it is OK if men chide women if they do not take care of children” • Increased likelihood in agreeing with “it is OK if men abandon women if they wish to” • Decreased likelihood in agreeing with “a mother who works can establish a relationship as warm and solid with her children as a mother who does not work” • Decreased likelihood in agreeing with “father’s and mother’s dedication is equally important for the learning and effective development of children” • Decreased perception that there are no gender inequality problems in her community • Prefers to work as an employee instead of managing or owning a business • More likely to be in charge of marketing and advertising decisions Gender roles: • Decreased likelihood in agreeing with “a mother who works can establish a relationship as warm and solid with her children as a mother who does not work”. • Decreased perception that there are no gender inequality problems in her community • Prefers to work as an employee instead of managing or owning a business • Less likely to be part of deciding her own wage Gender roles: • Increased likelihood in agreeing with “it is OK if men abandon women if they wish to”. • Decreased perception that there are no gender inequality problems in her community Component B (BMT+ML+TS /AF) • Prefers to work as an employee instead of managing or owning a business • Decreased likelihood that people ask her for advice often • Decreased likelihood of being in charge of sales and client relations • Decreased likelihood of often/always managing sales and client relationships without consulting anyone else • Decreased likelihood of often/always deciding whether to apply for a loan without consulting anyone else Gender roles: • No association • Increased likelihood that people ask her for advice often • Decreased likelihood of being in charge of sales and client relations • Decreased likelihood of often/always managing sales and client relationships without consulting anyone else • Decreased likelihood of being part of deciding her own wage • Decreased likelihood of being part of deciding what type of work she will do • Increased likelihood of being in charge of marketing and advertising decisions Gender roles: • No association • Increased likelihood that people ask her for advice often • Decreased likelihood of being in charge of sales and client relations • Decreased likelihood of often/always managing sales and client relationships without consulting anyone else • Decreased likelihood of being part of deciding her own wage • Decreased likelihood of being part of deciding what type of work she will do • Increased likelihood of being in charge of marketing and advertising decisions Gender roles: • No association Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 27 Networks Activity Component Short-Term Associations (First follow-up) Medium-Term Associations (Second follow-up) Longer-Term Associations (Third follow-up) Component A (BMT + ML) • Increased likelihood of participating in trade shows or fairs • Decreased likelihood of sometimes feeling (very) confident negotiating higher prices with buyers • No effects • No effects Component B (BMT+ML+TS/AF) • Increased likelihood of participating in trade shows or fairs • Increased likelihood of having implemented professional advice • Increased likelihood of having implemented professional advice • Increase in number of people participant can ask for advice • Increased likelihood in having implemented professional advice Business Knowledge and Practices Activity Component Short-Term Associations (First follow-up) Medium-Term Associations (Second follow-up) Longer-Term Associations (Third follow-up) Component A (BMT + ML) • Increased likelihood of performing physical validation of inventory levels • Increased likelihood of having accountancy documents prepared annually • Increased likelihood of agreeing that “sometimes I miss deadlines” • Increased likelihood of performing physical validation of inventory levels • Increased likelihood of having accountancy documents prepared annually • Increased likelihood of agreeing that “sometimes I miss deadlines” • Increased likelihood of performing physical validation of inventory levels • Increased likelihood of having accountancy documents prepared annually Component B (BMT+ML+TS/AF) • Increased likelihood of having accountancy documents prepared annually • Increased likelihood of having changes planned over the next 12 months • Increased likelihood of agreeing that “sometimes I miss deadlines” • Decreased likelihood of agreeing that “employees should be treated as family” • Increased likelihood of having accountancy documents prepared annually • Increased likelihood of having a written expense budget • Increased likelihood of agreeing that “sometimes I miss deadlines” • Decreased likelihood of agreeing that “employees should be treated as family” • Increased likelihood of having accountancy documents prepared annually • Increased likelihood of having a written expense budget Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 28 Highly Motivated Achievers The positive findings from the component analysis regarding the TS/AF group and the IDIs with positive outliers corroborate the activity’s usefulness for the high achieving and motivated participants. In fact, the story presented for this group is quite different from the average effects detected for the overall treatment group. This group indicates a clear recognition of the benefits of the intervention: “This program will stay with me for a long time. Because it’s the program that has changed my attitude to business. The Kyrgyz people didn’t provide such services before and it looked ridiculous. They didn’t provide any services and didn’t want to. Our Uzbek neighbors always build good relations with customers, respect their relatives, are always friendly. Now we have succeeded in these things through business” “There’s a big difference between a working woman and a woman sitting at home, they are as different as day and night. Now you can tell what you couldn’t tell before. We couldn’t share our thoughts even with the husband. The communication with people, with the group works well. I can judge by myself.” There is also a clear recognition that the skills acquired are long-term benefits that can be sustained. The following quotes are illustrative: “Five points. It was zero in the beginning, then I would give 5 points. I keep developing little by little. It is impossible to become a good leader at once.” “These skills will develop with time. They are like yeast. They will become bigger because they are basic skills. As soon as you tackle your first challenge, you will be able to tackle any challenges that may come your way.” “It has given me a lot, now I don’t do everything, I know exactly what I want, I know the direction, I know exactly what I will achieve, I set a goal and go to it, in small steps, but I think I will succeed. The program helped me with that.” Furthermore, women in this group feel that the WLSME activity has helped them across most of the specific categories of the intervention, and in particular on leadership issues: “Even if I don’t feel myself a leader, this program has made a leader out of me. I am considered a leader in my community, in the village. In the village, I take part in various social events because people support me as a woman who can do something, who can build relations with women, who can make others follow her.” “The program gave me new life, new views. I have changed my views. I am learning to behave as a leader. I have leadership skills now. Now I feel confident that I can succeed. Now I work as the head of 36 ayil9, in Leilek rayon. It is leadership, isn’t it?” Many women understand their newfound leadership tools in the context of the overall knowledge provided by the activity: “The program taught women to be leaders. We used to do small things before, and now we are leaders; we got our eyes opened, we learned to manage. The program teaches how to manage business. I got very good knowledge from the program. When you are in a business, you do not 9 In Kyrgyz, kinship community Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 29 think about how to graduate from some higher education institution, some courses and then launch a business. If you have money, you buy material, put a sewing machine and sew. The USAID program was held at the right time.” Finally, this group of women is better prepared to deal with money issues and financial institutions, a critical aspect for many entrepreneurs – not only those in the highly motivated achievers group: “Now I can work with banks based on my business, patent. Before the program, I didn’t have a patent, and after the training I received a patent and now I work legally. If your business is legal, you can get a loan for your business development.” “We learned how to effectively use money earned, prioritize spending and received at least some theoretical skills on how to start making savings and gain financial stability, at least to some extent. I’m not talking about independence, this is far away, at least stability.” CONCLUSIONS Six general conclusions can be drawn from the WLSME intervention. First, there are some small positive impacts on business growth through short-term investments in capital inputs and longer-term investments in labor inputs, though these investments do not translate into higher sale or profits. This may be due to a combination of low statistical power due to drops in the sample size over time and difficulty in measurement of these variables; the latter is frequently reported in related literature (McKenzie 2012). Second, several observed short-term impacts were not sustained over time, including for the sparse networks, decision-making, and business knowledge and practices effects detected in the first round. It implies that behavior is difficult to change due to the contextual realities of applying what was learned. It may be that a more intense intervention is required to overcome the contextual issues that push against the teachings of the intervention. This idea is corroborated by the observed ambiguity related to cultural and gender-related intra-household dynamics, which play a prevalent role in women’s activities. Third, access to finance remains a constraint for women entrepreneurs in the Kyrgyz Republic. The finding that investments in capital inputs in the first follow-up round were not sustained, coupled with the voiced need for loans during the IDIs (at 20-months post-activity) and increased involvement in deciding whether to apply for a loan in the third follow-up round, points to access to finance as an important business aspect for these SMEs. However, there is also a supply side gap, where most of the financing available is expensive and suited for trade businesses that can offer immediate credit repayments. Despite the efforts of the implementing partner to develop new business loan products for SMEs within the WLSME activity, this was not accomplished and the landscape has still not changed as of 2018. Fourth, effects on decision-making and business knowledge and practices vary along different business dimensions for each follow-up round. This lack of consistency could indicate underlying intra-household dynamics as the businesses grow to include more household members as employees. This finding is particularly salient because the issue of conflict or tension among family members is rarely taken into consideration in the design of this type of program. In most cases, the implicit assumption is that bringing additional income to the household will always be welcomed by other family members, in particular the husband. While the evaluation team has little doubt that this is the case in the longer run, the path to reach it does not appear to be necessarily smooth and some difficulties appear to arise in the short- and even medium-term. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 30 Fifth, the compressed timeline and difficulty in recruiting the targeted number of eligible participants to the WLSME activity could have resulted in the inclusion of entrepreneurs who were either not motivated or fully committed to the activity. In a way, the resulting sample may have been “negatively” biased at the origin. The plausibility of this argument is supported by the 66 percent completion rate of the BMT component. Moreover, only 66 percent of those who did complete the BMT component chose to participate in the ML activities. This resulted in most participants receiving incomplete versions of the WLSME activity, diluting the treatment dosage to which they were exposed. In addition, 31 percent of the sample was lost between the baseline and 24 months post-activity. The small net sample size could have resulted in insufficient power to detect some of the more outlying outcomes, such as sales and profits. Sixth, the prior conclusions, coupled with positive findings from the TS/AF component and the qualitative evidence, suggest that policies or criteria that target the “most promising entrepreneurs” may be important. As observed in these results, selection of the “right” entrepreneurs and giving them targeted technical assistance may dramatically increase the probability of significant positive outcomes. LESSONS LEARNED First, while this activity was carefully designed and based on the existing literature, it has uncovered an issue rarely addressed in this type of activity, namely the potential frictions that any women-related intervention may cause among household members, in particular male counterparts. Considering the context of the Kyrgyz Republic, a country with a highly male-dominant culture and ever more so in rural areas and among those with more economic needs, the sign reversion and lack of impact of several outcome variables where male dominance is challenged is, in hindsight, not surprising. Perhaps the original design should have accounted for the demand side on intra-household issues more adequately. Another example relates to trying to better assess the relative needs of women entrepreneurs, in particular in this intervention, to the fact that from the beginning, expectations of the intervention were high on the issue of bank loans. In informal conversations with participants as well as in qualitative interviews, this issue consistently came up. For some reason, participants expected that the intervention would lead them to obtain a bank loan. Potential reasons for this may have included assumptions made over the role of BT&P Bank as an activity partner and confusion over the small grants available to a subset of entrepreneurs who were part of the TS/AF component. As the intervention progressed, several participants became disappointed when they realized that a loan was not forthcoming and disengaged. Perhaps, if the intervention had been designed with this need in mind, it could have been simplified and focused on loan-related issues and basic financial issues related to them. Future interventions should take deeper consideration of country-specific contextual factors and the demand and supply constraints associated with them. Second, cost-benefit issues should be taken into account more explicitly in the design. Given the cost of this type of intervention and the potential differential effect on the “high achieving” women, it is important to better target the activity to those who will benefit the most. While it may be difficult to know ex-ante who these people may be, trying to identify them might be worth it from the perspective of the success of the intervention. The obvious difficulty is that the typical characteristics that define these individuals, such as intrinsic drive and inherent motivation, are difficult to observe. But there are at least two ways to try to do this. One way is to focus the intervention on women who have already participated in similar activities, as this may indicate inherent interest. Doing this, coupled with interventions that focus heavily on the demand side, would help design better-tailored activities that may have a greater impact. Another way of achieving this is to make it costly to the participants, either monetarily or in terms of effort, but still accessible. The latter was part of the WLSME Kyrgyzstan training model, requiring participants to pay a Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 31 fee to engage in each business management training course. This resulted in demand-driven participation and providing useful information to the implementing partner as to which topics were most valued by participants. It was an unusual approach in the Kyrgyz context (and the broader global WLSME portfolio of programs) and ran counter to the local norm of free trainings offered by most other SME development activities in the country, requiring a behavior shift by participants. Charging fees to subsidize the full cost of business trainings was done to entice more invested and serious business owners to engage, and to not undermine the local business development services market. This commitment was held despite the challenges it posed in recruiting sufficient numbers of participants consistent with the evaluation requirements. Future designs should take into consideration the pros and cons of different mechanisms to optimize the type of participants sought, while maintaining accessibility. An indication that this type of approach might work is provided in this comment by one of the treated participants: “If the training is free of charge, no one will attend, and if participants have to pay some fee, the majority will attend it. We didn’t pay that much, but how much did they spend for us? First course, second course, hotel. I have attended other programs, and they have been weak. But I estimate your program highly because they spent a lot on us.” Third, there are important time trends associated with different outcome measures. The WLSME activity faced a shortened implementation timeline due to an extended period for the impact evaluation design, participant recruitment, and baseline data collection. During this start-up phase, the implementer was not authorized to begin the intervention, cutting down on their implementation period. Thus, only two years were allotted for intervention implementation, which means that due to the graduating intensity program design, the majority of the women in the TS/AF activity participated for less than six months. A longer implementation period would have provided a more intense intervention dosage. However, the strength of this evaluation was that it measured outcomes through three follow-up rounds, up to 24 months post-activity. This enabled the evaluation team to understand what changes are made relatively quickly after the activity ends and which impacts take more time to materialize. Conducting multiple follow-up rounds and measuring effects more than a year post-activity is important to tracing the trajectories of impacts and measuring both short- and longer-term effects. Fourth, this intervention highlights a frequently perceived trade-off between program implementation and impact evaluation. While evaluators feel somewhat constrained by the leeway of implementers, implementers sometimes perceive limits to their ability to adapt on-going interventions iteratively. This tension between implementation and evaluation is understandable, but not irreconcilable if addressed from the design stage. Future projects may explicitly consider this trade-off in the evaluation design by modestly increasing the sample size, so if mid-intervention changes are required the evaluation will not suffer, or by slightly increasing the budget for additional contingencies. Another alternative is to pursue multi-phased programs which allow for a small pilot, outside of the formal RCT, thereby providing a period in which to collaborate, learn, and adapt before ‘locking down’ an intervention model that will necessarily remain consistent over the formal intervention period to maximize learning from the RCT. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 32 ANNEX A: EVALUATION STATEMENT OF WORK Statement of Work: Impact Evaluation of Women’s Leadership in Small and Medium Enterprises (WLSME) Projects I. Activity Description USAID’s WLSME initiative, commissioned by its Office of Gender Equality and Women’s Empowerment in the E3 Bureau, aims to address women’s relative absence in the SME sector in order to promote broad-based economic growth and poverty reduction through higher productivity and efficiency gains. This initiative focuses on reducing three critical barriers by implementing specific measures to produce structural change, so that women may benefit from labor market participation both in the short- and medium-term: (1) agency constraints, which impede adequate accumulation of human capital and managerial capital, and thus limit women’s knowledge and business practices; (2) external constraints, which place gender-specific barriers that limit the presence and success of women entrepreneurs; and (3) relationship constraints, which limit women’s access to information and, as a consequence, reduce the opportunities for women entrepreneurs to build and draw on social capital. In September 2012, USAID awarded three WLSME projects through a competitive process in the Kyrgyz Republic, India, and Peru. The recipients of the cooperative agreements will be referred to as “grantees” throughout this document. Each cooperative agreement has a performance period of three years, starting late September 2012, and has a total USAID-funded budget of around $1.5 to $2 million. The project interventions are scheduled to end around late summer of 2015. FHI 360, through the FIELD-Support Leader with Associates award, completed the research design for all three evaluations, and also completed baseline data collection (including data cleaning and summary analysis) as well as the leadership scale validation as a component of these evaluations for the Kyrgyz Republic and India. FHI 360 also completed part of the baseline collection for Peru, and the Multilateral Investment Fund of the Inter-American Development Bank will fund the completion of that impact evaluation separately. This SOW describes the remaining activities for the impact evaluations of the WLSME projects in the Kyrgyz Republic and India that will be carried out by the E3 Analytics and Evaluation Project to address specific constraints to the development of women’s leaderships in SMEs. This includes revisions to the evaluation design as appropriate and agreed with USAID, conducting endline data collection and analysis, and preparing the final evaluation report, as well as potentially assistance in disseminating evaluation findings. WLSME project descriptions provided by the grantees are summarized below. WLSME India: Cashew Value Chain CARE USA in India, in partnership with its sub-grantee the Loyola Institute of Business Administration (LIBA), is implementing the WLSME project to promote women's leadership and the sustainable growth of 210 cashew-processing micro- and SMEs owned and managed by women in the Panruti block of Tamil Nadu. The project has the following three objectives aimed at addressing the critical barriers related to: (1) human capital gap (Agency), (2) information and social capital gap (Relations), and (3) external constraints (Structures). Objective 1: Strengthen skills, capacities and capabilities of women to own and manage sustainable enterprises. Project activities include skills training for women entrepreneurs, building awareness and Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 33 knowledge of women entrepreneurs on various aspects related to their enterprises, support new SMEs and strengthen existing SMEs as sustainable enterprises, and facilitate access to financial services and government schemes. Objective 2 Facilitate effective relationships among women entrepreneurs and with value chain actors. Project activities include establishing a network of women entrepreneurs in SME involved in cashew processing in the Panruti cluster, facilitating exchange of information and support among women entrepreneurs, and facilitating linkages with key actors in value chain. Objective 3 Promote an enabling environment and more positive attitude and behavior toward women entrepreneurs from family members and other stakeholders. Project activities include facilitating positive attitude and support for women entrepreneurs from other household members, and engaging with service providers in the value chain to strengthen support services to women entrepreneurs. WLSME Kyrgyzstan ACDI/VOCA, in collaboration with its partner organization Bai-Tushum Innovations Fund (BT Fund), is implementing the WLSME project in Kyrgyzstan. The project is operating nationwide and targets women who operate enterprises in priority sectors (garment, tourism, and agro-processing), meet minimum employee and loan size requirements, and are identified as potential high-growth entrepreneurs. Project activities are sequenced, with the main Business Management Training (BMT) reaching about 960 clients. As activities become more tailored and specialized, they progressively focus on fewer women. The second nested component, Market Linkages, consists of stakeholder meetings, trade fairs, workshops on value chains and sub-sectors, semi-annual value chain stakeholder meetings to address the information and social capital gap. The most intensive mentoring and skills development activities will be reserved for the 100 most promising women entrepreneurs. II. Development Hypothesis USAID’s development hypothesis for the WLSME projects is displayed graphically in Figure 1, highlighting each of the intended results of the project and the presumed causal linkages (arrows). The diagram focuses on three parallel constraints that are hypothesized to impede business growth and entrepreneurial leadership, which the WLSME projects aim to address. It also includes the possibility of increased investments in SMEs that may occur as a result of increased access and reduced cost of finance from the BT Fund partnership in Kyrgyzstan, shown under the dotted line. However, this final path of interest is not being delivered exclusively to impact evaluation treatment group members. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 34 Figure 1: WLSME Project Theory of Change III. Existing Performance Information Sources Given the ongoing status of the impact evaluations, USAID has provided the evaluation team with the following relevant project and evaluation documentation: 1. WLSME Kyrgyzstan project documents from ACDI/VOCA • Annual Work Plans • Annual and Quarterly Reports • Market Assessment Report • Gender Assessment Report 2. WLSME India project documents from CARE • Annual Work Plans • Annual and Quarterly Reports • Concept Notes on IEC materials, Joint Counseling Sessions, Change Leader identification process, Role Model selection, and Implementation Steps to avoid contamination for the RCT 3. Evaluation documents from FHI 360 • Evaluation Design Protocols • Baseline Survey Questionnaires and Informed Consent forms • Baseline Reports • Baseline Survey datasets (STATA format) The above list, which is non-exhaustive, highlights the more important sources of information that have been shared with the evaluation team. The following additional documents have not yet been provided to the evaluation team but will be shared as the evaluation progresses: • All future quarterly project reports provided by ACDI/VOCA and CARE • Copies or detailed descriptions of project activities • Attendance spreadsheets pertaining to participation in each project activity Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 35 IV. Evaluation Purpose, Audience, and Intended Use Purpose and Uses There is little existing research to provide convincing empirical evidence about which interventions have the greatest chance of success in terms of creating female-led small and medium enterprises and helping them grow their businesses. Therefore, the purpose of these impact evaluations is to provide a learning, accountability, and decision-making platform by clarifying the most important constraints to women’s business growth and leadership, and thus the most effective means to unleash the potential of women’s entrepreneurship in SMEs. This evidence is expected to be useful to USAID staff to improve future programming in order to better address the barriers to women’s entrepreneurship and to enhance its strategy on how to effectively support the business success of women entrepreneurs at the SME level. This evidence will also be disseminated among practitioners and other governments and donors to contribute to the improvement of women’s economic empowerment in developing countries. USAID is also supporting a core learning network, or closed Community of Practice (CoP), among the three implementing grantees and the evaluation teams. The CoP provides a space to share experiences on project implementation and the impact evaluations, and will serve as a nexus for disseminating results from these evaluations. Audience The primary audience for these evaluations is USAID staff in the E3 Bureau, particularly its Offices of Trade and Regulatory Reform and Gender Equality and Women’s Empowerment. Findings and lessons learned from these evaluations may also be of interest to the business community, governments, donor agencies, and relevant practitioners in the field of women’s economic empowerment in developing countries. V. Evaluation Questions The evaluation questions below were identified by USAID as reflecting Agency learning priorities for addressing women’s relative absence in the SME sector, which is important to the promotion of broad￾based economic growth and poverty reduction in terms of higher productivity and efficiency gains. WLSME Kyrgyzstan 1. Primary Objective (combined impact T vs. C): Compared to participants in the control group, do participants who are randomly assigned to receive the program have higher mean values on the following, post-intervention outcomes: entrepreneurial leadership, business growth, business knowledge/practices, and social/business networks? 2. Secondary Objective (separate estimates across T arms): • Compared to participants who only receive Business Management Trainings, do participants also exposed to Market Linkages have higher mean values on the same set of outcomes listed under the Primary Objective? • Compared to participants who only receive Business Management Trainings, do participants also exposed to Technical Skills/Access to Finance have higher mean values on the same set of outcomes listed under the Primary Objective? Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 36 WLSME India 1. Primary Objective (combined impact T vs. C): Compared to a propensity score weighted sample of participants from the control region, do study participants who were assigned to one of the four treatment components have higher mean values on the following, post-intervention outcomes: entrepreneurial leadership, business growth, business knowledge/practices, and social/business networks? 2. Secondary Objective (separate estimates across T arms): Compared to participants in each of other treatment arms (control, component 1 only, component 2 only, and component 3 only), do participants who were randomly assigned to receive the combination of components 1, 2 & 3 of the intervention have higher mean values on the same set of outcomes listed under the Primary Objective? VI. Gender Considerations USAID’s Gender Policy (Automated Directives System 203.3.1.5) calls upon Agency staff and evaluation teams to examine evaluation questions from a gender perspective and to incorporate gender issues into study designs. As the WLSME projects are focused on women, they only intend to include male perspectives through the External Constraints components in the India project. As such, the evaluation team is not expected to collect data from male stakeholders (either spouses or male value chain actors). Thus, data collected in these evaluations will not be disaggregated by gender and will not look at the differential gender effects of the project components. Nonetheless, the main objective of the WLSME projects is to close the multiple existing gaps between women and men in SMEs. VII. Evaluation Design and Methods Impact Evaluation Design USAID had previously commissioned impact evaluations of the WLSME projects under its FIELD￾Support award with FHI360, under which an evaluation design was developed and baseline data were collected and reported. The FIELD-Support Cooperative Agreement concluded in September 2014, and USAID is now requesting that remaining activities for the impact evaluation be transferred to the E3 Analytics and Evaluation Project. The Evaluation Protocols developed by FHI 360 describe the original research designs. The evaluation team should adhere to these protocols as much as possible, and deviations should only occur in consultation with the grantees and with USAID approval. The team should also note that each project has fairly comprehensive performance monitoring systems in place as well, but these were designed to minimize overlap with the evaluation indicators. The two impact evaluation designs consist of a rigorous mixed-methods approach, with a randomized assignment to treatment complemented by qualitative data (including focus group discussions and in￾depth interviews). The experimental designs involve collecting data from treatment groups and a control (counterfactual) group at multiple points in time, in order to make causal inferences with adequate statistical power and to document the size of the intervention’s effects. Data Collection Methods Considering that baseline data collection and reporting has already occurred, endline survey research under the E3 Analytics and Evaluation Project for this evaluation will need to be conducted following the end of the respective project, in three rounds over two years. In addition, qualitative research such as Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 37 focus group discussions will be conducted as soon as the projects end to access general intervention processes and content. To gain a deeper understanding of if and how the interventions improved participants’ lives and businesses, in-depth interviews will be conducted after the 12-month follow-up survey is implemented and the study data analyzed. In responding to this SOW, the evaluation team should identify where sampling and other types of selection procedures will be used to identify the individuals from whom qualitative data will be collected, and to explain those methods and why they are appropriate. VIII. Data Analysis Methods In its response to this SOW, the evaluation team should indicate and justify its choices for sequencing the collection of quantitative and qualitative data. For example, focus group discussions may precede further quantitative research to inform survey questions or may follow quantitative research to help explain survey findings; alternatively, these lines of data may be collected and analyzed in parallel and only synthesized once data from all sources are available. The evaluation design should also explain what statistical tests will be conducted on data collected to address all evaluation questions, how qualitative data will be analyzed, and whether that analysis will allow the evaluation team to transform some data obtained from qualitative into quantitative form. IX. Strengths and Limitations The evaluation designs of the two WLSME projects reflect a rigorous approach to address the proposed evaluation questions and to contribute to the global knowledge on women’s entrepreneurship in the SME space. One key contribution is that both projects were specifically designed to test different pathways through which barriers affect women’s economic outcomes and business success, which is a great contribution to the evidence base on women’s entrepreneurship. Another strength of the envisioned evaluation designs is the use of multiple follow-up rounds and measuring outcomes over two years after the end of the project. This ensures that the evaluation will not miss potential impacts from the projects. However, the evaluation will need to consider and address several limitations related to statistical analysis and attribution. Anticipated challenges, along with how they may be addressed by the evaluation team, are described briefly below. • Small sample sizes reduce the ability to detect statistically significant differences between the groups. The original evaluation designs attempted to compensate for this by including multiple time points in the data collection and analysis plans. • Indirect contamination across treatment arms and control groups may be present since eligible beneficiaries assigned to different groups reside in the same geographic areas. While WLSME staff attempt to limit direct contamination through the careful delivery of project activities, women may still pass on the information through word of mouth. • Attrition presents another potential challenge, particularly since the intervention and data collection will take place over a three-year period. To help address this, the grantees have asked participants to notify the WLSME contact person if they move to other place, change contact information, or decide to leave the project. WLSME staff will call all women in the project (controls and treatments) on a regular basis, as time and resources permit, and update the database in case their contact information has changed, they have moved, or they decided to leave the project. • Selection bias, given that not all project components were randomized, also poses a limitation in answering the evaluation questions, specifically when comparing across treatment groups in Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 38 Kyrgyzstan and when comparing treatment versus control groups in India. While the addition of new statistical analysis methods can provide support for the findings, the bias still remains. X. Evaluation Deliverables It is anticipated that the evaluation team will be responsible for the deliverables listed in Table 1 for each evaluation. A final list of proposed deliverables and due dates will be included in the Evaluation Design Proposal to be prepared for each evaluation for USAID’s approval. Table 1: Preliminary Deliverables and Schedule for WLSME Impact Evaluations Deliverable Estimated Due Date 1. Evaluation Concept Paper (both evaluations), including methodological options to improve the evaluation design, and associated methods to the extent that options exist at this level. o/a 30 days from client approval of SOW 2. Evaluation Design Proposals for each evaluation, including description of the evaluation methodology, drafts of data collection instruments and a sampling plan, as relevant o/a 30 days from client approval to move forward with preparing Evaluation Design Proposal 3. Intermediate Reports covering follow-up data collection rounds, FGDs, and IDIs o/a 60 days from completion of field research 4. Draft Impact Evaluation Report for each evaluation including key findings, conclusions and recommendations for USAID and its grantees o/a 60 days from completion of field research 5. Oral Presentation on preliminary findings, conclusions and recommendations from quantitative and qualitative data collection o/a 60 days from completion of field research 6. Final Impact Evaluation Report for each evaluation including evaluation data sets, codebooks, etc. o/a 21 days following receipt of USAID feedback on Draft Evaluation Report 7. Debrief for grantees and partners (tentative) As agreed following USAID approval of Final Evaluation Report All documents and reports will be provided electronically to USAID no later than the dates indicated in the approved Evaluation Design Proposal. The format of the evaluation report should follow USAID guidelines set forth in the USAID Evaluation Report Template (http://usaidlearninglab.org/library/evaluation-report-template) and the How-To Note on Preparing Evaluation Reports (http://usaidlearninglab.org/library/how-note-preparing-evaluation-reports). XI. Team Composition Each evaluation will be delivered by a core evaluation team supported by technical and administrative U.S.-based evaluation and project management specialists. It is anticipated that the core evaluation team will be composed of a Principal Investigator who is an Evaluation Specialist, a Local Qualitative Researcher, and a Research Assistant. A survey research firm may also be contracted to support endline data collection. A final team composition, including proposed evaluation team members and their CVs, will be included in the Evaluation Design Proposal. Each team member will be required to provide a Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 39 signed statement attesting that they have no conflict of interest, or describe any potential existing conflict of interest, and will be made available at USAID’s request. Principal Investigator/Evaluation Specialist The Principal Investigator must have a doctoral degree in a relevant social science and at least three years’ experience conducting rigorous, experimental research in developing countries. The specialist should be able to demonstrate successful collaboration and leadership in cross-cultural contexts. Professional experience in research on gender equality and women’s economic empowerment in developing countries and in the countries/region being studied is preferred. Proficiency in any of the relevant languages (Tamil, Kyrgyz, Russian) is a plus. The Principal Investigator will also possess skills in management, supervision, leadership and networking, and ability to work creatively towards solutions. Local Qualitative Researcher The evaluation team will also include a Local Qualitative Researcher who will complement the Principal Investigator in qualitative research and will lead the focus group discussions and in-depth interviews. The Researcher must have an advanced degree in a relevant social science, such as economics, statistics or sociology, and at least three years’ experience conducting rigorous research in economic development or gender in developing countries. The specialist should be able to demonstrate successful collaboration and leadership in cross-cultural contexts. The specialist must also be proficient in any of the relevant languages (Tamil, Kyrgyz, and Russian). Research Assistant The Research Assistant should ideally be pursuing or have completed a graduate degree in a relevant social science such as economics or statistics, and ideally be trained in quantitative and qualitative random sampling and data collection methods. Proficiency in any of the relevant languages (Tamil, Kyrgyz, Russian) is a plus. Survey Research Firm Competent and experienced research firms will be selected to conduct endline data collection for the household survey, including data entry, for each evaluation. The firms will also support the piloting and translation (into Kyrgyz and Russian) of the survey instrument, as well as, transcription of focus group discussions and in-depth interviews. XII. USAID Participation While regular communication between the evaluation team and the designated USAID Activity Manager for this evaluation will be essential, USAID does not anticipate that any of its staff will serve as a full time team member on these evaluations, nor is it currently expected that USAID staff will join field data collection visits to project sites. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 40 XIII. Scheduling and Logistics The following tables provide the anticipated timeframe for evaluation activities and deliverables. Table 2: Estimated Timeline for WLSME Kyrgyzstan Tasks FY15 FY16 FY17 FY18 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Finalize instruments and protocols for FGDs & survey WLSME Kyrgyzstan Implementation Conduct FGDs (n=6) Follow-up Round 1 (at project end) FGD Summary Report of methodology and findings Brief summary analysis and data for follow-up 1 Follow-up Round 2 (at 12 months) Brief summary analysis and data for follow-up 2 Instruments and protocols for IDIs Conduct IDIs (n=40) IDI summary report of methodology and findings Follow-up Round 3 (at 24 months) Brief summary analysis and data for follow-up 3 Draft Endline Report Draft Evaluation Report Final Evaluation Report Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 41 Table 3: Estimated Timeline for WLSME India Tasks FY15 FY16 FY17 FY18 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 WLSME India Implementation (with extension) Instruments and protocols for FGDs and survey Conduct FGDs (n=6) FGD summary report of methodology and findings Follow-up Round 1 (at 3 months) Brief summary analysis and data for follow-up 1 Follow-up Round 2 (at 6 months) Brief summary analysis and data for follow-up 2 Follow-up Round 3 (at 12 months) Brief summary analysis and data for follow-up 3 Instruments and protocols for IDIs Conduct IDIs (n=40) Follow-up Round 4 (at 18 months) IDI summary report of methodology and findings Brief summary analysis and data for follow-up 4 Follow-up Round 5 (at 24 months) Brief summary analysis and data for follow-up 5 Draft Endline Report Draft Evaluation Report Final Evaluation Report The evaluation team will be responsible for procuring all logistical needs such as work space, transportation, printing, translation, and any other forms of communication. USAID will offer some assistance in providing introductions to partners and key stakeholders as needed, and will ensure the provision of data and supporting documents as possible. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 42 XIV. Reporting Requirements All members of the evaluation team will be provided with USAID’s mandatory statement of the evaluation standards they are expected to meet, shown in the following text box below, along with USAID’s conflict of interest statement that they sign and return to the E3 Analytics and Evaluation Project Home Office where necessary before field work starts. Data Management Plan The storage and transfer of data will adhere to the requirements laid out in ADS 579.10 The E3 Analytics and Evaluation Project should also follow Institutional Review Board (IRB) guidance on data security and confidentiality. All data collected at the field level should be managed by the evaluation team and overseen by the E3 Analytics and Evaluation Project Home Office team. Data should be filed in the appropriate format and processed in parsimonious, machine-readable format as they are collected. Final datasets are expected to be submitted to USAID in a format consistent with ADS 579. Metadata should be generated in the form of codebooks and data summaries as necessary. To ensure transparency and replicability, all data should be submitted as annotated datasets clearly defined with codebooks and annotated analysis of files. XV. Budget The evaluation team will propose a notional budget in its Concept Paper for these evaluations, including cost implications of the methodological options proposed. Full detailed budgets will then be prepared and included in each Evaluation Design Proposal for USAID’s approval. 10 See http://www.usaid.gov/sites/default/files/documents/1868/579.pdf USAID EVALUATION POLICY (APPENDIX 1) CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT • The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. • Evaluation reports shall address all evaluation questions included in the scope of work. • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 43 ANNEX B: LITERATURE REVIEW Existing approaches to supporting growth-oriented women entrepreneurs have been heterogeneous in their design and delivery, although they have provided some suggestive evidence on the key issues that should be taken into account (Cirera and Qasim 2014). Recent efforts recognize this and have attempted to provide a more unified effort on how to best insert women into the productive process while helping to maximize their contributions to the well-being of societies (Buvinic et al. 2013). In the context of the current empirical research on the barriers faced by women entrepreneurs and based on the existing literature, the WLSME initiative identified (1) agency limitations, (2) external constraints, and 3) lack of relationships as critical issues that should receive support to remove crippling limitations to women’s productive advancement and contribution to the economy. Agency Constraints Whereas most academic and development policy discussions about female entrepreneurs focus on credit constraints, many studies and discussions assume that entrepreneurs manage their businesses optimally. In fact, human capital is treated as fixed with a focus on the process of infusion of financial capital into micro-enterprises, not human or managerial capital, but assuming that entrepreneurs have the latter in optimal amounts (Karlan and Valdivia 2012). Clearly, this is not necessarily the case, as the relatively poor among the self-employed rarely have any formal training in business skills. In particular, it has been argued that one must develop “managerial capital” to help entrepreneurs affect their firm’s business practices, including improving strategic and operational decisions and productivity of factors of production by helping to use them more efficiently (Bruhn et al. 2012). Managerial capital appears to be a fundamental constraint for microenterprise development, as business training may enable entrepreneurs to better identify profitable business opportunities, leading to changes in business practices and ultimately to higher sales, profits and happiness (Berge et al. 2012). The Kyrgyzstan WLSME activity tries to reduce agency constraints by improving human capital of female entrepreneurs with particular emphasis on their managerial capabilities. Thus, the key question asked is the following: is lack of managerial capital a first-order impediment to firm results, profitability, and growth? In fact, it has been shown in other studies that small-firm entrepreneurs are constrained in the acquisition of these skills, especially if they require formal training (Caselli and Gennaioli 2005). In particular, the design of the treatment arms in this activity follows a systematic pattern that tries to condense the approaches by a growing number of microfinance organizations attempting to build the human and managerial capital of micro-entrepreneur activities, which previously have been vastly idiosyncratic and heterogeneous and, as a consequence, have provided limited external validity. This is perhaps the reason why the current literature on human and managerial capital shows a mixed record. For instance, Karlan and Valdivia (2012) and Cole et al (2011) show that basic microenterprise training seems to affect the command of accounting practices for microenterprises, but has limited or no effect on actual firm outcomes and performance, including profits and sales. Similarly, Bruhn and Zia (2013) and Giné and Mansuri (2014) find that training in managerial capital leads to improvements in business practices, but has only limited effect on business performance and sales. On the other hand, Drexler et al, (2012) show that training activities increase in impact if they are targeted to the owner’s level, as training has significant impact on real outcomes for micro-entrepreneurs who have low educational attainment and poor business practices prior to the intervention. Along the same lines, Field et al (2010) find positive treatment effects on upper- caste Hindus, but no such effects on either lower-caste Hindus or Muslims. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 44 External Constraints In spite of the importance of human capital and managerial capital, a consistent finding in recent academic research is that business training is vastly more effective for male entrepreneurs than for female entrepreneurs (Berge et al. 2012; de Mel et al. 2014; Gine and Mansuri 2014). The differences are striking. Even though female entrepreneurs benefit from training in terms of business knowledge, researchers are unable to find a positive effect on their business-related outcomes. In fact, whereas it has been found that there are no differences in business knowledge between males and females, the former report better business practices, lower business failures, higher investment, and even more household expenditures (Gine and Mansuri 2014). These findings point to the need for more comprehensive measures to promote the businesses of female entrepreneurs as any positive effect of the business training is contingent on gender. Furthermore, it appears that deeper factors than lack of business knowledge seem to constrain the development of female-owned microenterprises. In fact, female and male entrepreneurs fundamentally differ in terms of mindset and household constraints, which may indicate that more comprehensive measures are necessary to promote development among female entrepreneurs, paying greater attention to their motivation for being involved in business activities and to external constraints that may limit their opportunities (Berge et al. 2012). In the context above, the WLSME activities aim to remove external constraints by promoting an enabling environment and a more positive attitude toward women entrepreneurs, with particular emphasis on social norms, which is the most plausible explanation for the gender differences about the role of women in the workplace. Relationship Constraints Despite the fact that informal social mechanisms, such as word of mouth, may help reduce external constraints, they have limits. For instance, a significant share of women say that their (male) spouses are responsible for most of their business decisions, suggesting that female businesses show no improvement because women have little decision-making control. In fact, female entrepreneurs are less willing to share income information with a spouse than male entrepreneurs are, which may suggest that female entrepreneurs are taxed by their husbands and thus may have less to gain from expanding their businesses (Berge et al. 2012). In this context, it is important to develop specific formal direct channels where women entrepreneurs are able to interact with all of the actors in the productive process. The fact that women are also less willing than males to compete suggests that they, to a lesser extent, have an entrepreneurial mindset focused on business competition and growth (Berge et al. 2012). The WSLME initiative aims to reduce information and social gaps in the productive process of women entrepreneurs by facilitating effective relationships between women and the value chain actors and, in particular, by increasing cohesion in the productive process. In addition, it is expected that the components included in the activity can help increase the sense of empowerment in women entrepreneurs, particularly those with specific leadership skills. This occurs through taking advantage of a combination of women’s increased economic activity and control over income resulting from access to a larger network (Mayoux 2001; Kulkani 2011). In particular, these activities can help enhance the status of women entrepreneurs within the community, which are reinforced by the formation of the networks that are part of this activity. This is consistent with an empowerment paradigm that advocates for explicit strategies that support women’s ability to protect their individual and collective gender interests (Mayoux 2001). Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 45 ANNEX C: QUALITATIVE DATA COLLECTION SAMPLING AND PROCESS Focus Group Discussions In conjunction with the survey conducted for this impact evaluation, six FGDs were also held with WLSME participants as the activity ended. The FGDs were held across five regions and separated by sector, with three FGDs focused on participants from the tourism sector and three with participants from the garment sector. Recruitment accounted for a no-show rate of 50 percent and variability of participation across the activity components. The FGDs varied in size, from 9 to 14 participants, and comprised beneficiaries who had participated in the different WLSME activity components (see Table 14). The heterogeneity of exposure to the different activity components provided insights on the value of partial activity attendance relative to full activity attendance. All of the groups discussed a consistent but open-ended series of questions related to the results of the WLSME activity. These lines of questioning generally corresponded to the four sets of outcome variables; however, the format and venue of the FGDs was intended to provide more opportunities for follow-up questioning and open￾ended participant response. TABLE 14: FOCUS GROUP SAMPLE DISTRIBUTION Region Sector BMT Only BMT + Market Linkages BMT+ML+TS/AF Total Issyk-Kul 1 Tourism 0 4 5 9 Issyk-Kul 2 Tourism 0 8 2 10 Naryn Tourism 3 7 4 14 Bishkek Garment 3 0 9 12 Jalal-Abad Garment 2 3 4 12 Osh Garment 4 5 9 13 Total 12 27 31 70 The FGDs were held in a convenient and central location within each region. Participants’ transportation expenses were covered and light refreshments were provided. Prior to the start of each FGD, each participant received and signed an informed consent form ensuring confidentiality and voluntary participation. The FGDs were led by the local qualitative researcher from the E3 Analytics and Evaluation Project with logistical support from M-Vector, and lasted approximated 90 minutes. The FGDs were recorded as audio files, and the facilitators took notes throughout the meetings, but in deference to the cultural context, the FGDs were not videotaped. As a result, although participant identity was tracked to the best of the facilitator’s ability, in some cases it was not possible to ascertain the identity of the individual speakers at certain points in the audio recordings. As a result, precise numbers for individual respondents on specific topics are not always possible, and nonverbal affirmations cannot be identified. In the cases where the transcript is not clear as to whether a single respondent is voicing an opinion twice, or two separate respondents are voicing similar sentiments concurrently, the more conservative numeric estimate is the one cited (e.g., “one respondent voiced an opinion, and at least two other members of the FGDs agreed.”). The audio files were transcribed verbatim into Russian, and then translated into English. Translations were audited independently by another member of the evaluation team. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 46 In-Depth Interviews Forty IDIs were planned to be conducted with participants who demonstrated significant change in the positive direction for each of the four outcomes and who had at least completed the BMT component (see Table 15). Selection of participants was based on change in the outcome measures. Composite scores were created for each of the four outcome categories. Each score consisted of positive change, between baseline and the second follow-up round, on several variables under each category. Respondents were ranked by their score, and the top 10 participants per outcome category were selected. Where more than ten participants existed with an equal score, a simple random selection process was employed for that group. TABLE 15: IN-DEPTH INTERVIEWS SAMPLE DISTRIBUTION Outcome Category Sector (Garment / Tourism) BMT Only BMT + ML BMT + ML + TS/AF Total Business Growth 6 / 4 7 1 2 10 Entrepreneurial Leadership 7 / 3 5 0 5 10 Networks 8 / 1 2 1 6 9 Business Knowledge & Practices 5 / 8 5 2 6 13 Total 26 / 16 19 4 19 42 Potential participants were contacted via phone to gauge their availability and interest in participating in the interviews. Respondents who were unavailable or not interested in being interview were replaced the next ranked respondents. Once interview respondents confirmed a day and time for participation, a local qualitative researcher from the E3 Analytics and Evaluation Project traveled to the respondent’s home or business to conduct the interview. Prior to the start of the interview, each participant received an informed consent form which explained the purpose of the interview and key elements such as voluntary participation, confidentiality, benefits, and time commitment. Each participant returned a signed copy of this form before the start of the interview. Interviews lasted between 30 to 45 minutes. A small gift (worth USD $10) was given at the end of the interview as a token of appreciation. The interviewer took brief notes to generate a summarized set of observation notes. In addition, all IDIs were audio-recorded, transcribed verbatim, and then translated into English for data analysis by M￾Vector. Translations were audited independently by another member of the evaluation team. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 47 ANNEX D: SURVEY DATA COLLECTION AND QUALITY ASSURANCE Data Collection Process A local survey partner subcontracted by Management Systems International, M-Vector Research and Consulting, collected quantitative data for both follow-up rounds, with close collaboration, supervision, and quality oversight provided by the evaluation team. The surveys were administered face to face, at the participants’ places of business, home, or other location that was convenient for them. The survey took 40 to 60 minutes to complete. The enumerators obtained oral informed consent from each participant, prior to the start of the survey, to confirm that she was willing to participate. M-Vector provided a mobile phone card with 200 Soms (equivalent of USD $2.70) to respondents after the survey was completed each round as a token of appreciation. Prior to the start of data collection, enumerators were trained over the course of three days. Most of the enumerator team was consistent throughout the three follow-up rounds so the field team was very knowledgeable about the survey instrument improving the data collection consistency across the three rounds. Data quality assurance processes were put in place internally by M-Vector, and also independently by the evaluation team. Baseline data collection conducted by FHI 360 started in July 2013 on a rolling basis over a year as participants applied and were randomly enrolled in batches into the activity. The first follow-up survey conducted by the E3 Analytics and Evaluation Project took place at the end of the activity, between August and October 2015, with an 82 percent response rate. Non-response rates for this round were evenly distributed between treatment and control groups; however, within the treatment group, those who had dropped out of the activity (i.e., did not start or complete the BMT component) were more likely to not respond to the follow-up survey. Among those who did not participate in the follow-up survey, the main reasons given included moved or changed telephone number so could not be reached (9 percent), refusal to participate (6 percent), unavailable after multiple rescheduled appointments (2 percent), passed away (1 percent), and moved abroad (1 percent). The second follow-up survey, also conducted by the E3 Analytics and Evaluation Project, took place 12 months after the end of the WLSME activity, between August and October 2016, with an 84 percent response rate. Similarly to the previous round, non-response rates were evenly distributed between treatment and control groups; however, within the treatment group, those who had dropped out of the activity (i.e., did not start or complete the BMT component) were more likely to not respond to the second follow-up survey. Among those who did not participate in the second follow-up survey, the main reasons given included moved or changed telephone number so could not be reached (9 percent), refusal to participate (4 percent), unavailable after multiple rescheduled appointments (2 percent), and moved abroad (1 percent). The third follow-up survey, took place 24 months after the end of the WLSME activity, between August and October 2017, with a 77 percent response rate. Among those who did not participate in the third follow-up survey, the main reasons for non-response in the survey included lack of contact due to a change in contact information, business closure, and refusal to participate. Data Quality Assurance M-Vector followed standard operating procedures for data collection, including verification procedures conducted both at the site and at headquarters in Bishkek, double entry of survey data, and data query. The survey firm employed the following set of quality control procedures during each data collection round: Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 48 • The field manager and supervisors constantly managed the workflow to ensure all enumerators followed the agreed timeline and procedures. Field managers were in contact with the evaluation team to find proper solutions to any unexpected challenges. • Each supervisor reviewed all completed questionnaires on site, including reading through all questions and answers in the questionnaire to ensure that there were no blanks, skip mistakes, logical inconsistencies, etc. If the supervisor noticed missed questions, skipped questions, or unclear writing, questionnaires were marked and returned to interviewers. • Supervisors accompanied at least 10 percent of the interviews conducted. • Completed surveys were sent to the survey firm’s headquarters on a weekly basis, where an inspector reviewed each survey for completeness and adequacy prior to data entry. • Double data entry was performed by two individuals, and the second data entry was done without knowledge or cross reference to the first data entry. Any discrepancies between the two entries of data were resolved by a third person. • Datasets and progress reports were submitted to the evaluation team on a weekly basis. The progress report included number of contact attempts for pending surveys and reasons for pending status. The evaluation team provided additional oversight and monitoring during each data collection round: • Accompaniments of enumerators during interviews at regular intervals. The local coordinator on the evaluation team observed the enumerators’ familiarity with and comprehension of the questionnaire and clarity in asking questions. • The local coordinator also conducted back-checks on 10 percent of completed surveys for each round. Surveys to be back checked were selected randomly and stratified by enumerator to ensure each one was checked on an equal basis. During the back-check call, several validation questions were asked, including interview location, age, household size, type of business, receipt of mobile phone card, and friendliness of the enumerator. Only a few minor discrepancies were found. No back check resulted in significant variance from the reported data. • From the remaining surveys that were not back checked, a random sample (five percent) of the scanned paper surveys were compared with the database. This audit showed no meaningful data entry errors; minor discrepancies were fixed. • Each week, the evaluation team conducted additional checks to compare each enumerator’s average performance to the total sample averages in terms of interview length, number of completed codes, number of “do not knows,” scale usage, section skips, and ranges of numerical values. No significant outliers were found. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 49 ANNEX E: EMPIRICAL METHODS ANCOVA The evaluation team’s preferred method, analysis of covariance (ANCOVA), is a statistical method based on variance, multiple regression, and correlation analysis. It is used to increase the precision of comparison between groups and reduce the probability of Type II errors, i.e., when a false null hypothesis is not rejected (Miller and Chapman 2001; Huck 2012). ANCOVA is thought to improve statistical power and control as long as the relationship between the dependent variable and the covariate within each group is linear and parallel, the covariate is unaffected by other independent variables, and if data are collected under a completely randomized design and before any treatment is applied (Schwarz 2015; Huck 2012). When complying with these assumptions, ANCOVA can have a higher explanatory power than DID only if autocorrelation is low. In the context of this evaluation, ANCOVA takes advantage of the low autocorrelation of certain outcome variables in this study, such as business profits and sales, to improve power beyond what a DID approach can attain with the same sample size. Baseline data for these outcome measures have little predictive power for future outcomes, so it is inefficient to fully correct for baseline imbalances between treatment and control groups using DID. Instead, an ANCOVA model can adjust the degree of correction for baseline difference in means according to the degree of correlation between past and future outcomes actually observed in the data (McKenzie 2012). The ANCOVA specification used for estimations in this evaluation used information from the first and second follow-up survey, as well as the baseline: 𝑌𝑖,𝑡 = 𝛽0 + 𝛽1𝐷𝑖 + 𝛽2𝑌𝑖,𝑡−1 + 𝛽3𝑌𝑖,𝑡−2 + 𝛽4𝑌𝑖,𝑡−3 + 𝛿𝑋𝑖 + 𝜀𝑖,𝑡 In this case, 𝑌𝑖,𝑡−1 denotes the baseline value of the outcome variable, 𝑌𝑖,𝑡−2 is the first follow-up value of the outcome variable, 𝑌𝑖,𝑡−3 is the second follow-up value of the outcome variable, and 𝛽1 is the ANCOVA treatment effect. 𝑋𝑖 is a matrix of relevant covariates for identification to increase the efficiency of 𝛽1. Specifically, 𝑋𝑖 contains the following variables: age, marital status, higher education, business ownership, number of full-time workers from the household and also non-family workers, participation in previous training or seminars, and number of children under 18. The evaluation team also included sector and region-fixed effects. Difference-in-Differences The difference-in-differences (DID) estimator is one of the most popular methodologies for applied research in economics. To answer the hypotheses, DID estimates causal relationships among variables by comparing the difference in outcomes before and after an intervention between groups of beneficiaries and nonparticipants (Bertrand et al. 2004). The first “difference” in this method is the difference before the intervention (baseline) and after the intervention (endline). The second “difference” is between the beneficiary group (treatment) and nonparticipant group (control). Thus, two rounds of data are required. The main advantage of this approach is that in takes into account both observed and unobserved factors which reduces endogeneity problems and provides a tractable way to incorporate both types of variables in the analysis of the effects activity interventions have over beneficiaries (Bertrand et al. 2004; Khander et al. 2010). However, this method only remains unbiased as long as interventions are random, and the difference between treatment and control groups’ outcomes are time-invariant. This means that to avoid any over- or underestimation of an activity’s effects, it is crucial to ensure that both treatment Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 50 and control groups are similar (Ravallion et al. 2005; Khander et al. 2010). For this evaluation, the estimate of the overall activity benefits from the randomized assignment of the intervention; however, the estimates of the components do not. Unlike single differences-in-means, the DID method can be generalized to consider various periods in time. FIGURE 5: DIFFERENCE-IN-DIFFERENCES METHOD Source: Khandker, Koolwal & Samad (2010) Within this framework and in order to estimate the impacts of the WLSME activity, for each outcome of interest the evaluation team employed the DID specification proposed by Bernal and Peña (2014)11 as follows: Δ𝑌𝑖 = 𝛽𝑜 + 𝛽1𝐷𝑖 + 𝛿𝑋𝑖 + 𝜀𝑖 Here, Δ𝑌𝑖 is the first difference in the outcome variable between the baseline and the third round, 𝐷𝑖 is the treatment status dummy that remains unchanged between periods, and 𝛽1 is the treatment effect. The 𝑋𝑖 covariates matrix are evaluated in the baseline period and contains the same variable as those used in the ANCOVA model. This specification has the advantage of applying an analogous approach to a randomized controlled trial, as it focuses on the changes of one key variable of interest between periods, which is not the case of standard DID approaches, where period variations of the other explanatory variables also impact the outcome variable. The evaluation team also employed a DID panel model, following Imbens and Wooldridge (2007), to account for the longitudinal characteristic of the data and generalize the DID specification for several time periods. With many time periods, the DID panel estimation is specified as: 𝑌𝑖,𝑡 = 𝜆𝑡 + 𝑐𝑖 + 𝛽1𝐷𝑖,𝑡 + 𝛿𝑋𝑖,𝑡 + 𝜀𝑖,𝑡 ,𝑡 = 1, … , 𝑇 Where the same individuals are compared at baseline and each follow-up round, 𝜆𝑡 denotes the time fixed effects, 𝑐𝑖 are the individual fixed effects, and 𝑋𝑖 is a matrix of relevant covariates for each period and individual. 𝛽1 is the treatment effect. In other words, this technique uses all three follow-up surveys as well as the baseline information. The results for both DID specifications are very similar. The results presented in Annex F correspond to the DID panel model. 11 The Bernal and Peña (2014) approach allows the evaluation team to use the baseline and only one of the follow-up rounds. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 51 ANNEX F: FULL RESULTS Primary Question: Overall Impact (ANCOVA model) Row Outcome Variable Follow-up 1 Treatment Effect (std. error) Follow-up 2 Treatment Effect (std. error) Follow-up 3 Treatment Effect (std. error) Business Growth 1 Average sales on a good month (in ranges)+ 0.040 0.168 -0.067 (0.137) (0.097) (0.125) 2 Average sales on a bad month (in ranges)+ -0.069 0.060 0.084 (0.110) (0.129) (0.096) 3 Average sales on an average month (in ranges)+ -0.039 0.154 -0.030 (0.117) (0.111) (0.127) 4 Average profit on a good month (in ranges)+ 0.370 0.101 0.023 (0.228) (0.142) (0.117) 5 Average profit on a bad month (in ranges)+ 0.044 0.068 0.046 (0.095) (0.090) (0.168) 6 Average profit on an average month (in ranges)+ 0.081 0.052 -0.009 (0.156) (0.122) (0.125) 7 Number of good months in the last year -0.079 0.074 -0.214 (0.204) (0.103) (0.125) 8 Number of bad months in the last year 0.191 -0.060 -0.088 (0.119) (0.082) (0.109) 9 Sales in the last 12 months (Soms) -6,919 13,583 78,620 (16,731) (23,713) (45,191) 10 Likelihood of having a bank account for business only -0.038 -0.028 0.012 (0.032) (0.036) (0.022) 11 Number of household members who have worked in business in the last 12 months 0.0342 0.010 0.176** (0.105) (0.040) (0.072) 12 Number of paid household workers 0.0921 0.319** 0.275** (0.165) (0.122) (0.101) 13 Number of non-household people who have worked in business in the last 12 months -0.297 0.006 0.130 (0.586) (0.153) (0.298) 14 Number of paid non-household workers -0.102 -0.109 0.184 (0.588) (0.257) (0.500) 15 SME threshold (5 to 250 total workers) 0.007 0.000 0.046 (0.015) (0.022) (0.037) 16 SME threshold (5 to 250 paid workers) 0.019 0.013 0.202*** (0.037) (0.052) (0.045) 17 Number of months per year spent working in the business owned or managed 0.033 -0.282 -0.271* (0.215) (0.233) (0.121) 18 Number of days per week spent working in the business owned or managed 0.179 -0.104 0.053 (0.111) (0.092) (0.035) Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 52 Row Outcome Variable Follow-up 1 Treatment Effect (std. error) Follow-up 2 Treatment Effect (std. error) Follow-up 3 Treatment Effect (std. error) 19 Number of hours per day spent working in the business owned or managed 0.148 0.135 0.087 (0.217) (0.290) (0.147) 20 Likelihood of purchasing raw materials, goods, or equipment for business with a loan in the last 12 months 0.110** -0.031 -0.087 (0.050) (0.049) (0.060) 21 Likelihood of applying for a loan from a financial institution in the last 12 months 0.014 -0.026 -0.035 (0.039) (0.040) (0.041) 22 Likelihood of loan approval -0.026 -0.095 -0.137 (0.087) (0.092) (0.590) Entrepreneurial Leadership 1 Likelihood that prefers to work as an employee in a business instead of managing/owning one -0.017 0.006 0.015 (0.027) (0.037) (0.030) 2 Likelihood that people ask me for business advice (very) often -0.043 -0.007 -0.017 (0.039) (0.019) (0.022) 3 Likelihood that I (with my partner/spouse or another household member) am in charge of general business planning decisions -0.007 -0.006 -0.020 (0.032) (0.047) (0.021) 4 Likelihood that I (with my partner/spouse or another household member) decide what inputs to buy for production 0.027 0.007 -0.042 (0.036) (0.049) (0.028) 5 Likelihood that I (with my partner/spouse or another household member) am in charge of sales and client relations -0.024 0.018 0.002 (0.033) (0.037) (0.018) 6 Likelihood that I (with my partner/spouse or another household member) decide if I should apply for a loan -0.019 -0.035 0.082** (0.038) (0.037) (0.031) 7 Likelihood that I (with my partner/spouse or another household member) decide my own singular wage 0.005 0.050 -0.095** (0.034) (0.036) (0.036) 8 Likelihood that I (with my partner/spouse or another household member) decide what type of work I will do 0.021 0.006 -0.029 (0.032) (0.035) (0.021) 9 Likelihood that I (with my partner/spouse or another household member) am in charge of marketing and advertising decisions 0.031 -0.039 0.087** (0.037) (0.060) (0.027) 10 Likelihood that I (with my partner/spouse or another household member) am in charge of staffing of business decisions 0.045 -0.040 0.010 (0.036) (0.033) (0.036) 11 Likelihood that often (or always) makes general business planning decisions without consulting anyone else 0.026 -0.046 0.007 (0.038) (0.041) (0.043) 12 Likelihood that often (or always) decides what inputs to buy for production without consulting anyone else 0.033 0.026 -0.009 (0.039) (0.076) (0.049) Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 53 Row Outcome Variable Follow-up 1 Treatment Effect (std. error) Follow-up 2 Treatment Effect (std. error) Follow-up 3 Treatment Effect (std. error) 13 Likelihood that often (or always) manages sales and client relations without consulting anyone else 0.072* 0.015 -0.008 (0.039) (0.030) (0.033) 14 Likelihood that often (or always) decides whether to apply for a loan without consulting anyone else -0.011 0.051 -0.027 (0.044) (0.031) (0.064) 15 Likelihood that often (or always) decides own singular wage without consulting anyone else 0.026 0.042 0.016 (0.043) (0.032) (0.025) 16 Likelihood that often (or always) decides what type of work to do without consulting anyone else 0.052 -0.029 -0.019 (0.039) (0.033) (0.023) 17 Likelihood that often (or always) makes marketing and advertising decisions without consulting anyone else 0.054 0.014 -0.054 (0.044) (0.051) (0.042) 18 Likelihood that often (or always) makes staffing of business decisions without consulting anyone else 0.036 -0.019 0.032 (0.041) (0.050) (0.079) 19 Likelihood that (strongly) agrees that “women should do what men say” -0.028 -0.032 -0.021 (0.036) (0.047) (0.032) 20 Likelihood that (strongly) agrees that “women must share their income with their husbands” 0.014 -0.008 -0.040 (0.030) (0.024) (0.024) 21 Likelihood that (strongly) agrees that “it is OK if men abandon women if they wish to” -0.037 0.016 -0.008 (0.030) (0.027) (0.028) 22 Likelihood that (strongly) agrees that “it is OK if men chide women because they went out without any permission” -0.065* -0.017 0.005 (0.036) (0.028) (0.049) 23 Likelihood that (strongly) agrees that “it is OK if men chide women if they do not take care of children” 0.038 -0.015 0.025 (0.028) (0.024) (0.031) 24 Likelihood that (strongly) agrees that “the role of women is to earn money and take care of her family” -0.065* -0.019 -0.034 (0.038) (0.034) (0.038) 25 Likelihood that (strongly) agrees that “a mother who works can establish a relationship as warm and solid with her children as a mother who does not work” 0.017 -0.019 0.029 (0.018) (0.024) (0.019) 26 Likelihood (strongly) agrees that “father’s and mother’s dedication is equally important for the learning and effective development of children” 0.001 -0.013 -0.006 (0.014) (0.016) (0.019) 27 Likelihood that (strongly) agrees that there are no gender inequality problems in my community -0.014 0.059** -0.038 (0.027) (0.020) (0.027) Networks 1 Likelihood that there are social groups in my community 0.026 0.045 -0.035 (0.037) (0.032) (0.075) 2 Likelihood of being an active member of any social group 0.031 -0.102* 0.006 (0.053) (0.045) (0.066) Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 54 Row Outcome Variable Follow-up 1 Treatment Effect (std. error) Follow-up 2 Treatment Effect (std. error) Follow-up 3 Treatment Effect (std. error) 3 Likelihood of participation in Trade Shows or Fairs 0.032 -0.054 -0.009 (0.037) (0.034) (0.032) 4 Likelihood of implementation of professional advice (business adviser, lawyer or accountant) during the past two years 0.106*** -0.003 0.020 (0.034) (0.041) (0.018) 5 Number of people I can go to ask business advice 0.980** 0.604 -0.638 (0.393) (0.349) (0.571) 6 Likelihood that feels sometimes (very) confident negotiating lower prices with suppliers 0.027 -0.035 -0.017 (0.033) (0.053) (0.016) 7 Likelihood that feels sometimes (very) confident negotiating higher prices with buyers -0.030 -0.017 0.015 (0.030) (0.035) (0.018) Business Knowledge and Practices 1 Likelihood that no marketing activities implemented in the last 3 years -0.019 0.019 0.032 (0.020) (0.038) (0.026) 2 Likelihood that does not use internet for marketing purposes or to sell products/services -0.050 0.014 -0.056 (0.033) (0.030) (0.048) 3 Likelihood that does not formally keep track of business’ products and materials -0.007 0.012 0.020 (0.024) (0.018) (0.016) 4 Likelihood that does not perform a physical validation of inventory levels -0.003 -0.035 0.017 (0.031) (0.036) (0.022) 5 Likelihood that business runs out of inventory at least one time a month -0.004 -0.034 -0.046 (0.039) (0.022) (0.030) 6 Likelihood that tried to negotiate a lower price with suppliers during the last three months -0.023 0.002 0.013 (0.036) (0.047) (0.027) 7 Likelihood that does compared price and quality of inputs with other suppliers’ products during the last three months -0.050* 0.017 -0.039 (0.029) (0.043) (0.023) 8 Likelihood that fixed salary for the owner 0.012 0.045 -0.071* (0.035) (0.026) (0.031) 9 Likelihood that records salary of the owner in a notebook, registry or computer 0.018 0.112* -0.003 (0.056) (0.052) (0.006) 10 Likelihood that does not keep track of business purchases and sales -0.014 0.003 0.026 (0.024) (0.020) (0.022) 11 Likelihood that has a written expense budget 0.013 0.050 -0.018 (0.035) (0.031) (0.025) 12 Likelihood that has no written goals for next 12 months -0.054** 0.034 0.051 (0.026) (0.025) (0.035) Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 55 Row Outcome Variable Follow-up 1 Treatment Effect (std. error) Follow-up 2 Treatment Effect (std. error) Follow-up 3 Treatment Effect (std. error) 13 Likelihood that has no accountancy documents prepared annually 0.006 -0.042 0.021 (0.037) (0.031) (0.017) 14 Likelihood that has no changes planned over the next 12 months -0.005 0.036 -0.005 (0.021) (0.030) (0.047) 15 Likelihood that (strongly) agrees that “my workspace is well organized” 0.034 0.030 0.005 (0.031) (0.025) (0.054) 16 Likelihood that (strongly) agrees that “I often communicate clear objectives to my colleagues and employees” -0.008 0.009 0.013 (0.012) (0.014) (0.018) 17 Likelihood that (strongly) agrees that “I develop work plans at regular intervals” 0.018 0.032 -0.046* (0.025) (0.036) (0.022) 18 Likelihood that (strongly) agrees that “I sometimes miss deadlines” -0.013 0.014 -0.057 (0.040) (0.050) (0.043) 19 Likelihood that (strongly) agrees that “I believe employees should be treated like family” -0.009 -0.006 -0.051 (0.026) (0.031) (0.029) 20 Likelihood that (strongly) agrees that “I am sometimes late for appointments or meetings” -0.016 -0.028 -0.070* (0.038) (0.050) (0.035) 21 Likelihood that (strongly) agrees that “I often attempt to anticipate future circumstances and plan how I/my company will deal with them” -0.009 -0.021 -0.057* (0.017) (0.012) (0.028) 22 Likelihood that (strongly) agrees that “I am constantly collecting information about the market in which my company operates” 0.002 -0.018 -0.069** (0.019) (0.017) (0.027) Note: Coefficients were obtained by ANCOVA regressions with region and sector fixed effects. Age, marital status, education level, business ownership, number of workers, participation in previous trainings, and number of children under 18 were included as control variables. Robust standard errors in parentheses. Statistical significance is denoted by the following system: *** p<0.01, ** p<0.05, * p<0.1. Outcomes variables stated as “likelihood” can be interpreted as percentage point change by multiplying the coefficient (treatment effect) by 100. + These outcomes are ordinal variables with the following categories: 1) None; 2) Less than 5,000 Soms; 3) 5,001 – 10,000; 4) 10,001 – 20,000; 5) 20,001 – 40,000; 6) 40,001 – 60,001; 7) 60,000 – 80,000; 8) 80,001 – 100,000; 9) 100,001 – 150,000; 10) 150,001 – 200,000; 11) 200,001 – 500,000; 12) More than 500,000. The coefficient is a measure of increase towards the next higher category. While interpretation of this coefficient (as an ordinal variable) is not straightforward, the lack of statistical significance does not change if a multinomial logistic regression is used. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 56 Secondary Questions: Component Results (DID model) Follow-up 1 Follow-up 2 Follow-up 3 Row Outcome Variable Comp. A Comp. B Comp. A Comp. B Comp. A Comp. B Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Business Growth 1 Average sales on a good month (in ranges)+ 0.187 -0.100 -0.095 -0.039 0.044 0.141 (0.294) (0.351) (0.280) (0.169) (0.297) (0.138) 2 Average sales on a bad month (in ranges)+ 0.130 0.001 0.105 0.085 0.234 0.346 (0.294) (0.351) (0.198) (0.276) (0.238) (0.317) 3 Average sales on an average month (in ranges)+ 0.187 0.093 0.096 0.195 0.248 0.400** (0.247) (0.294) (0.241) (0.181) (0.266) (0.153) 4 Average profit on a good month (in ranges)+ 0.303 0.157 0.070 0.114 0.073 0.138 (0.254) (0.316) (0.124) (0.146) (0.157) (0.130) 5 Average profit on a bad month (in ranges)+ 0.252 0.185 0.202 0.143 0.155 0.222 (0.179) (0.225) (0.136) (0.181) (0.154) (0.180) 6 Average profit on an average month (in ranges)+ 0.230 0.119 0.084 0.099 0.100 0.189 (0.205) (0.250) (0.145) (0.131) (0.159) (0.109) 7 Number of good months in the last year 0.181 -0.223 -0.047 -0.327 -0.157 -0.333 (0.365) (0.378) (0.220) (0.244) (0.185) (0.200) 8 Number of bad months in the last year -0.147 0.495 -0.089 0.836*** -0.027 0.741** (0.311) (0.317) (0.512) (0.223) (0.483) (0.230) 10 Sales in the last 12 months (Soms) -37,192 -31,837 -1,384 -3,608 52,257 41,823 (59,839) (89,127) (45,440) (51,940) (35,759) (61,898) 11 Likelihood of having a bank account for business purposes only -0.159** -0.280*** 0.104* 0.197** 0.085 0.182** (0.063) (0.072) (0.054) (0.067) (0.052) (0.070) 0.168 -0.062 0.168 0.101 0.167* 0.097 Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 57 Follow-up 1 Follow-up 2 Follow-up 3 Row Outcome Variable Comp. A Comp. B Comp. A Comp. B Comp. A Comp. B Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) 12 Number of household members that have worked in business in past 12 months (0.186) (0.189) (0.091) (0.097) (0.079) (0.060) 13 Number of paid household members -0.058 0.151 -0.039 0.421 0.047 0.384 (0.259) (0.252) (0.290) (0.282) (0.306) (0.208) 14 Number of non-household people that have worked in business in past 12 months 1.044 1.322 0.642 1.632*** 0.615 1.589*** (0.721) (0.858) (0.563) (0.354) (0.488) (0.332) 15 Number of paid non-household workers 0.770 1.058 0.383 1.422** 0.405 1.355*** (0.826) (0.948) (0.475) (0.408) (0.437) (0.348) 16 SME threshold (5 to 250 total workers) 0.084 0.001 0.082 -0.006 0.073 0.054 (0.063) (0.007) (0.053) (0.020) (0.052) (0.053) 17 SME threshold (5 to 250 paid workers) 0.045 0.051 0.017 0.064 -0.007 0.068 (0.072) (0.086) (0.073) (0.085) (0.071) (0.074) 18 Number of months per year spent on the business owned or managed -0.386 0.326 -0.644 0.052 -0.748 -0.023 (0.495) (0.498) (0.457) (0.464) (0.467) (0.519) 19 Number of days per week spent on the business owned or managed 0.024 0.010 -0.415** -0.290* -0.314 -0.254* (0.206) (0.195) (0.146) (0.125) (0.184) (0.112) 20 Number of hours per day spent on the business owned or managed 0.668 0.474 0.254 0.169 0.200 0.119 (0.456) (0.488) (0.506) (0.427) (0.428) (0.311) 21 Likelihood of purchasing raw materials, goods, or equipment for business with a loan in the last 12 months -0.008 -0.051 -0.096 0.090 -0.068 0.096 (0.099) (0.098) (0.063) (0.105) (0.063) (0.088) 22 Likelihood of applying for a loan from a financial institution in the last 12 months -0.021 -0.101 -0.005 0.092 0.033 0.102 (0.084) (0.087) (0.099) (0.066) (0.096) (0.060) 23 Likelihood of loan approval 0.155* 0.025 -0.096 0.007 -0.063 0.002 (0.093) (0.086) (0.072) (0.074) (0.064) (0.064) Entrepreneurial Leadership Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 58 Follow-up 1 Follow-up 2 Follow-up 3 Row Outcome Variable Comp. A Comp. B Comp. A Comp. B Comp. A Comp. B Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) 1 Likelihood that prefers to work as an employee in a business instead of managing/owning one 0.272*** 0.129** 0.273** 0.120 0.228** 0.099 (0.066) (0.064) (0.082) (0.096) (0.068) (0.087) 2 Likelihood that people ask me for business advice (very) often -0.075 0.173** -0.065 0.181** -0.031 0.174** (0.084) (0.087) (0.068) (0.059) (0.053) (0.060) 3 Likelihood that I (with my partner/spouse or another household member) am in charge of general business planning decisions 0.031 0.003 0.002 0.010 -0.006 0.015 (0.073) (0.075) (0.044) (0.052) (0.051) (0.061) 4 Likelihood that I (with my partner/spouse or another household member) decide what inputs to buy for production 0.010 -0.114 0.000 -0.071 -0.020 -0.073 (0.079) (0.077) (0.038) (0.058) (0.044) (0.052) 5 Likelihood that I (with my partner/spouse or another household member) am in charge of sales and client relations -0.018 -0.123* -0.037 -0.094* -0.054 -0.094** (0.075) (0.074) (0.054) (0.041) (0.046) (0.038) 6 Likelihood that I (with my partner/spouse or another household member) decide if I should apply for a loan -0.021 -0.100 -0.075 -0.048 -0.058 -0.034 (0.080) (0.081) (0.061) (0.059) (0.057) (0.058) 7 Likelihood that I (with my partner/spouse or another household member) decide my own singular wage 0.153 0.097 -0.100 -0.164** -0.117* -0.130* (0.341) (0.232) (0.057) (0.062) (0.052) (0.061) 8 Likelihood that I (with my partner/spouse or another household member) decide what type of work I will do 0.027 -0.085 0.015 -0.095* -0.014 -0.078* (0.073) (0.071) (0.040) (0.045) (0.045) (0.037) 9 Likelihood that I (with my partner/spouse or another household member) am in charge of marketing and advertising decisions 0.093 0.049 0.123** 0.150** 0.070 0.156** (0.077) (0.076) (0.045) (0.056) (0.059) (0.046) 10 Likelihood that I (with my partner/spouse or another household member) am in charge of staffing of business decisions 0.065 -0.047 0.028 0.001 -0.004 0.037 (0.074) (0.074) (0.046) (0.036) (0.051) (0.039) 11 -0.040 0.016 -0.047 -0.045 -0.067 -0.035 Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 59 Follow-up 1 Follow-up 2 Follow-up 3 Row Outcome Variable Comp. A Comp. B Comp. A Comp. B Comp. A Comp. B Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Likelihood that often (or always) makes general business planning decisions without consulting anyone else (0.081) (0.081) (0.075) (0.095) (0.068) (0.105) 12 Likelihood that often (or always) decides what inputs to buy for production without consulting anyone else 0.024 -0.064 0.047 -0.079 0.046 -0.079 (0.082) (0.083) (0.056) (0.071) (0.069) (0.084) 13 Likelihood that often (or always) manages sales and client relations without consulting anyone else -0.098 -0.176** -0.078 -0.177** -0.094 -0.164** (0.079) (0.080) (0.098) (0.053) (0.099) (0.055) 14 Likelihood that often (or always) decides whether to apply for a loan without consulting anyone else -0.118 -0.143* -0.069 -0.126 -0.044 -0.093 (0.083) (0.085) (0.078) (0.132) (0.086) (0.131) 15 Likelihood that often (or always) decides own singular wage without consulting anyone else 0.134 0.043 0.139 0.031 0.121 0.026 (0.090) (0.092) (0.109) (0.095) (0.096) (0.097) 16 Likelihood that often (or always) decides what type of work will do without consulting anyone else -0.124 -0.061 -0.112 -0.080 -0.138 -0.053 (0.084) (0.085) (0.088) (0.046) (0.094) (0.060) 17 Likelihood that often (or always) makes marketing and advertising decisions without consulting anyone else -0.002 -0.064 0.060 -0.066 0.008 -0.046 (0.089) (0.088) (0.117) (0.137) (0.142) (0.154) 18 Likelihood that often (or always) makes staffing of business decisions without consulting anyone else 0.024 -0.046 0.054 -0.034 0.026 -0.021 (0.084) (0.082) (0.068) (0.117) (0.085) (0.118) 19 Likelihood that (strongly) agrees that “women should do what men say” -0.054 0.047 -0.067 0.025 -0.070 -0.002 (0.071) (0.076) (0.092) (0.082) (0.088) (0.087) 20 Likelihood that (strongly) agrees that “Women must share their income with their husbands” -0.037 -0.103 0.007 -0.017 -0.033 -0.028 (0.062) (0.066) (0.046) (0.109) (0.051) (0.104) 21 Likelihood that (strongly) agrees that “it is OK if men abandon women if they wish to” 0.130** -0.017 0.113 0.006 0.130* -0.011 (0.060) (0.061) (0.060) (0.058) (0.063) (0.069) Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 60 Follow-up 1 Follow-up 2 Follow-up 3 Row Outcome Variable Comp. A Comp. B Comp. A Comp. B Comp. A Comp. B Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) 22 Likelihood that (strongly) agrees that “it is OK if men chide women because they went out without any permission” -0.028 0.080 0.039 0.057 0.045 0.061 (0.078) (0.081) (0.063) (0.059) (0.060) (0.053) 23 Likelihood that (strongly) agrees that “it is OK if men chide women if they do not take care of children” -0.108** -0.092 -0.060 -0.064 -0.031 -0.019 (0.054) (0.062) (0.041) (0.054) (0.036) (0.062) 24 Likelihood that (strongly) agrees that “the role of women is to earn money and take care of her family” -0.051 0.038 -0.059 0.067 -0.110 0.033 (0.080) (0.084) (0.067) (0.060) (0.079) (0.081) 25 Likelihood that (strongly) agrees that “a mother who works can establish a relationship as warm and solid with her children as a mother who does not work” -0.075** -0.041 -0.057* -0.045 -0.034 -0.051 (0.036) (0.037) (0.026) (0.056) (0.028) (0.060) 26 Likelihood that (strongly) agrees that “father’s and mother’s dedication is equally important for the learning and effective development of children” -0.045* -0.035 -0.019 -0.004 -0.022 -0.005 (0.025) (0.029) (0.027) (0.027) (0.025) (0.028) 27 Likelihood that (strongly) agrees that there are no gender inequality problems in my community -0.045* -0.035 -0.178** -0.102 -0.164*** -0.117 (0.025) (0.029) (0.060) (0.092) (0.045) (0.085) Networks 1 Likelihood that there are social groups in my community -0.097 -0.042 -0.126 -0.067 -0.112 -0.076 (0.083) (0.085) (0.112) (0.134) (0.104) (0.133) 2 Likelihood of being an active member of any social group -0.090 -0.028 -0.008 0.019 0.017 0.010 (0.059) (0.067) (0.095) (0.069) (0.098) (0.085) 3 Likelihood of participation in Trade Shows or Fairs 0.300*** 0.395*** -0.017 0.040 -0.043 0.008 (0.084) (0.086) (0.069) (0.056) (0.074) (0.055) 4 Likelihood of implementation of professional advice (business adviser, lawyer or accountant) during the past two years 0.109 0.179** 0.051 0.170* 0.076 0.172** (0.072) (0.074) (0.070) (0.075) (0.055) (0.071) Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 61 Follow-up 1 Follow-up 2 Follow-up 3 Row Outcome Variable Comp. A Comp. B Comp. A Comp. B Comp. A Comp. B Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) 5 Number of people I can go to ask business advice 0.273 0.382 1.115 1.394** 0.072 0.404 (0.791) (1.007) (1.039) (0.552) (1.060) (0.691) 6 Likelihood that feels sometimes (very) confident negotiating lower prices with suppliers -0.076 0.006 -0.060 -0.004 -0.075 -0.014 (0.074) (0.079) (0.063) (0.113) (0.061) (0.098) 7 Likelihood that feels sometimes (very) confident negotiating higher prices with buyers -0.122* -0.003 -0.093 -0.011 -0.101 -0.037 (0.071) (0.076) (0.080) (0.082) (0.077) (0.087) Business Knowledge and Practices 1 Likelihood that no marketing activities implemented in the last 3 years 0.016 0.019 0.009 -0.033 0.029 -0.015 (0.021) (0.032 (0.023) (0.020) (0.020) (0.026) 2 Likelihood that does not use internet for marketing purposes or to sell products/services 0.039 -0.066 0.046 -0.040 0.061 0.051 (0.053) (0.049) (0.052) (0.046) (0.048) (0.044) 3 Likelihood that does not formally keep track of business’ products and materials 0.052 -0.057 0.055 -0.022 0.022 -0.045 (0.042) (0.058) (0.053) (0.060) (0.059) (0.047) 4 Likelihood that does not perform a physical validation of inventory levels -0.122** -0.116 -0.093* -0.149 0.061 -0.021 (0.049) (0.090) (0.041) (0.091) (0.048) (0.053) 5 Likelihood that business runs out of inventory at least one time a month 0.006 0.039 0.029 0.055 -0.074** -0.136 (0.074) (0.115) (0.064) (0.092) (0.030) (0.074) 6 Likelihood that tries to negotiate a lower price with suppliers during the last three months 0.018 0.039 0.083 0.127 0.032 0.048 (0.112) (0.080) (0.084) (0.088) (0.069) (0.092) 7 Likelihood that compares price and quality of inputs with other suppliers’ products during the last three months 0.015 0.004 0.043 0.064 0.075 0.107 (0.083) (0.041) (0.077) (0.046) (0.085) (0.071) 8 Likelihood that has fixed salary for the owner -0.047 0.046 -0.035 0.085 0.034 0.056 (0.098) (0.116) (0.075) (0.093) (0.073) (0.046) Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 62 Follow-up 1 Follow-up 2 Follow-up 3 Row Outcome Variable Comp. A Comp. B Comp. A Comp. B Comp. A Comp. B Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) 9 Likelihood that records salary of the owner in a notebook, registry or computer -0.062 -0.057 -0.043 -0.013 -0.004 0.102 (0.131) (0.102) (0.102) (0.107) (0.057) (0.080) 10 Likelihood that does not keep track of business purchases and sales -0.031 -0.078 -0.026 -0.045 -0.080 -0.032 (0.030) (0.044) (0.044) (0.059) (0.125) (0.121) 11 Likelihood that has a written expense budget -0.047 -0.047 -0.044 0.219*** -0.006 -0.046 (0.050) (0.050) (0.050) (0.039) (0.036) (0.052) 12 Likelihood that has no written goals for next 12 months 0.004 0.004 -0.004 -0.049 -0.056 0.207*** (0.030) (0.049) (0.033) (0.043) (0.049) (0.042) 13 Likelihood that has no accountancy documents prepared annually -0.228** -0.284** -0.218* -0.250** 0.017 -0.044 (0.088) (0.098) (0.103) (0.090) (0.035) (0.038) 14 Likelihood that has no changes planned over the next 12 months -0.014 -0.074* -0.038 -0.104 -0.210* -0.270** (0.025) (0.032) (0.044) (0.055) (0.108) (0.099) 15 Likelihood that (strongly) agrees that “my workspace is well organized” 0.055 -0.002 0.064 0.032 -0.017 -0.080 (0.050) (0.049) (0.049) (0.059) (0.042) (0.044) 16 Likelihood that (strongly) agrees that “I often communicate clear objectives to my colleagues and employees” 0.000 -0.007 -0.016 -0.020 0.035 0.018 (0.027) (0.022) (0.033) (0.044) (0.040) (0.048) 17 Likelihood that (strongly) agrees that “I develop work plans at regular intervals” 0.029 -0.052 0.030 0.001 0.004 0.001 (0.053) (0.051) (0.040) (0.034) (0.027) (0.031) 18 Likelihood that (strongly) agrees that “I sometimes miss deadlines” 0.158* 0.202** 0.110* 0.049* 0.017 -0.013 (0.071) (0.069) (0.051) (0.025) (0.038) (0.038) 19 Likelihood that (strongly) agrees that “I believe employees should be treated like family” -0.068 -0.070** -0.060 -0.059* 0.051 0.013 (0.040) (0.023) (0.056) (0.026) (0.037) (0.032) 20 Likelihood that (strongly) agrees that “I am sometimes late for appointments or meetings” 0.129 0.128 0.092 0.056 -0.062 -0.040 (0.113) (0.177) (0.081) (0.120) (0.059) (0.028) Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 63 Follow-up 1 Follow-up 2 Follow-up 3 Row Outcome Variable Comp. A Comp. B Comp. A Comp. B Comp. A Comp. B Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) Treatment Effect (std. error) 21 Likelihood that (strongly) agrees that “I often attempt to anticipate future circumstances and plan how I/my company will deal with them” 0.011 -0.018 0.004 0.001 0.046 0.051 (0.047) (0.056) (0.028) (0.018) (0.067) (0.112) 22 Likelihood that (strongly) agrees that “I am constantly collecting information about the market in which my company operates” 0.008 0.020 0.000 0.035 -0.002 -0.006 (0.036) (0.039) (0.046) (0.022) (0.028) (0.014) Note: Coefficients were obtained by DID regressions with region and sector fixed effects. Robust standard errors in parentheses. Statistical significance is denoted by the following system: *** p<0.01, ** p<0.05, * p<0.1. Outcomes variables stated as “likelihood” can be interpreted as percentage point change by multiplying the coefficient (treatment effect) by 100. + These outcomes are ordinal variables with the following categories: 1) None; 2) Less than 5,000 Soms; 3) 5,001 – 10,000; 4) 10,001 – 20,000; 5) 20,001 – 40,000; 6) 40,001 – 60,001; 7) 60,000 – 80,000; 8) 80,001 – 100,000; 9) 100,001 – 150,000; 10) 150,001 – 200,000; 11) 200,001 – 500,000; 12) More than 500,000. The coefficient is a measure of increase towards the next higher category. While interpretation of this coefficient (as an ordinal variable) is not straightforward, the lack of statistical significance does not change if a multinomial logistic regression is used. ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 64 ANNEX G: DATA COLLECTION INSTRUMENTS Survey Instrument Third Follow-Up Survey ACDI/VOCA WLSME Kyrgyzstan Training Program Impact Evaluation ACDI/VOCA and Bai Tushum Fund implemented a program to increase the number of women- owned small and medium enterprises (SMEs) in the Kyrgyz Republic by targeting assistance to high-growth female entrepreneurs (i.e. ‘micro-entrepreneurs’). The purpose of the program was to reduce the human and social capital and information barriers that limit the growth of women's enterprises, and to increase access to finance. LOCATION: Oblast District Sub-District (aiyl okmotu) Village INTERVIEW DETAILS: VISITS DATE TIME NEXT SCHEDULED VISIT RESULT FROM VISIT (*) START END DATE TIME FIRST SECOND THIRD * CODES Completed …………...…………………………. 11 Not at home/business …………………………... 22 Rescheduled ……………………………………. 33 Refused to participate ....…………………………44 Started, but incomplete …..……………………... 55 Disabled ….……………………………………. 66 Other (specify) __________________________ 77 QUALITY CONTROL: INTERVIEWER SUPERVISOR OPERATOR DATA ENTRY NAME: |___|___| NAME: |___|___| NAME: |___|___| NAME1: NAME2: |___|___| |___|___| DATE : |___|___| / |___|___| / |___|___| DATE: |___|___| / |___|___| / |___|___| DATE: |___|___| / |___|___| / |___|___| DATE 1: |___|___| / |___|___| / |___|___| DATE 2: |___|___| / |___|___| / |___|___| CORRECTED? 1. Yes 2. No CORRECTED? 1. Yes 2. No ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 65 SECTION 1: BUSINESS CHARACTERISTICS Q1.In the last 12 months, how many business trainings/seminars have you participated in? 1. One or more than one / INTERVIEWER: ASK PARTICIPANT TO NAME PROGRAMS, FILL TABLE BELOW 2. None  SKIP TO Q4 98. Don’t know  SKIP TO Q4 99. Refused to answer  SKIP TO Q4 Q1. Program Name (Past 2 years) 1. 2. 3. 4. 5. 6. Q2.Frequently, do you keep in touch with some of the women entrepreneurs you met during the WLSME program? 1. Yes 2. No SKIP TO Q4 3. Did not participate in WLSME program  SKIP TO Q4 98. Don’t know  SKIP TO Q4 99. Refused to answer  SKIP TO Q4 Q3.What types of topics do you talk about? And with how many entrepreneurs? Topics Q3.1 Topics Q3.2 With how many entrepreneurs have you talked about this topic? 1. Personal topics 1. Yes 2. No |__|__|__| 2. Topics related to your/their business 1. Yes 2. No |__|__|__| Q4.How many types of businesses do you own or manage, and which is your primary business? Q4.1. How many types of businesses do you own or manage? |___|___|  IF NOT “0”, SKIP TO Q4.2 Q4.1a. When did you close your business? |___|___| Month |___|___|___|___| Year Business Type Q4.2 Primary Q4.1b. Why did you close your business? __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 66 1. Garment production 2. Traditional garment production (national folklore dresses, kalpaks, and accessories) 3. Restaurant 4. Tourism guesthouse, including traditional houses (yurtas) 5. Tourism souvenir production (non-garment) 6. Agro Processing 7. Other (specify) ______________________________ Now I’m going to ask you a set of questions about your primary business: Q5.Who legally (i.e., on paper) owns the business? / INTERVIEWER, READ THE ALTERNATIVES AND MARK ONLY ONE / 1. Myself 2. My husband 3. A family member other than my husband__________________(specify relationship) 4. Joint ownership – myself and husband 5. A business partnership with other people who aren’t members of my family 6. No one. It is not officially registered 7. Other___________________________________________(specify) Q6.On a day-to-day basis, who usually MANAGES the business? / INTERVIEWER, READ THE ALTERNATIVES AND MARK JUST ONE / 1. Myself 2. My husband 3. A family member other than my husband__________________(specify relationship) 4. Joint managership – myself and husband 5. A non-family member 6. Other___________________________________________(specify) Q7.Do you have a bank account which you use ONLY for your business? 1. Yes 2. No 98. Don’t know 99. Refused to answer Q8. Considering your business, have you registered in the statistics, tax office, social fund, Ministry of Justice or Department of Justice? / INTERVIEWER, MARK All THAT APPLY/ 1. Yes, patent based simplified registration 2. Yes, Individual entrepreneur 3. Yes, LLC 4. Yes, cooperative 5. Other________________________(specify) 6. My business is not registered 98. Don’t know 99. Refused to answer Q9. In the last 12 months, have you worked for pay outside your business? (If No, Don’t Know or Refused to answer, skip to Q11) 1. Yes 2. No 98. Don’t know 99. Refused to answer ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 67 Q10.In which sector or type of work did you do? 1. Public sector worker (government) 2. Budget sector worker (such as doctor, teacher) 3. Private sector, dependent work 4. Other independent / self-employee work 5. Other (specify): ________________________________ Q11.In the last 12 months, how many people from your household, apart from yourself, have worked in your business? (If None, Don’t Know or Refused to answer, skip to Q13) |___|___| (people) 0. None 98. Don’t know 99. Refused to answer Q12.How many of them were remunerated with cash? |___|___| (people) 98. Don’t know 99. Refused to answer Q13.In the last 12 months, how many people from outside your household have worked in your business? (If None, Don’t Know or Refused to answer, skip to Q15) |___|___| (people) 0. None 98. Don’t know 99. Refused to answer Q14.How many of them were remunerated? |___|___| (people) 98. Don’t know 99. Refused to answer Q15.Number of Employees by Category / INTERVIEWER: ASK RESPONDENT HOW MANY EMPLOYEES HER BUSINESS HAS IN EACH CATEGORY / Type of Employee 1. Full-time 2.Part-time 3.Temporary Q15.1. From the household |__|__| |__|__| |__|__|  How many months? |__|__| Q15.2. Non-family |__|__| |__|__| |__|__|  How many months? |__|__| ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 68 Q16.On average, how much time do you spend on the business you own or manage? Measurement Amount of time spent at business 1. Months per year |__|__| 2. Days per week |__|__| 3. Hours per day |__|__| 98. Don’t know 99. Refused to answer Q17.In what physical place(s) does your business activity mainly operate? Type of Place Number of business locations 1. In your home with special installations |__|__| 2. In your home without special installations |__|__| 3. Owned commercial space (not your home) |__|__| 4. Rented commercial space (not your home) |__|__| 5. Owned uirta |__|__| 6. Rented uirta |__|__| 7. Other __________________________(specify) |__|__| 98. Don’t know 99. Refused to answer Q18.Currently, what is the biggest challenge you face now in managing your business? / INTERVIEWER, SHOW CARD Q18, CIRCLE ONLY TWO RESPONSES/ Sources of information The 1st most important The 2nd most important 1. Accessing financing 1 1 2. Getting support from my family 2 2 3. Finding a suitable business location 3 3 4. Finding information on how to start a business 4 4 5. Dealing with registration and regulatory requirements 5 5 6. Finding suppliers 6 6 7. I was worried about the development of business 7 7 8. Searching qualified staff 8 8 9. Personnel turnover 9 9 10. Sales market 10 10 11. Verifications of Supervisory Authority 11 11 12. Other______________________(specify) 12 12 98. Don’t know 98 98 99. Refused to answer 99 99 Q19.In the last 12 months, did you buy raw materials, goods, or equipment for your business? (If No, Don’t Know or Refused to answer, skip to Q21) 1. Yes, often 2. Yes, but rarely 3. No 98. Don’t know 99. Refused to answer ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 69 Q20.Were any of these purchases made on loan? 1. Yes 2. No 98. Don’t know 99. Refused to answer Q21.In the last 12 months, have you borrowed cash for your business? /INTERVIEWER, SHOW CARD Q21, MARK ALL THAT APPLY/ 1. From any family member or friend 2. Another business 3. Microfinance institution 4. Bank 5. Moneylender 6. Government / government agency 7. Religious group / charity 8. Cooperative / business association 9. Supplier / retailer/ local store 10. Other________________________(specify) 11. Business didn’t borrow in the past 12 months 98. Don’t know 99. Refused to answer Q22.In the last 12 months, have you applied for a loan from a financial institution? (If No, Don’t Know or Refused to answer, continue to section 3) 1. Yes 2. No 98. Don’t know 99. Refused to answer Q23.Were you approved? 1. Yes 2. No 3. Application is in the process of consideration 98. Don’t know 99. Refused to answer IMPACT OUTCOME INDICATORS (Sections 3 – 7) SECTION 3: BUSINESS RESULTS In the last 12 months, are you doing better, worse, or the same in terms of sales from your business? 1. Better 2. Worse 3. Same ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 70 Q24.On a monthly basis, how much does your business make from SALES (in soms)? / INTERVIEWER, SHOW CARD 0, FILL THE TABLE BELOW / Range, som Q30.1 Sales on a GOOD month Q30.2 Sales on a BAD month Q30.3 Sales on an AVERAGE month 1. None 1 1 1 2. Less than 5000 2 2 2 3. 5001 – 10,000 3 3 3 4. 10,001 – 20,000 4 4 4 5. 20,001 – 40,000 5 5 5 6. 40,000- 60,000 6 6 6 7. 60,000 – 80,000 7 7 7 8. 80,001 – 100,000 8 8 8 9. 100,001 – 150,000 9 9 9 10.150,001 – 200,000 10 10 10 11.200,001 – 500,000 11 11 11 12.More than 500,000 12 12 12 98.Don’t know 98 98 98 99.Refused to answer 99 99 99 Q25.On a monthly basis, how much PROFIT does your business make (in soms)? / INTERVIEWER, SHOW CARD Q25, FILL THE TABLE BELOW/ Range, som Q31.1 Profit on a GOOD month Q31.2 Profit on a BAD month Q31.3 Profit on an AVERAGE month 1. None 1 1 1 2. Less than 5000 2 2 2 3. 5001 – 10,000 3 3 3 4. 10,001 – 20,000 4 4 4 5. 20,001 – 40,000 5 5 5 6. 40,000- 60,000 6 6 6 7. 60,000 – 80,000 7 7 7 8. 80,001 – 100,000 8 8 8 9. 100,001 – 150,000 9 9 9 10.150,001 – 200,000 10 10 10 11.200,001 – 500,000 11 11 11 12.More than 500,000 12 12 12 98.Don’t know 98 98 98 99.Refused to answer 99 99 99 ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 71 Q26.In the last 12 months, how many of each type of months did you have? / INTERVIEWER: TOTAL NUMBER OF MONTHS FROM 27.1 – 27.4 MUST EQUAL 12 / 27.1 Good months _____  Which are your good months? / CIRCLE ALL THAT APPLY/ JAN / FEB / MAR / APR / MAY / JUN / JUL / AUG / SEPT / OCT / NOV / DEC 27.2 Bad months _____  Which are your bad months? / CIRCLE ALL THAT APPLY/ JAN / FEB / MAR / APR / MAY / JUN / JUL / AUG / SEPT / OCT / NOV / DEC 27.3 Average months _____  Which are your average months? / CIRCLE ALL THAT APPLY/ JAN / FEB / MAR / APR / MAY / JUN / JUL / AUG / SEPT / OCT / NOV / DEC 27.4 No business activities _____  Which months does your business not operate? / CIRCLE ALL THAT APPLY/ JAN / FEB / MAR / APR / MAY / JUN / JUL / AUG / SEPT / OCT / NOV / DEC Q27. In the last 12 months, how much did your business make from sales? |___|___|___| |___|___|___| som 98. Don’t know 99. Refused to answer SECTION 4: BUSINESS PRACTICES Marketing Q28. During the last 12 months, which of the following actions have you performed? / INTERVIEWER, READ THE ALTERNATIVES AND SHOW CARD Q28. MARK ALL THAT APPLY / 1. You visited somebody running a business similar to yours to see what prices they were charging. 2. You visited somebody running a business similar to yours to see the products that they were selling. 3. You asked your current customers if they would like the business to sell or produce other products. 4. You talked to some ex-customers to ask him/her about why she stopped buying in your business. 5. You asked a supplier what products are selling well in the market. 6. You attended events to exchange experiences, such as seminars and trainings 7. None 98.Don’t know 99.Refused to answer Q29. In the last 12 months, which of the following types of advertisement have you used? /CHECK ALL THAT APPLY/ 1. Flyers / Pamphlets 2. Signs at community events 3. Yellow pages 4. Newspaper / Magazines 5. Billboard signs not near your business 6. Word of mouth 7. Radio 8. Social Networks (facebook, twitter, namba, diesel) 9. Other (specify): _____________________ 10. None 98. Don’t know 99. Refused to answer ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 72 Q30.Which of the following statements apply to your branding practices? / INTERVIEWER, READ THE ALTERNATIVES AND SHOW CARD 0. MARK ALL THAT APPLY / 1. I own a patented logo 2. I have a logo without a patent 3. I have defined the colors and fonts I use in all of my marketing materials 4. I have a slogan / brand vision 5. I have a variety of logo, colors, and fonts I use 6. None 98. Don’t know 99. Refused to answer Q31.During the last 3 months, have you made any special offers to attract customers? 1. Yes 2. No 98. Don’t know 99. Refused to answer Access to External (international) Markets Q32.Currently, do you sell your product/service in external markets? 1. Yes, but through another company 2. Yes, but not formally 3. Yes, I export formally through a broker 4. Yes, I export formally, and directly 5. No, I don’t export 98. Don’t know 99. Refused to answer Q33.Currently, what is your business presence in the Internet? / INTERVIEWER, READ THE ALTERNATIVES AND SHOW CARD Q33. MARK ALL THAT APPLY / 1. I use my personal social networking account to market my business (e.g., Facebook, Odnoklassniki.ru) 2. My business has its own social networking website (e.g., Facebook, Odnoklassniki.ru) 3. My business has an informational website (information only. Clients cannot purchase from site) 4. My business has an operational website from which clients can purchase products/services 5. I don’t use the Internet to market my business or sell products/services 6. Other ______________________________________________(specify) 98. Don’t know 99. Refused to answer Inventory Q34. How do you keep track of your business’ products and materials? /CHECK ALL THAT APPLY/ 1. With a notebook 2. Specialized software 3. Excel spreadsheet 4. Other ___________________________(specify) 5. I don’t formally keep track 98. Don’t know 99. Refused to answer Q35.Do you perform a physical validation of inventory levels? ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 73 1. Yes, daily 2. Yes, weekly 3. Yes, monthly 4. Yes, quarterly 5. Yes, every 6 months 6. Yes, yearly 7. No 98. Don’t know 99. Refused to answer Q36. How often do you run out of inventory, goods raw materials, or other assets essential for your business? 1. Never, I always have enough 2. Not very often, once every 6 months 3. Once every 3 months 4. One or two times a month 5. Don’t need/doesn’t apply to my business 98. Don’t know 99. Refused to answer Q37. During the last 3 months, have you tried to negotiate a lower price with the suppliers for the inputs or goods that you buy from them? 1. Yes 2. No 98. Don’t know 99. Refused to answer Q38.During the last 3 months, have you compared with other suppliers the price and quality of the inputs and other goods offered by your supplier? 1. Yes 2. No 98. Don’t know 99. Refused to answer Salaries Q39. Do you pay yourself a fixed salary? (If No, Don’t Know or Refused to answer, skip to Q42) 1. Yes 2. No 98. Don’t know 99. Refused to answer Q40.Do you record this salary in a notebook, registry, or computer? 1. Yes 2. No 98. Don’t know 99. Refused to answer Costs and Records Q41.How do you record your business purchases and sales? 1. With a notebook 2. Specialized software 3. Excel spreadsheet 4. Other ___________________________(specify) 5. I don’t formally keep track 98. Don’t know 99. Refused to answer Q42.Do you have a written expense budget that indicates you how much you have to pay in: rent, electricity, equipment maintenance, transportation, publicity and other indirect cost of the business? ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 74 1. Yes 2. No 98. Don’t know 99. Refused to answer Q43. Do you withdraw products, services or cash from your business, besides your fixed salary, to help cover the necessary expenses from your home? (Do not include any type of salary you pay yourself or any person from your household) 1. Yes 2. No  SKIP TO Q45 98. Don’t know  SKIP TO Q45 99. Refused to answer  SKIP TO Q45 Q44.Do you record these withdrawals of products, services or cash into your registry of income and expenses? 1. Yes 2. No 98. Don’t know 99. Refused to answer Q45. Indicate which of following are your THREE biggest challenges to growing your business in the future? / INTERVIEWER. SHOW CARD Q45. RANK THE TOP THREE ONLY/ Challenge Top 3 1. Accessing financing 2. Getting support from my family 3. Finding a suitable business location 4. Finding information on new markets (e.g. exporting) 5. Finding qualified workers 6. Finding trusted workers 7. Finding mentors /advisors to give growth advice 8. Taxation issues 9. Transportation issues 10. Searching people who love their job (interested in) 11. Bring to the entrepreneurs statistic 12. Verifications of Supervisory Authority 13. Other_________________________(specify) 14. Other_________________________(specify) 98. Don’t know 99. Refused to answer Business Planning Q46.Do you have written business goals for next 12 months? / INTERVIEWER, CHECK ALL THAT APPLY/ 1. None 2. General sales goals 3. Sales goals by products or services 4. New products or services 5. Other_____________________________(specify) 98. Don’t know 99. Refused to answer Q47.What of the following documents do you or your accountant prepare annually? /INTERVIEWER, SHOW CARD Q47, CHECK ALL THAT APPLY/ ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 75 1. Balance sheet of gains and losses 2. Cash flow 3. Income and expenses 4. Tax return 5. Don’t prepare any document 6. Other___________________________(specify) 98. Don’t know 99. Refused to answer Q48.How do you plan to develop your business over the next 12 months? /INTERVIEWER, SHOW CARD Q48, CHECK ALL THAT APPLY/ 1. No changes planned 2. Expand/improve the business 3. Increase the number of workers 4. Decrease the number of workers 5. Expand the range of products/services 6. Reduce the range of products/services 7. Move to larger premises 8. Implement practices to be more environmentally sensitive (conserve energy and water, reduce, recycle, reuse) 9. Other___________________________(specify) 98. Don’t know 99. Refused to answer Q49.I will now read out statements, please tell me if you agree or disagree with each one. There are no right or wrong answer, please just tell me how these apply to you. / INTERVIEWER, READ OUT EACH STATEMENTS AND ASK ABOUT EACH ONE/ 1. Agree 2. Disagree 98. Don’t know 99. Refused to answer Q49.1 My workspace is well organized 1. AGREE 2. DISAGREE 98 99 Q49.2 It’s not my job to resolve employee conflicts 1. AGREE 2. DISAGREE 98 99 Q49.3 I often communicate clear objectives to my colleagues and employees. 1. AGREE 2. DISAGREE 98 99 Q49.4 I develop work plans at regular intervals 1. AGREE 2. DISAGREE 98 99 Q49.5 I sometimes miss deadlines 1. AGREE 2. DISAGREE 98 99 Q49.6 I believe employees should be treated like family 1. AGREE 2. DISAGREE 98 99 Q49.7 I always take the time to reflect on challenging situations, no matter how busy I am 1. AGREE 2. DISAGREE 98 99 Q49.8 I am sometimes late for appointments or meetings 1. AGREE 2. DISAGREE 98 99 Q49.9 I often attempt to anticipate future circumstances and plan how I/my company will deal with them 1. AGREE 2. DISAGREE 98 99 Q49.10 I am constantly collecting information about the market in which my company operates. 1. AGREE 2. DISAGREE 98 99 SECTION 5: ENTREPRENUERIAL LEADERSHIP AND EMPOWERMENT Entrepreneurial Leadership ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 76 Q50.If you could choose and would be paid the same amount of money, would you prefer to work as an employee in a company or business, instead of managing/owning your own? 1. Yes 2. No 98. Don’t know 99. Refused to answer Q51. How often do people ask you for business advice? 1. Never 2. Rarely 3. Sometimes 4. Often 5. Very often 98. Don’t know 99. Refused to answer Q52. How do you motivate your employees to give the best of themselves? /CHECK ALL THAT APPLY/ 1. Gifts for good performance 2. Team outings to promote fellowship 3. Periodic reviews of your employees’ performance and communication of those results 4. Functions and responsibilities are defined and written for each employee 5. Establish and communicate clear objectives 6. None of the above 7. Other (specify): _________________________ 98. Don’t know 99. Refused to answer Q53. Which of the following do you usually do when you delegate work to an employee? / INTERVIEWER, SHOW CARD Q53, CHECK ALL THAT APPLY/ 1. You speak directly with the employee and ask him / her if she is able to conduct the task 2. You outline every step and activity needed to accomplish the task and you ask your employee to follow that 3. You do not outline the steps or activities, instead you establish the end goal for your employee 4. You establish deadlines and organize periodic meetings to supervise the tasks 5. You do not provide any instructions to your employee, you only expect the result 6. You do not delegate tasks because you don’t need to 7. You do not delegate tasks because you don’t trust your employees. 8. Other (specify): ________________________ 98. Don’t know 99. Refused to answer Q54. How do you contribute to the professional development of your employees? / INTERVIEWER, SHOW CARD Q54, CHECK ALL THAT APPLY/ 1. You promote employees when there is a new vacancy in your business 2. You invest in professional development courses for your employees 3. You invest in skills training courses for your employees 4. You delegate challenging or complex tasks and responsibilities 5. You establish and communicate a career plan 6. None of the above 7. Other (specify): _________________________ 98. Don’t know 99. Refused to answer Q55. In which ways do you innovate within your business? / INTERVIEWER, SHOW CARD Q55, CHECK ALL THAT APPLY/ ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 77 1. You launch products or services that follow current trends 2. You redefine job functions based on the real / necessary work 3. You invest in equipment and infrastructure of your business 4. You reinvent the production processes 5. You ask your employees for their opinions on how to improve the business 6. None of the above 7. Other (specify): __________________________ 98. Don’t know 99. Refused to answer Business Decision-Making Q56. Now we’re going to talk about different types of decisions for your business. Please tell me for each one, who normally makes the decision? / INTERVIEWER, FILL THE TABLE AFTER Q57, READ OUT ALL ACTIVITIES ONE BY ONE/ Q57. How often do you make these decisions without consulting anyone else? /INTERVIEWER, SHOW CARD Q57, READ OUT ALL ACTIVITIES ONE BY ONE AND MARK SUITABLE ANSWERS. ONLY ONE ANSWER PER EACH TYPE OF DECESIONS/ № 1.General business planning 2.What inputs to buy for production 3.Decisions about sales and client relations 4.Whether or not you should apply for a loan 5.Your own (singular wage) 6.What type of work you will do 7.Marketing and advertising 8.Staffing of business Q56 1. Me alone 1 1 1 1 1 1 1 1 2. Me together with my partner/spouse 2 2 2 2 2 2 2 2 3. My partner/spouse alone 3 3 3 3 3 3 3 3 4. Another household member alone 4 4 4 4 4 4 4 4 5. Me with another household member 5 5 5 5 5 5 5 5 6. Someone outside the household 6 6 6 6 6 6 6 6 7. Decision not made (not applicable) 7 7 7 7 7 7 7 7 98. Don’t know 98 98 98 98 98 98 98 98 99. Refused to answer 99 99 99 99 99 99 99 99 Q57 1. Never 1 1 1 1 1 1 1 1 2. Rarely 2 2 2 2 2 2 2 2 3. sometimes 3 3 3 3 3 3 3 3 4. Often 4 4 4 4 4 4 4 4 5. Always 5 5 5 5 5 5 5 5 98. Don’t know 98 98 98 98 98 98 98 98 99. Refused to answer 99 99 99 99 99 99 99 99 Q58.Which of the following statements best describes the amount of time you dedicate to household tasks? /INTERVIEWER, READ OUT ALL STATEMENTS, BUT ONLY MARK ONE / ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 78 1. You spend more time than you would like on household tasks 2. You spend the right amount of time on household tasks 3. You spend less time than you would like on household tasks 4. You don’t do any household tasks 98. Don’t know 99. Refused to answer General Empowerment Q59. To what extent do you agree with the following statements? There are no right or wrong answers; please state your opinion on each one. /INTERVIEWER, SHOW CARD Q59, READ THE STATEMENT AND MARK SUITABLE ANSWER/ Strongly agree Agree Disagree Strongly disagree Don’t know Refused to answer Q59.1 “Women should do what men say” 1 2 3 4 98 99 Q59.2 “Women must share their income with their husbands” 1 2 3 4 98 99 Q59.3 “It’s okay if men abandon women if they wish to” 1 2 3 4 98 99 Q59.4 “It’s okay if men chide women because they went out without any permission” 1 2 3 4 98 99 Q59.5 “It’s okay if men chide women if they don’t take care of children” 1 2 3 4 98 99 Q59.6 “The role of women is to earn money and take care of her family” 1 2 3 4 98 99 Q59.7 “A mother who works can establish a relationship as warm and solid with their children as a mother who doesn’t work” 1 2 3 4 98 99 Q59.8 “Father’s and mother’s dedication is equally important for the learning and effective development of children” 1 2 3 4 98 99 Q59.9 There are no gender inequality problems in my community 1 2 3 4 98 99 SECTION 6: NETWORKS AND SOCIAL SUPPORT Participation in Society Q60.Are there any social groups(s) for women in your community, such as microfinance groups, informal savings groups, loan groups, or other informal groups or associations? (If No, Don’t Know or Refused to answer, skip to Q64) 1. Yes 2. No 98. Don’t know 99. Refused to answer Q61.Are you an active member of any of these groups? (If No, Don’t Know or Refused to answer, skip to Q64) ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 79 1. Yes 2. No 98. Do not know 99. Refused to answer Q62.How much input do you have in making decisions in any of these groups? 1. Group leader 2. A lot 3. Some 4. A little 5. None 98. Don’t know 99. Refused to answer Business Networks Q63.Are you an active member of any formal trade, business or professional associations? (If No, Don’t Know or Refused to answer, skip to Q66) 1. Yes 2. No 98. Don’t know 99. Refused to answer Q64.How much input do you have in making decisions in these associations? 1. Group leader 2. A lot 3. Some 4. A little 5. None 98. Don’t know 99. Refused to answer Q65.In the last 12 months, have you participated in Trade Shows or Fairs? 1. Yes 2. No 98. Don’t know 99. Refused to answer ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 80 Q66.Where do you normally get most of your information about business and legal matters? Choose the top two sources / INTERVIEWER, SHOW CARD 0, CIRCLE ONLY ONE IN EACH COLUMN/ Source of information The 1st most important The 2nd most important 1. Media 1 1 2. Government publications 2 2 3. Chamber of commerce 3 3 4. Business association 4 4 5. Lawyer 5 5 6. Accountant 6 6 7. Business development or enterprise center, including business association 7 7 8. Independent professional business adviser 8 8 9. Public opinion 9 9 10. Young people (migrated from other regions bring new innovative ideals) 10 10 11. International organizations in my region working in the same sphere 11 11 12. Other business owner - male 12 12 13. Other business owner – female 13 13 14. Mentor (if you work in a company) 14 14 15. Internet 15 15 16. I read specialized literature 16 16 17. From nowhere 17 17 18. Other______________________(specify) 18 18 98. Don’t know 98 98 99. Refused to answer 99 99 Q67. Anytime during the last 12 months, have you implemented advice about your business from any of the sources below? /CHECK ALL THAT APPLY/ 1. Lawyer/attorney 2. Accountant 3. Professional business adviser 4. Small business or enterprise centre, including business associations 5. Other business owner (female) 6. Other business owner (male) 7. Haven’t implemented 98. Don’t know 99. Refused to answer Q68. How many people do you know that you can go to for business advice? |___|___| (people) 98. Don’t know 99. Refused to answer ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 81 Self-Efficacy Q69.Please tell me, in general, how satisfied are you with… Statement Very Satisfied Satisfied Neither Unsatisfied Very Unsatisfied Q69.1 ….the life you lead? 1 2 3 4 5 Q69.2 ….what you can accomplish in your business? 1 2 3 4 5 Q69.3 ….your business network? 1 2 3 4 5 Q69.4 ….your sense of harmony with others (family, friends, neighbors)? 1 2 3 4 5 Q70.How confident are you negotiating lower prices with suppliers? Q71. How confident are you negotiating higher prices with buyers? 1. Not at all confident 2. A Little confident 3. Sometimes confident sometimes not 4. Confident 5. Very confident 98. Don’t know 99. Refused to answer SECTION 7: SOCIODEMOGRAPHICS AND HOUSEHOLD CHARACTERISTICS Q72. What is your age? |___|___| years Q73.What is your current marital status? 1. Single 2. Married, husband living at home 3. Married, husband living and working in a foreign country 4. Separated/Divorced 5. Widow Q74.Currently, how many people, including you, live in your household? |___|___| (people) /INTERVIEWER, FILL THE TABLE WITH NUMBER OF PEOPLE LIVING AND EATING IN THE HOUSEHOLD/ Number of people 0.1 Women-older than 15 years living in household |__|__| 0.2 Men-older than 15 years living in household |__|__| 1. Not at all confident 2. A Little confident 3. Sometimes confident sometimes not 4. Confident 5. Very confident 98. Don’t know 99. Refused to answer ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 82 Q75.In the past month, how many of these people living in your household have earned income, including you? |___|___| (people) Q76.Over the last 12 months, what was the total household income? (including salary, pensions, aids, alimony, income from agriculture, from running business) / INTERVIEWER, CIRCLE THE APPROPRIATE RANGE BELOW / 1. No income 2. Less than 2 000 3. 2 001 – 5 000 4. 5 001 – 8 000 5. 8 001 – 10 000 6. 10 001 – 12 000 7. 12 001 – 15 000 8. 15 001 – 20 000 9. 20 001 – 30 000 10. 30 001 – 40 000 11. 40 001 – 50 000 12. 50 001 – 70 000 13. 70 001 – 100 000 14. 10 001 – 150 000 15. 150 001 – 200 000 16. More than 200 000 98. Don’t know 99. Refused to answer ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 83 Q77. Do you, or another household member, have any of the following assets? /INTERVIEWER, ASK ABOUT EACH TYPE OF INVESTMENT. CHECK ALL THAT APPLY/ PROMPT: Do you have other assets that are worth more than 30 000 soms? Assets Q77.1 Savings account (bank or credit union) 1. Yes 2. No 98. Don’t know 99. Refused to answer Q77.2 Securities (bonds, stocks) 1. Yes 2. No 98. Don’t know 99. Refused to answer Q77.3 Property, other than current residence 1. Yes 2. No 98. Don’t know 99. Refused to answer Q77.4 Cars 1. Yes 2. No 98. Don’t know 99. Refused to answer Q77.5 Livestock 1. Yes 2. No 98. Don’t know 99. Refused to answer Q77.6 Washing machine 1. Yes 2. No 98. Don’t know 99. Refused to answer Q77.7 Fridge 1. Yes 2. No 98. Don’t know 99. Refused to answer Q77.8 Computer 1. Yes 2. No 98. Don’t know 99. Refused to answer Q77.9 Other (specify): 1. Yes 2. No 98. Don’t know 99. Refused to answer Q77.10 Other (specify): 1. Yes 2. No 98. Don’t know 99. Refused to answer Q78.The household where you live is: 1. Owned and totally paid for by you 2. Owned and paying mortgage 3. Rented 4. Borrowed from a relative or friend 5. Employer provided 6. Other _____________________(specify) 98. Don’t know 99. Refused to answer Q79. How many children do you have? |___|___| (Children) 99. Refused to answer Q80.How many children under the age of 18 live in your household at the moment? |___|___| (Children) 99. Refused to answer ID# |___|___|___|___| Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 84 IMPORTANT!!! • THIS PAGE MUST BE PRINTED ON ONE SIDE ONLY • THIS PAGE MUST BE SEPARATED FROM THE SURVEY AFTER COMPLETION OF THE SURVEY, AND NOT BE STORED WITH SURVEY DATA • THIS PAGE MUST NOT BE COPIED • THIS PAGE MUST BE SENT TO MSI AS SOON AS POSSIBLE AFTER DATA COLLECTION SECTION 8: FUTURE CONTACT INFORMATION Thank you for your participation in this survey. May I please get your contact information for when our team needs to contact you again in the future? • E-mail 1: ___________________________________ • E-mail 2: ___________________________________ • Your address: ____________________________________________________________ • Telephone numbers: |___|___|___|___| |___|___| |___|___| |___|___| |___|___|___|___| |___|___| |___|___| |___|___| • Do you expect moving to a new home next year? ➢ (0) No ➢ (1) Yes • To what address? ____________________________________________________________ • To what region? ___________________________________ • Regardless of any change of home, with which family member could we contact to have information about your location? • Relationship to you: ___________________________________ • Name of relative: ___________________________________ • Land Phone Number of relative: |___|___|___|___| |___|___| |___|___| |___|___| • Cell Phone Number of relative: |___|___|___|___| |___|___| |___|___| |___|___| • Address of the relative: ________________________________________________________ Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 85 In-Depth Interview Guide [BEFORE STARTING THE INTERVIEW, PLEASE MAKE SURE THE RESPONDENT HAS SIGNED A CONSENT FORM] Section I: Greeting Time: 5 Minutes Good morning/afternoon. My name is ______________ and I work for Management Systems International, an international development organization, in partnership with USAID. Today we will talk about the USAID Women’s Leadership in Small and Medium Enterprises (WLSME) program. I am interested in learning about your experiences with the program, how you benefitted from it, and what you think may be improved. Please note that there are no right or wrong answers. This is an opportunity for you to share your experiences. Please feel free to express your honest opinion. I will not be recording your personal information and will treat everything you say here with confidentiality. The results of this discussion will be considered in aggregate. Nothing you say here will be publicly attributed to you, and your name will not be given to anyone. I will be asking you questions to guide the overall direction of this discussion. The audio of this discussion is being recorded for our own record and analysis purposes. The rules of confidentiality also apply to the audio recording of this discussion. Our primary objective is to understand your perspective and we may ask you a few follow up questions in response to what you tell us. Thank you very much in advance for your participation. FACILITATOR NOTES: • THE QUESTIONS THAT FOLLOW ARE THE BASIC GUIDING QUESTIONS, WHICH SHOULD BE ASKED. HOWEVER, ADDITIONAL QUESTIONS CAN BE ASKED IF YOU FEEL THEY ARE APPROPRIATE AND RELEVANT TO THE PURPOSE OF THE INTERVIEW. MAKE SURE TO RECORD THESE QUESTIONS AND ANSWERS IN THE FORM BELOW. • PLEASE USE YOUR OWN JUDGEMENT AS TO HOW MUCH PROBING PARTICIPANTS NEED TO COME UP WITH AN HONEST ANSWER. BE CAREFUL TO ASK QUESTIONS IN AN OBJECTIVE MANNER THAT DOES NOT LEAD THEM TO ANSWER ONE WAY OR ANOTHER. • THROUGHOUT THE DISCUSSION, PLEASE NOTE WHICH ANSWERS WERE MENTIONED BY THE PARTICIPANT FIRST (NO PROMPTING) AND WHICH WERE MENTIONED ONLY AFTER YOU MENTIONED IT AS A PROMPT. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 86 Interview Questions Time: 45 Minutes 1. I would like to start by asking you about your experience with the USAID WLSME program, which took place between 2013 and 2015. Do you remember participating in this program? [If respondent does not remember, remind her that this program consisted of business training program and a business plan competition, as well as market linkages activities]. a. What did you think of the program? b. One a scale from 1 to 10, with 1 being the lowest score and 10 being the best score, how would you rate this program overall? Why did you give this score? c. Can you give me three specific things you found favorable in this program? d. Can you tell me about three specific things you found unhelpful in this program? 2. Did the WLSME program help you to feel more prepared to tackle business challenges in the future? a. [IF YES:] i. Can you explain in detail how this program helped you? Which parts of the program? ii. Which business challenges has the program helped you prepare for? b. [IF NO:] i. Why do you think it did not help you feel more prepared to tackle business challenges in the future? ii. How do you still feel unprepared to tackle business challenges in the future? 3. Did the program help you feel prepared to tackle future personal challenges, outside of your business? a. [IF YES:] i. Can you explain in detail how this program helped you? Which parts of the program? ii. Which personal challenges has the program helped you prepare for? b. [IF NO:] i. Why do you think it did not help you feel more prepared to tackle personal challenges in the future? ii. How do you still feel unprepared to tackle personal challenges in the future? 4. On a scale from 1 to 10, with 1 being the lowest score and 10 being the best score, how prepared do you feel to tackle any challenges that may come your way? 5. Do you think the things you learned from this program are only temporary (i.e. you will forget them with time) or will they stay with you for a long time? Why? Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 87 6. The information we collected from surveys with WLSME participants shows no improvements in businesses’ sales or profits. Why do you think this is? a. Would you say this is also the case with you and your business? If yes, what do you think has prevented the improvement of your business? 7. One of the objectives of the program was to empower participants, so that they have the ability to take control over their own life and make decisions that affect their own well-being and their families, as well as have the ability to make strategic life choices. Do you feel like the program empowered you, for example to have more decision-making power and to have more control over your resources and choices, in your business and your personal life? Please explain. a. [IF YES:] Do you think these changes are temporary (i.e. things will go back to the way they were before the program) or are they long-lasting? b. [IF NO:] Why do you think no change was made? What prevents you from feeling in control of your life and making decisions? What other things do you need to feel empowered? 8. We have found from the surveys that, on average, there have not been increases in WLSME participants’ decision-making power and that it is hard to change gender norms about women’s roles, but we also found an increased perception that there are no gender inequalities in the community. Why do you think this is? a. Can you give me a specific example in your community, family, or social network, where you’ve seen that women empowerment is constrained? b. On a scale from 1 to 10, where 1 is not difficult and 10 is very difficult, how difficult do you think it is to change women’s empowerment? i. [IF ANSWER IS 8 OR HIGHER:] Do you think these gender norms are static and cannot be changed? Why? ii. [IF ANSWER IS 7 OR LESS:] What do you think is needed to really change and improve women empowerment? Can you please give me three specific examples? c. What do you think women’s empowerment depends on? [PROMPT: Society? Culture? Family? Religion? Other things?] Concluding Remarks Time: 5 minutes Thank you for sharing your experiences and thoughts. I have learned a lot from you today. I recognize that you have many other obligations and I appreciate you taking the time to help with this process. Your feedback and insight will be useful in our research. Does you have any questions regarding this discussion or any final comments? Here is my contact information if you want to add anything or have any particular questions or concerns you would like us to address. Thank you once again for your time and help. I would like to give you this gift of appreciation for your time and participation today. Can you please sign receipt of this gift? Thank you. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 88 ANNEX H: TEAM COMPOSITION A five-person team carried out core activities for this evaluation, supported by the Home Office teams from the E3 Analytics and Evaluation Project. The specific qualifications and roles for each team member are listed below. Each evaluation team member signed a conflict of interest disclosure statement that is retained by the MSI home office and available upon request. In addition, the Kyrgyz survey research firm, M-Vector Research and Consulting, conducted the survey data collection and provided transcription and translation support on the FGDs and IDIs. Principal Investigator Dr. Alberto Chong, an evaluation specialist external to USAID, holds a Ph.D. in Economics and is currently a professor at the Department of Economics at the Andrew Young School of Policy Studies at Georgia State University. Dr. Chong has worked extensively with the Inter-American Development Bank and the World Bank, has published several academic papers, including impact evaluation results with respect to gender, microfinance, and the private sector, and has applied advanced econometric and quantitative evaluation techniques to data collected in developing country contexts. Dr. Chong was primarily responsible for improving the quality of the evaluation design and minimizing its limitations, particularly with respect to the evidence to be obtained regarding causality and the attribution of outcomes to the project. Evaluation Coordinator Irene Velez holds a Master’s Degree in International Development Policy and has over seven years of global work experience designing and implementing impact evaluations. She has technical knowledge of different experimental and quasi-experimental evaluation methods, as well as practical experience managing the execution of these evaluations. She has also conducted large-scale data collection efforts, including hiring, training, and supervising survey teams as well as providing supervision and quality assurance oversight to subcontracted local survey firms. Ms. Velez was primarily responsible for overseeing and coordinating the execution of the evaluation design, ensuring efficient and timely reporting, and monitoring fidelity of the evaluation design. She is also the main technical point of contact on the evaluation team for USAID and the implementing partner. Local Qualitative Researchers Zura Mendikulova holds a Master’s Degree in Economics and Business Administration and has significant experience in planning and overseeing qualitative data collection processes for evaluations and similar studies, including with teams that are working at a distance. Ms. Mendikulova has experience facilitating FGDs while properly engaging and prompting participants, as well as experience assisting in the design of open-ended questions. She also has previous experience analyzing group-to-group variation and synthesizing the content analysis of results across FGDs. Ms. Mendikulova is fluent in English, Russian, and Kyrgyz. For this evaluation, Ms. Mendikulova facilitated the focus group discussions and conducted the in-depth interviews. Altyn Kapalova holds a Master’s Degree in Management of Non-Profit Organizations and additional academic experience in Policy Analysis and Ethnology, and an internship at the Department of Anthropology at Texas A&M University. She has significant qualitative research experience, leading and overseeing focus groups and interviews, and has conducted research on SME development in Kyrgyzstan and women’s role in this sector. She is fluent in English, Russian, and Kyrgyz. For this evaluation, Ms. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 89 Kapalova conducted the in-depth interviews and provided data quality control and oversight during data collection. Quantitative Data Analyst Angelo Cozzubo holds a Master’s Degree in Economics and has five years of research experience in applied microeconomics and conducting econometric analysis of panel data and poverty assessments. For this evaluation, Mr. Cozzubo supported the analysis of quantitative data collected for this impact evaluation. Survey Research Firm Following a competitive procurement process, MSI subcontracted with M-Vector to provide survey research services for the evaluation team in Kyrgyzstan, which included conducting the survey field work, logistical support for the FGDs, and transcription and translation services for the FGDs and IDIs. M-Vector, which also supported baseline data collection with FHI 360, is an international company that provides a wide range of data collection and analysis services for international and donor organizations in Central Asia, Kazakhstan and Russia. M-Vector has offices in Bishkek and Osh in Kyrgyzstan and is extensively involved in the European Society for Opinion and Marketing Research. Home Office Support Home Office support by the E3 Analytics and Evaluation Project team was also provided to the core evaluation team, including technical reviews, research assistance, qualitative data analysis, administrative oversight, management of the survey research firm, and logistical support. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 90 ANNEX I: REFERENCES Abadie, Alberto, Alexis Diamond, and Jens Hainmueller (2010). “Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program,” Journal of the American Statistical Association, 105(490). Angrist, J. D., & Pischke, J. S. (2008). Mostly harmless econometrics: An empiricist's companion. Princeton university press. Berge, Lars Ivar Oppedal, Kjetil Bjorvatn, Kartika Sari Juniwaty, and Bertil Tungodden (2012). “Business Training in Tanzania: From Research Driven Experiment to Local Implementation,” Journal of African Economics, forthcoming. Bernal, Raquel and Eduardo Peña (2014). “Diferencias en Diferencias: Notas,” Manuscrito, Universidad de Los Andes. Bruhn, Miriam, Dean Karlan and Antoinette Schoar (2012). “The Impact of Consulting Services on Small and Medium Enterprises: Evidence from a Randomized Trial in Mexico,” Yale Economics Department Working Paper 100. Bruhn, Miriam and Bilal Zia (2013). “Stimulating managerial capital in emerging markets: the impact of business training for young entrepreneurs,” Journal of Development Effectiveness, Taylor & Francis Journals, vol. 5(2), pages 232-266, June. Buvinic, Mayra, Rebecca Furst-Nichols, and Emily Courey Pryor (2013). “A Roadmap for Promoting Women’s Economic Empowerment,” United Nations Foundation, New York. Caselli, Francesco, and Nicola Gennaioli (2005). “Credit Constraints, Competition, and Meritocracy,” Journal of the European Economic Association 3(2‐3): 679-689. Cirera, Xavier and Qursum Qasim (2014). “Innovation, Technology, and Entrepreneurship, Policy Note,” World Bank, no. 5, September, Washington, DC. Cole, Shawn A., Thomas Sampson, and Bilal Zia (2011). “Prices or Knowledge? What Drives Demand for Financial Services in Emerging Markets?” Journal of Finance 66, no. 6: 1933–1967. De Mel, Suresh, David McKenzie and Christopher Woodruff (2014). “Business Training and Female Enterprise Start-up, Growth, and Dynamics: Experimental evidence from Sri Lanka,” Journal of Development Economics, 106, 199-210. De Mel, Suresh, David McKenzie and Christopher Woodruff (2009). “Measuring Microenterprise Profits: Must We Ask How the Sausage is Made?” Journal of Development Economics, 88(1), 19-31. Drexler, Alejandro, Greg Fischer, and Antoinette Schoar (2012). “Keeping it Simple: Financial Literacy and Rules of Thumb,” Manuscript, London School of Economics. Duflo, E., Glennerster, R., & Kremer, M. (2007). “Using randomization in development economics research: A toolkit,” Handbook of development economics, 4, 3895-3962. Impact Evaluation Report: WLSME Activity in the Kyrgyz Republic 91 Field, Erica, Seema Jayachandran and Rohini Pande (2010). “Do traditional institutions constrain female entrepreneurship? A field experiment on business training in India,” American Economic Review Papers and Proceedings 100(2): 125-29. Gathani, Sachin, and Dimitri Stoelinga (2013). “Export Similarity Networks and Proximity Control Methods for Comparative Case Studies.” Journal of Globalization and Development, 3(2): 1-39. Gine, Xavier and Ghazala Mansuri (2014). “Money or Ideas: A Field Experiment on Constraints to Entrepreneurship in Rural Pakistan,” Policy Research Paper, World Bank 6959, Washington, DC. Hejazi, Seyed Amir Mousavi, Mohamad Mehdi Maleki, and M. J. Naeji (2012). “Designing a Scale for Measuring Entrepreneurial Leadership in SMEs,” International Conference on Economics Marketing and Management, 28: 71-77. Singapore: IACSIT Press. Imbens G. and Wooldridge J. (2007). “Difference-in-Differences Estimation,” Lecture Notes 10, What’s New in Econometrics? NBER, Summer 2007. Karlan, Dean and Martin Valdivia (2012). “Teaching Entrepreneurship: Impact of Business Training on Microfinance Clients and Institutions,” The Review of Economics and Statistics, 93, 2: 510-527. Kulkani, Vami (2011). “Women’s Empowerment and Microfinance: An Asian Perspective Study,” Occasional Paper 13, International Fund for Agricultural Development, Washington, DC Mayoux, Linda (2001). “Microfinance and the Empowerment of Women: A Review of the Issues,” International Labor Organization, Geneva. McKenzie, David (2012). “Beyond Baseline and Follow-up: The Case for More T in Experiments,” Journal of Development Economics, 99(2): 210-221. McKenzie, David and Christopher Woodruff (2014). “What Are We Learning From Business Training and Entrepreneurship Evaluations Around the Developing World?” World Bank Research Observer, Forthcoming. Waldinger, Fabian (2014). “Differences-in-Differences,” Lecture Notes, University of Warwick, United Kingdom. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20004