Eval_u0;tion of . • J￾U SAID Pr_ograms To Develop , ) ·Business Skills. in Russia I CABANA Corporation Evaluation of USA.ID Programs To Develop Business Skills in Russia TE~4MMEMBERS :".frman Alibekov Byron Battle Nathaniel Bowditch Eiren_a Chichmeli Ivan Gerasimov Gerald Wein (Team Leader) July, 2000 4350 North Fairfax Drive, Suite 500, Arlington, VA 22203 USA telephone: (703) 243 1700 fa.~: (703) 243 0471 TABLE OF CONTENTS Executive Summary ................................................................................................i Acronyms ..............................................................................................................vii I. Introduction................................................................................................1 A. Background.....................................................................................1 B. Purpose and Methodology.............................................................2 C. The CARANA Evaluation Team..................................................5 D. Organization of this report............................................................5 E. Acknowledgements.........................................................................5 II. Entrepreneurial Business Services/West: Citizens Democracy Corps (CDC)/Counterpart International................................................7 Findings...........................................................................................8 Recommendations.........................................................................11 III. Entrepreneurial Business Services/East: ACDI/VOCA, Winrock International and IRIS ............................................................................13 Findings.........................................................................................14 Recommendations.........................................................................18 IV. SME Support Network Strengthening Project: Institutional Reform and the Informal Sector (IRIS)................................................20 Findings.........................................................................................22 Recommendations.........................................................................25 V. Regional Small Business in the Russian Far East: the American Russian Center (ARC) of the University of Alaska at Anchorage ......26 Findings.........................................................................................28 Recommendations.........................................................................31 VI. The Morozov Project: the Academy of Management and Market (AMM).................................................................................33 Findings.........................................................................................34 Recommendations.........................................................................37 VII. Cross-cutting Findings about these USAID Programs.........................39 A. Cross-cutting Findings on Process.................................................39 B. Cross-cutting Findings on Impact .................................................44 VIII. Implications for Future USAID SME Support Strategy......................51 A. Findings and Recommendations with Respect to Strengthening Business Skills and Business Service Infrastructure .................................................................................51 B. Findings and Recommendations with Respect to Other Elements of the Mission’s Strategy .................................................53 ANNEXES A. Scope of Work B. Evaluation Team Members C. BSIs Visited D. People Interviewed E. Interview Protocol i Executive Summary I. Introduction USAID efforts to stimulate the growth of small and medium-sized enterprises (SMEs) are central to its efforts to facilitate Russia’s transition to a market economy and to build democratic pluralism. USAID investments have included programs to enhance business skills, to develop systems for the provision of credit to entrepreneurs, and to improve the policy and regulatory environment in which business must function. This report reviews four programs with five grantees or contractors that address the need to improve business skills within the SME community. These programs include: · The Entrepreneurial Business Services Project (EBS) through the Citizens Democracy Corps (CDC) in western Russia and through ACDI-VOCA and Winrock International in eastern Russia · The SME Support Network Assessment and Strengthening Project (SME Network) through the University of Maryland’s Center for Institutional Reform and the Informal Sector (IRIS) · The Morozov Project (Morozov) through The Russian Academy of Management and Market (AMM) · Regional Small Business Development in the Russian Far East through the University of Alaska’s American Russian Center (ARC) Each of these programs provides business training and consulting to SMEs through Russian business support institutions (BSIs). However, the strategies for strengthening BSIs and for providing those services to SMEs vary across the projects. This evaluation assesses these programs, addressing questions of their impact on the intermediary BSIs, their impact on the SMEs, and their overall cost-effectiveness. The evaluation also reviews and makes recommendations about USAID’s overall program of business skills development. The following sections relate to the major sections of this report and summarize the evaluators' findings and recommendations with respect to each of these questions. (Section VII of the report is not discussed in this executive summary.) Page numbers are shown in parenthesis indicate the location in the text where each point is discussed. II. The Entrepreneurial Business Services Project (EBS) through the Citizens Democracy Corps (CDC) in western Russia Findings · CDC did an effective job in the selection of its LRPs. (p. 9) · CDC has done an effective job of training and coaching its LRPs to support the volunteer program. (p. 9) ii · CDC has effectively utilized the media to support LRP marketing activities. (p. 9) · CDC has been very effective at placing volunteers. (p. 9) · CDC's recruitment and preparation of volunteers has, for the most part, been effective. (p.10) · American volunteer technical assistance to Russian SMEs is very valuable. (p. 10) · Russian clients remain in contact with effective volunteers. (p. 10) · EBS/West is a moderate cost, high impact program. (p. 10) · The project has created a good system to evaluate volunteer results and to track client progress in implementing volunteer recommendations. (p. 11) · Some LRPs have successfully fielded volunteers in excess of their original quotas. Recommendations · USAID should extend the CDC grant for two years, if possible increasing resources to permit an increase in the numbers of volunteers and to allow the selection of additional LRPs in cities not currently being served. (p. 11) · Consider phasing-out CDC's in-country office. (p. 11) · Improve volunteer selection. (p. 11) · LRPs should work with clients to improve company-to-volunteer pre-assignment information and communication. (p. 11) · Given USAID’s broader goal of developing BSIs, CDC should place greater emphasis on building LRP consulting capacity in general, and not solely as a marketer of American volunteers. (p. 11) · Reflecting the need to generate revenue for the LRPs and to educate clients about consulting, CDC should place stronger emphasis on fee generation by LRPs. (p. 12) · USAID and CDC should collaborate in an effort to reduce the reporting required of LRPs. (p. 12) III. The Entrepreneurial Business Services Project (EBS) through ACDI-VOCA and Winrock International in eastern Russia Findings · The ACDI/VOCA consortium selected LRPs with diverse backgrounds. Some have considerable consulting experience. Several others have little or no experience in consulting, and this has slowed project implementation. (p. 14) · All but three LRPs (the ARCs) appear to be self-sustainable organizations. The project is likely to enhance their potential to remain sustainable. (p. 15) · The project's geographic coverage is uneven. (p. 15) · The ACDI/VOCA consortium placed a strong emphasis on LRP development. (p. 15) · Although the numbers of SMEs reached is modest, American business volunteers have a very strong positive impact on the SMEs with which they work. (p. 16) · The ACDI/VOCA consortium has had a slow start with volunteer placement but has accelerated its activities in recent months and should accomplish its targets. (p. 16) iii · The ACDI/VOCA consortium has put in place an excellent system to assess impact on SME clients. (p. 16) · The ACDI/VOCA consortium has implemented a powerful information sharing intranet network. (p. 17) · Russian clients remain in contact with effective volunteers. (p. 17) · EBS East is a moderate cost, medium impact program. (p. 17) · Multiple assignments for volunteers are sometimes difficult to arrange and burdensome on the volunteer, but make an important contribution to cost￾effectiveness. (p. 17) Recommendations · USAID should expand coverage in major cities East of the Urals, with special emphasis on Eastern Siberia. (p. 18) · The ACDI/VOCA consortium must, as a matter of the highest priority, keep working to expand and broaden in its volunteer databases in the U.S. and shorten the response time between a volunteer request and its fulfillment. (p. 18) · The program needs to improve its in-country marketing. (p. 18) · USAID should extend the EBS/East grant for two years, if possible increasing resources to permit an increase in the numbers of volunteers and to allow the selection of additional LRPs in cities not currently being served. (p. 18) · Consider phasing-out ACDI/VOCA’s in-country office. (. 18) · The ACDI/VOCA consortium should place greater emphasis on building LRP capacity to function as fully self-sustainable SME support institutions. (p. 18) · ACDI/VOCA should place more emphasis on fee generation by LRPs. (p. 18) · ACDI/VOCA should place more emphasis on fee generation by LRPs. (p. 19) · USAID and ACDI/VOCA should review the multiple assignment policy. (p. 19) · LRPs should work with clients to provide more company-to-volunteer, pre￾assignment documentation and information. (p. 20) IV. The SME Support Network Assessment and Strengthening Project (SME Network) through the University of Maryland’s Center for Institutional Reform and the Informal Sector (IRIS) Findings · IRIS’ selection of a very diverse group of BSIs offers the opportunity to experiment with new types of business service organizations, but it also reduces the potential to produce strong BSIs that carry out traditional business-skill enhancing activities. (p. 22) · Training provided to BSIs by the IRIS SME network project has been of good quality and appropriate to the needs of the majority of participants. (p. 23) · Technical assistance provided to IRIS BSIs through the EBS program has been productive. (p. 23) iv · The small grants program has had a significant financial and programmatic impact on the BSIs. However, the evaluation team questions whether some of the new services being developed will have a significant positive impact on SMEs or on BSI sustainability is uncertain. (p. 23) · IRIS SME network participants have benefited from informal networking opportunities provided by project training seminars. (p. 24) · IRIS BSIs are engaged in advocacy activities with local and regional officials. (p. 24) · Overall, the participating BSIs have benefited, but it is too early to determine how significant these benefits will be over the longer term. (p.24) · It is likely that the combination of training, technical assistance and grants will strengthen BSIs’ ability to provide effective services to the SME community. However, it is too early to see much impact at this time. (p. 25) · The IRIS SME Network project is a moderate cost, moderate benefit program · IRIS’ collaboration with the EBS program is mutually beneficial and makes both programs more cost-effective. (p. 25) Recommendations · If the IRIS program is to be continued, USAID should reconsider whether the selection of such a diverse assortment of BSIs is appropriate. The evaluation team believes that, if the program is extended and new BSIs are incorporated, selection criteria should emphasize more private sector-oriented BSIs with a demonstrated capacity to provide consulting services for a fee to a range of SME clients. (p. 25) · The grant review process should be tightened to emphasize activities for which there is clear SME demand and for which SMEs will pay. (25) · To increase impact and the assessment of impact, training sessions should normally end with participants developing “action plans” showing, as appropriate, a timetable and specific steps that need to be taken to implement ideas introduced in the training. IRIS should follow-up to determine whether or not plans are implemented. (p. 25) V. Regional Small Business Development in the Russian Far East through the University of Alaska’s American Russian Center (ARC) Findings · ARCs are performing at a high level. (p. 28) · ARCs have had a significant impact on their host communities. (p. 28) · ARCs are not sustainable as currently managed and financed. (p. 29) · The continuing dependence on Americans as trainers and managers, while contributing to the quality of the program and its short-term impact, is counterproductive to the achievement of longer-term capacity development and self￾sufficiency. (p. 29) · ARCs provide effective training for SME participants. (p.30) · ARC’s assessment of impact is largely anecdotal. (p. 30) · The creation of ARC “Alumni Business Circles” is a promising development. (p. 31) v · ARC is a high cost, high benefit program. (p. 31) Recommendations · USAID should ask ARC/Anchorage to articulate a vision and strategy for the program in which the three existing ARCs become independent Russian BSIs. (p. 31) · To implement this new strategy, the team recommends that USAID extend the ARC grant for a two-year transition period. (p. 31) · As contract resources are freed from support to the existing ARCs, consideration should be given to utilizing and extending the Alaskan experience with its many parallels to the RFE. (p. 31) VI. The Morozov Project (Morozov) through The Russian Academy of Management and Market (AMM) Findings · Over the seven years of this program, USAID assistance has helped AMM to develop an extensive network of BSIs delivering good quality business training. (p. 34) · Through the recent financial crisis, AMM helped to hold the Morozov system together, countering the closure of a number of centers with the opening of others and encouraging MRCs to open subcenters in smaller cities and towns. (p. 34) · The overall impact of AMM’s current program of support on MRCs’ capacity is positive, but modest. (p. 34) · AMM offers good training-of-trainers courses. The content of courses is generally appropriate, although it does not always meet the needs of the diverse MRCs. (p. 35) · The impact of AMM’s current program of support on MRCs varies by region. (p. 35) · MRCs benefit from informal networking at conferences and training sessions. (p. 35) · MRCs are self-sustaining. Current AMM activities are helpful but not critical to MRCs’ ability to remain financially viable in the future. (p. 35) · Morozov business training programs administered by the MRCs continue to have a major impact on SME development. (p. 36) · Although Morozov’s impact continues, the impact of the current USAID-AMM two￾year program is modest. (p. 36) · The current USAID-AMM Project is a low cost, low benefit program. (p. 36) · AMM has not succeeded in finding alternatives to USAID funding to support its core curriculum development and training of trainers activities. (p. 36) Recommendations · USAID should consider terminating, or dramatically altering, the current grantee arrangement with AMM. (p. 37) · If USAID decides to continue its support for AMM and/or the MRCs, the evaluation team recommends that AMM and USAID give priority to extending the Morozov. (p. system to unserved or under-served areas, to redressing current regional imbalances vi that lessen benefits to MRCs east of the Urals, and to expanding the products and services that AMM provides to MRCs. (p. 37) VII. Implications for USAID's Future SME Strategy Strengthening Business Skills and Business Service Infrastructure Findings · This review finds that the five programs reviewed are entirely consistent with the Mission’s stated objectives and strategy. (p. 51) · This study has found that many more than 70 self-sufficient BSIs exist. (p. 52) Recommendations · Give lower priority in USAID programming to business training. (p. 52) · Continue to support the provision of business consulting programs. (p. 52) · Selectively support the development of other business support services. (p. 53) · Explore interest in the development of regional BSI networks. (p. 53) · Under difficult resource constraints, focus activities selectively on promising regions. (p. 53) Other Elements of the Mission’s Strategy Findings · The responses suggest that the most serious impediments are, most importantly, the business environment and, secondly, financing. (p. 54) Recommendations · Give increased priority to work at the regional level to improve the business environment, including the development of business associations, the training of public sector officials, and the creation of regional BSI networks. (p. 54) · Expand efforts to strengthen credit systems, either through banks or non-banking institutions, to meet the needs of SMEs, with a focus on improving financial mechanisms and providing microenterprise credit. (p. 55) SME Evaluation: Acronyms ARC American Russian Center of the University of Alaska ARCs The business support institutions administered by ARC ACDI/VOCA formerly the Agricultural Cooperative Development Institute and Volunteers in Cooperation for America, now simply ACDI/VOCA BCC Business Collaboration Center BDI USAID’s Office of Business Development and Investment BSI Business support institution BSN Business Support Network BSO Business support organization BVP Business Volunteer Program CA Cooperative Agreement CDC Citizens Democracy Corps COTR Contract Officer’s Technical Representative DOC U.S. Department of Commerce EBRD European Bank for Reconstruction and Development EBS Entrepreneurial Business Services Program GKRP State Committee for Small and Medium Enterprise Support IRIS Center for International Reform and the Informal Sector LRP Local Russian Partner (a BSI partners under the EBS program) MRC Morozov Regional Center (a BSI in the Morozov program) NGO Non-governmental organization NSDP Novgorod SME Support and Development Project RII USAID’s Regional Investment Initiative SBWG Small Business Working Group of the U.S.-Russia Bi-national Commission SME Small and medium-sized enterprises SNAS SME Support Network Assessment and Strengthening Project (of IRIS) SO Strategic objective of USAID SUNY State University of New York TA Technical assistance TACIS The European Union’s economic assistance agency UFSQ Unified Field Survey Questionnaire (IRIS) USAID U.S. Agency for International Development VA Volunteer Advisor July 2000 SME Evaluation - 1 - I. Introduction A. Background USAID’s support for Russian small and medium-sized businesses (SMEs) is a key component of its efforts to support Russia’s transition to a market economy and a democratic society. That support has been oriented toward: · Creating a supportive business environment · Improving access to financing · Providing up-to-date business information · Developing SME business skills Focusing on business skills development (but also contributing to the other three objectives), USAID currently supports four major projects through five grantees: · The Entrepreneurial Business Services Project (EBS) through the Citizens Democracy Corps (CDC) · The Entrepreneurial Business Services Project (EBS) through ACDI-VOCA · The SME Support Network Assessment and Strengthening Project (SME Network) through the University of Maryland’s Center for Institutional Reform and the Informal Sector (IRIS) · The Morozov Project (Morozov) through The Russian Academy of Management and Market (AMM) · Regional Small Business Development in the Russian Far East through the University of Alaska’s American Russian Center (ARC) Each of these grantees implements its business skills development activities working directly with, or through, intermediary Russian business services and training organizations.1 These organizations are collectively referred to by USAID and in this report as business support institutions (BSIs). The number of BSIs supported by each of the USAID programs is shown in Table 1. Table 1. Number of BSIs by Program PROGRAM CDC ACDI/VOCA IRIS ARC MOROZOV TOTAL BSIs 9 10 33 3 59 114 As 14 BSIs receive support from two of this USAID-financed program, the 114 BSI support activities shown in this table actually support 100 BSIs. The overlap in support, shown in Figure 1 below, was the result of conscious decisions by the grantees in consultation with USAID. 1 In the case of ARC, the Russian BSIs are seen both as Russian institutions and as sub-centers of the University of Alaska’s American Russian Center. July 2000 SME Evaluation - 2 - Figure 1 BSIs Participating in USAID Programs and the Overlap Among Them IRIS—33 BSIs Morozov—59 MRCs 6 CDC—9 LRPs 3 ACDI/VOCA 10 LRPs 1 1 ARC 3 ARCs 3 B. Purpose and Methodology As part of its on-going review and assessment of the impact of its program, the USAID Mission to Russia (USAID/Russia) decided to undertake an evaluation of these activities. USAID/Russia contracted with the CARANA Corporation to conduct this evaluation between April – June 2000. This study has two principal purposes: · Looking at the past and present, to assess the progress of the above-mentioned projects, determining what is working well and what is not and what impact the programs are having on SMEs and on institutions (BSIs) that support SMEs · Looking toward the future, to suggest how USAID/Russia might best utilize its limited financial resources to support the development of Russia’s SME sector. The scope of work for this study (see Annex A) required the evaluation team to answer a great many questions about the operations, impact, sustainability2 and cost-effectiveness of the programs under review. Because of USAID’s interest in impact and sustainability, the team has carefully reviewed but is not reporting in depth on process indicators, i.e., the number of training sessions, people trained, etc. (These numbers are reported in detail in the various grantees’ monthly, quarterly and annual reports.) Rather, this report will 2 In this report, the term “sustainability” refers to the capability of organizations to provide services on a long-term basis without USAID funding. It is understood that NGOs and not-for-profit BSIs may require non-commercial financial support, either through government support, donor support, and/or fund-raising activities. July 2000 SME Evaluation - 3 - attempt to look behind the numbers to see why programs have proceeded well or not done so, and it will indicate, to the extent possible, what impact these activities are having. Each of the USAID grantees3 has numerous anecdotal stories that attest to the impact of its program. These stories are often compelling. However, apart from the rather questionable data on the number of jobs saved and created, none of the reviewed programs can as yet provide meaningful comprehensive data on impact. This is not surprising: rigorous impact assessments are time consuming and expensive, and the results are often debatable. The evaluation team’s assessment of impact has necessarily been largely qualitative rather than quantitative. It is based largely on the review of documents and interviews and on the judgment of the team members. To respond to the scope of work’s many questions, the CARANA team has organized much of this report around three critical questions about each program: · What has been the impact of these programs on developing effective, sustainable BSIs? Is there evidence that BSI training and consulting programs have improved during the period of the current program? Have BSIs initiated new services as a result of current programs? If so, are they reaching significant numbers of SMEs and generating revenue for the BSIs? Have BSIs improved their internal management? If so, with what results on service delivery and costs? Have the prospects for the BSIs’ long-term self-sufficiency improved as a result of the current program? How could the impact on BSIs be increased? · What has been the impact of the program on SME performance? Have current programs resulted in impacts at the SME level? Are participating BSIs reaching more SMEs? Is there evidence that SMEs are receiving improved programs that result in better decision-making, more investment, increased efficiency, or better marketing and sales? Are SMEs benefited by new services? If so, are those services helping SMEs to be more competitive? How could the impact on SMEs be increased? · What has been the overall cost-effectiveness of the program? What are the costs of the programs? How do the costs compare with the impacts that have been identified? How could cost-effectiveness be increased? The team’s methodology included the following steps: · Review of program documentation. The team began its efforts by reviewing an enormous quantity of program documentation. These included, inter alia, relevant USAID documents and strategy statements, the statement of work for each grantee; 3 This report will refer to the implementing organizations as grantees, although the team is aware of that fact that USAID provides its funds to these institutions as cooperative agreements or contracts. July 2000 SME Evaluation - 4 - USAID cooperative agreements; work plans; monthly, quarterly and annual progress reports, and other diverse reports and studies that have resulted from each grantees activities. · Initial meetings at grantee headquarters. The team met with the headquarters staff of several of the grantee organizations to gain a preliminary understanding of their mission definition, how their operations in Russia have been organized, how results are measured, lessons are learned, problems are encountered, etc. · Meetings with grantee’s Russian teams. Upon arrival in Russia, and following meetings with the USAID mission, the team met with the local headquarters of four of the five grantees, namely IRIS, CDC, ACDI/VOCA, and Morozov. The purpose was to corroborate our understanding of the objectives of each group as well as to coordinate our visits to the local BSIs. · Development of the interview protocol. The team developed an interview protocol for use in meetings with BSIs and BSI clients. The development of the protocol was an important team building activity, ensuring that each member had the same understanding of what questions the team needed to ask and the kinds of data that it needed to review to answer those questions. These points, and the interview protocol, were reviewed with the USAID mission to make certain that a clear consensus existed on the purposes of the assessment. (The team’s interview protocol is included in this report as Annex E.) · Field interviews. The team visited grantee BSIs, clients and government officials in 11 cities (Moscow, St. Petersburg, Novgorod, Rostov-on-Don, Voronezh, Yekaterinburg, Omsk, Novosibirsk, Valdivostok, Khabarovsk, and Yuzhno￾Sakhalinsk) during the period of April-May, 2000. Included in these visits were 27 of the 100 BSIs (27%) participating in the USAID programs, representing 35 of 114 (31%) of the program sites, as indicated in Table 2. Table 2. Number of BSI Programs the Evaluation Team Visited, by Program Program BSIs Visited CDC 5 ACDI/VOCA 6 IRIS 14 ARC 2 Morozov 8 Total 35 (A list of the BSIs visited is included as Annex C.) Overall, more than 150 people were interviewed, including USAID officers, representatives of grantees in the U.S. and in Russia, directors and other officers of July 2000 SME Evaluation - 5 - the BSIs, BSI clients, federal, regional and municipal government officials. (A list of people interviewed is included as Annex D.) Most of the data collected in the preparation of this report do not lend themselves to compilation. For example, although the evaluation team asked each BSI for numbers of people it reached through its training, consulting and other activities, responses tended to be lengthy, complex and subject to numerous caveats and conditions. These data cannot be tabulated, and indeed can be interpreted judgmentally only with care. With limited data on impact, the team found it particularly difficult to assess the cost-effectiveness of these programs, and comparisons of effectiveness and cost-effectiveness across projects whose outputs and impacts are different are extremely difficult to make. Nevertheless, in compliance with its scope of work and recognizing that USAID must make difficult decisions, often with less data than the evaluation team had at its disposal, the report does provide the team’s best judgment on these issues. The many judgments that the team did reach and that are reported in this document were the result of long hours in which the team members summarized, interpreted and argued over their observations from the review of literature and interviews. C. The CARANA Evaluation Team The CARANA team comprised six professionals, three U.S. nationals, and three Russians. Each of these individuals has worked with CARANA in numerous projects. All have worked in Russia previously and all have SME experience. Team members were Arman Alibekov, Byron Battle, Nathaniel Bowditch, Eirena Chichmeli, Ivan Gerasimov and Gerald Wein (Team Leader). Annex B provides brief biographical sketches of the team members. D. Organization of this report This remainder of this report is organized as follows: · Sections II through VI provide the team’s findings and recommendations on the five programs reviewed in this report. · Section VII, Cross-cutting Findings about USAID’s Skills Development Programs, presents a set of themes and observations that pertain to all or most of the five programs. · Section VIII, Implications for Future Programming, presents the teams thoughts and recommendations about future USAID strategy with respect to SME development. E. Acknowledgements The team expresses its appreciation to the many people who provided their time, ideas, guidance and information. Each of the five grantees provided the team with extensive July 2000 SME Evaluation - 6 - briefings, documentation, and assistance in scheduling meetings. Dozens of BSI staff members and consultants met with the team, provided information on their programs, and shared their perspectives on the SME sector and its future. USAID staff provided guidance on the Mission’s strategy and programs, people to interview and many documents. July 2000 SME Evaluation - 7 - II. Entrepreneurial Business Services/West: Citizens Democracy Corps (CDC)/ Counterpart International The Citizens Democracy Corps (CDC), in conjunction with Counterpart International, is implementing an American SME technical assistance volunteer program in Western Russia. The program comprises three components: provision of technical assistance and training to SMEs by American business volunteers, selection and capacity-building of BSIs to market and support the American volunteer program in-country and more generally to strengthen the BSIs as viable institutions, and creation of an overall program information system. Over the two years of the program, CDC is committed to providing American volunteers to a minimum of 700 SME clients in Western Russia. CDC first selected eleven BSIs (later reduced to nine) from western Russia to participate in the program as Local Russian Partners (LRPs). CDC’s current LRPs include five for￾profit institutions and four not-for-profit organizations. CDC has trained these LRPs to market American volunteer business consultants to SMEs, to prepare scopes of work for those assignments, and to monitor and evaluate the results. CDC is responsible for overall program management, recruitment and orientation of volunteers, and transporting them to and from their volunteer job assignments. To be selected for the program, BSIs had to meet a set of criteria that included being in operation as a consulting or training firm for at least one year, having experience with U.S. Government support programs and having a good relationship with the local government. Most CDC LRPs are private companies in business to provide consulting services to SMEs. These LRPs survive largely on revenue from sales. A few CDC LRPs are non-commercial, non-governmental organizations (NGOs) partly “owned” by a municipality or regional governmental body and partially subsidized by grants from the Russian government and/or international donors. CDC has a sub-agreement with Counterpart International to use its diagnostic tools for evaluating the level of organizational development of each of the participating LRPs. During the start-up phase of the program, representatives from Counterpart International and a local firm, Partner, visited each LRP, performed a participatory evaluation and developed a customized training and technical assistance plan (TATAP) that has served as a guide in strengthening the LRPs. CDC assigned each LRP a target for marketing volunteers and trained the LRPs in how to generate demand, support volunteers and assess results. CDC also provides monthly training on subjects of importance to the LRPs’ consulting business. Training covers subjects as creating demand through seminars, competing for contracts and grants, assisting SMEs to find financing and trade partners, developing products and services that help clients get results, performing in-company diagnostic studies, developing action plans, and tracking results. July 2000 SME Evaluation - 8 - In addition to receiving training, the LRPs receive a fee from CDC for each American volunteer assignment resulting from the LRPs work. This fee, based on how well the LRP carries out its various volunteer support activities, ranges from $500-700 per primary assignment and about $100 per secondary assignment. (Primary and secondary assignments are explained below.) LRPs may also charge their clients a fee for the volunteer’s services. The grant agreement encourages the collection of fees for volunteer services and stipulates “there will be considerable effort made toward having SMEs pay a flat fee to the Russian BSO/BSI.” LRPs market the volunteer program to small and medium-sized enterprises in the service sector, light manufacturing, non-financial services, and other businesses that are close to consumer markets. Strategic business sectors for the EBS West program include distribution, transportation, retail sales and operations, construction and building materials, light manufacturing, business services, and food processing. Each EBS LRP designates a program manager to oversee its participation in the program. The program manager is responsible for the assignment development process culminating in a scope of work. The scope of work includes company background, a description of its operations, the goals of the proposed volunteer assignment and the expected deliverables/results. Program managers are also responsible for volunteer support and for post-assignment reporting, including completion of a volunteer assignment evaluation and an assessment of the client’s implementation of the volunteer’s recommendations. This tracking of clients’ implementation of volunteer recommendations continues for six months after completion of the assignment. CDC has created a website (www.ebs.ru) that provides comprehensive information about the program in Russian and English. The site serves Russian companies, existing and potential volunteers and other users. It also contains a limited-access database containing scopes of work and other strategic information for program managers. In April 1999, USAID gave CDC a two-year grant expected to total $4.855 million. That amount was first raised by $100,000 to accommodate USAID’s desire to expand programming in Samara, then subsequently reduced because of cuts in USAID’s overall budget for Russia. The amount now available for CDC’s two-year program is $4.428 million. CDC’s target for American volunteer assignments was maintained at its original level despite the budget reduction. According to the terms of the grant agreement, each American volunteer consultant is expected to perform multiple assignments, typically one longer “primary” assignment and one or two short “secondary” assignments. CDC has committed to USAID that it will achieve an average of 2.6 assignments per American volunteer fielded. Under the terms of the agreement, USAID absorbs all volunteer costs with the exception that Russian clients are asked to contribute to the cost of the volunteer by being responsible for accommodation, transportation and interpretation. July 2000 SME Evaluation - 9 - Findings What has been the impact of the program on developing effective, sustainable BSIs? · CDC did an effective job in the selection of its LRPs. CDC selected 11 BSIs (subsequently pared down to nine) to serve as LRPs. In making its selection, CDC emphasized institutions engaged in for-profit consulting. LRPs’ engagement with private sector clients, understanding of the role of consultants and experience in marketing and managing consulting assignments have been important factors in their success as CDC local partners. · CDC has done an effective job of training and coaching its LRPs to support the volunteer program. CDC’s program managers have trained the LRPs to identify clients, articulate consulting needs and serve as a conduit for information between the clients and volunteers, and the program managers are continuing to play a major role in these processes. As LRPs appear to have acquired many of the skills necessary to support the volunteer program, the role of CDC’s program managers is expected to decrease in the future. The CDC program has also provided training and consultants to develop LRP capacity in other areas. CDC volunteers’ secondary assignments have frequently targeted LRP capacity building. Second, through its collaboration with IRIS’ SME Network training program, LRPs may attend selected training seminars set up for IRIS’ 33 BSIs. At the time of April 2000, IRIS reports that only six participants from CDC LRPs had availed themselves of this opportunity. Finally, some LRPs have coupled or “twinned” a Russian consultant to an American volunteer so that the former can increase his/her skills. · CDC has effectively utilized the media to support LRP marketing activities. CDC’s in-country management has demonstrated strong marketing skills, training LRPs and using the media to reach an understandably skeptical audience about the unique potential contributions of American volunteer consultants. Volunteer placement has been greatly facilitated by CDC’s effective mass media promotion as a back-up to LRP marketing activities. · CDC’s LRPs are sustainable. CDC chose BSIs that were already reasonably healthy and sustainable. The program has generated substantial increased income for these participating institutions, and, more importantly, the LRPs have increased their client￾base and consulting skills so that they can generate income in the future. What has been the impact of the program on SMEs? · CDC has been very effective at placing volunteers. CDC’s volunteer recruitment and matching process improved steadily during the first year of the grant, and CDC exceeded its volunteer placement targets. As of the end of May 2000, CDC had fielded over 200 volunteers and claimed to have completed more than 700 July 2000 SME Evaluation - 10 - assignments (exceeding the target set for the entire two-year project). It is particularly impressive that CDC has achieved its target ahead of schedule and despite a reduction in USAID funding. It is clear that the CDC “formula” has the capacity to field a substantially greater number of volunteers and volunteer assignments if additional funding were available to support this. CDC also attributes some of its success to a marked increase in the demand for Western expertise -- partly the result of USAID, TACIS and other business assistance programs -- following the 1998 financial crisis. · CDC’s recruitment and preparation of volunteers has, for the most part, been effective. CDC LRPs noted that, while improvements are visible, matching assignments with volunteer skills in a timely way remains a challenge. Clients commented that the expectations of volunteers coming to work with them were not always on target, suggesting that preparation for assignments -- both in Russia and the U.S. -- could be improved. · American volunteer technical assistance to Russian SMEs is very valuable. The team’s meetings with clients confirmed that, with rare exception, business owners greatly appreciated the contributions of their volunteer consultants. Many SME clients spoke of their initial skepticism about the value of a consultant – especially an American consultant with limited knowledge of Russian business practices. Time and again, these same clients described to team members how American volunteer consultants with real-world operating experience had filled a particular Russian business information vacuum with realistic, practical, and strategic advice geared to today’s global free market economy. Most of these clients also gave examples (which CDC has also described in its “success stories”) of improvements in organization, management, production and sales. · Russian clients remain in contact with effective volunteers. Interviews with SMEs that had received volunteers revealed that the client and the volunteer often maintain some relationship after the visit, typically by e-mail. The team believes that this continuing dialogue enhances the impact of the assignment. Two interviewed former clients reported that they were discussing or had successfully concluded joint venture discussions with their volunteers. What has been the overall cost-effectiveness of the program? · EBS West is a moderate cost, high impact program. In addition to the impact on BSIs and SMEs noted above, the presence of a significant number of American businesspeople providing assistance as volunteers in many Russian cities generates considerable goodwill toward the United States. The USAID grant amounts to about $245,000 per BSI (LRP) per year, although this figure is less meaningful (than in the case of other USAID projects being reviewed) because a large share of EBS resources are used to provide direct assistance to SMEs. In the first ten months of operations (through March 2000), CDC had completed 580 assignments. CDC’s reported expenditures for that period were $1.827 million. These figures suggest a cost per volunteer of about $11,650 while the cost per assignment would work out to about July 2000 SME Evaluation - 11 - $3,150 (with an actual ratio of 3.7 assignments per volunteer). CDC’s dynamic leadership, media savvy, experience, effective volunteer recruitment system and team approach to working with LRPs, operating in Western Russia’s stronger economy and business environment, have yielded laudable results. · The project has created a good system to evaluate volunteer results and to track client progress in implementing volunteer recommendations. · Some LRPs have successfully fielded volunteers far in excess of their original quotas. One or two CDC LRPs have performed extraordinarily well, leading some to ponder seriously the possibility of increasing the “sales” territory granted to these aggressive firms. While the team applauds the success of these LRPs, it is also true that -- with a limited budget – an LRP that exceeds its volunteer quota causes other LRPs whose learning curve may be longer and other geographic regions to receive fewer volunteers. Recommendations · USAID should extend the CDC grant for two years, if possible increasing resources to permit an increase in the numbers of volunteers and to allow the selection of additional LRPs in cities not currently being served. As indicated above, the team finds that this program has significant impact directly on SMEs, and it provides a type of resource for which there is effectively no substitute. If resources permit, it would be useful to provide this unique resource to more Russian SMEs. · Consider phasing-out CDC’s in-country office. As LRPs reach full maturity in their capacity to manage the demand side of the American business volunteer program, CDC’s in-country office should be phased out. If it is necessary to maintain the Moscow office (e.g., to support the development of new LRPs), CDC should train a replacement for its American director. · Improve volunteer selection. The team endorses CDC’s plan to place a Russian recruitment officer in its home office to facilitate the effective matching of volunteers to the needs of Russian clients. · LRPs should work with clients to improve company-to-volunteer pre-assignment information and communication. This will help the volunteers to prepare more effectively and to have a better picture of what to expect both in business and in day￾to-day life when s/he arrives in Russia. · Given USAID’s broader goal of developing BSIs, CDC should place greater emphasis on building LRP consulting capacity in general, and not solely as a marketer of American volunteers. One recommended way to accomplish this objective is to increase the “twinning” of Russian consultants with American volunteers. In addition, CDC might encourage secondary assignments with LRPs, i.e., have volunteers work more closely with the LRPs, suggesting methods to improve July 2000 SME Evaluation - 12 - internal organization, marketing and other business tasks. The LRPs that have been most successful in identifying assignments and that have exceeded their quotas – that have uncovered a considerable demand for consulting services – would appear to be particularly appropriate institutions in which to increase emphasis on expanding consulting with Russians. USAID’s grant agreement with CDC needs to give increased priority to the development of this capacity. · Reflecting the need to generate revenue for the LRPs and to educate clients about consulting, CDC should place stronger emphasis on fee generation by LRPs. As the effectiveness of the American business volunteers becomes better known in the local business community, charging more substantial fees should be possible. · USAID and CDC should collaborate in an effort to reduce the reporting required of LRPs. It is particularly important excessive paperwork not be allowed to divert LRP program managers’ attention from tasks that directly support the program. July 2000 SME Evaluation - 13 - III. Entrepreneurial Business Services/East: ACDI/VOCA, Winrock International and IRIS4 In April 1999, USAID awarded the ACDI/VOCA-Winrock-IRIS consortium a two-year grant to implement the EBS/East program. USAID funding for the grant totals $3,363,863. Under this program, the ACDI/VOCA group provides American business volunteers to selected SMEs in that part of Russia lying east of the Urals. The ACDI/VOCA program comprises the same three components as EBS/West: provision of technical assistance and training to SMEs by American volunteers, selection and capacity building of LRPs to market the program in-country and more generally to strengthen the BSIs as viable institutions, and creation of an overall program information system. The program has also created an LRP/client database and a program Web site (www.ebs￾sfe.ru). Over the two years of the program, ACDI/VOCA was to provide 290 American volunteer consultants to 725 clients in Eastern Russia through a network of ten LRPs. As USAID was forced to reduce the program budget, ACDI/VOCA secured USAID approval to reduce these targets by 20 percent, to 231 volunteers and 578 clients. Like CDC in the western part of the country, USAID required that ACDI/VOCA utilize Russian LRPs to generate demand and support volunteers in the field. ACDI/VOCA documentation indicates that it selected LRPs with “ongoing business support operations that have developed an already successful record of business support services.” Specific responsibilities of the LRPs include marketing American volunteer consultants to prospective SME clients; helping clients to assess their needs; preparing scopes of work for volunteers; working with the clients to ensure that volunteers receive adequate housing, transportation and interpretation services, and monitoring and evaluating results. Responsibilities of the ACDI/VOCA consortium members include overall program management, selection and training of LRPs, review and approval of scopes of work, recruitment and orientation of volunteers, and transporting them to and from their volunteer job assignments. In carrying out these responsibilities, ACDI/VOCA and Winrock are each responsible for working with and overseeing five of the group’s LRPs. The consortium assigned each LRP an annual quota for marketing volunteers, i.e., a target for the number of volunteers that it should place annually. Each volunteer works with several clients. According to the terms of the grant agreement, EBS/East American volunteer consultants should carry out an average of 2.5 assignments. These assignments may be of approximately equal length, or there may be one major assignment and one or more very short (e.g., one day) assignments. Occasionally, short assignments are LRP￾hosted seminars for a group of SMEs. Volunteers typically remain in Russia for two to three weeks. 4 This paper will refer to this consortium as ACDI/VOCA. July 2000 SME Evaluation - 14 - To identify candidates to fill the scopes of work developed by the LRPs, ACDI/VOCA and Winrock each maintain a database of individuals who have expressed interest in volunteering to work in an overseas assignment. Each organization utilizes its own database to fill assignments coming from their respective LRPs. However, ACDI/VOCA staff indicate that each organization will ask the other for help, as needed, when recruitment is difficult. USAID finances all project costs except for volunteers’ accommodation, transportation and interpretation. To the extent feasible, these cost are borne by the SME clients receiving the volunteers’ assistance. LRPs costs are covered through a set of five incentive fees paid by ACDI/VOCA and Winrock following the successful completion of various volunteer support activities. The five fees total approximately $600 per assignment. Thus, an LRP might earn as much as $1,800 for bringing in a volunteer who carries out three assignments. LRPs are also permitted to ask clients to pay a fee (that is retained by the LRP), and several are doing so. The ACDI/VOCA consortium provides training to the LRPs’ staff in the development of consulting assignments for the volunteers and in an array of topics that will help them to increase their consulting capacity. This is done through a combination of technical assistance provided by in-country consortium staff, BSI training courses run by IRIS, and by EBS volunteers. Each LRP has designated a program manager to oversee its participation in the program. The program manager is responsible for the assignment development process culminating in a scope of work. The scope of work includes company background, a description of its operations, the goals of the proposed volunteer assignment and the expected deliverables/results. LRP managers are also responsible for volunteer support and various post-assignment reporting duties, including a volunteer assignment evaluation and a follow-up to determine whether the client has implemented the volunteer’s recommendations and with what results. ACDI/VOCA has developed an impact evaluation system that includes an initial LRP follow-up visit to the client -- within three to six months following the volunteer’s consultancy -- at a point in time when the volunteer thinks that action should have been taken. A second visit occurs approximately 12 months after the completion of the volunteer’s assignment. Findings What has been the impact of the program on developing effective, sustainable BSIs? · The ACDI/VOCA consortium selected LRPs with diverse backgrounds. Some have considerable consulting experience. Several others have little or no experience in consulting, and this has slowed project implementation. The LRPs in the EBS/East program are committed to the SME sector, and they have strong records of support to July 2000 SME Evaluation - 15 - the SME community. However, these LRPs have a less private sector, more non￾commercial NGO “flavor” to them than the LRPs chosen by EBS/West. The issue here is not whether they are for-profit or not-for-profit, but whether they think of themselves as businesses selling a valuable service to other businesses or as social service organizations: the team’s observations suggest that the former make better LRPs. Compared with the LRPs of CDC, ACDI/VOCA’s LRPs started the program with less of an established client base for consulting, and they faced a longer and steeper “learning curve” before they could produce acceptable scopes of work. Several factors appear to underlie ACDI/VOCA’s selection of LRPs. The inclusion, for example, of the three ARC centers, whose very limited consulting experience makes them less than ideal institutions for developing EBS consulting assignments, should help those centers which are of obvious USAID interest to move toward financial viability. It is also true that ACDI/VOCA had a more difficult time finding strong candidates for LRPs in the east than did CDC in the west where there are many more BSIs and more SME activity. · All but three LRPs (the ARCs) appear to be self-sustainable organizations. The project is likely to enhance their potential to remain sustainable (or to achieve it). Participating LRPs, with the exception of the three ARCs and perhaps the newly formed LRP in Vladivostok, appear to have been viable institutions at the outset of the project. As described above, the project gives the LRPs extensive training, technical assistance, additional income and the opportunity to expand their client base. In addition to the income the LRPs generate in payments from the project, one LRP reported that it charged as much as $100/day for volunteers. However, the program’s long-term impact on LRP sustainability will be determined less by the revenues the LRPs generate from fielding American volunteers than by the growth in their capacity to sell consulting with Russian consultants. In this regard, the program is making an important contribution to enhancing LRPs’ sustainability by increasing their capacity to market consulting. The program is not making a great deal of progress (and was designed less to do so) in developing Russian consultants. · The project’s geographic coverage is uneven. There are no LRPs (and therefore no volunteers) in Eastern Siberia, while two cities in the Far East have two LRPs each. Additional resources would be required to extend the program to unserved areas. · The ACDI/VOCA consortium placed a strong emphasis on LRP development. ACDI/VOCA has provided extensive technical assistance to the LRPs to enable them to carry out the many activities required to develop consulting assignments, to support volunteers and to assess impact. In addition, each LRP was able to participate in two of the workshops IRIS organized for its own BSIs. IRIS records show that 21 individuals from ACDI/VOCA LRPs have availed themselves of this opportunity. The ACDI/VOCA consortium also convened an annual workshop that provided an opportunity for information-sharing and team building. Of greatest benefit to the building of BSI capacity in EBS/East has been the policy of committing three volunteer primary assignments to the strengthening of each participating LRP. These July 2000 SME Evaluation - 16 - programs represent a serious commitment to LRP development, which should pay dividends in terms of LRP capacity and long-term sustainability. What has been the impact of the program on SMEs? · Although the numbers of SMEs reached is modest, American business volunteers have a very strong positive impact on the SMEs with which they work . The team’s meetings with clients confirmed that, with rare exception, business owners benefited greatly from their volunteer consultants. This view often contrasted with their initial skepticism about using a consultant, particularly an American consultant with limited knowledge of Russian business practices. Clients spoke enthusiastically about the benefit of having had a volunteer with many years of practical experience. Both clients and LRP representatives indicated that there are no comparable Russian consultants, at least not outside of Moscow and St. Petersburg. Although ACDI/VOCA’s impact assessment instrument is just now being applied for the first time, strong anecdotal evidence suggests that volunteers often have a profound impact on the SMEs they assist. (See, for example, ACDI/VOCA “success stories.” · The ACDI/VOCA consortium has had a slow start with volunteer placement but has accelerated its activities in recent months and should accomplish its targets. The consortium’s year one quotas were 120 volunteers and 300 assignments. During the first 12 months of the project (through April 2000) the consortium actually fielded 65 volunteers (54 percent of the target) and completed 186 assignments (62 percent of the target). If, as the consortium has sometimes suggested, the targets are interpreted as approved requests instead of assignments completed, ACDI/VOCA actually exceeded its targets. There were145 volunteers and 334 assignments approved at the end of April 2000, 120 percent and 111 percent respectively of the first year targets. The team believes that the main causes were: initially weaker LRP capacity (especially in marketing) and, correspondingly, strong program emphasis on upgrading LRP capacity; the less developed economy east of the Urals and the consequent weaker demand for consultants; and the more challenging operating environment East of the Urals, and difficulty recruiting volunteers. The considerable number of approved assignments indicates that the consortium and its LRPs have made considerable progress during the first year in overcoming the obstacles in Russia. Progress appears to have also been made in overcoming recruiting obstacles. As a result, during the four-month period from May to August 2000, the consortium expects to field an additional 83 volunteers who will carry out 233 assignments. With this accelerated pace, overall goals for serving SMEs with American business volunteers should be achieved. · The ACDI/VOCA consortium has put in place an excellent system to assess impact on SME clients. A series of post-volunteer interview questionnaires to be administered by LRP EBS managers has the potential to effectively track volunteer effectiveness and to measure impact. This system appears to be a good tool to encourage clients to July 2000 SME Evaluation - 17 - implement the recommendations of the volunteer consultant and to generate follow￾up consulting assignments for the LRP · The ACDI/VOCA consortium has implemented a powerful information sharing intranet network . Despite initial installation difficulties, the ACDI/VOCA website is an effective bilingual information source for Russian companies, volunteers and others. The site also contains a second, password-protected management information system. This has allowed managers within EBS East LRPs to interact with one another, rapidly share formal and informal evaluations of volunteer effectiveness and exchange ideas with respect to marketing, developing scopes of work and sharing of volunteer expertise. · Russian clients remain in contact with effective volunteers. It was not uncommon for client SMEs to report that they have a continuing relationship with a former volunteer, typically by e-mail. Because of the great potential to achieve substantial results from longer-term volunteer-client relationships, USAID and ACDI/VOCA may wish to review their policy on repeat visits by volunteers to previous clients. What has been the overall cost effectiveness of the program? · EBS East is a moderate cost, medium impact program. The USAID grant amounts to $93,000 per BSI (LRP) per year. Assuming that costs in the first year are close to the planned budget, the cost per assignment – utilizing the figure of 184 completed assignments as of April 2000 – was approximately $5,000. Although EBS/East operates in a region that is difficult, the team believes that the disappointing volunteer placement numbers are due primarily to factors for which the consortium must take responsibility: questionable LRP selection and delays in volunteer recruitment. However, the EBS/East LRPs appear to have now “learned their trade,” and they are now producing volunteer requests at a more rapid rate. Effective volunteer recruitment is now the overriding priority. As the number of volunteers rises, the cost per assignment will fall and the cost-effectiveness of the program will increase. · Multiple assignments for volunteers are sometimes difficult to arrange and burdensome on the volunteer, but make an important contribution to cost￾effectiveness. Several program managers indicated that finding multiple assignments for a volunteer can be difficult. Multiple assignments are more likely to be a problem when the initial or primary assignment requires a specialization (e.g., paint production technology or quality control in the production of plastic bottles) not likely to be of widespread interest to other potential clients. Moreover, multiple assignments involving competing firms cause the clients to question whether their confidential information might find its way to that competitor. Finally, managing multiple assignments is sometimes difficult for the volunteer and/or bothersome to the client. In spite of these concerns, the EBS grantees and the LRPs have made the July 2000 SME Evaluation - 18 - multiple assignment system work, and this has substantially increased the impact of the EBS program. Recommendations · USAID should expand coverage in major cities East of the Urals, with special emphasis on Eastern Siberia. As indicated above, there are a number of areas that have no access to the project’s resources. · The ACDI/VOCA consortium must, as a matter of the highest priority, keep working to expand and broaden in its volunteer databases in the U.S. and shorten the response time between a volunteer request and its fulfillment. ACDI/VOCA and Winrock might also take steps, if not to share their databases, to at least turn more quickly to their partner’s database if they have difficulty finding an appropriate volunteer within their own system. · The program needs to improve its in-country marketing. Although the ACDI/VOCA LRPs are now producing a sufficient number of eligible scopes of work to meet project targets, the team believes that it would be useful to continue efforts to stimulate demand. The existence of more demand than the program can meet will allow the program to encourage a sense of competition for scarce volunteer resources, permit the consortium to pick and choose from among potential clients, facilitate the notion that SMEs should pay a fee (in addition to in-country support costs) for volunteers, and contribute to the establishment of a broader LRP-based consulting business that would include the use of Russian consultants. · USAID should extend the EBS/East grant for two years, if possible increasing resources to permit an increase in the numbers of volunteers and to allow the selection of additional LRPs in cities not currently being served. As indicated above, the team finds that this program has significant directly on SMEs and it provides a type of resource for which there is effectively no substitute. If resources permit, it would be useful to provide this unique resource to more Russian SMEs. · Consider phasing-out ACDI/VOCA’s in-country office. As LRPs reach full maturity in their capacity to manage the demand size of the American business volunteer program, ACDI/VOCA’s in-country office should be phased out. If it is necessary to maintain the Khabarovsk office (e.g., to support the development of new LRPs), ACDI/VOCA should train a replacement for its American director. · The ACDI/VOCA consortium should place greater emphasis on building LRP capacity to function as fully self-sustainable SME support institutions. One recommended way to accomplish this objective is to increase the “twinning” of Russian consultants with American volunteers. · ACDI/VOCA should place more emphasis on fee generation by LRPs. As the effectiveness of the American business volunteers becomes better known in the local July 2000 SME Evaluation - 19 - business community, charging more substantial fees should be possible. Charging reasonable consulting fees is important to LRPs’ long-term financial viability. · ACDI/VOCA and USAID should work together to reduce the volume of paperwork and reporting required. It is particularly important to reduce the reporting that must start with the LRP program managers as that reporting is currently diverting their attention from key program tasks. Eliminating monthly reporting would be a valuable first step. · USAID and ACDI/VOCA should review the multiple assignment policy. Despite the fact that the ACDI/VOCA consortium has met its multiple assignment target, experience has shown that multiple assignments can be problematic. Multiple assignments are most likely to create difficulties when the volunteer’s stay is particularly short, the primary client has very high expectations, and/or the volunteer is performing his/her first consulting assignment or first trip to Russia. A reduction in multiple assignments would also provide more opportunity for volunteers to assist in finding new clients, to do more LRP capacity building (which often is a secondary assignment), and to devote more attention to “twinning” with Russian LRP consultants. · LRPs should work with clients to provide more company-to-volunteer, pre￾assignment documentation and information. Several clients reported that volunteers often arrived with an under-rated set of expectations about the capability of their Russian clients. July 2000 SME Evaluation - 20 - IV. SME Support Network Strengthening Project: Institutional Reform and the Informal Sector (IRIS) According to the USAID-IRIS contract, the goal of the IRIS network project is “to … strengthen those BSIs that are in the best position to provide long-term expertise to entrepreneurs after USG funding is completed.”5 . The IRIS SME network project is currently operating under a two-year, $2 million per year grant that expires in August 2000. As a first step in its project, IRIS conducted a countrywide BSI assessment and selection process. This required about nine months of the two-year contract. IRIS assessed 296 BSIs in 89 locations across Russia. To be selected for inclusion in the project, BSIs had to be legally registered entities, in operation for at least one year, have had some previous affiliation with USAID programs, and “…demonstrate a commitment to the continued servicing of the SME sector and be likely to become sustainable.”6 The contract further indicated that the selection of BSIs should be as broad a representation of regions outside of Moscow as possible. USAID also asked that gender issues be taken into account. IRIS screened these BSIs and created a short list of 70 institutions. The IRIS team invited each of these 70 BSIs to make a presentation about their organizations and their business plans to a panel of experts. The panel chose a diverse group of 33 BSIs for participation in the project. These included: · 13 consulting and training institutions · 4 technological innovation support institutions · 3 business incubators · 3 vocational training institutions · 2 women's entrepreneurial associations · 1 each of the following: commodity producer association; apex body for regional business centers; trainer of unemployed and home-based businesses; environmental protection consultancy; software development firm; legal practice; association of construction companies; chamber of commerce. The IRIS project provides three types of inputs to upgrade the capacity of these organizations – training, grants and technical assistance. · Training. During the past year IRIS provided eight two-to-three day seminars in Moscow for BSI managers. Training topics included management consulting, marketing and promotion, high technology (intellectual property rights), 5 USAID contract with IRIS, dated 11 September 1998; p. 3. 6 Ibid; p. 6. July 2000 SME Evaluation - 21 - information technology and the Internet, and human resources management. Training courses were developed on the basis of BSI requests and were well received by participants. Topics that have been of particular interest to the BSIs include SME financing mechanisms, advocacy, building the “SME Support Network,” and legal issues related to SMEs. IRIS conducted pre- and post-training surveys to assess changes in knowledge, and these surveys document positive outputs, IRIS reports that participants are sometimes asked to develop “action plans” as a seminar activity; other times they are encouraged to do so after they return home. Although there is no formal follow-up to assess impact, anecdotal evidence suggests that the training is regarded as helpful to the BSIs. · Grants. USAID has provided $885,000 to IRIS to implement a small grants competition among its 33 BSIs. Grants are intended to strengthen existing BSI services or to create new services for the SME community. Each grant proposal is reviewed by a committee of experienced professionals, including members from USAID. Grants, which to date have averaged $22,000, are to be used for project development activities only, not for overhead and salaries. Grant-funded projects have helped individual BSIs to create a Web site and Internet related products; develop and produce training materials; exhibit SME products; develop legal services; develop employment services; provide information services (on legislation, credit, office space, equipment, etc.); offer business incubators, and conduct seminars on construction technology. · Technical assistance. Finally, the 33 IRIS BSIs are entitled to receive Targeted Technical Assistance through a partnership with the EBS program. Under this EBS-funded arrangement, one American business volunteer is being made available to each IRIS BSI. (In return, six EBS LRP management staff are entitled to attend each IRIS training seminar). IRIS BSIs in the RFE are all taking advantage of this program (through ACDI/VOCA and Winrock). Of the IRIS BSIs in the west, most are seeking a volunteer, although eight have indicated that their technical assistance needs have been met through their participation in the IRIS training programs. Networking and advocacy training have been additional priorities. IRIS is implementing a demonstration advocacy project with BSIs in two communities “to bring together the SME community and administrative agencies in their respective regions to analyze current and pending rules regarding SME development.” Long-term IRIS goals are to expand the number of SME network BSIs, possibly establish information “hubs” to more closely link BSIs and to create a National SME Advocacy Council. July 2000 SME Evaluation - 22 - Findings What has been the impact of the program on developing effective, sustainable BSIs? · IRIS’ selection of a very diverse group of BSIs offers the opportunity to experiment with new types of business service organizations, but it also reduces the potential to produce strong BSIs that carry out traditional business-skill enhancing activities. The criteria that USAID established for BSI selection are broad, meaning that many Russian BSIs could have qualified. Obviously, additional criteria were incorporated in the selection process to allow the IRIS review committees to select the final 33 participating institutions. IRIS reports indicate that final selection was based on an assessment of “strengths and weaknesses, institutional development capacity, client outreach, quality of services developed and provided, and their business or educational commitment.”7 Even with these additional criteria, it is clear that IRIS (and USAID) had a number of critical choices. For example, since there are many types of organizations that support small and medium-sized businesses, one of the key strategic decisions relates to the type of BSI that would be included in the project. The project selected a diverse group of BSIs, including a number of “traditional” BSIs that provide training in business skills, consulting and information services, and others that promote technological development and adaptation, gender equity, the interests of particular industries, the development of software useful to business, legal services for business, etc. The three IRIS BSIs that the team visited in Vladivostok, for example, included groups whose interests were primarily in representing the interests of the construction industry, in promoting women in business and in the application of science to business (but strongly oriented toward the science side). These are worthy goals, and these organizations can play constructive roles. Their contributions, however, are quite different from those more traditional BSIs that focus on business skills development. As USAID participated in and approved the selection of BSIs, the team assumes that the Mission supported the selection strategy that was utilized. The team understands that USAID’s interests in broad geographic distribution and in addressing gender issues were also important factors in the selection of several of the BSIs. Although a broad range of business support organizations is desirable, the evaluation team believes that there would have been advantages in selecting a more homogeneous group of traditional BSIs providing business skills training and consulting services. This view is based on the belief that Russia’s need for business skills development is particularly critical, that the BSIs working in this area are having a significant impact, and that the United States has particular capacity to provide assistance in this area. The diversity among the participating BSIs also complicates the provision of some types of development assistance. For example, it is clearly more difficult to provide effective training to a group with quite diverse 7 IRIS Quarterly Report dated January 31, 2000; page 2. July 2000 SME Evaluation - 23 - backgrounds and interests. The team also felt that one or two of the “less traditional” BSIs were quite weak, raising the question of whether, even after training and other project inputs, they would be able to become strong contributors to the SME community. · Training provided to BSIs by the IRIS SME network project has been of good quality and appropriate to the needs of the majority of participants. IRIS’ training program for BSI managers and staff is based on a survey of BSI needs, and careful thought and preparation have gone into the organization of the seminars. Given the newness of many of the recipient organizations, this training in subjects like management consulting, marketing, use of the World-Wide-Web and human resources management has been valuable to a majority of participants. However, given the diversity with the 33 IRIS BSIs, it is inevitable that some topics and the level of instruction will not be appropriate for each participant. Several interviewed BSI participants were critical of particular courses. Also, as noted above, the team had difficulty understanding, and the participants had difficulty explaining, how some training was relevant to some participants. For example, although the scientist who represented the Vladivostok Association of Employees of High-Tech Companies benefited from the training on micro-credit said that he appreciated the course that he attended on micro-financing, the relevance to him was not apparent. Other concerns about the training include clarity of the message and the need to encourage participants to define appropriate follow-up actions. In many cases, the team found that BSIs were enthusiastic about the training but could not identify changes they were making as a result. Sometimes changes were being made that were not intended. For example, several BSI participants, interviewed immediately following the micro-credit seminar, had become extremely enthusiastic about micro￾credit and had decided to launch such programs – an undertaking that the team sees as inappropriate for these organizations. These incidents underscore the importance of IRIS sending clear messages in the training programs and clearly defining appropriate BSI follow-up activities. · Technical assistance provided to IRIS BSIs through the EBS program has been productive. Twenty-three IRIS BSIs have so far placed requested Targeted Technical Assistance through an EBS volunteer. Those BSIs that have received volunteers reported very positive experiences, suggesting that the volunteers had shown ways to improve marketing and services. · The small grants program has had a significant financial and programmatic impact on the BSIs. However, the evaluation team questions whether some of the new services being developed will have a significant positive impact on SMEs or on BSI sustainability is uncertain. Given BSIs’ small size and very limited budgets, the infusion of a $15,000 to $30,000 grant has been a major event in the lives of these organizations, enabling them to strengthen existing services or to launch initiatives that would not otherwise have been possible. July 2000 SME Evaluation - 24 - However, the team questions whether some of the grants (e.g., the many Internet￾based services, employment services, legal assistance with copyrights and patents) are establishing services for which there is significant SME need and which will generate significant income for the BSI. BSI efforts to develop Web sites and other Internet services (e.g., to market the products of SME clients, to find investors, to sell used equipment that might be useful to SMEs) seem unlikely to be fruitful or tangential to the more practical and pressing needs of SMEs to improve planning, marketing, accounting and quality control. The team found many of the grant proposals (read in both Russian and English) obtuse on key questions of operations, operating costs, projected demand, pricing and revenue. (Better information on these issues may have been provided to the review committee in response to questions from committee members; the team did not have the opportunity to review that data.) Obviously, the BSIs and IRIS are more optimistic than is the evaluation team about these grant-funded activities, and IRIS believes that there have already been “conclusively demonstrated innumerable positive and far-reaching results and impact.” The evaluation team believes that more time will be required before the results of these grants emerge. Although IRIS plans an evaluation, the team is concerned that the project’s planned project completion date will be too early to assess the impact of these activities. · IRIS SME network participants have benefited from informal networking opportunities provided by project training seminars. Echoing a sentiment heard from many BSI managers in other USAID grantee networks, BSI managers spoke of how much they valued the interaction opportunities with colleagues which is provided by the training seminars. · IRIS BSIs are engaged in advocacy activities with local and regional officials. IRIS encourages and helps to train its 33 BSIs to advocate on issues affecting SMEs with municipal and regional government officials. The team found that most BSIs are engaged in some advocacy activities with the local authorities in their communities and regions. Although the role of individual BSIs in advocacy is normally quite limited, the IRIS demonstration advocacy project shows that BSIs, working in collaboration with others, can a play a key role in advocating SME interests. In any event, BSIs engagement in advocacy both provides an additional voice for pro-SME reforms and helps BSIs to understand the environment in which they carry out training and consulting activities · Overall, the participating BSIs have benefited, but it is too early to determine how significant these benefits will be over the longer term. Most IRIS BSIs were self￾sustaining before the project began. The project is likely to improve the quality and types of their services and to further enhance their prospects for sustainability in the future. July 2000 SME Evaluation - 25 - What has been the impact of the program on SMEs? · It is likely that the combination of training, technical assistance and grants will strengthen BSIs’ ability to provide effective services to the SME community. However, it is too early to see much impact at this time. All technical assistance and grants to BSIs occurred over the past 12 months, and the team uncovered little evidence of direct SME impact from the project. What has been the overall cost-effectiveness of the program? · The IRIS SME Network project is a moderate cost, moderate benefit program. USAID’s two-year grant amounts to $61,000 per BSI per year. (This figure does not include the cost of EBS volunteers that provide technical assistance to IRIS BSIs.). The team believes, as USAID and IRIS obviously did when they undertook this project, that the combined effect of training, grants and technical assistance should sharply increase the capacity of the cooperating BSIs to promote SME development. However, the evaluation team had considerable difficulty identifying that impact. Although the evaluation team has several concerns (described above) about project implementation, it is likely that the difficulty in identifying project impact at this time is largely the result of (1) the rather short period this project has been in operation, (2) the fact that the better part of a year was needed for the BSI survey/selection, and (3) the additional time required to design and implement training, arrange technical assistance. In sum, it is premature to assess impact and cost-effectiveness. · IRIS’ collaboration with the EBS program is mutually beneficial and makes both programs more cost-effective. Recommendations · If the IRIS program is to be continued, USAID should reconsider whether the selection of such a diverse assortment of BSIs is appropriate. The evaluation team believes that, if the program is extended and new BSIs are incorporated, selection criteria should emphasize more private sector-oriented BSIs with a demonstrated capacity to provide consulting services for a fee to a range of SME clients. · The grant review process should be tightened to emphasize activities for which there is clear SME demand and for which SMEs will pay. · To increase impact and the assessment of impact, training sessions should normally end with participants developing “action plans” showing, as appropriate, a timetable and specific steps that need to be taken to implement ideas introduced in the training. IRIS should follow-up to determine whether or not plans are implemented. (The use of action plans is recommended in USAID’s “best practices” for training.) July 2000 SME Evaluation - 26 - V. Regional Small Business in the Russian Far East: the American Russian Center (ARC) of the University of Alaska at Anchorage The American Russian Center in Anchorage (ARC/Anchorage) is an initiative of the University of Alaska at Anchorage. The USAID ARC program was established to support small and medium enterprise development in the Russian Far East (RFE). The program has four goals: · Promote SME development by providing direct training and technical assistance. · Provide business extension services to Russian entrepreneurs on key topics, such as managerial accounting and access to credit/investment. · Facilitate educational and entrepreneurial exchanges between Alaska and RFE institutions. · Enhance US-Russian business connections. To implement this program, ARC Anchorage manages three Russian-registered business support institutions in the RFE: · The Russian American Business Education Center (RABEC), serving the Magadan Region · The Russian American Business Training Center (RABTC) in Yuzhno Sakhalinsk, serving the Sakhalin Island region · The Russian American Education Center (RAEC) in Khabarovsk, serving the Khabarovsk Region and other RFE locations While each of these centers (which this report refers to as ARCs) has a somewhat different mix of activities, their primary function is to provide courses and seminars in business planning and management to existing and future Russian entrepreneurs. ARC has created a basic, six-week business training course, entitled “Western Approach to Business,” which is taught at least twice per year at each of the three centers. The ARCs also offer a condensed version of this course in other cities in the RFE. Trainers in the basic business course include about equal numbers of Russians from the local ARC staff and consultant roster and from American trainers with real-world business experience. A central feature of the basic business course is the requirement that each participant prepare a business plan. These business plans are judged by an independent panel, and the winners are given the opportunity to attend an advanced business training program in Anchorage. ARC/Anchorage conducts the advanced business training program course in Anchorage three times a year for 20-25 people each time. Participants pay their own travel expenses, and the project pays for project development and implementation, hotel costs (at a very low off-season rate) and per diem ($10/day for meals and expenses). ARC strongly discourages participants from viewing this trip (for many their first outside of Russia) as July 2000 SME Evaluation - 27 - an opportunity for sightseeing and shopping, threatening to send home anyone who provides less than full and consistent participation. As part of the program, participants are required to develop a business plan for the creation of a business in Anchorage. Although the team initially questioned the relevance and usefulness of this exercise, ARC believes that having the Russians learn what the U.S. business environment is like will cause them to exert pressure to change their systems at home. ARC also uses Alaska as a learning laboratory when specially tailored business attachment opportunities are organized in Anchorage for Russian businesspeople. For example, the director of a four-store Khabarovsk retail business was paired, during an eight-day trip to the U.S., with counterparts at COSTCO, JC Penney and Sears stores in downtown Anchorage. ARCs increasing conduct short seminars on a range of subjects. These include, for example, credit unions, international accounting standards, Internet for entrepreneurs, fund-raising techniques, proposal writing, basics of retailing, and “The Seven Habits of Highly Effective People.” They also undertake client-focused seminars for companies, municipalities and regional Dumas. ARCs’ training courses and seminars reach a large number of Russians. ARC reported that in the three-year period, 1997-1999, it conducted some 357 courses and seminars, averaging approximately three per month per ARC center that reached 8,649 entrepreneurs. The two oldest ARCs, Yuzhno Sakhalinsk and Khabarovsk, have operated the basic business course since 1993, training 1,214 people, of whom 241 have gone on to attend the advanced small business training course in Anchorage. The ARCs have shown increased interest in providing consulting services, and the Yuzhno Sakhalinsk and Khabarovsk centers have each hired a full-time professional consultant. ARC reports that the centers have provided almost 1,500 free “consultations” and “consulting services” to entrepreneurs since 1998. All three ARCs have also become Local Regional Partners (LRPs) for the EBS/East program administered by ACDI/VOCA. Involvement in this American volunteer consulting program enhances the ARCs client base and experience in consulting. In addition to small business courses, seminars and consulting support, the ARCs carry out a variety of other activities for the business sector and the broader community. These include, for example, provision of a business support library and an NGO support center, Internet instruction, fundraising and grant writing training, and municipal and regional Duma support. July 2000 SME Evaluation - 28 - USAID has supported ARC’s small business training activities since 1994. The current two-year contract is for $2.2 million and will run through June of 2001. The major components of the budget are as follows: Amount Percent Activity $546,900 25% ARCs’ operations costs (Magadan—$127,500; Yuzhno Sakhalinsk--$205,800 and Khabarovsk— $213,600). $75,366 3% Basic business training courses in the RFE (salary and airfare for American trainers). $359,027 17% Advanced small business training in Anchorage (program staff, instructors, translator, Russian participant costs). $704,180 32% ARC core operations: Anchorage staff & office ($532,180) and a Regional Director (American) based in Yuzhno Sakhalinsk ($172,000). $490,473 23% University of Alaska overhead. Each of the three ARCs has a Russian director who reports to the ARC Executive Director in Anchorage. Although the authority of the Russian directors has increased over time, ARC/Anchorage retains overall responsibility for program direction and management. Findings What has been the impact of the program on developing effective, sustainable BSIs? · ARCs are performing at a high level. The results achieved by the ARCs are impressive. Training courses reach a very large number of clients, providing a wide range of effective business instruction. Evidence from the clients interviewed indicates that ARC training is often an important factor in their eventual career choices, business success and business management breakthroughs. The emphasis on development of training plans makes the courses highly practical and more likely to lead to action. Based on these interviews, the team believes that the ARCs’ training activities are as effective as those of any USAID-supported BSIs that it visited. · ARCs have had a significant impact on their host communities. Magadan and Yuzhno Sakhalinsk – even Khabarovsk though it is far larger – are small frontier cities with relatively little experience to prepare themselves for the transition to a market economy, democratic pluralism and the “new world order.” These cities have few institutions that can encourage and ease the transition. The ARC centers, through their training and the support they provide for a variety of other Western projects, July 2000 SME Evaluation - 29 - provide many Russians in the RFE with their first meaningful exposure to Western approaches, particularly to free enterprise as it is practiced in the West. As a result, the ARCs probably have a community impact out of proportion to their modest size. · ARCs are not sustainable as currently managed and financed. ARCs receive only about half of their operating revenue from fees generated, with the remainder supplied through the USAID grant. This is due in part to a high cost structure, including the use of expensive American trainers and staff salaries that are reported to be considerably above comparable rates in the local economy. At the same time, fees are kept low, and potentially lucrative ARC consulting for larger businesses has been rejected on the basis that it might shift the focus from the smallest enterprises that can least afford to pay. Essentially, the ARC program is structured to meet the perceived needs of start-up and small businesses rather than market demand. The ARC model for BSI operations is not unlike many BSIs in the U.S. that focus on the smallest businesses and receive support from the federal government’s Small Business Administration (SBA). Unfortunately, there is no SBA-equivalent sponsor in sight to subsidize the ARCs and other Russian BSIs. For this reason, ARC has given increased attention to the development of new, revenue producing “products” and to funding diversification. Results include grants from Eurasia and Soros Foundations, a Russian government contract to retrain out-of-work coal miners, a customer service seminar for a large retailer, the relationship with the EBS/East program, and a training partnership with USAID’s microcredit program. Nevertheless, a considerable financial gap remains. Indefinite USAID support is not the answer to financial sustainability for the ARCs. Aside from the financial cost to USAID and the U.S. Government of continuing a high level of support for the three RFE centers, that focus limits the capacity of ARC and USAID to collaborate on other programs that would bring valuable Alaskan experience and expertise to bear on other problems in the RFE and that would further increase the range of ties between Alaska and the RFE. · The continuing dependence on Americans as trainers and managers, while contributing to the quality of the program and its short-term impact, is counterproductive to the achievement of longer-term capacity development and self￾sufficiency. ARC’s American staff presence in the RFE has been reduced from a high of ten to its current level of one, and consideration is currently being given to eliminating that remaining position. ARC deserves credit for this Russianization effort. The continued extensive use of American trainers and the prominent role of ARC’s American managers have both positive and negative consequences. On the positive side, the use of American trainers to conduct courses on topics in which Russian experience is limited (e.g., strategic planning, marketing and managerial accounting) enhances the quality of training. The use of American trainers apparently also helps to July 2000 SME Evaluation - 30 - attract SME clients to the ARCs’ courses. Additionally, ARC/Anchorage’s direction of the program has helped to ensure that the emphasis on quality and practical instruction would continue and that USAID financial and reporting standards would be strictly observed. At the same time, the evaluation team believes that greater emphasis should now be given to the development of Russian capacity to assume full responsibility for the three ARC centers. After seven years of program implementation, ARC/Alaska continues to think of and treat the three ARCs in the RFE as an extension program of the University of Alaska, to make final planning and budgetary decisions in Anchorage, and to use Americans to provide almost half of the basic business training. The evaluation team believes this level of American involvement, perhaps warranted in the past, at this point is unnecessary, unduly expensive and counterproductive to the emergence of strong Russian business support institutions. The total “Russianization” of USAID’s other BSIs shows that they can be managed and staffed entirely by Russians. As regards expenses, the costs of operating the ARCs are certainly many times the level of other BSIs. Finally, the evaluation team believes that requiring Russian ARC directors to report to an American executive director (stationed in Anchorage) makes it more difficult for the Russian ARC leaders to emerge as strong, competent leaders, and the current high level of staff and budgetary support reduces the pressure to reduce costs, raise fees, develop new and profitable business products and seek support from local and international sources.8 What has been the impact of the program on SMEs? · ARCs provide effective training for SME participants. As noted above, ARC courses reach large numbers of current and potential businesspeople with a high quality program. It is not unusual to encounter graduates of the basic business course -- especially if they also participated in the advanced business training course in Anchorage – for whom the ARC training experience was a life-changing event. For many trainees, this is the first time they have come face-to-face with the idea of planning to pursue a career goal. It is also the first time for many to be exposed to a practical, step-by-step, business planning process. The fact that much of the growing demand for ARC’s specialized training seminars is coming from former trainees is another indicator of program effectiveness. · ARC’s assessment of impact is largely anecdotal. ARC monitors impact through informal contact with former students. This feedback is used for “success stories” and is reflected in periodic reporting to USAID. There is no programmed, routine follow￾up with training program graduates to find out if and how they use what they learned. 8 ARC/Anchorage and the University of Alaska’s accounting office have been operating under the understanding that the University was accountable to USAID for all expenditures of the ARCs, including those financed with local currency generated from program fees. The team believes that this view of its financial responsibility has contributed to ARC/Anchorage’s desire to maintain tight control over the program. July 2000 SME Evaluation - 31 - For instance, most trainees complete a business plan as a principal product of their training, but ARCs do not formally track how many of those plans are implemented. · The creation of ARC “Alumni Business Circles” is a promising development. These training course alumni groups managed by the ARCs bring together previous program participants for monthly meetings and special events. In Yuzhno Sakhalinsk, the ARC is seeking to register its alumni association so former trainees can work on legislation and “show their power” to the regional Duma. What has been the overall cost effectiveness of the program? · ARC is a high cost, high benefit program. The USAID grant amounts to $367,000 per BSI per year, reflecting the high cost structure of the ARCs and the high level of U.S. participation in the program. As indicated above, such evidence as exists suggests that benefits are also high, particularly in terms of impact on SMEs. The program also contributes to the important collaboration between Alaska and the RFE, people-to￾people and university-to-university cooperation. Nevertheless, the team believes that the cost-effectiveness of this program can be improved substantially. Recommendations · USAID should ask ARC/Anchorage to articulate a vision and strategy for the program in which the three existing ARCs become independent Russian BSIs. This vision could well include a continuing involvement of the University of Alaska, but this should be a relationship that is defined as much by the Russians as the Americans. The team believes that a reasonable target might be to make the centers financially independent of USAID/ARC resources within two years. Prompt attention would need to be given to training additional Russian instructors, curtailing costs, developing and marketing profitable new products, and seeking funding (contracts or grants) from local government and international donors. Consideration may need to be given also to gradually adjusting fees upward. This recommendation for a change in the nature of ARC/Anchorage’s relationship to the three centers should not be seen as a recommendation to sever that relationship. Indeed, the evaluation team would strongly support some type of long-term partnership between ARC/Anchorage and the centers in the RFE. · To implement this new strategy, the team recommends that USAID extend the ARC grant for a two-year transition period. · As contract resources are freed from support to the existing ARCs, consideration should be given to utilizing and extending the Alaskan experience with its many parallels to the RFE. Two new initiatives seem highly promising: creation of an MBA-like program to further business education (possibly utilizing successfully tested distance education models) and development of one or two new RFE BSI centers (following the new model which would make them self-sufficient within three July 2000 SME Evaluation - 32 - years). With respect to the development of university-level business education, ARC has already been helping to establish business schools at Magadan International University, Sakhalin State University and the Transportation University. If financing permits, these efforts might be greatly expanded. July 2000 SME Evaluation - 33 - VI. The Morozov Project: the Academy of Management and Market (AMM) The Morozov project was founded in 1991 by the Moscow-based Academy of Management and Market (AMM). USAID assistance began in1994. The mission of the Morozov Project is to contribute to the strengthening of Russia’s market economy and to the development of entrepreneurship. The strategic tasks are to implement a large-scale program of business entrepreneurship training and other business support services. These services are provided by a Russia-wide network which currently has 59 independent Morozov Regional Centers (MRCs). MRCs are the equivalent of business support institutions (BSIs) supported through the other USAID projects. AMM and the MRCs are non-profit institutions. Morozov activities have historically included: · Training (entrepreneurs, consultants and trainers at the MRCs, regional & local government officials) · Business consulting (emphasizing small and microbusinesses) · Investment promotion (accessing loan and equity financing; creating business partnerships between Russian and foreign companies) · Information support for entrepreneurs · encouragement of business innovation · Initiatives to develop the financial sustainability of the MRCs. MRCs are typically housed in government-financed educational entities, such as technical and public service universities and vocational secondary schools. MRCs provide a variety of short-term business training programs geared to the needs of small enterprises and to individuals contemplating starting a business. Most MRCs that the team visited are involved in a vigorous program of training for entrepreneurs and managers in small, medium and large businesses. These training programs function along side of and often in coordination with the normal programs of the institution hosting the MRC. However, the MRCs primarily provide continuing education to adults already in the workforce. A small number of MRCs also serve as curricula development centers for the whole project. Some centers provide business consulting, usually as a limited complement to or follow-on to training. A limited number of MRCs serve as business incubators and exhibition centers. AMM’s role under the current USAID cooperative agreement is to train MRC trainers and consultants and to develop and disseminate training curricula and materials. Data from AMM show that in the 1998-2000 period it has provided 63 training of trainers programs drawing 1407 participants (an average of 22 participants per program). Reflecting the greater knowledge and experience of the MRC personnel now coming to the training, AMM’s strategy for training trainers has evolved from rather general business education in the earlier years to today’s shorter, more focused and practical training. Further, current AMM training of trainers tends not to be a one-way information delivery, but rather a dialogue between the AMM trainers and MRC participants. July 2000 SME Evaluation - 34 - In some ways the relationship between AMM and the MRCs resembles a franchise operation. AMM provides the technology, trains staff and offers a name or trademark that has market power to its franchisees, the MRCs.9 However, a critical element of a franchising system is missing: MRCs are not paying AMM any appreciable amount for its services. AMM personnel indicate that plans to charge MRCs were upset by the 1998 financial crisis (which severely affected the MRCs, causing several to close) and by USAID’s requirement that AMM put all of its training materials in the public domain, thus lessening AMM’s ability to charge MRCs. The 1998 financial crisis placed the MRCs and AMM under considerable financial stress. AMM reports that 13 training centers were forced to cease operation. These closures were offset by the opening of an equal number of new centers. AMM is currently operating under a two-year project budget of $1.7 million, of which $460,000 comes from counterpart financing. USAID’s contribution is $1.26 million, of which $270,000 goes to a subcontract with the State University of New York (SUNY). The current grant will expire in August 2000. Findings What has been the impact of the program on developing effective, sustainable BSIs? · Over the seven years of this program, USAID assistance has helped AMM to develop an extensive network of BSIs delivering good quality business training. USAID’s and AMM’s objectives, creating effective, sustainable BSIs in a large number of Russian cities, have been largely accomplished. There is, of course, ample room for the system and individual MRCs to expand and improve in the future. · Through the recent financial crisis, AMM helped to hold the Morozov system together, countering the closure of a number of centers with the opening of others and encouraging MRCs to open subcenters in smaller cities and towns. AMM’s contribution to the geographic expansion of the Morozov system to new communities may be as important as the role it played in strengthening existing MRCs. · The overall impact of AMM’s current program of support on MRCs’ capacity is positive, but modest. MRC management and trainers indicated that they valued the training and curriculum materials they had received, and these had contributed significantly over the years of the program to the development of their capacity. However, the perceived value of current support is less, with a number of respondents indicating that current training of trainers and curriculum development activities do not have a major impact on what they do or on what they teach. Trainers indicated 9 Morozov does enjoy a degree of name recognition and credibility within the Russian business community. However, when asked about the benefit his institution derived from the Morozov name, one BSI director indicated that the name was known in his city only because of the efforts of his MRC and that AMM had not carried out efforts to build name recognition. July 2000 SME Evaluation - 35 - that AMM/Morozov materials are used, as are materials that trainers obtain from other sources and produce themselves. · AMM offers good training-of-trainers courses. The content of courses is generally appropriate, although it does not always meet the needs of the diverse MRCs. There has been significant impact from AMM’s training of trainers over the years. Currently, these courses are shorter and more specific. Quality remains high. It is not always possible to design a course that meets the diverse needs of MRCs that vary in experience and which operate in quite different markets. One MRC in the eastern part of the country indicated, for example, that the course given on marketing for the service industry was designed for conditions in European Russia and had little relevance to the East. · The impact of AMM’s current program of support on MRCs varies by region. The participation of MRCs in AMM’s training of trainers courses diminishes as distance from Moscow training site increases, with MRCs in Siberia and the Russian Far East (RFE) sending only a few participants each year to the short Moscow-based courses. BSI directors in those more distant regions indicate that the cost of travel to Moscow (half of which is paid by AMM) is too high for them to afford more frequent participation. In the words of one MRC director, “Morozov looses it network behind the Urals.” In contrast, the high total number of participants in AMM training of trainers workshops during the past two years (reported above) suggests that participation (and presumably impact) are very much higher in the western part of the country. · MRCs benefit from informal networking at conferences and training sessions. MRC leaders see substantial value of bringing their business trainers together with those of other MRCs to share experiences – especially in a country as large as Russia. Learning about the experiences of other institutions with similar objectives and programs helps MRC directors and trainers to develop new programs and to avoid costly mistakes. AMM-sponsored training sessions in Moscow afford this opportunity for informal networking. · MRCs are self-sustaining. Current AMM activities are helpful but not critical to MRCs’ ability to remain financially viable in the future. Interviews with MRC directors indicate that MRCs generate sufficient revenue to pay their operating costs, exclusive of the real (but hidden) cost of facilities they enjoy from their host educational institutions. AMM’s earlier training of trainers courses and curriculum development certainly helped the MRCs to build the training capacity upon which their current revenue is based. Current AMM activities in those areas help MRCs to remain up-to-date but do not appear to be critical to MRCs’ ability to remain financially viable. AMM is also marketing training services to the Russian government. To the extent that these efforts are successful and utilize MRC capacity (as opposed to AMM capacity) to deliver services, these efforts also have the potential to enhance MRCs’ prospects for long-term sustainability. July 2000 SME Evaluation - 36 - What has been the impact of the program on SMEs? · Morozov business training programs administered by the MRCs continue to have a major impact on SME development. The Morozov program, with its large number of MRCs and broad geographic coverage, has been a major force for business development in Russia. AMM estimates that the Morozov system currently trains about 50,000 businesspeople per year, representing about 45 percent of all business education provided in the country. The Morozov system is a valuable Russian asset and a tribute to the work of AMM and to USAID. · Although Morozov’s impact continues, the impact of the current USAID-AMM two￾year program is modest. With the current USAID-AMM program’s limited impact on the training programs of previously existing MRCs, the resulting impact of current USAID support on SMEs served by those Centers must also be limited. Although the overall number of MRCs has remained fairly constant over the past two years, it is noteworthy that the creation of new MRCs in communities not previously served can be expected to produce a significant positive impact on SMEs in those communities. What has been the overall cost-effectiveness of the program? · The current USAID-AMM Project is a low cost, low benefit program. The USAID grant has the lowest per BSI cost ($13,000/year) of the five programs reviewed by the team. A number of MRC directors questioned whether the low level of benefits might not reflect too high a proportion of the program’s resources being devoted to AMM core operations. This criticism may reflect the fact that the USAID-AMM program is designed, appropriately the team believes, to provide services rather than financial support to the MRCs. However, it also reflects a certain frustration that some MRCs, particularly those east of the Urals, feel about the level of AMM services they receive. · AMM has not succeeded in finding alternatives to USAID funding to support its core curriculum development and training of trainers activities. The question of AMM’s financial sustainability has been before USAID for some time. USAID’s 1996 Morozov project evaluation concluded that too much of USAID funding was supporting AMM core operations and that its plans for funding diversification were not practical. AMM has down-sized its staff and made some progress in attracting new business. For example, it recently signed agreements with the Soros Fund, TACIS and the Ministry for National Affairs, and it negotiated an agreement with the central government for a nationwide municipal official training program on SME support. However, AMM efforts to find alternative sources of funding to replace USAID support for teacher training and curriculum development have not been successful. AMM believes that its progress has been significantly constrained by the 1998 financial crisis and its aftermath, and it reported to the evaluation team that it has assurances of increased financial support from the national government beginning in 2001. July 2000 SME Evaluation - 37 - Recommendations · USAID should consider terminating, or dramatically altering, the current grantee arrangement with AMM. The experience of the past several years indicates that either AMM is not aggressively seeking alternative funding (which does not appear to be the case) or there currently are no alternative sources. If there are no alternatives, this suggests either that no institution (other than USAID) sees sufficient value to justify this program, or that others will not step forward with resources while continued USAID funding remains available. It will be difficult to know with certainty which is the case unless or until USAID terminates funding and forces the issue. The risk, of course, is that the reduction of USAID funding could result in the disintegration of the Morozov system. This would mean, presumably, that AMM would cease its curriculum development and its training of trainers activities; the system would loose its ability to speak to and solicit support from the donor community and the Russian federal government; the potential to expand business education into new geographic areas would be more fully dependent on local impetuous and resources; and opportunities for networking among the MRCs would be reduced. The existing MRCs, which already function as independent entities, would probably continue to function, perhaps with new names. The better of the MRCs would continue to develop new and better programs. The team believes, however, that a more likely scenario in the event of a cessation in USAID assistance is that AMM and the MRCs will find a way to support a continuation of Morozov as a system. · If USAID decides to continue its support for AMM and/or the MRCs, the evaluation team recommends that AMM and USAID give priority to extending the Morozov system to unserved or under-served areas, to redressing current regional imbalances that lessen benefits to MRCs east of the Urals, and to expanding the products and services that AMM provides to MRCs . BSI directors working with one or more of the USAID-financed programs are taking initiative in extending training and other business support services to Russia’s smaller cities and towns. The Morozov system has been prominent in this geographic expansion of services and should continue to do so in the future. AMM and USAID could continue to support this expansion, financing specific activities to create new MRCs or sub-centers. If USAID decides to finance such activities, the team suggests that it consider a system that reimburses AMM for results rather than financing staff and other inputs. With respect to regional imbalances, the team suggests that, at the very least, AMM’s travel subsidy policy be altered, eliminating the 50 percent formula and paying all costs above a cap (e.g., the first 1500 rubles) that each MRC would be expected to pay to send a faculty member to an AMM course. This would mean that AMM would provide greater subsidies to those facing greater travel costs, and it would equalize the July 2000 SME Evaluation - 38 - cost each MRC would pay to participate. A preferable approach would include the development of curriculum and training materials that better reflect the diversity in business conditions across the country. It is clear from the team’s interviews that many MRC directors would like a regionally-based program, and they would like to be full partners with AMM in its development. AMM might also consider expanding the types of assistance products that it provides to the MRCs. In this regard, AMM might review IRIS’ list of seminar topics (and the content of those seminars) and consider providing similar seminars to the MRCs. If AMM can expand services that have value to MRCs, this should enhance AMM’s ability to move toward a franchising system in which it charges fees to MRCs for its services. If MRCs are unable or unwilling to pay for AMM services, AMM might consider selling its products and services to any BSI wishing to participate. July 2000 SME Evaluation - 39 - VII. Cross-cutting Findings about these USAID Programs In its review of USAID’s five current programs to develop business skills in Russia, the evaluation team has sought to identify characteristics that cut across some or all of the programs. This chapter will present those crosscutting findings that deal with process and impact. A. Cross-cutting Findings on Process Characteristics of Russian business support institutions The five programs reviewed in this study provide assistance to 100 business support institutions. These BSIs represent approximately 20 percent of the 500 BSIs identified in the IRIS survey and a still smaller percentage of the universe of Russian BSIs. In light of the fact that none of these institutions (at least in their current form) existed a decade ago, it is clear that Russia has made considerable progress in establishing an infrastructure of institutions that can provide training and other business support services. USAID programs, both past and present, have made a significant contribution to this positive development. The BSIs supported under USAID projects are a diverse group, making it difficult to generalize about them. This difficulty is compounded by the lack of consistent definitions about BSI program outputs and sometimes by the BSIs’ own difficulty in describing their programs. For example, since training courses (appropriately) vary greatly in content and duration, comparisons of the number of people different BSIs trained per year may be very misleading. Similar problems exist for consulting; some BSIs report brief, one-on￾one consultations conducted, often with no specific fee, in connection with training courses as “consulting,” while others consider consulting to be only longer-term problem￾solving activities conducted for individual firms for a fee. Despite these difficulties, the team will share a number of impressions about the BSIs. The sample of 27 BSIs that the evaluation team visited suggests that these institutions are small, typically with a staff of four to eight full-time employees. Most of the BSIs visited make fairly extensive use of consultants to carry out their activities. In the words of one of the BSI directors, organizations, including BSIs, are essentially either “money earners or money spenders.” BSIs assisted by USAID programs include both of these types; they range from very businesslike organizations that pay close attention to costs and revenues to very NGO-like service providers that view their roles more in social than financial terms. At the “more private sector” end of the spectrum are BSIs that are essentially small consulting companies. These are profit-driven businesses that sell professional services to SMEs – typically in the areas of business planning, finance and marketing. Their clients tend to be established businesses struggling to grow. Among the USAID programs reviewed in this study, the EBS-West program (CDC) is most typified by this type of BSI. July 2000 SME Evaluation - 40 - At the other end of the spectrum are “non-commercial” organizations that see their role more in terms of “helping people.” These BSIs tend to focus on start-ups and very small firms whose needs are for basic skills training and whose capacity to pay is limited. These BSIs are usually not-for-profit, non-governmental organizations (NGOs) that receive their resources from a combination of fees, free facilities, and grants from governmental and donor organizations. The typical NGO BSIs are training centers, often affiliated with and located within universities. Many Morozov centers, particularly those in the slower growth areas of Russia where fee-for-service consulting is less common, fit into this category, as do the three ARC centers (which are also BSIs of ACDI/VOCA￾Winrock). In between the NGO and commercial categories is a third group of BSIs. These are evolving from the “spend money” category towards the “earn money” category. Driven by financial necessity and the urging of donors and grantors to be more self-sustainable, these BSIs are becoming more market oriented in a variety of ways. Basic business training courses are increasingly being supplemented by seminars developed exclusively for individual business clients. One-on-one, fee-based consulting is also becoming more common, as are services (some fee-driven and some supported by grants) typically built around specialized staff skills and interests. For example, the ARC Center (RAEC) in Khabarovsk now provides customer relations training to the employees of a large chain of department stores as well as specialized management training for a wide range of NGOs. Selection of BSIs within USAID-financed programs There is clearly a need for many types of organizations to support the development of SMEs. These include training and consulting groups, associations to promote the interests of particular types of business, professional groups, advocacy groups, etc. In this sense, diversity is a virtue. However, too much diversity within a single project can make it difficult to identify common objectives and to build coherent programs that adequately serve the diverse membership. The evaluation team found the diversity of organizations within several of the USAID programs raises the question of whether the selection criteria were sufficiently rigorous and whether some of those selected to participate would be able to participate fully and perform effectively in the programs. In this regard, the evaluation team had several concerns related to BSI selection: · IRIS. The IRIS program selected an eclectic group of BSIs that includes some organizations quite different than the traditional training and consulting-focused BSI. It is too early in the IRIS program to determine whether these outliers will benefit significantly from IRIS training sessions geared to a more traditional BSI organization. · Morozov. The 59 Morozov BSIs share a focus on training in basic business skills. As AMM forewarned the team, there is a considerable difference in BSI capacity. Inclusion of weaker BSIs in the Morozov system makes it more difficult to July 2000 SME Evaluation - 41 - establish the Morozov name as one of consistently high quality; at the same time, their inclusion in the program may bring them up to a higher standard. · EBS. In comparing the two EBS programs, the team believes that one reason for CDC’s success in placing volunteers is that it chose as its local partners business￾like BSIs that were already doing consulting for small to medium-sized enterprises. In contrast, at least four of ACDI/VOCA-Winrock’s BSIs focus on start-ups and very small enterprises, emphasize training rather than consulting, and act more like NGOs than for-profit businesses. Experience over the past year suggests that the those BSIs have a longer road to travel before they are able to generate the demand for consulting services that fires the EBS program. ACDI/VOCA BSIs appear to have now increased their capacity to develop consulting assignments for American volunteers and to support those volunteers. It remains to be seen whether they will also be able to develop their own Russian consultant-based program as a result of their involvement in the program. Geographic distribution of USAID-supported BSIs Russia east of the Urals, roughly two-thirds of the geographic area, has only about one￾third of the BSIs included in USAID programs. In part, this is a reflection of economics, demography and geography. The area west of the Urals has more people and more economic activity; the area east of the Urals is immense but less densely populated. European Russia, aside from having more BSIs, tends to have stronger, more private sector-oriented BSIs. Irrespective of whether one is looking at EBS, Morozov or IRIS, the BSIs in the West tend to be more dynamic and sophisticated.10 In at least one case, Morozov, the program itself contributes to this geographic disparity, albeit in a modest way, by providing higher levels of support services in the western region. There is also some disparity in the geographic coverage within the Asian part of Russia. Five of the EBS/East BSIs are grouped within a few hundred kilometers of one another in the Far East (two each in Vladivostok and Khabarovsk and one in Sakhalinsk). Three more EBS/East BSIs are within a few hundred kilometers of one another in Southern Siberia (Omsk, Tomsk and Novosibirsk), and IRIS has three BSIs in Vladivostok. On the other hand, a number of sizable cities in the RFE and Eastern Siberia – Komsomolsk, Irkutsk, Bratsk, Ulan-Ude, Chita and Jakutsk – are not included in these programs. Networks and Networking The team has found it useful to distinguish between the projects’ “networks” and “networking.” “Networks” refers to the formal association of BSIs. “Networking” refers to the informal sharing of experiences, ideas and technologies among individuals from different BSIs. 10 Obviously, there are exceptions to this generalization. July 2000 SME Evaluation - 42 - Each of the five USAID programs reviewed in this report includes an element of network building. The five project “networks” that the team observed were created to administer a set of services to the participating BSIs. Those networks are top-down organizations, with the program content, direction, and the timing and content of events largely decided from above. Membership (with the partial exception of Morozov) is determined largely on the basis of the parameters defined by USAID and the organization administering the program. These “networks” have been effective in administering USAID assistance and providing a valuable, albeit temporary, forum for “networking” among sub-sets of their BSI members. However, the networks have little raison d’etre outside of administering the USAID programs. As a result, the evaluation team believes that the likelihood that these networks will survive when USAID assistance to them ceases is low. Of the five programs included in this review, Morozov has been in existence the longest and would appear to have the strongest potential for sustainability as a network. However, even here, the network exists primarily to administer the AMM’s curriculum development and training of trainers program – activities that the member BSIs have been unwilling or unable to support financially. Thus, it is by no means clear that even this network will survive without an external subsidy. A great deal of “networking” occurs at the programs’ training sessions and meetings. Essentially, each participating BSI utilizes these gatherings to create its own “informal network” of BSIs. These groups are formed on the basis of perceived self-interest and require no outside support. Some of these relationships will grow and mature, perhaps leading in some cases to formal ties or associations. In others, where the benefits are not apparent, they will die. BSI staff interviewed for this study often commented on the great value of informal networking. Several interviewees viewed it as the single most important benefit derived from their participation in project activities. It is interesting to note that the formal networks provide the opportunity for informal networking and thus define networking limits. For this reason, it is more likely that an IRIS BSI in Yekaterinberg will be networking with another IRIS BSI in Novosibirsk or St. Petersburg than with a Morozov BSI in Yekaterinberg. The evaluation team believes that an alternative forum that provided the opportunity for networking among the BSIs in a particular region might be quite useful (See discussion in Section IV, below.) Information Systems The team has observed that all grantees use up-to-date information systems for their day￾to-day operations. Given the distances and time zones where the grantees operate, all grantees rely heavily on the Internet in their operations and communication. Both EBS West and EBS East have databases posted on the Internet that are accessible by the program managers in the fields. The access is secured via a password system. July 2000 SME Evaluation - 43 - One of the major parts of the information system of the grantees is their websites. These websites serve different purposes: provide the general information about the programs as well as specific information like Success Stories from EBS program, used as a marketing tool for promotion of the programs in the local communities, used to recruit volunteers in case of EBS program, provide information on the regions of Russia where the programs operate, offer links to other Internet resources on the SME support and development. The work of the grantees has also contributed to the development of the information technologies of their partner organizations: heavy reliance on the Internet has stimulated the LRPs to move their information system forward and some have even opened their websites that provide useful information about their company, the area that they operate in, their services, etc. some grants, e.g. under SME Support Project or Morozov project, were channeled to the development of websites and other related IT areas in the regions. However, beyond the communication purposes, i.e. use of Internet, the team feels that the programs should not over-emphasize the information technology and its advancement. The SME community has expressed in numerous studies and in the interviews that the team has conducted that the major problem areas are more mundane -- taxation, bureaucracy, financing sources. Cooperation among USAID’s grantees Collaboration among USAID grantees and contractors can yield increased impact. In this regard, the excellent cooperation between the two EBS programs and the IRIS program is particularly noteworthy. The EBS programs provide volunteers to IRIS BSIs, and the IRIS program allows staff of EBS BSIs to participate in up to three IRIS training sessions. This collaboration strengthens both programs. Impact evaluation within USAID programs In recent years, USAID has focused considerable attention on program impact. The five projects reviewed in this report have yielded many “success stories” that have been quite impressive. The reliability of the data generated by some other efforts to report on impacts (e.g., jobs created and jobs saved) seems highly dubious. Moreover, the team saw little effort to follow-up on training, either training of BSI staff or training of SMEs, to determine its effectiveness. Although following-up to assess the impact of training can be time-consuming and expensive and the interpretation of follow-up data is difficult, it would be useful to selectively do so. For example, it might be useful to set out some BSI “change objectives” for IRIS training and then to assess the extent that BSIs had changed their operations. Also, it would be useful to know how many of the business plans developed in the ARC basic business course are implemented. July 2000 SME Evaluation - 44 - Perhaps because impacts are focused within a smaller group, the EBS program has developed more effective assessment systems. The evaluation team was particularly impressed with the efforts of ACDI/VOCA to develop a system for impact assessment. This system, which includes two follow-up visits to clients that have received assistance from an American volunteer, is just now beginning to generate data. Quality and quantity of reporting Each of the five grantees is responsive to USAID’s requests in terms of the quantity and type of reporting. The team found the content of reporting generally reasonable, although providing more detail on inputs and outputs than USAID needs to know. The quantity of required reporting is daunting. The frequency of reporting, some of which is monthly, seems excessive. The team questions whether any information generated on these programs is so time sensitive that it requires monthly reporting. The underlying problem with too much reporting is that it takes large amounts of staff time to prepare, both of the grantees and their BSI partners, decreasing the level of resources devoted to program implementation. In the EBS programs, for example, the BSIs’ noted that reporting takes considerable time away from the key tasks of finding clients and developing scopes of work. The team also questions whether the quantity of reporting is not more of a hindrance than a help to USAID officers. The quantity of reports coming in on the five reviewed programs seems more than could be read and absorbed effectively by USAID’s heavily-burdened BDI staff, which also has responsibility for many other projects and which itself must generate various internal USAID reports. B. Cross-cutting Findings on Impact This section will provide the team’s observations with respect to BSIs’ training, consulting, advocacy, self-sufficiency and capacity development. BSI training Most of the BSIs participating in USAID-financed projects provide training courses, and it is the principal activity of somewhat more than half of these institutions. Clients are varied, with ARC and most Morozov BSIs focusing largely on start-ups and business that are quite small and other programs reaching a somewhat more diverse group. As the Russian SME sector grows and matures, the demand for training is gradually shifting from the basic business skills training to more specialized and advanced training and to training focused specifically on the needs of individual firms. The team saw evidence that the BSIs are responding to these market changes, expanding their training programs to meet these needs. This seems to be true for BSIs operating within each of the USAID-assisted programs. The team’s assessment of the quality of training is necessarily dependent on indirect evidence – interviews with BSI staff, trainers and clients; evidence of demand for courses July 2000 SME Evaluation - 45 - and repeat business. These indicators suggest that the level and quality of BSIs’ business training activities is quite appropriate for Russian conditions. As a result of the efforts of AMM and others, trainers indicate that there are a variety of course plans and curriculum materials from which they can choose. The reported level of demand for BSI courses and the frequency of repeat clients suggest that clients perceive that they are benefiting. One possible weakness in BSI training is that training may be somewhat theoretical, since trainers often lack practical business experience. This problem is particularly difficult to resolve in the short-run in Russia since there are few nationals with many years of experience solving business problems in a competitive environment. The team’s interviews suggest that Russians who have been successful in the past decade in starting and running businesses are busy running those businesses and making money and are unlikely to offer their services as trainers in the near future. The ARC program attempts to deal with this issue by bringing in U.S. trainers who are experienced businesspeople and by requiring that participants carry out practical assignments such as the development of a business plan. This use of American trainers undoubtedly enriches ARC training. However, the fact that these trainers’ experience is not in the Russian environment, that translators must be used, and that the costs are high make this an imperfect solution not replicable by other programs. Although the BSIs do sometimes use end-of-training evaluations to get feedback on the content and process of training, none of the programs includes systematic follow-up to assess the impact of training on participants. The fact that SMEs pay for training programs suggests that they are having a positive impact that is at least commensurate with the fees charged. Given the larger number of Morozov training centers and their focus on training, it is likely that Morozov is having the largest and geographically most widespread impact. USAID’s past and present investments in the Morozov program have clearly contributed to this impact. If one looks at impact per BSI, the three ARC centers, which provide a large number of courses and reach thousands of participants, appear to have had the largest impact on their communities. BSI consulting The evaluation team found that almost all BSIs carry out activities that they describe as consulting. These activities range from free consultations with walk-ins and participants in training courses and seminars to contracts with individual firms. Consulting is the principal activity of perhaps a third of the BSIs assisted by USAID programs, particularly those working with CDC and to a lesser but considerable extent with IRIS and ACDI/VOCA. BSIs that have traditionally focused on training appear to be moving slowing towards consulting. The market for consulting services is widely viewed as quite thin. Many BSIs told the evaluation team that few of their actual or potential SME clients had any experience with consulting or perceived it as something that they might need. EBS BSIs working in July 2000 SME Evaluation - 46 - Siberia and the RFE report that they need to work hard to convince firms to accept EBS volunteers, while CDC reports that there has been a strong upswing in SME demand for volunteers in western Russia. Although many of the skills required to do business training are the same as those required to do business consulting, the latter often requires more experience and problem￾solving skills. Moreover, Russian businesspeople with whom the team spoke sometimes expressed concern that the Russian consultants did not have sufficient knowledge of the industry in which their businesses operated. Thus, the “low demand” for consulting services that many BSIs described may in part be a reflection of businesses’ lack of confidence in the BSIs’ capacity to offer quality consulting services. The gradual movement of Russian training-oriented BSIs into consulting seems entirely appropriate, and the efforts of USAID-assisted programs to aid in this transition are also appropriate. The training (of IRIS and others) to help BSIs move in this direction is highly valued. It would be helpful if IRIS were to assess the changes in BSI programs in this (and other) area(s) before the end of its project. The EBS program, of course, focuses specifically on consulting, providing American volunteers to work as consultants with selected Russian firms. As indicated in Section II above, feedback from Russian businesses on the work of these volunteers is mostly very positive. The evaluation team does believe, also as indicated in Section II, that more emphasis needs to be given in this program to developing Russian consultants. BSI advocacy As BSIs are not set up to represent SMEs and by nature are not advocacy organizations, the evaluation team was somewhat surprised to find that almost all BSIs are engaged in activities with local officials to try to improve the environment for SMEs. Nevertheless, the BSIs appear to enjoy good access to and relationships with government officials. BSI directors often sit on local committees and advisory groups that help to draft or comment upon legislation and regulations. Among the issues that BSIs are discussing with local officials, lengthy and cumbersome bureaucratic procedures (e.g., for registration of new businesses) and excessive government inspections were frequently mentioned. Several BSIs are working with local officials to develop a “registry of inspections” to reduce the number of times government agencies inspect individual enterprises. BSI involvement in these issues is positive as it strengthens the voice of the SME community, and it helps to keep the BSI staff up-to-date with current issues. However, as BSIs have not been selected by the SME community to represent its interests, the team has some doubts about the impact of these BSI activities on local legislation and regulation. July 2000 SME Evaluation - 47 - Self-sufficiency of BSIs The evaluation team found that BSIs were willing to reveal little if any information about their revenues and costs. Given the almost universally-held belief in Russia that all businesses (including NGOs) keep at least two sets of financial books to avoid exorbitant taxes, the evaluation team’s difficulty in this regard is not surprising. There may be other reasons as well for the lack of financial information. It may be that BSIs’ accounting systems do not permit easy tracking of sources and uses of funds. Also, there are a number of BSIs, particularly in the Morozov system, whose accounts are not easily separated from those of the educational institution in which they are housed. Nevertheless, the following observations provide some insight into the BSIs’ financial situation, suggesting that most of these institutions operate on very limited budgets. · IRIS’ BSI directors indicate that the grants they receive, typically in the range of $15,000-$30,000, have had a major impact on their cash-flow. · The EBS program’s $1,000-$2,000 payment per American volunteer is sufficient incentive to get BSIs to carry a variety of time-consuming tasks -- contacting potential clients, preparing scopes of work, developing profiles of the client, facilitating pre-visit dialogue between the client and the volunteer, supporting the volunteer in-country, and assessing the outcomes. These modest payments, sometimes supplemented by fees from the SMEs receiving the volunteers, apparently represent a significant share of the BSIs’ income. One ACDI/VOCA￾Winrock BSI told the team that income resulting from the EBS program, through EBS payments and fees collected from the Russian firms receiving volunteers, represent 80% of its revenue. Although the team does not believe that this level of dependence on EBS is common, it is clear that payments of less than $2,000 are quite significant payment relative to other income. · In the Morozov program, the fact that MRCs east of the Urals find the payment of airline tickets to Moscow a major impediment to sending staff to training sessions again suggests that overall budgets are small. Despite their very limited income, BSIs assisted by USAID programs (with the exception of the ARC centers) are essentially self-sufficient. Self-sufficiency is defined here to mean that a BSI generates sufficient revenue from the sale of services, mostly to businesses,11 possibly supplemented to a modest extent by stable local government support, to pay its operating costs. Government support typically occurs through the provision of free office space, often from a government educational or scientific institution that houses the BSI. These subsidies represent a small percentage of the BSI’s income and are stable, i.e., not subject to volatile legislative processes or political shifts. 11 Some BSIs also sell training services to government institutions, and some receive grants from international organizations and/or subsidies from Russian government agencies. July 2000 SME Evaluation - 48 - Of the five USAID programs reviewed, only the ARC program provides operating subsidies to its Russian BSIs. In the other four USAID-assisted programs, expenditures for BSIs are essentially payments for services (e.g., through EBS) or subsidies for capacity enhancing activities (e.g., the training of BSI staff through Morozov, IRIS and EBS and new or improved services through IRIS). An important question is: To what extent do these USAID-supported programs contribute to the long-term financial viability of their client BSIs? As indicated in previous sections of this report, the team believes that the answer to this question is positive, although not as markedly as one might hope. · ARC. ARC has encouraged and helped its BSIs to increase their ability to generate fees and obtain grants from sources other than USAID. Nevertheless, operating subsidies to ARC’s BSIs with USAID resources, including direct cash transfers and the provision of management assistance and trainers, remain very substantial. ARC’s continued provision of large operating subsidies reduces the pressure on its BSIs to aggressively seek new funding and to reduce expenditures. ARC also provides assistance to its BSIs that is more appropriately considered investment: the provision of equipment, the training of trainers, and the training of BSI management staff. These investments will clearly help to build long-term financial sustainability. · IRIS. IRIS training and grants should increase BSIs’ income generating capacity. However, in reviewing a sample of the grants, the revenue generation potential of the new or improved services seemed often not to be a major consideration in grant selection or design. · EBS. The key to these BSIs’ longer-term financial viability is their capacity to carry out consulting with Russian (not American) consultants. Clearly, the BSIs are learning relevant skills -- how to market consulting, identify SME problems that may be good targets for consulting, develop their client bases, etc. These are important assets. However, relatively little attention has been paid to developing a cadre of Russian consultants. Further, the EBS BSIs’ common practice of asking SME clients to pay only some of the local expenses but not a fee for their American volunteer consultant may be setting a precedent that will make it more difficult later to sell consulting at market prices. One EBS BSI director, whose organization had suffered terrible set-backs when a previous USAID project ended, commented, “It is a terrible mistake to give away things that you later hope to sell.” · Morozov. To the extent that AMM training of trainers and curriculum development activities improves MRCs’ programs, they will contribute to the latter’s longer-term viability. Efforts to enhance the Morozov name nationally would also help MRCs to market their services. Several MRC directors commented that local Morozov name recognition was entirely the result of efforts of the MRCs themselves. July 2000 SME Evaluation - 49 - Self-sufficiency is an important and appropriate goal for the BSIs. BSIs have succeeded in achieving this goal either by developing a balanced cost and fee structure and/or by supplementing fee revenue with grants and subsidies. It is to the credit of USAID, its intermediaries and its BSI partners that so many BSIs are now operating with minimal government subsidies and providing services of a reasonable quality to the SME community. BSI capacity development USAID’s multi-year program of institutional development assistance has been an appropriate and effective strategy for developing the capacity of targeted service delivery intermediaries. Thanks in part to past and present USAID programs, Russia now has a considerable number of BSIs that are self-sustaining, and these institutions are doing a credible job of providing training and (to a lesser extent) consulting services to help SMEs, enhancing business skills across the country. The team’s observations suggest that progress under current programs continues to be made in enhancing BSI capacity, but progress appears to be uneven and advancing at only a modest rate. This conclusion reflects the greater capacity and maturity of the BSIs themselves and their increased ability to generate their own capacity development through learning from experience and listening to others. The benefits in terms of BSI capacity development from current programs can be seen primarily in process rather than impact indicators. Each of the programs trains BSI staff, and the numbers and variety of topics are impressive. However, it was not clear from the data reported or from the interviews conducted that this training had greatly increased the BSIs’ capacity to provide effective training, consulting or other services to SMEs. IRIS grants are allowing recipient BSIs to introduce new services, but it remains to be seen how important those new services will be to the SME community. BSIs have procured computers and other equipment and set up websites, but it was difficult to see that the SMEs were deriving major benefits from those investments. In EBS programs, BSIs have developed and are effectively using capacity to support American volunteer programs. This is a significant advance, reducing the cost of the volunteer program. However, a more important step in long-term BSI capacity development would be the development or expansion of their capacity to do consulting with Russian personnel. Overall, after visiting 35 program sites, the evaluation team heard many positive comments and is convinced that the USAID programs are having a positive impact. At the same time, the team would be hard-pressed to name many BSIs in which the USAID￾assisted programs are having a dramatic or profound impact on the capacity to provide effective services for the SME community. (See discussion in Section IV of implications for future USAID strategy.) It is important to emphasize, as was done in Section II, that some of the USAID programs are intended to do more than impact on BSI capacity and should not be assessed solely on that basis. In particular, the EBS program yields a very significant positive impact July 2000 SME Evaluation - 50 - directly on a select, albeit small, number of SMEs. It is also noteworthy that producing this impact requires a type of input, experienced businesspeople, for which there is essentially no substitute in Russia. ARC would argue, similarly, that its program is less about developing long-term BSI institutional capacity than about directly helping SMEs. It also directly benefits Russian businesses, and there are no Russian substitutes for some its inputs. However, after seven years and a very considerable USAID and University of Alaska investment, the evaluation team believes that the focus needs to be more strongly focused on developing Russian capacity to replace most of ARC’s American inputs. The ARC program also yields a third type of benefit, collaboration between Alaska and RFE, which is of considerable economic and political value. However, Alaska has many types of relevant experience and expertise to offer the RFE, and bringing these to bear would yield additional economic and political benefits. Unduly extending the current ARC program may, by tying up USAID’s limited financial resources and the leadership and organizational capacity of ARC/Anchorage, effectively limit the opportunity to bring those other unique Alaskan resources to bear on RFE development problems. July 2000 SME Evaluation - 51 - VIII. Implications for Future USAID SME Support Strategy USAID’s strategy for assisting Russia to make the transition to a democratic society with a free market economy is laid out in its planning document, “USAID/Russia Country Strategy, 1999-2003.” That document identifies strategic objectives (#1.3): “the accelerated development and growth of private enterprise.” Key impediments to achieving that objective are identified as (1) the lack of a clear regulatory policy for business creation; (2) the paucity of business skills; (3) the lack of access to credit; (4) the lack of affordable business services, and (5) the lack of a business information culture. The means that USAID will use to address these impediments include: · Business training for individual entrepreneurs in marketing and western management practices · On-site client-focused business consulting services · Formation of sustainable business associations, trade organizations and business support centers · Access to credit from non-bank sources · Land privatization to help make additional collateral available The Mission’s “Strategy for Small and Medium Business Development” identifies the key elements of the strategy to be: · Continue to strengthen the ability of grassroots organizations to advocate for SMEs; · Inform economic decision making at all levels of government, targeting efforts particularly to the regions; · Maintain focus on strengthening management skills and business service infrastructure; · Enhance efforts to link technical assistance and training of businesses to sources of financing; · Expand successful non-bank micro-finance programs; and · Improve the availability of business and policy information. A. Findings and Recommendations with Respect to Strengthening Business Skills and Business Service Infrastructure This review finds that the five programs reviewed are entirely consistent with the Mission’s stated objectives and strategy.12 These five programs are essentially about “business training for entrepreneurs in marketing and western management practices,” the development of capacity for and the delivery of “business consulting services,” and the “formation of business support centers.” Morozov and IRIS, and to a lesser extent the 12 USAID’s strategy is articulated in “USAID/Russia Country Strategy, 1999-2003,” dated April 5, 1999 and in “USAID/Russia Strategy for Small and Medium Business Development in the Russian Federation” (undated). July 2000 SME Evaluation - 52 - other three programs, are designed to produce sustainable business support centers that train Russians in western business practices and provide focused business consulting services. ARC is focused on providing business training directly to Russians, but it is also developing centers that are gradually moving toward sustainability. EBS is directly involved in providing “focused business consulting services,” but it is also developing the capacity of Russian business support organizations to support the provision of American consultants and to expand its Russian consulting activities. The Team believes that USAID could continue indefinitely to identify and support new BSIs, or to strengthen its existing client BSIs, and this assistance would be useful. But this raises the question: When is enough? What is USAID’s exit strategy? USAID/Russia’s Country Strategy addresses the same question: “How do we know when we’re done?”13 The Country Strategy goes on to identify six indicators that would suggest that enough has been accomplished. The first and most important of these is “70 Russian Business Support Institutions are self-sufficient.” This study has found that many more than 70 self-sufficient BSIs exist. However, most are small, and many are weak. Business consulting is still in its infancy, often limited to free “consultancies.” At what point, then, will there be a sufficient infrastructure of self-sustaining BSIs -- organizations capable of growing and adapting to changes in their environment and enhancing business skills across the country -- to justify USAID’s “declaring success” and moving its assistance resource to other areas? With respect to assistance to build self-sustaining BSIs, the evaluation team believes that time may be close at hand. However, the question of further development of BSI capacity must be addressed in the context of budget availability and investment options. In a tight budget environment, USAID must consider which business services are most needed, which are closest to being adequately provided, and which can be provided on a sustainable basis (i.e., will generate income adequate to cover costs. Recommendations On this basis, the evaluation team suggests the following: · Give lower priority in USAID programming to business training. Certainly USAID programs could continue to improve the quality and extend the number of programs that currently exist. However, Russians institutions now have considerable experience in this area, and the more capable of them will continue to grow and improve even without USAID assistance. Where training is not being done by Russians, emphasis should be on the development of that capacity. · Continue to support the provision of business consulting programs. Until Russian capacity in increased, American business volunteers can provide a unique and 13 USAID/Russia Country Strategy, 1999-2003; April 5, 1999; page 16. July 2000 SME Evaluation - 53 - important contribution. At the same time, USAID programs should exploit every opportunity to develop Russian consulting capacity. · Selectively support the development of other business support services. The first step in this regard would be to carefully and critically assess the IRIS program’s efforts to support new services. That assessment should include the costs and the revenue generated. These results, both the negative and the positive, need to be widely disseminated so that the larger BSI community can learn from the mistakes and the successes. Some continued USAID experimentation and subsidization of investments in this area might be desirable. However, if these services are productive and information about them is available, the market will see that they are expanded to new communities. Morozov might be an appropriate institution to market the more successful of IRIS’ approaches over the longer term to the BSI community. · Explore interest in the development of regional BSI networks. This evaluation revealed that the existing project networks, created and effectively administering programs from above, also serve to enable informal networking among smaller groups of BSIs. This networking is a major method of transferring technology from one institution to another. Larger regional BSI networks or associations, formed from below, might thus be able to play a much-expanded role in information and technology sharing. A BSI association of this kind might also be effective in advocacy on behalf of BSIs and SMEs. The evaluation team recommends that USAID consider whether and how it could be a catalyst for the formation of such networks or associations. The team believes that a successful strategy might start with the selection of a small number of outstanding BSIs in key regions that could explore the level of interest in this idea with other BSIs. If regional BSI associations developed, some very modest USAID support for studies (e.g., of local regulation), office equipment and limited technical assistance (e.g., on research and advocacy) might be appropriate. The evaluation team would not view it as appropriate to provide operating budget resources. · Under difficult resource constraints, focus activities selectively on promising regions. With budget reductions, the USAID must increasingly make difficult choice among promising investments. The team would recommend that the Mission increasingly focus SME strengthening activities on selected regions where the business environment is most conducive to private sector growth. B. Findings and Recommendations with Respect to Other Elements of the Mission’s Strategy Although this study did not include USAID programs outside of business skills development and consulting, the evaluation team asked interviewees about their views about critical constraints to SME development. The responses suggest that the most July 2000 SME Evaluation - 54 - serious impediments are, most importantly, the business environment and, secondly, financing. The team found that the major concerns about the environment were the extraordinary amount of regulation and inspection and the lack of clarity about them. Registering a new enterprise, for example, may take a year (if one does not take extra-legal short-cuts). Not only does the process require a large number of steps, but there is almost never a document that spells out the whole process. Inspections, established for supposedly legitimate reasons, are reportedly used more often to elicit income for the inspector and his/her supervisors. USAID’s strategy correctly recognizes that improvements in the business environment are essential to the success of its SME development effort. The team found that the major concerns about the financial system centered on the scarcity of credit for small enterprises, particularly microenterprise credit, and the terms at which credit is extended. With regard to the latter, limitations on the length of lending seem as much of a concern as are high interest rates. A number of respondents also voiced concern about the lack of financial mechanisms such as leasing and franchising. Recommendations These concerns, often expressed by the team’s large number of interviewees in the business community, suggest additional USAID programming priorities might include: · Give increased priority to work at the regional level to improve the business environment. Business associations. USAID strategy already recognizes the importance of forming business associations that can represent the interests of the SME community and advocate changes before local government bodies. In an advocacy role, these organizations can exert substantially more economic and political influence than can BSIs and are thus likely to be more successful. Training of public sector officials. One additional way to influence the thinking of thousands of public sector officials might be to work through the Russian public service training (university level) schools (of which there are some 13 across Russia). Each of these institutions trains thousands of current and future public sector officials annually. The team’s interview with the director of the school in Novosibirsk suggests that this institution would welcome assistance in developing curriculum materials (e.g., case studies on the successful use of mapping the registration process) to help public sector officials recognize current problems and learn how to correct them. Regional BSI networks. A second idea might be to build a different kind of network of BSIs (as suggested above) with advocacy as one of its functions. Although individual BSIs have limited effectiveness in advocacy, a large network of BSIs might be effective in this area. Although none of the existing top-down July 2000 SME Evaluation - 55 - BSI networks are candidates for such an effort, a large bottom-up network, probably limited to a region, might be able to speak with sufficient authority to command the attention of local and oblast officials. · Expand efforts to strengthen credit systems, either through banks or non-banking institutions, to meet the needs of SMEs. Improving financial mechanisms. The team suggests that USAID explore whether more can usefully be done to strengthen credit systems, either through banks or non-banking institutions, to meet the needs of SMEs. Programs to stimulate the development of leasing, franchising and other financial mechanisms should also be considered Microenterprise credit. The evaluation team recommends that this program be expanded, perhaps in partnership with top-performing BSIs (as is now occurring in Yuzhno Sakhalinsk between USAID’s implementing agency and the ARC). The team also believes that the time is ripe in Russia for USAID to pursue other small enterprise funding initiatives, perhaps as part of a consortium of public and private funders. Given the difficult legal and regulatory environment faced by non-profit organizations, consideration might be given to opening commercial bank windows for micro-lending. ANNEXES A. Scope of Work B. Evaluation Team Members C. BSIs Visited D. People Interviewed E. Interview Protocol Annex A Page 1 of 9 Scope of work Page 1 Evaluation of USAID/Russia’s Small Business Development activities SCOPE OF WORK FOR EVALUATION OF SMALL BUSINESS SUPPORT ACTIVITIES I. ACTIVITIES TO BE EVALUATED This evaluation will examine USAID/Russia’s principle Small and Medium Sized Enterprise (SME) support activities. The following four USAID/Russia SME support activities will constitute the primary focus of this evaluation: - The Morozov Project implemented by the Academy of Management and Market (AMM) since 1993 provides business education and training via a network of 60 sustainable training centers located throughout Russia. - The American – Russian Centers (ARC) Project implemented by the University of Alaska since 1993 contains three basic components: promoting entrepreneurship and small business development in the Russia Far East (RFE); encouraging American – Russian business connections; and facilitating educational and cultural exchanges between Alaska and the RFE. The ARC Project currently maintains centers in Yuzhno Sakhalinsk, Khabarovsk, and Magadan in the RFE. - The Entrepreneurial Business Support (EBS) Program recently initiated in May 1999 as the follow on to the Business Volunteer Program. Like its predecessor, the EBS Program provides businesses throughout the Russian Federation with short-term technical assistance from US Business Volunteer experts. Overall, the EBS program is designed to promote development and growth of private enterprise in Russia by (a) enhancing the sustainability of SME’s, (b) strengthening a group of 20 selected Russian Business Support Institutions (BSI’s) across Russia, and (c) developing local capacity among Russian BSI’s to independently administer similar volunteer technical assistance programs in the future. The program is implemented by Citizen’s Democracy Corps (CDC) in Western Russia and by a consortium lead by ACDI/VOCA in the Russian Far East (RFE) and Siberia. - The SME Support Network Strengthening Project, implemented since September 1998 by IRIS, fosters the creation of a network of 33 Business Support Institutions (BSI) throughout Russia capable of rendering business support services to the Russian private sector and engaging in advocacy efforts on behalf of the Russian SME sector. These SME support activities contribute to USAID/Russia’s Strategic Objective 1.3. Accelerated Development and Growth of the Private Sector. In addition, these activities also contribute specifically to Intermediate Result (IR) 1.3.3 Successful Models of Private Ownership and Modern Management Widely Replicated, and IR 1.3.4 Sustainable Network of Business Support Institutions Rendering Services to Entrepreneurs and Enterprises. I. BACKGROUND Because small and medium sized enterprises are recognized as the primary engine for economic growth and job creation in most free market economies, development of Russia’s nascent SME sector is critical if Russia is to successfully complete the transition to a free market economy. Over Annex A Page 2 of 9 Scope of work Page 2 Evaluation of USAID/Russia’s Small Business Development activities the past 6 years, USAID/Russia has actively promoted SME development throughout Russia. These efforts include providing direct technical assistance to small and medium sized Russian businesses through the deployment of American business volunteers, establishing and supporting numerous business training centers, and providing assistance to SME’s through a network of Russian Business Support Institutions located across the Russian Federation. USAID/Russia’s first SME support program began in January 1993, when the American Russian Center (ARC) of the University of Alaska established three centers in the Russian Far East to promote the concept of entrepreneurship and to support small business development. Shortly thereafter, USAID/Russia initiated the Morozov Project, expanding SME support activities across the Russian Federation through a network of 60 regional Morozov Centers. Since its inception in 1994, USAID/Russia’s Morozov Project has provided 10,000 small businesses throughout Russia with technical assistance. Furthermore, the Morozov Project has also trained 60,000 entrepreneurs and local government officials, and provided consulting services to 12,000 individuals across Russia. In addition to the ARC and Morozov Projects, US business volunteer programs have also played a critical role in USAID/Russia’s support strategy for the Russian SME sector. Under the business volunteer programs, US business consultants and managers are placed with local firms and businesses throughout the Russian Federation offering individualized consulting services on a volunteer basis. Since 1997, these business volunteer programs have been managed through cooperative agreements with Citizens Democracy Corps (CDC), International Executive Services Corps (IESC), and ACDI/VOCA. To date, approximately 1,300 Russian firms and businesses have directly benefited from the consulting services offered by these programs. USAID/Russia’s current business volunteer program, the Entrepreneurial Business Service (EBS) Program, was recently initiated in May, 1999. This latest business volunteer program is implemented by CDC in Western Russia and ACDI/VOCA in the Russian Far East and Siberia. However, the EBS Program differs from previous business volunteer programs, because a select group of 20 local Russian BSI’s has been chosen to implement the program in conjunction with CDC and ACDI/ VOCA. Thus, the training that these 20 Russian BSI’s receive from CDC and ACDI/VOCA should enable them to independently administer USAID business volunteer programs in the future without the assistance of US PVO organizations. The SME Support Network Strengthening and Assessment (SME-Net) Project, implemented by IRIS since September 1998, represents the final element of USAID/Russia’s current SME support strategy. Under the SME-Net Project, IRIS systematically surveyed and assessed over 300 BSI’s across Russia that had previously participated in USAID sponsored activities or directly received USAID assistance. Upon completion of this intensive survey and assessment phase, IRIS selected 33 of the strongest BSI’s to receive additional technical assistance and training and to compete for $850,000 in small grants funding. The ultimate goal of the SME-Net project is to create a network of sustainable BSI’s across Russia capable of rendering high quality business support services to the Russian private sector. Additionally, this network of BSI’s should also be able to engage in advocacy efforts on behalf of the SME sector at both the federal and regional level. The anecdotal evidence indicates that the four SME support activities outlined above have had a significant impact upon the Russian SME sector. However, because the Morozov Project and the SME-Net Project will be ending in August 2000, and because USAID/Russia’s Office of Business Annex A Page 3 of 9 Scope of work Page 3 Evaluation of USAID/Russia’s Small Business Development activities Development and Investment (BDI) is currently contemplating follow on projects, it is an opportune time to formally evaluate USAID/Russia’s SME support strategy and activities. III. INFORMATION SOURCES See attached chart IV. PURPOSE OF EVALUATION This evaluation has three principle purposes: 1. To assess USAID/Russia’s overall SME support strategy; 2. To measure the impact of the four main projects through which USAID/Russia implements its SME support strategy; and 3. To evaluate the sustainability of those Russian BSI’s and training centers receiving assistance through USAID/Russia’s SME support activities. The evaluation will examine USAID/Russia’s SME overall support strategy and provide recommendations and suggestions for modifying and refining this strategy if necessary. The evaluation will reveal the strengths and weaknesses of each of the four SME support activities outlined above. From a strategic perspective, the evaluation will analyze the role that each of these activities play in carrying out USAID/Russia’s SME support strategy and the extent to which these activities complement one another. In addition, because the SME-Net and Morozov projects will be ending shortly, the evaluation will provide recommendations concerning the design of possible follow on activities. In this respect, the evaluators should examine whether these interventions remain a priority and if so, whether it is possible to combine and consolidate some of these activities and how this should be done. The evaluation will also reveal lessons learned and best practices of the program to help determine future, long-term USAID strategy in the field. With regard to Russian BSI’s and training institutions receiving assistance from USAID, USAID/Russia staff is interested in measuring the sustainability and effectiveness of these local Russian organizations. Because USAID and other international donors do not envision providing assistance to Russia indefinitely, it is critical that local Russian organizations working to strengthen the SME sector become both self sufficient and capable of providing high quality business support services to Russian SME’s. Developing local Russian support capacity is the only way to guarantee that SME’s throughout Russia will continue to be promoted and supported after international donor assistance has ended. V. EVALUATION QUESTIONS The evaluation should address three general question areas: (I) Are USAID/Russia’s four main SME Support Activities effectively providing assistance to Russian BSI’s, training institutions, and small and medium sized enterprises? (II) Are the Russian BSI’s and training institutions that are participating in USAID/Russia’s SME support activities sustainable without USAID funding? (III) Do the existing elements of USAID’s strategy remain high priorities? Or, should Annex A Page 4 of 9 Scope of work Page 4 Evaluation of USAID/Russia’s Small Business Development activities USAID/Russia modify its overall SME support strategy? If so, how should the strategy be modified and what kinds of activities should be involved in implementing that modified strategy? The following list of questions is not meant to be exhaustive, but illustrative of the issues that should drive this evaluation. (I) Are USAID/Russia’s four main SME Support Activities effectively providing assistance to Russian BSI’s, training institutions, and small and medium sized businesses? 1. Have USAID/Russia SME Support activities had a significant impact upon the economic development of the regions and communities where these activities are located? 2. Which of the four SME Support activities have produced the most positive results and why? 3. What approaches or mechanisms have worked best for providing technical assistance to BSI’s and training institutions? Why? 4. What approaches or mechanisms have worked best for providing technical assistance to SME’s. Why? 5. What are the greatest accomplishments and lessons learned of these SME support activities? Are they replicable? 6. What are the major constraints facing the contractors and grantees of these activities? 7. How have each of the grantees or contractors performed relative to the requirements of their cooperative agreements or contracts? 8. Have those Russian BSI’s and training institutions participating in USAID/Russia’s SME support activities been able to expand their businesses, training programs, or services as a result of USAID assistance? If so, please provide examples. 9. How do Russian SME’s value the consulting services and training provided by those BSI’s and training institutions that are receiving USAID/Russia assistance? Is there significant market demand for the fee based services that these BSI’s or training institutions offer? 10. How successful have the contractors and grantees been in assisting their partner BSI’s or training institutions to expand their businesses and become sustainable independent of USAID funding? 11. What mechanisms are contractors and grantees using to assist their partner BSI’s and training institutions achieve sustainability? 12. Could the contractors or grantees do more to assist their partner BSI’s or training institutions to achieve sustainability? If so, how ? 13. How flexible have the programs been in responding to changing conditions and varied demands for SME assistance in Russia? (II) Are the Russian BSI’s and training institutions that are currently participating in USAID/Russia’s SME Support activities sustainable without USAID funding? 1. What percentage of BSI’s and training institutions participating in USAID/Russia SME activities can be considered sustainable without USAID assistance? 2. For those BSI’s and training institutions that are not sustainable without USAID assistance, what is the greatest impediment to reaching sustainability? 3. Why are certain BSI’s and training institutions sustainable independent of USAID resources, and why do others fail to reach this level of sustainability? 4. How effective are the BSI’s and training institutions in providing high quality business Annex A Page 5 of 9 Scope of work Page 5 Evaluation of USAID/Russia’s Small Business Development activities support services (i.e. business planning, financial management consulting, marketing and PR services, etc.) to the Russian private sector? Is their significant market demand for the fee based services that these BSI’s or training institutions offer? If not, why not? 5. What are the primary areas in which these BSI’s and training institutions focus their consulting and training services? 6. How effective are these BSI’s and training institutions in marketing and promoting their services to private clients? 7. Do the BSIs and the training institutions have business plans, and operating procedures and policies that allow them to achieve their goals? 8. Do the goals and policies of the BSIs and the training institutions exploit market opportunities or rely heavily on donor support? 9. With respect to the EBS activity, will the Russian partner BSI’s be capable of independently administering USAID business volunteer programs in the future, without the assistance of a US Partner PVO, once the current EBS activity ends? 10. With respect to the SME-Net activity, upon completion of the activity in September 2000, will the 33 participating BSI’s be capable of maintaining a sustainable network of BSI’s across Russia that provides quality business consulting services and engages in advocacy efforts on behalf of the Russian SME sector? 11. With respect to the Morozov activity, absent USAID funding, would the Morozov network, and the individual Morozov Centers that comprise that network, be sustainable? (III) Should USAID/Russia modify its overall SME support strategy? If so, how should the strategy be modified and what kinds of activities should be involved in implementing that strategy? 1. What is the most effective way to build on the success and progress achieved by USAID/Russia’s current SME Support Activities? How should future programs be designed to capitalize on past progress and avoid past failures? 2. If a significant number of USAID’s participating Russian BSI’s and training institutions still have yet to reach sustainability, should additional resources be allocated in an attempt to make these local organizations sustainable? And, how can future USAID/Russia programs achieve greater success in helping local Russian BSI’s and training institutions become self sustainable? 3. Do USAID/Russia’s current SME support activities complement one another. If so, do synergies exist among these activities? 4. Are certain USAID/Russia SME support activities duplicative of one another? 5. Is it possible to increase efficiency without jeopardizing results by combining certain USAID/Russia SME support activities? If so, which of these activities should be combined and how should this be done? The team hired to conduct this evaluation should collect the most important and interesting success stories (not more than three-five) which may be identified in the discussions with project personnel and activity managers. Collected success stories should be presented in the report but should not influence the objectivity of the overall evaluation. If necessary, the team may place found success stories in a separate annex attached to the report. VI. EVALUATION METHODS Annex A Page 6 of 9 Scope of work Page 6 Evaluation of USAID/Russia’s Small Business Development activities The evaluators in collaboration with the USAID/Russia will finalize the overall evaluation methodology. However, USAID expects that at a minimum the evaluators will: 1. Review and analyze the existing performance information; 2. Interview representatives of the home offices of organizations included in the review; 3. Interview field staff of USAID, the implementing organizations, organizations implementing associated programs, including private sector organizations, and Russian Federal Government and local government counterparts; 4. Conduct site visits to a representative number of cities and regions in the Russian Federation, including at least three areas outside of Moscow and St. Petersburg and at least one in the Russian Far East. VII. SCHEDULE Approximately eight weeks are estimated to complete this evaluation with an assumption of a five￾day workweek. If necessary, a six-day workweek is authorized. A representative work schedule is indicated below, but it may be modified on mutual agreement between the outside team and the Evaluation Coordinator from USAID/Russia. Activity Description Location Approximate Dates Outside team selection Selection of contractor, negotiations Sign contract USAID/Russia will provide general background, program and other documentation. Moscow Washington February 22 – March 27 March 20 – 24 Background Finalize schedule, review background documents and performance information, design a list of interviews, develop survey instrument(s), if necessary, and report outline. Finalize and discuss the methodology and the scope of work with Evaluation Coordinator (by e-mail). Washington March 27 – 31 Interviews I Interviews with AID/Washington staff and staff from home offices of organizations-providers. Select sites to visit and draft the schedule. Start logistical arrangements. Washington April 3 – 7 Interviews II Interviews with Mission and Provider’s staff, subcontractors, counterparts, NGOs, and organizations implementing affiliated programs. Russia April 10 – 14 Annex A Page 7 of 9 Scope of work Page 7 Evaluation of USAID/Russia’s Small Business Development activities Review methodology and refine, if necessary. Finalize travel schedule with Evaluation Coordinator. The team may wish to split into two sub-teams and visit different regions where the project has been implemented. Site Visits Conduct site visits. Begin drafting reports. Report structure discussion with the USAID/Russia Evaluation Team. Agreement by the Evaluation Coordinator. Before departure to Washington, prepare the first draft of the report, and provide exit briefing to USAID/Russia. Russia April 17 – May 5 May 11 Analysis, Report Draft final report design, additional interviews, if necessary. Report draft submitted to USAID/Russia for comments. USAID/Russia reviews and comments final draft. Incorporate the comments into the report, finalize and submit to USAID/Russia. Washington May 15 – 29 May 30 May 30 – June 9 June 12 –16 The final report is expected to be submitted to USAID no later than June 19, 2000. VIII. REPORTING AND DISSEMINATION REQUIREMENTS The final report will include an overall assessment of the issues listed in the section “IV. Purpose of Evaluation” and will address the questions listed in the section “V. Evaluation Questions”. Other information to be included in the report will be determined in consultation with USAID staff over the course of the evaluation. The final report will be submitted to USAID/Russia on diskette in MS Word with ten hard copies as well. The structure and format of the report will be proposed by the evaluator and approved by the Evaluation Coordinator at the beginning of the evaluation. The evaluation report will primarily be for internal use by USAID project management and ENI/UDH in USAID/Washington. It may, at USAID’s determination, be disseminated to outsiders. IX. TEAM COMPOSITION AND PARTICIPATION A team comprised of two US consultants and two Russian consultants will carry out the evaluation with one of these experts acting as team leader. Additionally, one support staff person will support the team as an interpreter and logistics coordinator. Fieldwork may be augmented by USAID Mission staff, as available. The members of the team are as follows: Annex A Page 8 of 9 Scope of work Page 8 Evaluation of USAID/Russia’s Small Business Development activities - Team Leader: Responsible for coordinating and directing the reporting effort, including preparation and submission of the draft and final report. The incumbent should have extensive overseas program evaluation experience, including USAID experience, preferably in the ENI region. He/she must be thoroughly familiar with techniques of program appraisal. As team leader, the incumbent should possess excellent organizational and team-building skills. - Three small business development experts: Must possess both overseas and evaluation experience. Two of them should be familiar with USAID programs in the area of small business development. These consultants should have a combination of consulting experience that includes small business development, business promotion and management. These consultants should also be familiar with the role that both federal and local governments, non￾governmental organization, communal service providers and financial institutions play in the development of a strong small business sector. At least one of these experts should have real life small business experience, not necessarily from a donor's perspective. - Interpreter and Logistics Coordinator: He/she should have knowledge of terminology related to small business activities. He or she will translate conversation between the evaluation team and Russian-speaking program participants, as well as any Russian language documents provided to the evaluation team. Experience in simultaneous translation is desired. This person will be also responsible for all necessary actions as a Logistic Coordinator (i.e. schedule, meeting arrangement, tickets, etc.). X. BUDGET The current Cooperative Agreements and contracts do not budget funds for an evaluation. PD&S funds will be used to finance this evaluation. Team members from USAID if necessary will be funded from their contracts. An estimated budget for this evaluation is attached. Annex A Page 9 of 9 Scope of work Page 9 Evaluation of USAID/Russia’s Small Business Development activities ATTACHMENT INFORMATION SOURCES Project Contractor Agreement # Period Information Available IRIS SME Support Project IRIS OUT-PCE-I-00-97-00042-00 DO#807 9/1/98-8/31/00 Contract proposal, Workplan, monthly, quarterly reports, BSI selection data Entrepreneurial Business Services West CDC 118-A-00-99-00055-00 5/1/99-4/30/01 Cooperative Agreement, Workplan, monthly flash reports, quarterly reports Entrepreneurial Business Services East ACDI/VOCA 118-A-00-99-00057-00 5/1/99-4/30/01 Cooperative Agreement, Workplan, monthly flash reports, quarterly reports Morozov Project Phase I Phase II Research Foundation of the State University of New York AMM 110-0005-G-00-4031-00 118-A-00-98-00190-00 7/94-4/98 9/98-8/00 Cooperative Agreement, Monthly flash reports, quarterly reports, project evaluation, final report Cooperative Agreement, Monthly flash reports, quarterly reports Business Development in the Russian Far East American Russian Center at the University of Alaska-Anchorage 118-A-00-97-00194 1/93-7/99 Grant Agreement, Workplan, Monthly flash reports, quarterly reports Novgorod SME Support and Development Project (NSDP) Nathan MSI Group OUT-PCE-I-806-98-00016-00 DO#806 9/29/99-5/15/00 Workplan Consulting Services for Russian Entrepreneurship (CSRE) ACDI/VOCA 118-A-00-97-00124-00 4/10/97-5/31/99 Cooperative Agreement, Workplan, Monthly flash reports, quarterly reports Business Collaboration Center (BCC) CDC 118-0005-A-00-5309 9/29/95-9/30/99 Cooperative Agreement, Workplan, Monthly flash reports, quarterly reports, newsletters Client Focused Business Volunteer Program (BVP) IESC 118-A-00-97-00109-00 5/97-5/99 Monthly flash reports, quarterly reports, final report Enterprise and Economic Development Project (EED) CDC 118-A-00-97-00157-00 2/1/97-4/30/99 Monthly flash reports, quarterly reports, CDC publications Small Business Opportunities Opportunity International 110-A-94-00040-00 9/94-6/00 Monthly flash reports, quarterly reports Small Business Incubator SUNY, AARW 118-0005-A-00-5194-00 3/95-10/98 Monthly flash reports, quarterly reports, Russian Business Incubators Review Annex B Page 1 of 2 Evaluation Team Members Arman Alibekov is an Associate with CARANA Corporation and has been engaged in numerous economic development projects in the Former Soviet Union. In recent years Mr. Alibekov has been intensively involved in privatization and deregulation of the CIS economies and enterprise restructuring. In Russia, he served as a resident advisor to regional governments on economic reforms, competitor analysis and strategy formulation for economic development and investment promotion. Mr. Alibekov holds an MBA degree and was a recipient of Edmund S. Muskie and Freedom Support Act Fellowship. Byron Battle, a senior associate with CARANA, directs that firm’s Private Sector Development Practice Area with a special emphasis on small and medium enterprise development, business association strengthening, and investment promotion. He has led numerous assignments for CARANA in Russia, Central Asia, and Africa. He has directed CARANA’s projects under the REPAIS program which have focused on working with regional governments in Russia to improve the local business environment, improve business association activities, and encourage investment, both local and foreign. Prior to CARANA, he worked for many years with the international development consulting division of Arthur D. Little, Inc. Ivan Gerasimov, a lawyer, graduated from the Khabarovsk State Academy of Economics and Law, majoring in civil legislation and business law. He also graduated from Khabarovsk State Pedagogical University, majoring in English. Mr. Gerasimov’s career has been mostly devoted to practicing law. For a brief time he worked as an Assistant to Food Program Coordinator for CRS/Khabarovsk office. He left CRS to become the legal counsel for a timber export company. For the past four years he has been working as legal counsel for various, mostly private companies in Khabarovsk. He currently serves as a chief of the Legal Support Department of Far Eastern Railroad Company. Mr. Gerasimov has also carried out a variety of translating assignments for CARANA Corporation in Khabarovsk and accompanied CARANA’s consultants implementing investment promotion and association development projects. Nate Bowditch has held leadership positions on many sides of the development equation: enterprise development, investment promotion, small business microcredit, venture capital, tourism promotion, natural resources planning, historic preservation, city planning and community development. He has directed municipal and state government agencies in the United States and headed five non-governmental organizations on three continents. He has divided his career between the Northeastern United States and both Asian and African countries where he has lived and worked for 13 years. One career highlight was his position on the Cabinet of Maine Governor John R. McKernan, Jr. where he oversaw the State’s business development, tourism marketing, community development and community planning programs. More recently, Mr. Bowditch spent four years in Cape Coast, Ghana coordinating back-to-back United Nations and USAID Annex B Page 2 of 2 sponsored ecotourism projects involving three Ghanaian government ministries, five American organizations, a Ghanaian regional government, two Ghanaian cities, numerous small businesses and many rural communities. He was then named a Fulbright Senior Research Scholar to research and write a book on Ghanaian business management culture. The resulting work, The last Emerging Market, was published by Praeger (see ). Mr. Bowditch is currently an advisor to five business and entrepreneur associations in the Russian Far East and serves as a Senior Fellow at the New England Board of Higher Education in Boston, Massachusetts where he directs the New England Public Policy Collaborative. Gerald Wein, the team leader for this evaluation, is a development economist and program manager. He holds masters degrees in economics from the University of California (Berkeley) and in public administration from Harvard. Mr. Wein worked for USAID as an economist, project planning and evaluation specialist, and manger. His experience includes fifteen years as a resident development specialist in six Latin America and Africa countries and short-term assignments in more than forty countries in all developing regions of the world. He has served in a variety of management positions, including as the Deputy or Acting Mission Director of USAID programs in Nicaragua, Tunisia and Ecuador and, after leaving USAID, as the Director of two worldwide projects that provided technical assistance on health sector finance and policy reform to more than 35 countries. Currently, Mr. Wein is an independent consultant. His recent work includes the design and management of six project evaluations carried out for USAID/Russia by the CARANA Corporation. Annex C Page 1 of 2 BSI Program Sites Visited by the Evaluation Team, by Grantee Program Location BSI Name CDC Ekaterinburg International Consulting Center Novgorod Novgorod TecnoPark Rostov-on-Don ESC St. Petersburg Foundation for SME Support St. Petersburg Regional Economic Development Agency (ILIP) ACDI/VOCA Khabarovsk Far Eastern Chamber of Commerce & Industry Khabarovsk Russian American Education Center Novosibirsk International Consulting Center Omsk Business Center, Siberian Prof. Pedagogical College Vladivostok Corps of Executives Far East Russia (CEFER) Yuzhno Sakhalinsk Russian American Business Training Center (RABTC) IRIS Ekaterinburg International Consulting Center Ekaterinburg Chamber of Manufacturers Ekaterinburg Women’s Entrepreneurship Center Khabarovsk Territory of Law Novgorod Business and Training Center (BTC) Novgorod Regional Agency for SME Support (SMEDA) Novosibirsk International Consulting Center St. Petersburg Foundation for SME Support St. Petersburg Regional Economic Development Agency (ILIP) Vladivostok Confederation of Businesswomen Vladivostok Vlastra Association of Construction Firms Vladivostok High Technologies Association Voronezh RENAKORD Voronezh International Consulting Center ARC Khabarovsk Russian American Education Center (RAEC) Yuzhno Sakhalinsk Russian American Business Training Center (RABTC) Morozov Ekaterinburg International Institute of Distance Education Khabarovsk Khabarovsk BTC Moscow Institute for Entrepreneurship and Investment Novgorod Novgorod BTC Novosibirsk Siberian Academy for Public Administration Omsk Business Center, Siberian Prof. Pedagogical College St. Petersburg Higher Economic School Voronezh Educational and Consulting Center for Agribusiness Annex C Page 2 of 2 BSIs Visited by the Evaluation Team, by City and Grantee Morozov Ekaterinburg International Institute of Distance Education CDC/IRIS Ekaterinburg International Consulting Center IRIS Ekaterinburg Chamber of Manufacturers IRIS Ekaterinburg Women’s Entrepreneurship Center ACDI/VOCA Khabarovsk Far Eastern Chamber of Commerce & Industry ACDI/VOCA-ARC Khabarovsk Russian American Education Center IRIS Khabarovsk Territory of Law Morozov Khabarovsk Khabarovsk BTC Morozov Moscow Institute for Entrepreneurship and Investment CDC/ IRIS Novgorod Novgorod TecnoPark IRIS-Morozov Novgorod Business and Training Center (BTC) IRIS Novgorod Regional Agency for SME Support (SMEDA) ACDI/VOCA-IRIS Novosibirsk International Consulting Center Morozov Novosibirsk Siberian Academy for Public Administration ACDI/VOCA-Morozov Omsk Business Center, Siberian Prof. Pedagogical College CDC/ IRIS Rostov-on-Don ESC CDC/IRIS St. Petersburg Foundation for SME Support CDC/IRIS St. Petersburg Regional Economic Development Agency (ILIP) Morozov St. Petersburg Higher Economic School ACDI/VOCA Vladivostok Corps of Executives Far East Russia (CEFER) IRIS Vladivostok Confederation of Business Women IRIS Vladivostok Vlastra Association of Construction Firms IRIS Vladivostok High Technologies Association IRIS Voronezh RENAKORD IRIS Voronezh International Consulting Center Morozov Voronezh Educational and Consulting Center for Agribusiness ACDI/VOCA-ARC Yuzhno Sakhalinsk Russian American Business Training Center (RABTC) Annex D Page 1 of 5 People Interviewed ACDI/VOCA & Winrock Frederick W. Smith Assistant Vice president—Europe & New Independent States (ENI) Anne Dudte, Project Coordinator—ENI Division Naya Kenman Associate Director Rich Cleland Programmer Jeffrey S. Singer Russia Country Representative, Moscow Carol L. Hamblin, Project Director, Entrepreneurial Business services (East), Khabarovsk Nina I. Danilyuk Project Director, Khabarovsk Elena V. Prikhodko Program Manager, Khabarovsk Ludmila Khakhaleva Program Manager, Khabarovsk Sergei Kouptsov Informational Technology Specialist, Khabarovsk Natalia Marchenko Chief Accountant, Khabarovsk Tatyana L. Alekseyeva EBS Program Manager, Russian American Education Center, Khabarovsk (LRP) Marina N. Kirsanova EBS Project Manager, Russian American Business Training Center, Yuzhno Sakhalinsk (LRP) Igor Yu. Vostrikov Vice President, Far Eastern Chamber of Commerce, Khabarovsk (LRP) Natalia N. Shershunova EBS Program Manager, Far Eastern Chamber of Commerce & Industry, Khabarovsk, (LRP) Bakery Khabarovsk (client) Public Relations Firm Yuzhno Sakhalinsk (client) Olga Shagautdinova Owner, Right Shore, Khabarovsk (client) Irina O. Taenkova Director, Regional Association of Family Planning, Khabarovsk (client) Anna Popova Executive Director, CEFER (LRP), Vladivostok Alexander Popov President, CEFER (LRP), Vladivostok Nikolai Kornuyk Deputy Director, CEFER (LRP), Vladivostok Victor Podcaura Director/Manager, ICAR-V (client), Vladivostok Irina Chernova Program Manager, ACDI/VOCA, Novosibirsk Evgenij Chasin Director, Omsk Vocational School (LRP), Omsk Vladimir Stepanov Director General, EcoPlast (client), Novosibirsk Andrei Nakonechnyi Director General, Soverel (client), Novosibirsk Elena Bogdashina Director General, Drugstore chain 'Lekarstva Sibiri" (client), Omsk Vladimir Ivan'kin Director, International Consulting Center (LRP), Novosibirsk Inna Chikunova Program Manager, International Consulting Center (LRP), Novosibirsk Annex D Page 2 of 5 ARC Russell B. Howell Director, American Russian Center, Anchorage Benjamin Hanson Regional Director, American Russian Center, Sakhalin Natalya G. Ivanova Director, Russian American Business Training Center (RABTC), Sakhalin Vladimir N. Tuporshin Director, Business Software Co. Ltd., Sakhalin (client) Alla L. Serebrova Sakhalin Project Manager, ECOLINKS, Sakhalin (client) Tamara N. Silukova Director, Russian American Education Center (RAEC), Khabarovsk Marina V. Shesterkina Deputy Director, RAEC, Khabarovsk Nina N. Ogneva Director, Trade House “Intur,” Khabarovsk—accompanied by Anna & Sergei--(clients) Lyudmila A. Bliznyukova General Director, Far Eastern Center of Tourism development, Khabarovsk (client) Dan Berkshire Instructor, Basic Business Course & Chief of Party, CARANA Corporation, Sakhalin CDC Michael A. Levett President Linda E. Nemec Vice President S. Amelia Smith, Esq. Director, Contract Compliance Edward J. Tarpinian Chief of Party, Entrepreneurial Business Services (West), Moscow Alla Noskova Program Manager, Moscow Zhenya Rudometova Office Manager, Moscow Anatoly Plotkin Director, International Consulting Center, Ekaterinburg Sergei Zykov Director of MBA Program, Golden-Platinum Institute (client), Ekaterinburg Michael Dunaev Director for Public Relations, Golden-Platinum Institute (client), Ekaterinburg Larisa Guseva Director of the School, Golden-Platinum Institute (client), Ekaterinburg Vitalij Abroshikov Director General, Stroimarket (client), Ekaterinburg Dmitij Kudryavtsev Deputy Director, International Consulting Center (LRP), Ekaterinburg Sergey Balanev General Manager, Foundation for SME Support (LRP), Saint Petersburg Anastassia Mymrina Business Consultant, Foundation for SME Support (LRP), Saint Petersburg Igor Kuprienko Deputy General Manager, Foundation for SME Support (LRP), Saint Petersburg Igor Rokhlikov Director, REDA (LRP), Saint Petersburg Elena Sivolob Chief of International Planning Center, REDA (LRP), Saint Petersburg Annex D Page 3 of 5 Michail Zimnev Technical Director, Polikom Pro (client), Saint Petersburg Peter A. Gordon Consultant (Volunteer), Polikom Pro (client), Saint Petersburg Maxim Soroka Director, ViTec (client), Saint Petersburg Alexander Cherepitsa Director, Novgorod TecnoPark (LRP), Novgorod Olga Doubonossova Director, ESC (LRP), Rostov-on-Don Anna Leyderman Deputy Director, ESC (LRP), Rostov-on-Don Olga Zagouskina Program Manager, CDC, Rostov-on-Don Leonid Tincker Director, MEDIS (client), Rostov-on-Don IRIS Theodora Turula Project Manager, Maryland John Nielson Chief of Party, Moscow Irina Tikhomirova SME Evaluation Advisor, Moscow Elena Yanboukhtina SME Development Advisor, Moscow Yevgeny Shinkorenko Director, Territory of Law, Khabarovsk Olga Dyachkova Territory of law, Khabarovsk Andriyan Nikolaev HIDE Co., Ltd., Khabarovsk (client) Lyudmila Davydenko President, Vlastra Association of Contractors, Vladivostok Irina Tumanova President, Confederation of Business Women, Vladivostok Alexander Kulikov Director, High Technologies Association, Vladivostok Anatoly Plotkin Director, International Consulting Center, Ekaterinburg Valentina Samsonova Director, Regional Foundation for Women Entrepreneurship Support, Ekaterinburg Dmitij Kudryavtsev Deputy Director, International Consulting Center (LRP), Ekaterinburg Sergey Balanev General Manager, Foundation for SME Support (LRP), Saint Petersburg Anastassia Mymrina Business Consultant, Foundation for SME Support (LRP), Saint Petersburg Igor Kuprienko Deputy General Manager, Foundation for SME Support (LRP), Saint Petersburg Igor Rokhlikov Director, REDA (LRP), Saint Petersburg Elena Sivolob Chief of International Planning Center, REDA (LRP), Saint Petersburg Vladimir Ivan'kin Director, International Consulting Center (LRP), Novosibirsk Inna Chikunova Program Manager, International Consulting Center (LRP), Novosibirsk Vladimir Brylin Director, Ekaterinburg Chamber of Manufacturers (LRP), Ekaterinburg Olga Kolpakova Deputy Director, Ekaterinburg Chamber of Manufacturers (LRP), Ekaterinburg Denis Orlov General Director, SMEDA (LRP), Novgorod Boris Makeev Director, International Consulting Center (LRP), Voronezh Alexander Plotnikov Director, RENACORD (LRP), Voronezh Annex D Page 4 of 5 Morozov Michael O. Shishkin Vice Rector & Financial director of the Morozov Project Veniamin Sh. Kaganov Vice Rector (AMM) & Director, Institute for Entrepreneurship and Investments Peter Shlykhov Director, Morozov Project, Khabarovsk Mostovoy Nikolay Dean, Institute of Industrial Field Technologies, Maintenance, Business and Law, Khabarovsk Technology University Vadim Y. Saburov General Director, Model Forest Industrial Union, Khabarovsk (client) Far Eastern Technical University Valentina Chasovitina Director, Omsk Banking School of CBR (LRP), Omsk Nikolai Vernov Rector, International Institute of Distance Education (LRP), Ekaterinburg Pavel Neverov Director, International Institute of Distance Education/Recruitment Agency (LRP), Ekaterinburg Anatoly Lebedev General Director, UralRelCom (client), Ekaterinburg Evgenij Chasin Director, Omsk Vocational School (LRP), Omsk Evgenij Ryabov Director, Institute for Entrepreneurship & Investments/Center for information projects (LRP), Moscow Eugeny Boiko Rector, Siberian Academy for Public Administration (LRP), Novosibirsk Irina Goulieva Leading Specialist, Siberian Academy for Public Administration (LRP), Novosibirsk Eugeny Rapuntsevitch Deputy Director, Siberian Academy for Public Administration/ZAO 'Siberia-Education-Business' (LRP), Novosibirsk Valentin Galenko HES Director, Higher Economic School (LRP), Saint Petersburg Alexander Kizhin Deputy BTC Director, Higher Economic School (LRP), Saint Petersburg Vladimir Bogdanov Director, Novgorod BTC (LRP), Novgorod Vassiliy Starshov General Director, Borovitchi BTC (LRP), Novgorod Vladimir Belousov Head of the Center, Voronezh Oblast Educational and Consulting Center for Agribusiness (LRP), Voronezh state Agricultural University, Voronezh Andrei Belousov Executive Director, Exhibition Center, Voronezh State Agricultural University (LRP), Voronezh Annex D Page 5 of 5 USAID Carol Peasley Mission Director Mark Ward Deputy Mission Director Rick Scott Director, Program & Project Development Office Brook Isham Deputy Director, Program & Project Development Office Denis V. Korepanov Unit Chief, Evaluation and Performance Monitoring, Program & Project Development Office Nikita Pisnyachevsky Program & Project Development Office Kevin Armstrong Director, Office of Business Development & Investment (BDI) Tom McAndrews Chief, Microenterprise & Management training, Office of Business Development & Investment (BDI) Raymond L. Lewman Chief, Business Development, Office of Business development & Investment (BDI) Stephen Pelliccia Senior Business development Advisor, Office of Business Development & Investment (BDI) Other Andrew Wilson Coordinator, Regional Initiative, Russian Far East Irina Isaeva Deputy Coordinator, Regional Initiative, Russian Far East Andrea E. La Fayette Director, Targeted Grants Program, Khabarovsk Eugeny N. Shulepov Chairman, Khabarovsk Union of Entrepreneurs Roman Romashkin Sakhalin Regional Administration Economic Committee Yulia A. Murzina Sakhalin interpreter & interview team member Genadi A. Sichkarenko Head of Department for SME support, Administration of Primorsky Region Konstantin Rumyantsev Head of Representative Office, EBRD, Novosibirsk Vladislav Alexeev Chief of Entrepreneurship Development and Support Department, Novgorod region Administration Economic Committee, Novgorod Vladimir Polinevich Head, Department for SME Support, Administration of Novosibirsk Region, Novosibirsk Alexander Sidorov Head of International Relations Department, FFSBS, Moscow Gapour Tatiev Department of International Relations, FFSBS, Moscow Andrey Yunak Senior Expert, Department for International Cooperation and External Economic Relations, Ministry for Antimonopoly Policy and Support Entrepreneurship, Moscow 1 Annex E Page 1 of 3 INTERVIEW PROTOCOL: BUSINESS SERVICE INSTITUTIONS (last updated: 4/15/00) Introduction Who we are USAID’s purpose of this exercise Evaluation schedule Confidentiality I. Information about the BSI and its programs 1. Name: 2. Location: 3. Started (year):____ 4. For profit:____ Not-for-profit:____ 5. Currently supported by (which USAID projects): EBS:ACDI/VOCA-Winrock:___ EBS:CDC:___ IRIS: ___ Morozov: ___ ARC:___ Other (identify):_________ 6. Names and positions of person(s) interviewed: (Skip these questions if you already have the business plan from IRIS.) 7. Do you have a business plan?____ Last updated when?_____ II. Services Provided In this section of our discussion, we would like to understand the principal services that you provide to support SMEs. Our questions will deal with to whom those services are provided, how those services are financed, and the contribution (if any) played by USAID-sponsored programs. 8. What is the most important type of service you provide for SMEs? 9. In what principal SME service areas (in the past year): Marketing:____ Business planning:____ Finance:____ Human resource management____ Other:_____(describe) 10. Is this service supported by a USAID program? If so, please explain? 11. To how many SME clients have you provided this service in the past year? 2 Annex E Page 2 of 3 12. What is the approximate size of those SME clients? <10 employees:___ %10-50 employees:___% > 50 employees: ___% 13. What major economic sectors are represented by those clients? Manufacturing: ___% Services:___% Agriculture:___% Retail:___% Government___% 14. Do you measure the impact of this service on the recipients?____ How? 15. Roughly what percent of those clients return for additional services? __ 16. Do you charge fees for this service?_____. How much? _____ 17. Do these fees equal, or exceed, your costs? ___. Please explain. 18. Roughly what % of staff time is devoted to this service?_____ 19. Roughly what % of total institutional revenue does this service generate? _____ What is the next most important service you provide for SMEs? (Repeat questions) 20. In what principal SME service areas (in the past year): Marketing:____ Business planning:____ Finance:____ Human resource management____ Other:_____(describe) 21. Is this service supported by USAID?____ If so, please explain? 22. How many SME clients have received this service in the past year? 23. Can you approximate the size of those SME recipients? <10 employees:_____ % 10-50 employees:____% > 50 employees: _____% 24. Can you approximate the business categories of those recipients? Manufacturing: ____% Services:____% Agriculture:____% Retail:____% Government____% 25. Do you measure the impact of this service on the recipients?__How? 26. Roughly what % of those clients return for additional services? 3 Annex E Page 3 of 3 27. Do you charge fees for this service?_____ How much? 28. Do these fees equal, or exceed, your costs? ___. Please explain. 29. Roughly what percentage of staff time is devoted to this service?_____ 30. Roughly what percentage of your organization’s total revenues are generated by this service?____ 31. Have we missed important services you provide to SMEs and/or others (e.g., SME advocacy, TA to government, other)? Questions that pertain to your program as a whole (not just one program): 32. Has your program led others (e.g., those who you trained) to offer other services to the SME community? 33. Changes in SME demand: How has demand for your services changed over time? If so, how has your program responded? 34. For those BSIs participating in networks: Has your participation in the network led to collaboration with other (BSI) centers? If so, how? 35. Have the USAID-sponsored programs had other impacts that we have not discussed? III. The Future 36. What are the one or two greatest barriers to the growth of SMEs in Russia today? 37. What are the one or two things that firms most need (e.g., training in business methods, market information, bulk purchasing or raw materials, incubators, technology or manufacturing assistance, tender advice) that the SME sector most needs? 38. Based on your experience, if you had 50% more money, what additional things would you do? 39. If you had 50% less resources than you now have, what would you cut? 40. What are the key factors determining your institution’s future sustainability and growth? Where is your future revenue going to come from? 41. What mistakes of the past need to be avoided in the future? 42. Success story: What techniques have been particularly successful? Can you mention an example?