BULGARIAN PARTICIPANT TRAINING PROGRAM LEGACY ASSESSMENT Prepared for: USAID/Bulgaria Bureau for Economic Growth, Agriculture and Trade (EGAT) Office of Education U.S. Agency for International Development November 30, 2004 Draft 11 Page 2 ACRONYMS AUBG American University of Bulgaria BARDA Bulgaria Association of Regional Development Agencies BIA Bulgarian Industrial Association BIIA Bulgarian Institute of Internal Auditors BJA Bulgarian Judges Association CSD Center for the Study of Democracy CI Counterpart International ECESP Eastern & Central European Scholarship Program EU European Union FLGR Foundation for Local Government Reform HAPI Hungarian-American Partnership Initiative IBI International Banking Institute ICT Information and Communication Technologies IPAEI Institute of Public Administration and European Integration ISPA Instrument for Structural Policies for Pre-Accession (EU) LGI Local Government Initiative MSI Management Systems International NACC National Association of Court Clerks NAMRB National Association of Municipalities in the Republic of Bulgaria NGO Non-Governmental Organization NIJ National Institute of Justice OSI Open Society Institute PAO Public Affairs Office (U.S. Embassy) PHARE Poland Hungary Assistance for Reconstruction of the Economy (EU) PI Performance Improvement PTP Participant Training Program SAPARD Special Accession Programme for Agriculture and Rural Development (EU) SEED South East European Development (Act) SJC Supreme Judicial Council SO Strategic Objective TA Technical Assistance TCT Third Country Training UNDP United Nations Development Programme USAID United States Agency for International Development USG United States Government UST United States Training VEG Volunteers for Economic Growth WB World Bank WL World Learning, Inc. Draft 11 Page 3 TABLE OF CONTENTS I. Executive Summary II. Background III. Purpose of Assessment IV. Impact of the Participant Training Program V. Objectives of a USAID Training Legacy Institution VI. Criteria for a Sustainable Legacy Institution VII. Options for Legacy Mechanisms VIII. Stakeholders IX. Conclusions X. Recommendations XI. Summary and Next Steps APPENDICES APPENDIX A Assessment Methodology APPENDIX B Impact of the Bulgaria Participant Training Program APPENDIX C Organizations Contacted APPENDIX D Statistics on Bulgarians Trained APPENDIX E Scope of Work Draft 11 Page 4 I. Executive Summary The Participant Training Program (PTP) in Bulgaria has been a vital component of the USAID country program since PTP was initiated eleven years ago. This assessment reviewed the PTP impact and sets forth potential options for maintaining the legacy of a very successful training program after Bulgaria graduates from USAID assistance in 2007. With nearly unanimous feedback from former participants, institutions served and USAID technical assistance contractors and grantees, the PTP appears to have been successful in most respects. Most importantly, the PTP has had a measurable beneficial impact on Bulgaria’s development toward a market-oriented, democratic society as it moves toward membership in the European Union (EU). During the course of its presence in Bulgaria, USAID has assisted in the establishment and strengthening of several institutions with a capacity to train and administer training programs. These organizations will remain and provide services that foster Bulgaria’s development transition. They represent an outstanding legacy of the USAID program. It is recommended that the mission undertake initiatives to improve the performance of a select number of these institutions so as to further enhance their capability and sustainability after the USAID program ends. Four potential legacy approaches were considered: (1) alumni association; (2) NGO: foundation or association; (3) for-profit organization; and (4) autonomous university affiliate. A USAID alumni association appears to lack sustainability as a service organization, in part because its broad and diverse pool of former participants tends to counter focused objectives. However, the Georgetown scholarship program (ECESP) is attempting to establish a sector-based alumni association, which warrants monitoring as a possible model. Also, the modest cost of updating the foreign trained participant database and supplying this data to the Embassy for public diplomacy purposes is worthwhile. Affiliation of the training contractor capacity with an indigenous training NGO or university is the best course of action for a sustainable legacy organization that will continue to serve multisectoral training abroad. It could provide value added to any training organization by bringing quality in international training planning and methodology, networking, processing and support to the affiliate, and possibly establish working relationships with training organizations other than its base partner. Such an organization could emulate the successes of the Hungarian-American Partnership Initiative (HAPI), but would need to address shortcomings noted in the HAPI structure. A for-profit legacy organization has some limitations in the Bulgarian marketplace and development sphere, making it a less desirable approach than an NGO or university setting. There is significant complementarity between university training programs and the skills potentially offered by the USAID training contractor. On the other hand, any relationship between the two needs to ensure autonomy from the university system. Draft 11 Page 5 II. Background USAID began its program of assistance to Bulgaria in 1991. As the new country programs in Bulgaria and the region evolved, USAID/Washington engaged a training contractor, World Learning (WL), to support the increasing training needs. WL opened its Participant Training Program (PTP) in Sofia in 1993 to support country program training needs across all Strategic Objectives. PTP was designed to support Bulgarian institutions, professionals and leaders by providing access to new ideas, knowledge and skills that help meet development objectives in the USAID portfolio. In the eleven years of PTP presence in Bulgaria, approximately 4,430 persons have been trained in the United States, in third countries and in Bulgaria. Approximately one￾quarter received training in the U.S. and one-half in third countries. The participants are nearly equally divided by gender, with slightly more women than men trained under the program. In recent years a higher proportion has been trained in European countries, reflecting Bulgaria’s needs to prepare for accession into the European Union, which is planned for 2007. Other details and statistics concerning the training location and subject matter can be found in Appendix D. At one stage, Bulgaria served as the Third Country Training (TCT) office for the region. The PTP has accumulated a wealth of knowledge and contacts in the region as it has matured, and is a model training support program that can and should be emulated elsewhere. This assessment focuses on the potential for preserving Bulgaria’s training experience after the country graduates from USAID development assistance in 2007. III. Purpose of Assessment The purpose of this assessment is twofold: (1) to analyze the overall impact of the Participant Training Program (PTP) in Bulgaria for the period 1993–2003; and (2) to assess the prospects of a legacy training organization that can provide ongoing training services in the post USAID-presence years. The examination of impact will determine to what extent the program goals and objectives have been met, identifying program successes and lessons learned. The consideration of a legacy training organization entails an analysis of anticipated future demand for training and training support services of the nature now being provided by the USAID training contractor, opportunities for improving collaboration with other donors, and recommendations for improving the sustainability of indigenous training organizations prior to USAID’s planned departure in FY 2007. The rationale for this assessment hypothesizes that the large and diverse pool of Bulgarian participants trained under USAID programs can serve as the basis for a USAID legacy in Bulgaria. It also seeks to identify a possible organizational arrangement that might maintain the USAID training experience. Draft 11 Page 6 IV. Impact of the Participant Training Program Training has been an integral part of the USAID development program in Bulgaria for over eleven years. Inherently cross-cutting, the training program has contributed significantly to USAID’s initiatives in every Strategic Objective (SO). Since training is a vital component of nearly every mission activity, its results have a direct bearing on achievement of planned results. Overall, PTP training has contributed substantially toward Bulgaria’s economic, social and democratic transition. According to TraiNet, the Agency’s training data repository, 4,430 Bulgarian participants have been trained since 1993, in such diverse fields of study as agriculture and natural resources, democracy and governance, rule of law, economic growth, management skills, and civil society (See Annex D). However, measuring the number of participants trained does not begin to convey the true impact training has had in Bulgaria. Through its training delivery mechanisms, including technical assistance (TA) contracts and the Participant Training Program implemented through World Learning, USAID has transferred new skills, knowledge and attitudes to individual trainees in their various professional fields; but perhaps of greater importance for those who have had training in the U.S. is the exposure these participants have had with U.S. business and government culture, which is characterized by innovative problem-solving, analytical acumen, practicality, openness, interactivity, exchange and transparency. These distinctly American qualities are introduced by participants to others through their personal and professional connections to other Bulgarians. Thus, the “American experience” becomes a shared experience from which other local people may benefit. A direct product of the training, as identified by former participants, is the benefits from relationships created among individuals working in different agencies and at different levels brought together through training. Several instances were cited of participants who had developed relationships and networks with colleagues that, prior to the training event, did not exist. Through these newly established bonds, it became possible to modify legislation and regulations or jointly initiate development activities. Training has fostered national unity by joining participants in common goals and objectives leading to short-term and long-term results and accomplishments. In addition, a large number of former participants have opened and maintained a dialogue with European and U.S. counterpart organizations, providing ongoing opportunities for exchange of ideas and approaches to both regional and domestic development issues. Training has produced many success stories on local and national levels, and has provided an opportunity for Bulgaria to begin offering its model programs as roadmaps for success to other Eastern European and Eurasian countries, as they follow Bulgaria in their accession to the European Union. A more detailed discussion of the PTP impact in Bulgaria can be found in Appendix B. V. Objectives of a USAID Training Legacy Institution Draft 11 Page 7 The primary objectives of a PTP legacy institution are to continue to promote and disseminate the training methodologies and processes introduced through USAID development programs and to establish and maintain networks of institutions and individuals who have received U.S. sponsored training. Exposure from U.S. trained individuals now working in Bulgaria’s government, business, civil society and the community will help convey a knowledge of American concepts and management systems, which should lead to increased effectiveness and efficiency in national development. Moreover, it is intended that a USAID legacy will continue to generate a favorable image of the United States, its policies, principles and culture with the Bulgarian people for years to come. The prospects of a viable legacy institution depend upon operating on sound business objectives and underlying sustainable principles. A. Business Service Objectives Optimally, a USAID training legacy institution will be a sustainable indigenous organization whose operating service objectives would include the following: 1. The coordination and facilitation of domestic and international training events, offering both administrative and logistical training support services for in-country training, third-country training, and U.S. training. These services would be available to Bulgarian organizations, governmental and non-governmental, to send Bulgarians abroad, or to promote model Bulgarian domestic programs on a multisectoral basis; 2. The capacity to identify and coordinate with the most effective training providers worldwide in order to meet the specific training needs of Bulgaria, including requirements for EU accession and conformity with membership stipulations; 3. The capacity to provide performance improvement consulting services to both develop and refine training requests and analyze these requests in the context of overall performance needs of organizations; 4. The development and maintenance of a database for training consulting services, identification of appropriate training organizations and individual consultants to satisfy training and associated requests, and to provide consulting services to the extent the expertise is available from the legacy institution staff; 5. The capacity to plan and design results-oriented training programs to meet the needs of the requesting organization; 6. The capacity to plan, conduct and facilitate conferences as requested; and Draft 11 Page 8 7. The capacity to host potential USAID training alumni associations. Compensation would be received for all of the above-listed business service objectives, aimed at sound operational and management principles and at fostering sustainability. B. Sustainability Considerations A cardinal aspect of sustainability for the legacy institution is to effectively market its services and expertise to potential training clients, both domestic and international. Accordingly, such an organization must be able to plan, design and manage training programs that may be both complex and geographically dispersed. In an increasingly competitive and global marketplace, the long-term sustainability of a legacy institution also will depend upon its international networking capabilities. With accession to the EU placing increasing demands on Bulgaria and its regional neighbors, excellent prospects exist for sharing the Bulgarian transition experience with those countries facing comparable challenges in the near future. Consequently, the legacy institution’s ability to identify and research regional needs and to develop attractive proposals that meet those needs will be critical in maintaining a robust training business strategy. The recognized expertise of the organization’s in-house staff will be paramount in developing and maintaining a reputation of excellence in training and human performance improvement, as will a targeted service focus and excellent management practices. Financial sustainability of a legacy institution necessitates an adequate stream of income, based on demand for services. Sole dependency on uneven and unreliable donor support for specific events does not provide a suitable foundation for long-term stability. Compensation for services and a business approach are essential elements of a sustainable institution. One consideration for enhancing sustainability might be for USAID to establish a trust or endowment fund to provide a base for long-term financial support to the legacy organization. A clear strategy and a sound business plan would be required for the legacy institution. USAID/Bulgaria has positive experience in participating in a trust managed as an endowment to help fund the safeguard of Bulgaria’s protected areas. Establishment of a fund as an endowment would be attractive to partnering with an affiliate organization, since it helps assure self-sufficiency as an autonomous unit in the organization. The preferable approach would be to link the legacy institution to an existing organization that served former USAID participants. Currently, USAID supplies the demand for needed training in targeted areas, and provides resources for training through its activity grants and contracts, including the PTP contract. In order for the training capacity developed through USAID programs to become sustainable, there will have to be a demand from within the country to identify, develop and conduct ongoing training. Draft 11 Page 9 Once Bulgaria has graduated from USAID support, indigenous organizations and government agencies will have to recognize the value of training to their organizations and make the investments needed to engage appropriate training advisors. Post-USAID support for training in development programs will be primarily through the European Union pre-accession and post-accession funds. This European assistance will be provided directly to the government, which will be responsible for managing the funds and identifying implementing organizations. The government, ideally, will need to become more receptive to the idea of conducting organizational assessments and training needs assessments of local institutions, which in turn should help sustain the training capacity that has been developed through USAID (and other programs). Another area of future training needs will be support for non-Bulgarians, who come to Bulgaria to receive guidance from the Bulgarian development experience. Several countries in Europe and Eurasia look to Bulgaria as a learning model in the transition process. This training would be provided by USAID missions and other donors in the region. In addition, some opportunities may exist for private sector training of non￾Bulgarians on a fee-for-service basis. VI. Criteria for a Sustainable Legacy Institution In analyzing the prospects for a legacy institution, a number of critical factors must be carefully considered, regarding both the institution itself and Bulgaria’s development environment. A. Financial sustainability: The institution must be able to generate sufficient revenue to fund ongoing operations. It is generally accepted that the large increases in EU funding, especially in 2005 and 2006 will supply adequate resources for carrying out the national development plan, which involves much training. B. Sufficient demand for services: Concern is being expressed about the serious shortfall in skilled people across most sectors, which has limited the absorptive capacity of Bulgaria to meet its transition requirements. Consequently, there should be continuing strong demand and opportunities for an appropriate training organization. Marketing of services is an essential part of creating demand. C. Targeted focus: Successful training providers target their efforts toward discrete subject areas. However, a training service provider would need to maintain a range and depth of contact networks across sectors and internationally to fulfill training needs as they occur. At the same time, services need to be focused on training and training-related activities. D. Management capacity: An indispensable criterion for a legacy institution will be its responsiveness to clients, service orientation and quality of work. Well￾managed experience and quality output will, to a great extent, attract and sustain business relationships. Draft 11 Page 10 Other criteria to be considered in selection of a legacy institution include knowledge of Bulgarian and regional donors, public and private training programs; collaborative experience with other donor training programs and activities implemented in Bulgaria; experience in maintaining past trainee and partner networks; and visibility as a local training service provider, including on the Internet. As part of this assessment, a review was made of the experience of the Hungarian￾American Partnership Initiative (HAPI), a USAID legacy organization. Created in 2000 by a USAID competed cooperative agreement, HAPI is implemented by a Hungarian foundation (NGO) with the goal of using the pool of human and organizational talent generated by the USAID program to assist other countries in the region through their economic, political and social transition process. HAPI services focus on training, study tours and a scholarship fund that supports regional training. HAPI receives strategic direction from an Advisory Council of Hungarian government and non-government representatives. The USAID grant provided for two years of start-up expenses and scholarships. While it is too early to judge HAPI’s long-term sustainability, it appears that more effort at marketing its services in the region is necessary. Observers have noted that HAPI has not given sufficient emphasis to business management and income generation activities. Also, by not providing in-country training services, it is possible that potential business is forgone; however, this may be seen as competing against HAPI members that are local training providers. VII. Options for Legacy Mechanisms The assessment team considered a number of possible institutional options for retaining the outstanding services made available by the USAID training services provider following the closure of the country program. Categories of potential legacy institutions include an: A. alumni association; B. NGO: foundation or association; C. for-profit organization; and D. autonomous university affiliate. To ascertain viability, each must be measured against the criteria identified in the Section VI above. A. Alumni Association The establishment of an alumni association for all Bulgarians who have received training through USAID projects conducted by technical assistance (TA) implementers, the PTP program implemented by World Learning, the Georgetown University Eastern and Central European Scholarship Program (ECESP), and U.S. Embassy training programs could provide a forum for continuing professional relationships. Such an association could also create a unified embodiment of the “American Experience” and serve to reinforce American values and approaches to training and organizational development that have been transferred to Bulgarians through the USAID development assistance program as well as other USG training initiatives. In the event the U.S. Embassy decides to support an alumni association in the post-USAID era, it likely would attract the interest of a nucleus of former participants. Through an alumni association, former participants may continue to network with other Bulgarian professionals working in similar or related Draft 11 Page 11 fields, and take advantage of a national cadre of experts to advance common causes, acting together as agents of change to meet the challenges facing Bulgaria in its EU accession aspirations and nation-building efforts. While offering the potential for the opportunities described immediately above, an alumni association would require ongoing promotion and support by an indigenous organization to be sustainable. To incur the cost and exert the administrative effort to maintain an alumni organization, the hosting organization would need to derive some value-added benefit from such an initiative. The goals and objectives of maintaining an alumni association would require a direct tie to the organizational goals of the host for it to be a meaningful and, therefore, sustainable initiative. The assessment team is unaware of a successful, self-supporting alumni association outside of the university community. Interviewees suggested, and the team agrees, that an alumni association cannot be sustainable unless it is energized and motivated by a core group of alumnae. Further, it requires an unambiguous set of objectives that sets a course both for direction and ongoing revenue to maintain itself. This may be particularly difficult in the case of former participants of U.S. supported training programs because of their wide subject area diversity and lack of sectoral focus. Indications are that the single underlying commonality of being trained under U.S. programs is insufficient to sustain an alumni association without outside institutional affiliation. One of the primary actions of an alumni association of USG-trained participants would be to establish and maintain a personnel database. This entails staff and associated costs. For example, the Embassy has one person working nearly full time on its training and exchange programs database. Consequently, even serving the function as a forum as suggested above necessitates an income base. Beyond this, it is obvious that long-term sustainability will be affected by normal attrition in the absence of significant continuing U.S. training activities. The Embassy PAO office indicated interest in the possibility of establishing a single database that includes USAID-trained participants. While having value for public diplomacy, and possibly commercial, purposes, it is not clear that such a database would serve national development needs, as Bulgaria increasingly moves toward EU integration. Nevertheless, the proposition of Embassy maintenance of the database overcomes the cost issue and helps ensure continuity. A precautionary note is signaled by the experience in the prestigious Fulbright Commission exchange program. Some 600 Bulgarians have participated in the program since its inception in Bulgaria in 1993. Fulbright alumni have established and registered an alumni association. It has about 100 members, 20 to 30 of whom are active. The Commission does not have a comprehensive database of its former participants; rather it maintains contact with the association membership through quarterly newsletters and its web site, as well as the core active group. This underscores the necessity for a member￾level demand driven approach to an alumni association. Draft 11 Page 12 The assessment team believes that there is an opportunity for the USAID training contractor to create a database of foreign trained former USAID participants over the coming year and to call them together to determine if there is genuine interest in establishing an alumni association. The model of sector-level units, being considered by the USAID-sponsored Eastern & Central European Scholarship Program (ECESP) could be examined, while recognizing that ECESP is attempting to set up a regional alumni association with country level member associations. The Bulgarian association has been registered and created three sector-level groups. B. NGO: Foundation or Association Since Bulgaria encompasses foundations and associations under a single law and they have NGO status, they are discussed as one. The differences between foundations and associations are mainly in their founding and governance structure, e.g. associations are established by three or more legal persons, while in the case of foundations there is a single founder. According to the legislation, NGO goals are focused on non-economic activities and societal needs including civil society, health, education, culture, sports, human rights, environmental protection, etc. Irrespective of their founding principles, most NGOs compete for the limited funding opportunities that come from donor programs in Bulgaria. All NGOs face the challenges associated with Bulgaria’s forthcoming accession to the EU and the related withdrawal of bilateral donors from the country. The situation appears more difficult for NGOs active in the area of civil society, while others closer to business and government needs or involved in specific economic services find greater funding prospects. At the same time, Bulgarian legislation does not provide a framework for donor funding of NGOs, and does not provide incentives for business donations for societal needs. Nevertheless, there is an increasing trend for business donations to NGOs, especially in smaller towns. A lack of transparency also is considered a serious obstacle for access to funding. NGOs often find it easier to apply for funding abroad, rather than from Bulgarian donors. With regard to consulting services offered by NGOs, the business community generally prefers to hire an individual consultant instead of engage an NGO for its expertise. Bulgarian NGOs have successfully established public-private partnerships with ministries and public bodies. An example is the Bulgarian Industrial Association (BIA), an association of the business community. BIA conducts vocational and educational training throughout the country for the Ministry of Labor and Social Policy and its local Employment Agencies with the aim of reintegrating unemployed into the workforce. This model could be pursued by USAID-assisted training organizations, especially in the area of local government. Through its development program, USAID has helped create and strengthen numerous indigenous NGO foundations and associations. They, indeed, represent legacy institutions of the USAID country program, and most will likely endure long after USAID presence is gone. In many cases these organizations have made great strides toward sustainability and have developed promising marketing strategies for seeking Draft 11 Page 13 funding from other donors and developing clientele in the Bulgarian economy, including government and non-government organizations, some of which are for-profit. These organizations exist in several critical sectors including government administration, banking, justice, local governance, water and sanitation, and broadcast and media. A list of some of these organizations is in Annex B. Some NGOs active in the area of local government training have found a particular niche that complements the training activities of the state-owned Institute of Public Administration and European Integration (IPAEI). NGOs have targeted specific needs of the municipalities for practical and legislative skills, increased citizen participation and local economic development. The networking of such indigenous NGOs with several municipalities is helping to build the necessary human capacity at regional and local levels. More mature structures like the Foundation for Local Government Reform (FLGR) have invested in training-of-trainers programs, resulting in a large pool of trainers. FLGR is addressing sustainability through multiple funding sources, actively submitting proposals on development initiatives and establishing close relationships with local governments. As EU integration increasingly puts huge demands on stakeholders at the local level, training such as that provided by FLGR on future needs in areas like planning, management, budgeting, accounting and the legislative process is necessary for the changing role of local government. Another group of USAID partners is dedicated to helping establish an efficient, qualified and fair judiciary that supports democratic processes and combats corruption. The Judicial Development project established a National Institute of Justice (NIJ), which has responsibility for training clerks, judges and prosecutors throughout the country. Although still a young institute, NIJ has developed an effective training capacity and soon will be in a position to provide required regular training of courts members. Study tours and in-country training are key tools used by NIJ to build the necessary capacity. There is space, however, for regional partnerships between universities and courts to further develop the capacity of judicial human resources. In several NGOs, the marketing strategy comprises regular publication and dissemination of newsletters or information bulletins to partners, members, former and potential clients. Other marketing and promotion channels are the broadcast media, interviews or paid advertisements in the newspapers and exhibits at public meetings and business exhibitions. Internet also is seen as an increasingly valuable tool for marketing training activities and services. Most of the USAID-assisted NGO foundations and associations are sector-specific legacy organizations and individually would not be able to establish a fully representative embodiment of the total USAID legacy. Several provide a full package of training services—curriculum and training module development, training of trainers, training consultancies and logistics and administrative support (see Appendix B). However, none are as proficient or thorough as World Learning in international training planning, networking, processing and support throughout the training event. Draft 11 Page 14 Should the skill base produced by the training contractor decide to remain intact after 2007, one option for it to consider is to affiliate with a training-oriented NGO that is well established and provides quality services. This new affiliate could serve as an umbrella organization to offer its training services, methodologies, partner network and other assets generated through training contractor experience. Possibly this unit could establish formal working relationships with NGOs and others that serve sectors other than its base partner. Obviously, competition will exist among all existing NGOs for specific training and for funding opportunities. Sustainability will depend upon the organizational arrangement and the ability of its staff, without USAID financial support, to develop its portfolio and offer unique, highly demanded training services. A further option could be for the residual WL contract staff to establish its own stand alone NGO for training support services. Although the core of expertise would exist, the idea of creating a new organization and all that entails would be a higher risk venture than partnering with an existing training support NGO, and offer fewer advantages. It is the opinion of the assessment team that the stand alone NGO option should not be pursued, or if so it should be accorded a low priority among choices. A potentially more viable variation of this legacy option would be for World Learning to consider investing in a long-term, independent presence in Bulgaria by building on its experience with a comprehensive, competitive array of training support services for Bulgaria and the region. Services could include coordinating a consortium of sector￾based training entities, similar to the HAPI approach. The assessment team did not discuss this option with World Learning, so is not aware of the extent of their interest in this possibility. C. For-Profit Organization Numerous for-profit indigenous service organizations exist in the Bulgarian economy. These organizations would be able to provide most of the training support and consulting services identified as requisites for a legacy organization. Further, by their for-profit nature, these organizations likely have the marketing expertise to develop an ongoing enterprise centered on these services. The business community in general probably would not make an important source of clientele for a legacy institution. Larger enterprises including many former state-owned enterprises have internal business and management training capacity, many of them with links to successful, well-operated foreign companies. It is unlikely many would seek training from a legacy training institution. Smaller businesses probably need access to more training, but this demand is being supplied by existing for-profit consulting and training firms. It should be noted that small and medium enterprises trying to survive in the marketplace probably are not inclined to invest in much training of their staff. The BIA mentioned above as an association has a membership comprised of for-profit businesses. Its strong training capacity in ICT and business-related topics puts it in a position to provide this type of training to the business community. Draft 11 Page 15 One consideration for a legacy organization would be to establish an affiliate relationship with the American Chamber of Commerce and create its own profit center. The advantage to such an arrangement would be closer access to a business clientele base and identification more directly with an American image. On the other hand, as Bulgaria moves more closely to the European stage, this affiliation could be somewhat limiting. Also, EU funds being managed by the government may be less easily available to a for￾profit training institution, since NGOs might be more favored. In principle, the assessment team found little advantage for a USAID legacy institution to be formed as a for-profit business. D. Autonomous University Affiliate There are two types of universities in Bulgaria, private and state-owned. The state￾owned are established by a legislative act and receive an annual state subsidy, which ensures the necessary financial resources for the educational process and respective scientific activities. For private universities a chief source of income is student tuition, whereas tuition is only symbolic at state-owned universities. In both types of universities, international programs and projects constitute an important source of income. Also, both obtain additional income from scientific projects, consultancy work, sponsorship and training activities. An important aspect of the Bulgarian university system is academic autonomy, which involves academic freedom, academic self-governance and independence. This provides a framework for entrepreneurial activities, especially at state-owned universities. Despite the conservative and complex administrative structures at universities, specially established research departments in state-owned universities have a certain level of flexibility in undertaking supplementary activities, such as additional training courses, educational or consultancy services and contracts with the business community and other legal entities. However, development of these supplementary activities in state-owned universities depends upon the entrepreneurial, innovative and managerial skills of their staff. In comparison, private universities are characterized by greater initiatives and innovative approaches to educational and training processes. Some even go beyond national borders and attract students from neighboring countries to develop as important regional educational institution in the Western Balkans. The financial, administrative and policy autonomy of a legacy organization associated with a university would need to be carefully examined. Whereas private universities meet educational demand and ensure high quality courses to position themselves on the training market, in state-owned universities the initiative for developing new courses or training courses in response to development and societal needs is an exception. Opinion surveys of employers do not paint a positive picture of universities. They highlight the emerging necessity to develop functional skills and respond to actual labor market needs by the educational institutions. Draft 11 Page 16 Nevertheless, Bulgaria has much to offer in the academic universe. A number of universities have departments with keen insights into the human capital needs of Bulgaria as it moves toward global competitiveness following EU accession. Recognizing the inter-dependency of the academic, business, and government communities, these departments are actively anticipating the future academic needs of the country and developing course modules in areas of national interest. In many instances these courses are expanded into Bachelor, Master, and/or PhD programs. For example, the dynamic development of ICT requires constant updating of courses to follow the pace of change. Partnerships with businesses and other ICT user stakeholders regarding demand for skills and competencies increase the employability of graduates. Sofia University and some other universities have established ICT centers of excellence recognized by business, government and others. They also offer internationally recognized IT certification courses. Universities recognize the opportunities arising from the EU accession process. They actively participate in EU programs like Socrates, Leonardo da Vinci and the Sixth Framework for Research and Technology. Yet large categories of stakeholders affected by EU accession are not reached and inadequately trained human capital is expected to result in a shortfall in absorptive capacity to meet transition needs. Only a few universities, e.g. American University in Bulgaria (AUBG), have established a prominent capacity to support EU integration through coursework, certification courses and consulting services focused on EU funding, project development and management. Both state-owned and private universities have highly-skilled professionals with experience and knowledge for training course design and conducting quality training. There is some difference in cost for services, however. Courses at state-owned universities are lower cost, but private universities use donor support to reduce such costs to trainees. The daily rate of trainers from state-owned universities is less, as a rule, so their services are often preferred for training courses. Although Bulgarian universities do not offer study tours in the format of WL, they have much experience in receiving and sending guest lecturers to and from other countries, and organizing student exchange programs in the framework of the Erasmus program, other EU programs and bilateral agreements. The mobility of students and professors is an important tool for knowledge transfer within Europe, and for building skills. WL legacy skills would appear to be highly complementary with university competencies. An important requisite of a legacy institution is the availability of linkages with other institutions in the country and abroad. Networking with international partners and national organizations is quite well developed in many universities, especially those working on international research or educational programs. Alumni networks are relatively unknown at Bulgarian universities. An exception is the AUBG. In order to fully meet the requisite needs of a legacy institution, an independent department or affiliate would have to be created within the university structure to ensure the integrity of the legacy objectives and provide a clear separation to protect the funding Draft 11 Page 17 and operation of this initiative from other wider university demands. If such an arrangement can be assured, a legacy training institution in affiliation with a university could be a mutually acceptable partnership. Affiliation with a center of excellence at a university might offer substantial advantages in regard to flexibility and autonomy. Two such centers for consideration are the: · Center for European Programmes and Elieff Center at the American University in Bulgaria; and · Center for Information Society Technologies at the Sofia University. VIII. Stakeholders Given that training is a vital element of nearly all USAID-assisted development programs, the spectrum of stakeholders extends across a large portion of Bulgarian society. Not only the thousands of individuals trained, but also the numerous institutions that directly benefited from training during the thirteen years of USAID assistance represent major stakeholders affected by the training experience. This includes people and institutions in government, business and civil society involved in many sectors. Former USAID participants form the foundation of a training legacy of the U.S. development assistance program in Bulgaria. Some 2,800 Bulgarians that have been trained in various U.S. Embassy public diplomacy, democracy and media programs complement this core group. With regard to a post-USAID legacy, the key stakeholders will be local institutions established and supported through the development program and their clients and users. A test of a legacy institution will be its sustainability in terms of financial viability and demand for its services. Recipients of services will become downstream stakeholders to the training efforts of USAID. Similarly, partner organizations, government and the donor community that use local institutions, such as those identified in Annex B, for meeting development objectives become stakeholders by benefiting in the results of USAID trained and supported organizations. In summary, there is potential for an enormous number of beneficiaries and stakeholders to gain from the well-placed training and associated institutional support that has originated from USAID. IX. Conclusions A. Impact 1. The training program has had a significant impact on USAID/Bulgaria’s development program. USAID’s Participant Training Program has had an important role in Bulgaria’s transition and nation-building, both in short-term actions and longer-term incremental development changes. Draft 11 Page 18 2. Training has been a contributing factor for meeting USAID program objectives, not an end in and of itself. 3. USAID has established strong legacy institutions that provide training services, which will remain after USAID departs. 4. The training contractor, World Learning, has demonstrated unique qualities, especially for international training. 5. The impact of USAID’s legacy in training in Bulgaria is multi-faceted. It includes: · A trained cadre of experts in business, civil society and all branches of government including local government; · A vast network of former USAID training participants contributing to the development of the country; · Institutional development of training organizations and trainers in various sectors; · Introduction and effective use of adult training methodologies; and · International professional networks permitting exchange and transfer of ideas and knowledge. 6. While the impact of training and these legacy outcomes are already established and should be further strengthened during the remaining years of USAID presence, their identity with USAID likely will become increasingly blurred over time. B. Alumni Association 1. Alumni associations do not have an established model of success outside of the university environment. Sustainability would be a challenge to such an organization. C. Training Support Institution Options 1. NGOs (foundations and associations) have demonstrated an ability to carry out training functions effectively on a sectoral basis. 2. NGOs and universities have a capacity for meeting a wide variety of in-country training needs. 3. The best alternative for retaining USAID training contractor expertise and its special capabilities after the USAID country program is completed appears to be an operating partnership with a local training services NGO. 4. A totally new stand alone institution to fulfill the current USAID training contractor function does not appear to be a preferred or viable option, in comparison to partnering with another organization. 5. A second choice as an alternative could be affiliation with a university as a partner. USAID training contractor expertise could be beneficial to both parties, but needs to be approached with a careful analysis of the university relationships to ensure that the training function is not stifled. 6. Opportunities for a partnership with a for-profit organization appear to be limited and less preferable than with an NGO or university affiliation. Draft 11 Page 19 X. Recommendations A. Alumni Association 1. The USAID training contractor should be tasked with updating and consolidating data from all U.S. training participants as a legacy of the USAID training program. Data may be provided to the PAO to further U.S. diplomatic goals, as appropriate. 2. Utilizing the updated alumni database, a meeting of alumni or series of meetings on a sectoral basis should be held to determine the sincere interest of a member driven alumni association. If commitment is genuine with a clear understanding of purpose, activities, obligations and effort, then former participants could consider formation of an alumni association. (As stated earlier, the assessment team found no example of a financially viable, sustainable alumni association outside of a university setting.) 3. The Georgetown University ECESP is currently in the early stages of establishing an alumni association of participants in its training program, which warrants USAID observation as to the association’s approach and viability. B. Training Support Institution 1. USAID should select 3 to 5 institutions that it has created and/or strengthened that provide sector-specific training services, conduct an organizational assessment, and based on the results, consider providing assistance to close any critical performance gaps that may inhibit long-term sustainability as an institution. (This would require only a modest investment and further enhance the organization’s chances for sustainability.) 2. Should the WL-generated staff skills, possibly with the support of WL, desire to retain its nucleus of institutional expertise after 2007, it should initiate discussions on potential partnership with selective NGOs and universities in order to weigh the possible interest, benefits and issues of such relationships. 3. USAID should stimulate the formation of sector-specific training networks through a set-aside of small grant funding accessible through the PTP contract or its follow-on contract. 4. USAID should monitor the HAPI model to assess its strengths and shortcomings as a possible approach to a legacy training institution. If there is interest in pursuing a modified HAPI legacy organization in Bulgaria, USAID would need to commit financial resources, either as a grant or an endowment, for startup support to help ensure that it is viable. Since the mission stated it is not interested in establishing a new institution, the best approach is to seek a “home” institution, along the lines of HAPI. XI. Summary and Next Steps Draft 11 Page 20 The assessment team considered a number of options for a training legacy institution, which are summarized in Section IX above. However, it is the view of the team that the best approach for an ongoing training support legacy is that the expertise, knowledge and network of the USAID training contractor become affiliated with an existing Bulgarian training NGO or, if appropriate, a university. Not only would this foster synergy between them, it would increase the prospects of institutional viability. The unique strengths of the USAID training contractor lie in support for training abroad, whereas in-country training has become increasingly mature and professional by sector level providers, thanks to the USAID and other training programs carried out through respective country assistance portfolios. As a first step in forming a broad-based training legacy organization, the current professional USAID training contractor staff needs to determine their interest in affiliating with an NGO or university in the post-USAID era. In the absence of such interest, there does not appear to be significant motivation among sector-oriented Bulgarian training providers to initiate a multisectoral training support function. This step is predicated on the basis of the training contractor’s particularly effective methodology and networking capabilities. If a decision is made to proceed, possible institutional affiliations need to be examined (see X.B.2). USAID needs to decide whether it chooses to see the accumulated training contractor expertise maintained, or to simply accept the various training entities it has created or strengthened during its program presence as its legacy institutions. Presuming USAID wants to support the former, the next step is to make a resource commitment (see X.B.4). Finally, with respect to a training support institution, if USAID decides to assist continuation of the training contractor function as part of its legacy, a design team should be engaged to propose the process for implementation of this legacy mechanism. Maintaining the USAID training contractor function does not address the broader question of preserving a visible organization of former USG trained participants. While the assessment team is not confident this can be done in a sustainable manner, steps suggested in the recommendations section should be followed to determine if there is a realistic basis for an alumni association. These steps can be carried out with minimal cost. Draft 11 Page 21 APPENDIX A ASSESSMENT METHODOLOGY The assessment team applied a series of approaches to obtain essential insights and information to conduct the overall review and analysis of USAID/Bulgaria’s training experience and environment. This led to the projection of possible scenarios after 2007, in the period of a post-USAID program presence. Prior to departure from Washington, a review of program documents, the mission Graduation Strategy and training material was conducted. A meeting was held with training staff and others to examine the scope of work in greater detail and to consider the Bulgarian training experience from a Washington perspective. Time permitted contact with only one USAID implementing partner prior to departure. The in-country methodology involved a range of individual interviews of organizations, meetings with focus groups and discussions with development organizations not associated with the USAID training program. In order to analyze interview responses across organizations and institutions, a set of common questions was developed to ensure similar queries were made in individual interviews and meetings. The various target audiences and contacts included the following: · USAID Director and Strategic Objective teams · World Learning, Inc., the USAID training services provider · NGOs and foundations supported under the PTP · USAID technical assistance contractors that utilize the PTP · U.S. Embassy Public Affairs Office · Bulgarian university community · Bulgarian private sector organizations · Foreign donor community organizations · Bulgarian government agencies · Former USAID participants The SO teams were interviewed as teams, rather than individually. Three focus groups were held: (1) a group of NGOs and foundations; (2) a cross-section of former participants; and (3) a group of former participants comprised of senior government officials. Other organizations were consulted on an individual basis. This methodological approach allowed the assessment team to receive input from a broad representation of stakeholders associated with training in one aspect or another, and to gather insights into the Bulgarian training environment as well as its concrete experience. Regular briefings and updates were provided to the USAID program officer responsible for managing the assessment. A draft report and out briefing was given to the senior USAID staff prior to departure. The final report will be completed upon receipt of comments from the USAID mission. Draft 11 Page 22 APPENDIX B IMPACT OF THE BULGARIA PARTICIPANT TRAINING PROGRAM Training has been an integral part of the USAID development program in Bulgaria for over eleven years. Inherently cross-cutting, the training program has contributed significantly to USAID’s initiatives in every Strategic Objective (SO). Since training is a vital component of nearly every mission activity, its results have a direct bearing on achievement of planned results. Overall, PTP training has contributed substantially toward Bulgaria’s economic, social and democratic transition. Overall Impact According to TraiNet, the Agency’s training data repository, 4,430 Bulgarian participants have been trained since 1993, in such diverse fields of study as agriculture and natural resources, democracy and governance, rule of law, economic growth, management skills, and civil society (See Annex D). However, measuring the number of participants trained does not begin to convey the true impact training has had in Bulgaria. Through its training delivery mechanisms, including technical assistance (TA) contracts and the Participant Training Program implemented through World Learning, USAID has transferred new skills, knowledge and attitudes to individual trainees in their various professional fields; but perhaps of greater importance for those who have had training in the U.S. is the exposure these participants have had with U.S. business and government culture, which is characterized by innovative problem-solving, analytical acumen, practicality, openness, interactivity, exchange and transparency. These distinctly American qualities are introduced by participants to others through their personal and professional connections to other Bulgarians. Thus, the “American experience” becomes a shared experience from which other local people may benefit. A direct product of the training, as identified by former participants, is the benefits from relationships created among individuals working in different agencies and at different levels brought together through training. Several instances were cited of participants who had developed relationships and networks with colleagues that, prior to the training event, did not exist. Through these newly established bonds, it became possible to modify legislation and regulations or jointly initiate development activities. Training has fostered national unity by joining participants in common goals and objectives leading to short-term and long-term results and accomplishments. In addition, a large number of former participants have opened and maintained a dialogue with European and U.S. counterpart organizations, providing ongoing opportunities for exchange of ideas and approaches to both regional and domestic development issues. Training has produced many success stories on local and national levels, and has provided an opportunity for Bulgaria to begin offering its model programs as roadmaps for success to other Eastern European and Eurasian countries, as they follow Bulgaria in their accession to the European Union. Draft 11 Page 23 Training Delivery Mechanisms Each year USAID prepares a training plan that identifies the purpose, nature and other details of each implementing contractor and grantee’s training needs for the coming year. This serves as the basic document for forthcoming training requirements. Training is provided to Bulgarian participants through USAID’s implementing technical assistance (TA) contractors and grantees and through the Participant Training Program (PTP). While TA contractors conduct training activities directly through their ongoing technical assistance programs, a significant portion of their training is provided through the USAID PTP contract with World Learning, Inc. The combination of these two approaches has resulted in a powerful impact on Bulgaria’s development initiatives. Much, but not all, of the in-country training is handled directly by TA contractors and grantees, typically using partner organizations to conduct the training. The approach often entails bringing in foreign expertise to prepare training modules and train local trainers to conduct the training courses. The PTP is most often used by implementers to conduct study tours in the U.S. and third countries as part of an ongoing technical assistance activity in the context of meeting its broader development objectives. Through PTP the USAID also provides support to selected participants in order to implement the action plans developed at the end of each training activity. A competitive small grants component of the World Learning task order provides funding for participants to build on the training activity by taking the initiative to make changes in their organization or in their communities. The PTP training contractor, WL, is generally acknowledged by the USAID staff, TA contractors, former participants and the Bulgarian training community to be highly effective in assisting USAID conduct its development program. WL services include: ÿ Reviewing training requests ÿ Providing feedback to USAID, its implementing contractors and recipients regarding: o Content o Timing o Venue o Group composition o Training provider o Scheduling (when to do what types of activities) ÿ Finding training providers, including competing when appropriate between countries ÿ Identifying and selecting interpreters Draft 11 Page 24 ÿ Providing appropriate background, orientation and briefing materials to all parties: trainees and trainers and interpreters ÿ Managing travel and related support ÿ Coordinating event logistics ÿ Follow-up with participant action plans World Learning has a competitive edge in preparing, conducting and coordinating study tours in Europe and the U.S., and finding the most effective training providers. However, most TA contractors believe that study tours which are conducted outside of the context of an overall development package are expensive and do not have the impact to warrant the expense. Objective post-training evaluations were supported under a separate USAID contract (Aguirre), but funding is no longer available. Still, WL staff makes an effort to follow up with trainees and maintain contact to the greatest extent possible given staffing constraints and willingness of participants to stay in touch. Sofia previously was the site of a regional TCT training office. In terms of maintaining a post-USAID presence, consideration could be given to re-establishing this regional function in Sofia. A strong regional networking capability currently exists and could be a cost effective way to serve regional training needs, as well as permit the training contractor to extend services to Bulgarian organizations and EU resources. Establishment of Legacy Institutions USAID already has established a clear-cut legacy through the creation or strengthening of indigenous training organizations, which include government entities, NGOs, and universities. Examples include: ÿ 3 Net Association ÿ American University of Bulgaria (AUBG) ÿ Broadcast Training Center (BTC) ÿ Bulgarian Institute of Internal Audit (BIIA) ÿ Center for Independent Living ÿ Financial Supervision Commission ÿ Foundation for Local Governance Reform (FLGR) ÿ Index Foundation ÿ International Banking Institute (IBI) ÿ National Institute of Justice (NIJ), formerly the Magistrates Training Center ÿ Regional Association of Municipalities ÿ Sofia University “St. Kliment Ohridski” Characteristics of these institutions are: ÿ Most training organizations and organizations that provide training as a key function (except universities) conduct training in a specific area of interest; ÿ Trainers are given training in training methodology as well as subject matter; Draft 11 Page 25 ÿ There is a trend toward training that is relevant for EU accession and post￾accession needs; ÿ Increasing attention is being given to sustainability issues as bilateral donor development assistance programs are winding down in Bulgaria; and ÿ Few TA contractors and partner organizations have established training ties to the academic community, indicating universities do no supply programs for their specific development needs. The assessment team did find increasing flexibility in the universities to adapt to changing needs in Bulgaria’s economic and social transition. Certificate and degree programs have been introduced in areas such as ICT, EU accession requirements and education management. However, a recent survey on the labor market conducted through a USAID project found large gaps in education and training, which must become more responsive to the evolving labor market in meeting human capital needs of the country. Overall, there will be a strong ongoing demand for jobs and development-related training. Impact of Training on Participants Two focus group meetings were held with former participants, one of which was with senior level government and business individuals. Participants gave an exceptionally high rating to the PTP, both from the perspective of making a significant impact on the lives of individual participants and the program’s contributions to development objectives. The former participants gave a number of specific examples; some of the generic benefits of the training cited were: ÿ Training helped give a clear understanding and perspective of the topic; ÿ U.S. exposure to successful cases gave dedication and persistence to local NGO development; ÿ Catalytic role of the training contractor was crucial in getting the appropriate mix of participants in the study tour; ÿ Following training, joint efforts by participants have resulted in initiatives heretofore unattainable; ÿ International contacts were established and maintained; ÿ USAID training is designed with priority given to participant professional needs and interests; ÿ Training has a practical orientation; ÿ Training is approached in a flexible manner to get the best quality and most appropriate results; and ÿ Administration (compared with other training providers) of USAID training is relatively simple. Constraints to Maximum Effectiveness 1. Training is wants-based and dependent primarily upon USAID contractors to specify the training need. It is not clear that all project training requirements are developed in the context of the overall organizational needs, and therefore may not be contributing to the maximum possible extent to the organizational Draft 11 Page 26 strengthening of the host country partner organizations. Engaging WL staff earlier in the process may lead to greater accomplishments in terms of organizational effectiveness and growth for these organizations. 2. Due in part to the lack of organizational context of the overall training approach, it is difficult to measure the true organizational impact of training. Training is a “contribution, not a solution” to an organization’s challenges, and while WL staff have done due diligence in conducting evaluations with the limited resources available, it is impossible to measure the overall organizational change relating to the partner organization’s development objectives and operational effectiveness unless other aspects of organizational performance are considered (i.e. job expectations, feedback, motivation, environment and tools), and tangible measurable results are identified against which to measure effectiveness of the training program interventions. USAID Training Success Stories Numerous examples of success stories have been cited where the technical assistance contractor and the training contractor have collaborated to obtain observable and measurable results through the Participant Training Program. Several are outlined below. 1. Judicial Code of Ethics Training (February 2004) This study tour was part of the USAID Judicial Development Project and brought together judges of the Supreme Judicial Council and the Union of Judges to review and assess the Austrian and Spanish approaches to the question of ethical conduct among judges. This study tour resulted in a closer partnership between these two bodies and upon completion of the activity a code of ethics that had been drafted and recommended several months earlier was quickly passed. The achievement came from a careful, representative selection of participants and the training contractor’s expertise in soliciting proposals and selecting the most appropriate organizations and countries to make this activity a success. 2. Training in More Effective Municipal Councils (June 2003) This study tour to Denmark and Poland was organized in support of the Local Government Initiative (LGI) to promote effective interaction between municipal councils and the public. Participants included representatives of municipal councils and national￾level policy makers, all of whom became strong advocates for an amendment to the Law on Local Government and Local Administration that enables citizens to participate directly in the work of municipal councils as a result of the training. One enterprising participant, a Mayor, decided to make his municipality a model for the country and championed the practice of publishing council decisions in regional media. The creation of a strong, representative advocacy group is having a gradual, yet positive, impact on local development.. Draft 11 Page 27 3. Institute of Internal Auditors Training (May 2003) Following a study tour to Prague to observe and analyze the activities of the Czech affiliate of the International Institute of Auditors, the president of the (then) Bulgarian Association of Internal Audit Practitioners, who participated in the study tour, coordinated the application process for establishing the Bulgarian Institute of Internal Auditors (BIIA). The new institute was formally recognized as an affiliate of the international organization in June 2003, and will help bring Bulgaria’s auditing standards to international norms. It is an example of successfully transferring knowledge on organizational qualities, so as to move a national professional group to a level of international standards and recognition. 4. Ombudsman Role Training (July 2003) Under the USAID initiative, Coalition 2000, the PTP arranged a study tour of an influential Bulgarian delegation with ombudsmen officials in Madrid and Barcelona, which resulted in the introduction of an amendment to the Law on Local Self￾Government and Self-Administration that passed Parliament, within a month after the participants returned. This amendment authorized Municipal Councils to elect public mediators. A number of local NGOs have since arranged seminars in several towns on the subject, and many other towns are seeking guidance and information from the Sofia Ombudsman. The training helped convince the Bulgarian delegation of the need and value of an ombudsman role associated with local government in Bulgaria. Draft 11 Page 28 APPENDIX C ORGANIZATIONS CONTACTED USAID Mission to Bulgaria Ms. Debra McFarland Mission Director Ms. Nora Ovcharova Program Specialist Mr. Gene Gibson Senior Advisor, Rule of Law and Governance Mr. Mihail Boyadjiev Advisor, Rule of Law Mr. Petar Kovachev Governance Advisor Ms. Svetozara Petkova Commercial Law Specialist Mr. David Lieberman Chief, Economic Growth and Restructuring Office Ms. Rayna Dimitrova Program management Specialist - Health and Labor Programs Ms. Ivanina Beleva Program Management Specialist - Energy and Environment EMBASSY OF THE UNITED STATES OF AMERICA Mr. Matthew R. Lussenhop Counselor for Public Affairs, American Center WORLD LEARNING USAID E&E Bureau Regional Participant Training Program Mr. Matt Brown Bulgaria Country Director Mr. Kostadin Evstatiev Program Coordinator Draft 11 Page 29 AMERICAN UNIVERSITY IN BULGARIA Dr. Olga Borissova Director, Centre for European Programmes and Elieff Center BEARINGPOINT USAID Commercial Law Reform Program Mr. Chris Thompson Chief of Party Ms. Ralitsa Petrova Program Specialist Ms. Kalina Lazarova Project Coordinator BROADCAST TRAINING CENTER USAID ProMedia Project Mr. Anton Tenev Program Manager BULGARIAN INDUSTRIAL ASSOCIATION Ms. Teodora Borissova Internet Training Manager Ms. Elena Tasheva Expert, Center for VET BULGARIAN INVESTMENT AGENCY Ms. Iva Stoikova Director, Investment Marketing and Analysis CARANA CORPORATION USAID Labor Market Project Ms. Gergana Rakovska Deputy COP CENTER FOR THE STUDY OF DEMOCRACY, COALITION 2000 Ms. Zhivka Damyanova Senior Adviser Draft 11 Page 30 DELEGATION OF THE EUROPEAN COMMISSION TO BULGARIA Mr. Ruud Van Enk Counsellor Head of PHARE/ISPA Section EAST-WEST MANAGEMENT INSTITUTE USAID Judicial Development Program Ms. Virginia Leavitt Deputy Chief of Party Judicial Training Specialist FOUNDATION FOR LOCAL GOVERNMENT REFORM Ms. Ginka Kapitanova Executive Director Ms. Nikoleta Efremova Training Team Leader FULBRIGHT PROGRAM Bulgarian-American Commission for Educational Exchange Assoc. Prof. Julia Stefanova, Ph.D. Executive Director GABROVO COMMUNITY DONATION FUND Ms. Ralitsa Genkova Executive Director GLOBAL FINANCE Sofia Representative Office Mr. Daniel Tomov Investment Manager ICT DEVELOPMENT AGENCY Ministry of Transport and Communications Mr. Orlin Kouzov CEO Draft 11 Page 31 INSTITUTE FOR SOCIAL ENVIRONMENT DEVELOPMENT Phone: 359-2-958-2433 Ms. Antoaneta Tzoneva Ex-Ombudsman of Sofia INSTITUTE FOR INTERNATIONAL COOPERATION OF THE GERMAN ADULT EDUCATION ASSOCIATION Dr. Maria Todorova NATIONAL ASSEMBLY OF THE REPUBLIC OF BULGARIA Mr. Valeri Dimitrov, MP Chairman of the Economic Policy Commission 3 NET ASSOCIATION Mr. Vladimir Galabov Ms. Iren Stephanova Mr. Plamen Dimitrov OPEN SOCIETY INSTITUTE SOFIA Ms. Cveta Petkova Program Director, Roma REGIONAL ASSOCIATION OF MUNICIPALITIES “TRAKIA” Mr. Ivan Varlyakov, Ph.D. Executive Director RTI INTERNATIONAL USAID Local Government Initiative Mr. Henry P. Minis, Jr. Chief of Party SOFIA COURT OF APPEALS Mr. Evgeny Staikov Chairperson Draft 11 Page 32 SOFIA UNIVERSITY “ST. KLIMENT OHRIDSKI” Assoc. Prof. Dr. Roumen Nikolov Vice Dean, Faculty of Mathematics and Informatics STARA ZAGORA COMMUNITY DONATION FUND Ms. Daniela Dimitrova Executive Director THE INDEX FOUNDATION Ms. Ludmila Mincheva Board Chair UNITED NATIONS DEVELOPMENT PROGRAMME Ms. Maria Zlatareva – Pernishka Assistant Resident Representative UNITED DUTCH FOUNDATIONS FOR CENTRAL AND EASTERN EUROPE (Samenwerkende Fondsen Midden- en Oost-Europa) Ms. Maria Petkova Coordinator WORLD BANK Ms. Boryana Gotcheva, PhD Senior Social Sector Operations Officer Draft 11 Page 33 APPENDIX D STATISTICS ON BULGARIANS TRAINED TOTAL BULGARIA PARTICIPANTS TRAINED 1993-PRESENT Bulgarian Trainees By Year of Training and Location Year IC TC US N/A Total 1993 10 10 1994 15 15 1995 18 223 241 1996 20 1,224 1,244 1997 17 276 293 1998 12 91 116 219 1999 2 25 156 41 224 2000 70 188 258 2001 95 92 193 380 2002 111 97 204 412 2003 420 119 110 649 2004 190 147 97 434 2005 36 12 48 N/A 3 3 Total 818 601 1,106 1,905 4,430 Bulgarian Trainees By Gender and Location Gender IC TC US Blank Total Female 184 281 516 130 1,111 Male 193 320 567 252 1,332 Group (M/F) 20 1,476 1,496 Blank 441 3 47 491 Total 818 601 1,106 1,905 4,430 Bulgarians by Major F ield of Stu dy and Location Major Field of Study IC TC US Blank Total Agriculture & Natural Resources 249 38 76 417 780 Arts & Humanities 0 0 0 0 0 Business Development 218 151 106 117 592 Education 0 60 61 19 140 Management Skills 122 94 381 775 1,372 Not Specified 57 18 0 37 112 Physical & Natural Sciences 1 7 10 2 20 Population, Health & Nutrition 2 10 51 18 81 Social Sciences 167 192 379 509 1,247 Draft 11 Page 34 Technical Skills 0 11 17 6 34 Blank 2 20 25 0 47 Total 818 601 1,106 1,905 4,430 Bulgarians by Specific Field of Study and Location Field of Study IC TC US Blank Total Accounting 52 0 34 21 107 Administration, Development 0 0 4 0 4 Advertising 0 0 3 5 8 Agribusiness Management 0 0 0 45 45 Agricultural Biotechnology 60 0 2 0 62 Agricultural Business 0 8 18 92 118 Agricultural Cooperatives 0 0 6 0 6 Agricultural Development 150 0 0 0 150 Agricultural Policy 18 0 1 0 19 Agricultural Training 0 0 18 0 18 Agriculture & Natural Resources 0 0 6 0 6 Agriculture, misc 15 0 0 0 15 Banking 46 6 28 28 108 Banking & Financial Training 0 0 7 0 7 Banking, Finance 46 6 3 0 55 Building & Construction Trade 0 0 0 4 4 Building and Construction 0 0 0 1 1 Business 60 16 21 1 98 Business & Labor Law 0 0 6 1 7 Business / Finance General 0 9 10 0 19 Business / Investments and Securities 0 12 15 0 27 Business Advertising 0 0 2 5 7 Business Development 0 0 1 0 1 Business Management 22 14 21 15 72 Business Management & Admin 0 0 2 38 40 Business Marketing 0 0 0 1 1 Business Marketing Research 0 0 0 1 1 Business Policy 0 37 23 121 181 Business, General 0 5 11 6 22 Business/Finance General 0 0 1 0 1 College & University Admin 0 0 5 5 Communications & Broadcasting 0 0 2 9 11 Communications/General 0 11 13 6 30 Communications/Television Journalism 0 0 4 0 4 Community Development 10 0 0 0 10 Comparative Law/Legal Systems 0 0 0 14 14 Computer Maintenance Tech 0 0 0 1 1 Conference Attendance 0 8 0 0 8 Construction & Building Trades 0 0 0 25 25 Construction Regs & Inspection 0 0 0 2 2 Consultation 0 0 0 2 2 Draft 11 Page 35 Cost & Financial Management 0 0 1 0 1 Counseling & Guidance 0 0 0 2 2 Court Administration 0 1 34 0 35 Criminology/Law Enforcement 47 0 3 6 56 Dairy 0 0 0 50 50 Development Planning 0 0 0 206 206 Diagnostic Radiology 0 0 0 2 2 Eastern Europe/Russian History 0 0 0 1 1 Econometrics 0 0 0 24 24 Economic Development 49 25 14 148 236 Economic Planning 0 1 11 12 Economics 0 1 13 1 15 Economics/Money and Banking 0 1 0 1 Educ - Culturally Disadvantaged 0 14 0 0 14 Education 0 9 0 3 12 Education Development 0 0 1 0 1 Electronic Engineering 0 0 0 1 1 Employment & Unemployment 0 0 0 4 4 Energy 0 7 4 0 11 Energy Regulation 0 10 7 0 17 English Literature 0 0 0 1 1 Entrepreneurial Development 0 24 0 0 24 Environment/Conservation 0 20 24 0 44 Farm Management 0 0 23 23 Finance 0 16 34 11 61 Finance, Accounting 0 0 2 0 2 Finance/Banking 0 0 1 0 1 Fisheries - Salt Water 0 0 0 1 1 Food 25 0 1 0 26 Food Processing 0 0 0 8 8 Foreign Policy 0 0 0 2 2 General Agriculture 0 0 0 32 32 General Business 0 0 0 14 14 General Communications 0 0 0 1 1 General Conservation 0 0 0 3 3 General Economics 0 0 0 2 2 General Finance 0 0 1 0 1 General Law 0 0 14 3 17 General Marketing 0 0 0 4 4 General Political Science 0 0 0 2 2 General Public Administration 0 0 0 2 2 General Social Work 0 0 0 1 1 Governance General 0 18 0 0 18 Govt Organization & Mgmt 0 12 17 0 29 Grant -making 0 10 0 0 10 Health Education 0 11 1 0 12 Health Facilities Administration 0 0 1 0 1 Health Services Delivery 0 10 0 0 10 Home Economics 0 0 1 0 1 Hospital Administration 0 0 0 2 2 Draft 11 Page 36 Hotel & Resort Administration 0 0 0 3 3 Human Rights Education 0 10 0 0 10 Information Science & Systems 1 0 0 0 1 Insurance 0 0 21 1 22 International Business Admin 1 0 10 1 12 International Economics 0 0 0 1 1 International Law 0 0 1 1 2 International Trade 0 0 0 1 1 Investments & Securities 2 11 46 147 206 Journalism 0 0 6 1 7 Journalism, Media, Communications 0 0 12 0 12 Judiciary System 0 13 17 0 30 Labor 1 13 1 6 21 Labor Ministry Development 0 8 0 0 8 Labor Organizations 0 9 0 0 9 Labor Relations 0 0 1 0 1 Labor Union Administration 0 0 0 2 2 Law Enforcement 0 0 10 0 10 Law, General 50 31 21 0 102 Law, Legal Administration 0 1 0 0 1 LAW, MISC 0 0 10 0 10 Law/Observation of US Legal Systems 0 0 26 0 26 Legal Education 0 0 24 0 24 Macroeconomic Theory 0 0 0 1 1 Management Skills 0 0 12 12 24 Management Training 0 0 25 128 153 Manufacturing, Industry 0 0 1 0 1 Marketing Economics 0 0 0 102 102 Marketing Research 0 0 0 18 18 Marketing, General 0 0 0 1 1 Media Develpment 1 0 8 0 9 Medical Records 0 0 0 5 5 Microeconomic Theory 0 0 0 25 25 Migration 0 0 1 0 1 Minorities/Minority Relations 3 25 1 0 29 Misc Agriculture 0 0 0 166 166 Misc Architecture 0 0 0 3 3 Misc Law 0 0 0 1 1 Misc Public Admin 0 0 0 4 4 Monetary Policy 0 0 0 3 3 Municipal Associations 1 0 0 0 1 Municipal Governments 6 20 0 119 145 NGO Management 0 1 0 0 1 Observation -US Courts/Admin 0 0 0 4 4 Observation -US Government 0 0 0 1 1 Occupation Therapy 0 0 0 4 4 Off-site Bank Supervision 0 0 4 0 4 On -Site Bank Supervision 0 0 0 6 6 Operations Research 0 0 0 10 10 Optometry 0 0 0 1 1 Draft 11 Page 37 Other Special Training 0 0 0 35 35 Peace Corps/Community Devt 0 12 0 0 12 Political Development 0 10 0 0 10 Political Science 0 0 10 0 10 Printing & Publishing 0 0 0 2 2 Production & Marketing 0 0 12 3 15 Public Administration 4 34 108 0 146 Public Finance & Tax Policy 0 0 54 6 60 Public Health 2 0 50 1 53 Public Law 0 0 20 1 21 Public Relations 17 0 24 1 42 Public Welfare 0 0 20 0 20 Real Estate 0 0 0 1 1 Scientific Research 0 0 6 0 6 Small Business Development 0 7 0 31 38 Social Change Issues 0 2 0 0 2 Social Security 0 10 0 0 10 Social Services 4 12 30 0 46 Taxation 0 0 8 10 18 Telecommunications 0 0 29 0 29 Tourism 65 14 1 0 80 Tourism & Travel 0 0 0 8 8 Training, Special Training, Other 57 10 0 0 67 Transportation 1 0 0 0 1 US Elections 0 0 3 0 3 US Political Systems 0 0 0 8 8 Women's Activities 0 8 10 0 18 (blank) 2 20 25 0 47 Total 818 601 1,106 1,905 4,430 Draft 11 Page 38 APPENDIX E SCOPE OF WORK IMPACT EVALUATION OF USAID/BULGARIA PARTICIPANT TRAINING PROGRAM AND ASSESSMENT OF LOCAL TRAINING CAPABILITIES I. Introduction The focus of this evaluation will be 1) the USAID-funded Participant Training Programs (PTP), implemented for the past ten years under contract by World Learning Inc (WL), and 2) local alternatives for PTP legacy mechanisms. Over the years the PTP program has evolved from a myriad of stand-alone programs into a true crosscutting activity, which provides short training interventions in support of the long-term strategic objectives of all USAID activities. Currently the PTP, in close coordination with USAID and its implementing partners, handles the majority of training programs and study tours that take place in the U.S., in other countries, or in Bulgaria. Based on the PTP Impact Evaluation, the team shall provide recommendations about the appropriate in the local context post-USAID legacy mechanism and tools, including, but not limited to local partner organization’s capable of hosting PTP-type activities, for consideration and ultimate informed decision by the Mission. II. Background The PTP program complements the objectives of the ongoing USAID projects by providing for the required practical skills, knowledge and attitude to be acquired by the prospective local experts, who are seen as the drivers of change. Additionally, training programs and observation of successful models abroad motivate and empower people to advocate for changes – these are based on the principle of “help people help themselves” and focus on facilitating sustainable development at local institutions. PTP has proven to be a flexible mechanism for quick and efficient response to deficiencies related to learning and attitudinal gaps, and/or to any politically tainted requests. The PTP program is expected to run through the USAID Graduation year of 2007 when Bulgaria is scheduled to join the EU. A formal impact assessment has never been conducted for the last 5 years, so USAID is planning to commission an independent assessment to validate reports on achievements and to evaluate longer-term program impacts. Additionally, during the outer years USAID will focus mainly on strengthening local partner institutions in an attempt to leave lasting and sustainable legacies behind. As the Mission is currently exploring alternatives of PTP legacy mechanisms to continue with 1) implementation of training programs as needed, 2) coordination of training capacities and needs in Bulgaria and in the region, and 3) maintaining the links among all past and Draft 11 Page 39 present US trainees (Georgetown ECESP, PTP, Fulbright and Ron Brown fellowships, IV Programs, etc.), we ask the team to recommend PTP legacy options, incl. but not limited to the option of a local training provider to take over/serve as PTP legacy. III. Objectives of the Evaluation The evaluation has the following principle objectives: 1) The evaluation will specifically identify existing challenges to achieving long-term sustainability of program outcomes, for consideration and appropriate actions under the legacy mechanisms. 2) The team shall further assess the local alternatives for PTP legacy tools and mechanisms, incl. but not limited to short-listed local training providers, with a view of their capabilities to become legacy partners for the following illustrative/non-exhausting list of functions: Implementation of training programs as needed, coordination of training capacities and needs in Bulgaria and in the region, serving as headquarters of all US￾trained alumni associations, maintaining the links among all past and present US trainees (Georgetown ECESP, PTP, Fulbright and Ron Brown fellowships, IV Programs, etc.), and as a Third Country Training (TCT) provider with extensive knowledge of local and in-country training providers. 3) Based on the PTP efficiency and impact evaluation, the assessment of local legacy alternatives and the projected future demand, the contractor shall provide USAID with a set of recommendations on how to link PTP activities with strengthening the proposed legacy tools. IV. Evaluation Questions The evaluation should address the needs, purpose, and methodologies behind the creation and maintenance of legacy partners. A. PTP Legacy Alternatives/Capabilities of Indigenous Training Providers ß What is the host country HCD environment – number and level of training providers? ß Has local capacity been built through the PTP Program? ß Are local training providers capable to serve as a TCT office, maintain a database of US alumni and serve as a secretariat of a potential Alumni association? Suggested criteria (not to be limited to the below) - HCD and Training as overarching Mission and Goal - Qualified personnel – logistics providers and training experts Draft 11 Page 40 - Prior training experience - Extensive knowledge of BG and regional donor, public and private programs and potential trainees’ needs? - Sources of funding? Share of own revenue derived from training￾related services? ß Have there been any collaboration efforts with other donor training programs and activities implemented in Bulgaria, which pertain to legacy building? Outside of Bulgaria? If so, what were the outcomes? If no, what might be the challenges to establish such relations? ß What are the lessons learned after 10 years of training in 2 directions: – has a community of training professionals been built – have the networks of sector experts been created – if yes: formal or informal? B. Recommendations re: Improved PTP effectiveness and Legacy tools ß What are the challenges/obstacles to achieving sustainability and/or having long-term impact of HCD training programs? ß Based on the comprehensive understanding of PTP achievements and impact, the host country environment and the projected demand, what would be the most appropriate legacy tool? ß Have the training programs affected the relationships between former trainees in a mixed group – consolidated their relations, new bridges, or lifetime enemies? ß How viable is an Alumni association model as a legacy mechanism – as a standalone vehicle, or rooted within an existing training provider? ß What should be the short-term immediate future objectives of the PTP program as a balance between traditional programs and support for the legacy tool? ß Assistance in promoting the PTP legacy mechanism in the region, as well as to donors. V. Evaluation Methods A. Literature/Desk Review Desk review of the available background information and report – the WL PTP Bulgaria task order, Quarterly reports and success stories, the annual brochures of WL training programs. Draft 11 Page 41 After this stage the evaluation team shall draft a plan for the research to be undertaken together with an outline of the report – for USAID review and approval. B. Qualitative and Quantitative Analysis ß Rapid Appraisal methods: key informant interviews, mini-surveys based on structured open-ended questionnaire, direct observations ß Non-probability sampling (the sample shall be credible, i.e. there should be logical selection of the sample to be surveyed) ß Selection Criteria for Trainees and Organizations to be interviewed: ß Diverse partner institutions size (large, medium, small, VIPS and mid level professionals, local authorities, SME businesses and NGOs– across SOs, men-women, etc.) - Trainings implemented in all the fiscal years reviewed (FY98- FY02, potential comparison between trainings implemented 1-3 years ago vs. 4-5 years ago as a way to seek longer versus shorter-term impact; also:) - Proportional representation of programs in all USAID SO areas? - Maximum variety of implemented activities - Feasibility in terms of time and effort In conducting the assessment a participatory approach will be used to ensure stakeholders’ ownership of the results and findings and their commitment to follow-up course of action for strengthening the legacy vehicle. VI. Team Composition The evaluation team shall comprise of three to four independent evaluators – an expert in HCD and training programs development, and an Evaluations expert with HCD experience – two foreign experts and one or two local independent consultants with extensive knowledge about PTP-like types of donor programs/their components. Criteria for the positions: ® HCD and Training Programs expert - at least 10 years in development and management of PTP-type of programs, experience with donors and overseas development programs is required, preferably w/ USAID in the SEE region ® Evaluation Expert: at least 10 years of experience evaluating projects. Demonstrated qualitative and quantitative research skills. Survey and questionnaire design and data analysis/interpretation. Local experts – Extensive experience (min 5 years) with a mix of the following: Bulgarian public administration, educational institutions, donor PTP type of programs, civil society and economic think tanks, etc. Familiarity Draft 11 Page 42 VII. Illustrative Schedule Preparation for departure /meetings at AID/EGAT/W – 4 days Fieldwork in Bulgaria Two weeks Follow-up in the US One week Travel 4 days The evaluation is expected to last appr. 4 weeks, 2 weeks in the field (Nov. 11-24, 2004). Week 1: Preparatory Work: The team shall spend the first week in finalizing the methodology, agenda, evaluation work plan, as well as on conducting a desk review of all the available indigenous capacities, annual ceremony brochures, success stories and statistical data. During this period, it is also supposed to develop a list of interviewees (focus groups), institutions & locations to visit, and duration of the site visits. An interview/questionnaire shall also be developed for discussion with USAID. A detailed outline of the draft report shall be submitted to USAID at end of week One. The remaining time will be spent on collecting data and identifying data sources. Week 2 & 3: Field Work: The team shall conduct site visits and interviews at the institutions of the in-country training providers. This phase will include field interviews, or focus groups where applicable with USAID staff and partners, as well as informational interviews with the PTP Country Director, WL program Assistants, USAID TA projects, the local training providers. Upon completion of first week of the fieldwork, the evaluation team shall debrief relevant USAID officials, and submit mid-term progress report. Week 4 : Report Writing: The team shall produce a complete draft of its evaluation report within 7 days of completing the field work involved in the evaluation. Following internal USAID review, a final report shall be developed within another week, reflecting the Mission’s comments and recommendations. VIII. Personnel and Level of Effort Level of effort Team leader: HCD Specialist – 24 days Participant Training and Institutional capacity Expert 18 days Education/Training Spec. 24 days Local HCD and training experts (2) - 20 days Total 86 days IX. Logistics and Technical Direction Draft 11 Page 43 Logistics: Travel and per diem shall be provided for by the Evaluation Contractor. The Mission will provide the team with contact info for short-listed local training providers. Translator (if needed) shall be provided for by the Contractor. WL office will facilitate contacts with stakeholders and provide logistical support on a limited basis. Supervision: The Evaluation team will be working under the direct supervision of the USAID/Bulgaria Program Officer, Ms Ivanka Tzankova and the USAID/Bulgaria Training Officer, Ms Nora Ovcharova. While in USAID/W, Mr. Jeffrey Shahan will provide overall background at USAID/EGAT/W. X. Existing Performance Information Sources Quarterly and Semi-annual reports on the regional PTP program are submitted to the USAID/Wash CTO. A web-based success story page is maintained by the Washington offices. The WL Bulgaria office is staffed with of 6 participants who work on specific programs. Other basic relevant documents include the Bulgaria task order, project proposals, reports, and other reports. XI. Deliverables and Reporting Requirements - An Evaluation Plan and Outline - based on the desk review, the plan shall outline the agenda for filed work and meetings suggested. (The local experts shall facilitate scheduling of the meetings, with USAID PTP support for the initial1-2 days) - A report outline – at the end of the First week: Quantitative analysis - A first progress report – end of first fieldwork - A first draft report – end of second filed workweek - A complete evaluation report - end of fourth week - A final evaluation report – within a week after USAID/Bulgaria comments, to include the final evaluation report with findings, conclusions, and recommendations in English. The final report shall be submitted to the USAID Mission by December 15, 2004. XII. Annexes: Definitions of important aspects of the PTP Program, supporting materials in the form of samples of stakeholder agreements, action plans, thank-you letter, etc. docs re: impact.