SUMMATIVE EVALUATION BULGARIA Public Computer and Communication Centers (PC3s) by Jeanne D. Ti.. Consultant in Information Sciences & Technologies . Location of Bulgarian PC3s Sponsored by USAID Implemented by AED Sept 2000 - May 2002 IQC Task Order # -1-10-96d0018-00 ACKNOWLEDGMENTS This evaluation effort is the work of many people whom the author acknowledges here for their help, support, advice, professionalism, and good cheer. My thanks to: Borislav Tahdjiyski and Georgi Shopov of Club Economics 2000, Sofia, for making professional collaboration such a pleasure, conversations, interviews, notes, and safe driving Iliana Nikolova, AEDISofia Country Director for the PC3 Telecenter project, for rigorous planning, inexhaustible energy, and meticulous documentation Stanislava Stancheva for endless typing, transcribing, collating, translating, and interpreting with enthusiasm. Georgi Gachev for meticulous data hygiene and Excel expertise. Also, safe driving and calm in an earthquake. Tina Peneva for interpreting with charm and patience despite upcoming examinations. Eduardo Contreras-Budge, AEDLeamLink Evaluation Specialist, for keeping me on the straight and narrow path of evaluating things usefully. TABLE OF CONTENTS EXECUTIVE SUMMARY ............................................................................................... " . 1 TASK ORDER REQUIREMENTS ........................................................................................................................................... 3 PROJECT ACHIEVEMENTS .................................................................................................................................................. 4 UNANTICIPATED OUTCOMES ............................................................................................................................................. 4 EVALUATION PROCESS ...................................................................................................................................................... 5 Planning ....................................................................................................................................................................... 5 Indicator Issues ............................................................................................................................................................ 5 PROJECT IMPLEMENTATION ....................................................................................... .......-. 6 STRENGTHS ....................................................................................................................................................................... 6 WEAKNESSES .................................................................................................................................................................... 7 Lessons Learned ........................................................................................................................................................... 7 PROJECT DIRECTION AND MANAGEMENT .......................................................................................................................... 8 . . Communrcat~on policies andpractices ........................................................................................................................ 8 Incentive structures ..................................................................................................................................................... 8 Futurepositioning for the PC3 telecenters .................................................................................................................. 9 PC3 OPERATOR SELECTION .............................................................................................................................................. 9 Lessom Lemned ......................................................................................................................................................... I0 FQUIPMTN'T SPECIFICATION, ACQUISITION, AND DELIVERY ............................................................................................. 10 Systems training ......................................................................................................................................................... I0 Equipment Troubles ................................................................................................................................................... I0 Lessons Learned ......................................................................................................................................................... I I PREPAID CARD/COUPON PROCEDURES ............................................................................................................................. 11 Purposes ..................................................................................................................................................................... I1 Coupon Users ............................................................................................................................................................. I1 Paying Users .............................................................................................................................................................. I2 Prepaid cmd distribution ........................................................................................................................................... 12 Distribution Effects .................................................................................................................................................... I2 Prepaid cmd management ......................................................................................................................................... 13 . . Prepaid card quant~tres .............................................................................................................................................. 13 Lessons Learned ......................................................................................................................................................... I4 PC3 OPERATOR TRAINING .................................... - .............. -....-.-...-.-.---.-"--.. - . 14 PROGRAM ........................................................................................................................................................................ 14 PARTICIPANT 'klMNG ............................................................................................................................................... 15 Lessons Learned ......................................................................................................................................................... IS CONTENT DEVELOPMENT ................................................................... -......-..-.-- - 15 CONTENT DEYELOPMENT PROCESS ................................................................................................................................. 16 Lessons Lemned ......................................................................................................................................................... I6 OPERATOR INTERVIEWS ................................................................................ 17 CLUB ECONOMICA 2000 SURVEY DATA ................... .............................. ,- 18 SUSTAINABILITY FINDINGS .............................................................................................................................................. 19 EFFECTS OF BUSMESS CONDITIONS ................................................................................................................................. 20 COMP~ON .................................................................................................................................................................. 20 BUSINESS PLANS ............................................................................................................................................................. 21 PC3 INCOMES .................................................................................................................................................................. 21 SKILL EVOLUnON AMONG USERS ................................................................................................................................... 22 COMPUTER USE ................................................................................................................................................................ 23 OPERATOR CHARACTERISTICS ........................................................................................................................................ 23 ASSUMPTIONS AND OUTCOMES ............................................................................................................................... 23 FUTURE PROSPECTS .................................................................................................................................................... 24 PROBABLE FUTURE MODELS FOR INDIVIDUAL ........................................................................................................ 24 BUILDINGNATIONAL ANDDONOR RELATIONSHIPS ......................................................................................................... 25 COLLABORA~ONS WITH UMBRELLA ORGANIZATIONS. ASSOCIATIONS. AND NGOS ........................................................ 26 MODEL REPLICATION REQUIRE~S ............................................................................................................................ 26 USAID AND AEDLEARNLINK PROJECT OUTCOMES .................................................................................................. 27 BIBLIOGRAPHY .............................................................................................................................................................. 28 ANNEX 1: PC3 OPERATORS' DIRECTORY ............................................................................................................. 30 ANNEX 2: PC3 IMPORTANT DATES .......................................................................................................................... 31 ANNEX 3: SERVICES OFFERED BY THE PC3 TELECENTERS ............................................................................ 32 ANNEX 4: PUBLIC-GOOD SERVICES OF THE PC3 TELECENTERS .................................................................. 34 ANNEX 5: EVALUATION DATA SOURCES ............................................................................................................... 35 ANNEX 6: SUMMATIVE EVALUATION FRAMEWORK ..................................................................................... 36 ANNEX 7: CLUB ECONOMlCA SCOPE OF WORK ................................................................................................. 42 ANNEX 8 : PC3 OPERATOR IN-DEPTH INTERVIEW INSTRUMENT ................................................................. 43 ANNEX 9: PC3 OPERATOR IN-DEPTH INTERVIEW FINDINGS ........................................................................ 45 ANNEX 10: PC3 PREPAID CARDS DISTRIBUTION AND ADMINISTRATION ................................................ 48 ANNEX 11: BOARD MEMBERS AT EACH PC3 TELECENTER ............................................................................ 50 ANNEX 12 : INSTITUTIONAL PARTNERS OF EACH PC3 TELECENTER ........................................................ 52 ANNEX 13: PC3 OPERATOR TRAINING PROGRAM ............................................................................................. 54 ANNEX 14: PC3 OPERATOR TRAINING AGENDAS .............................................................................................. 55 ANNEX 15: TRAINING SERVICES SUMMARY. ALL PC3S .................................................................................... 57 ANNEX 16: PC3 CONTENT DEVELOPMENT FORMAL PROCESS ...................................................................... 58 ANNEX 17: PC3 "HOW TO ... ?" REFERENCE TOPICS ........................................................................................... 59 ANNEX 18: 'PC3 INTERNET CATALOGUE" ........................................................................................................... 62 ANNEX 19: CONTENT VALUED BY PC3 CLIENTS ................................................................................................ 63 ANNEX 20: PC3 SUSTAINABILITY: FORM OF OWNERSHIP AND ITS IMPLICATIONS .............................. 64 FIGURE 1: PC3 COUPON USER SUMMARY DATA ................................................................................................ 65 FIGURE 2: PREPAID CARD UTILIZATION DATA ........................................... 66 FIGURE 3: "OTHER* COUPON USERS AND SERVICES .................................. --- 67 FIGURE 4: PC3 SERVICES USE ................................................................................... 68 FIGURE 5: PC3 PAYING CUSTOMERS .- ............................................................... I 70 FIGURE 6: PC3 TRAINING SERVICES AND PERFORMANCE ...................... - .-- ...--- - -- 71 FIGURE 7: PC3 TELECENTER PERFORMANCE .......................................... " .........-.- I . 73 FIGURE 8: PC3 INCOMES ............................................................................................... 74 FIGURE 9: PC3 INCOME TOTALS AND BY TYPE OF USE ............................................. 75 FIGURE 10: INTERVIEWED USERS' REASONS FOR USING PC3 TELECENTER SERVICES 76 FIGURE 11: ENVIRONMENTAL FACTORS AFFECTING PC3 GROW - ...-....--- - 77 EXECUTIVE SUMMARY During September 2000 and May 2002, to promote and extend the benefits of information and communication technologies (ICTs) and Internet access outside urban centers, AEDILeamLink established Public Computer and Communication Centers (PC3s) in 10 small towns lacking modem ICTs. The project supported small entreprenem-carefully selected among about 90 qualified applicants-in the creation and startup phases of locally owned and operated telecenters. The operators developed their own initial business plans and shared investments, while the project , delivered equipment, constant technical support, and training. In an innovative strategy, the project also provided operators with a number of prepaid cards to be distributed among prospective target users for promoting PC3 usage and creating a client base. This demand-based subsidy also would help the nascent PC~S survive an initial To enrich client resources, a project goal was facilitating more Bulgarian-language Internet or CD-ROM based content, scarce at startup time. Project evaluation focused on sustainability issues, assessing the impact of subsidies @repaid cards) and examining this business-oriented model. The findings fiom this evaluation reveal how the initial PC3 business plans were adapted by the 10 PC3 operators due to local conditions, clientele, and immediate sustainability requirements. Most initial users were students doing Internet hg. With notable local variations, services are evolving towards a diversified clientele with specific service needs, one of which is ICT training. Local government partnerships are being explored, as well as some with public and private institutions. The idea of a serious, non-smoking cybercafe, complete with games, is appealing, especially in tourist areas, and another is evolving as a local ISP node, serving competitors with "gaming clubs." To ensure sustainability, operators adopted wireless for obtaining the fastest Internet speed, which - - became a competitive advantage. DU&~ the project time framer~ul~aria rapidly pro&ssed in telecommunications, and there was a dramatic external increase in the amount and quality of Bulgarian-language content. Thus, the project refocused on ICT training materials still unavailable. Operators learned to ada~t their senice provision to meet local needs and PC3 abilities, including A A - training. For instance, some had stronger Web design skills; others, desktop publishing capabilities. Despite the client base, service and product diversity, training differences, and diversity of business all PC3s relied on the prepaid cards as income suppoi Given the short operatio& lifetime at proiect closure, it is not verifiable how successN PC3s will be in a business based solely on paying - - -~ ~ clientele, though recent PC3 income growth is encouraging. A longer term evaluation or follow-up would provide such data. In addition to the careful selection of PC3 entrepreneurs, it is important to recognize the importance of the BulgariadSofia Project Director and staff in consistently providing day-to-day support to the centers. She managed with dynamism, conscientiousness, and good humor the delivery of appropriate and sufficient training that increased fiom three planned workshops to seven plus a final study tour of Hungarian telecottages. In addition, she created training manuals, including a 300-page commercial quality one, facilitated cohesiveness and a mutual support group among operators, and solved problems for a significantly greater number of centers than anticipated at first. Her dynamism, university background and resources helped propel this project through a rapid and successful start￾UP. Clients, neighbors, and Mayors who were interviewed indicated satisfaction with the PC3s and liked their hospitable operations and smoke-free environments. Among the many lessons learned, the most prominent for sustainability was the use of the prepaid cards for stabilizing initial PC3 telecenter business income and attracting the paying client, albeit intended at the onset for educational and under-served groups. Findings reveal the potential for diversifying the value of the prepaid cards for different services, changing the allocation criteria, and incorporating computerized management of the cards, since the paper-based prepaid card system proved rather unwieldy. Better ways to use the prepaid cards to subsidize use by the under-served need to be devised, and an elusive question of how to serve "the public good" still remains. Evaluation indicated that the project unmistakably met its goals, although unanticipated actions and impact proved very important and effective, and some intended effects did not happen. For example, the project design expected development of spinoff micro-businesses. They did not materialize, but half the telecenters became ISPs and built municipal LANs, thus increasing Intemet penetration in Bulgaria. The PC3 telecenters apparently are becoming sustainable businesses that have increased the value of social goods by training over 4100 Bulgarians in basic computer and Intemet skills. This USAID-supported experience illustrates that a small (one person or husband and wife teams)ACT business enterprise is a viable model for expanding ICT access and developing human social and economic capacity under conditions identified more fully in the following evaluation. AEDILeamLink Staff Bulgarian Public Computer and Communication Centers (PC3s) BACKGROUND Previous telecenter projects in developing and transitional economies have received strong and continuing donor or national government support. The Bulgaria PC3 project, on the contrary, set out to see whether a relatively small amount of short term donor seed money for eauiument - software licenses, connectivity, training, and risk minimization activities would be enough to start up sustainable small private enterprises providing ICT services in small Bulgarian municipalities. It grew out of recommendations in a 1999 USAID-funded survey of the Internet idmstmcture in Bulgaria conducted under the Presidential Initiative on Intemet for Economic Development.' This report suggested telecenters as one means of spreading public Internet access and modernizing IT skills in Bulgaria. The Bulgarian populace is literate and educated. Bulgaria was the Communist era "Silicon Valley," but people's IT skills are out of date by 30 to 40 years. People are aware that they must learn to survive in the evolving market economy, but their business skills are rudimentary. All of these circumstances shaped the project. Task Order Requirements The Task Order required the Contractor (AED) to establish within 9 months a group of 3 to 10 private sector "Public Computer and Communication Centers" (PC3s) in selected middle-sized Bulgarian towns lacking Intemet access. Within one year, the project had opened ten. [See Annex 1 .] Each Bulgarian PC3 was to offer a "broad range of ICT-based services" suitable to local needs as determined by the local entrepreneurial PC3 operator. Technical assistance, two to three operator training sessions, and support for a portion of start-up costs was to last for up to 15 months, later extended to 20 months. At that point, the established PC3s were to show evidence of becoming self￾sustai~ing.~ The Task Order also called for specific methods and activities to be used. Each has yielded useful lessons learned: a competitive award process to identify prospective PC3 operators, select, and subcontract with finalists a system of vouchers or coupons so that the PC3 operators could offer a limited number of free introductory services to desirable user groups, for "common good" purposes, and then claim reimbursement from project funds based on submission of used, certified coupons a monitoring system throughout the project to capture useful data on the growth and usage of each PC3 plus the demographics, attitudes, and behavior of its users; the free coupons were also to function as a data collection instrument. ' USAID. Center for Economic Gmwih. Initiative on Internet for Economic Development Inibal Survey of &ma. September 1W0. 1999, p. 21-5. It indudes a scope of work for telecenters in HaM, which was adapled for the Bulgarian PC3 teleQnter prcja design: see Annex F. 'A comparison of first and last month incomes for all PC3s done for this evaluation shorn an overall in- of over 200%. See Figure 8, PC3 Incomes. 3 Project Achievements This pilot project has successfully demonstrated that the multipurpose telecenter can be a potentially sustainable and replicable demand-driven small private enterprise model for human capacity development in a transitional economy country, where many communities lack Internet connectivity and remain isolated from economic and social change. Despite constraints of time, budget, and staff, the project succeeded-and surpassed several goals--during a very short time period (15 - 20 months)? The project goal was to establish a minimum of three telecenters, and there are ten. Two to three operator training sessions were to be provided, and seven were delivered. The prepaid card scheme, which should have been automated, was a manual process, both labor intensive and error prone; the card scheme nevertheless demonstrated a powerful effect on business behavior and yielded the longest list of useful lessons learned. Each PC3 added multiple new services during the project period, while only one new service at each center was required. Unanticipated actions and effects proved very important and effective, and some intended effects did not happen. For example, the project design supposed development of spinoff micro-businesses. They did not materialize, but half of the telecenters became ISPs and built municipal LANs, thus increasing Internet penetration in Bulgaria. New services were added instead of the required one new service at each s enter.^ Another project achievement is attributed to its having been implemented not simply as a telecenter project but as a human capacity and small business development project with a telecenter label. Because the project has unmistakably met its goals, this report has envisioned a reader interested in the model's replicability and therefore focuses on lessons learned during program planning, program implementation, operator selection, training, and prepaid card administration. Descriptive annexes, graphs, and most tables are attached at the end of this report. Unanticipated Outcomes The project design incorporated an inherent contradiction never resolved and even noted by applicants during the pre-bid conference: "Are the ... social and business goals of the PC3 Project in contradiction?" The struggling new small businesses in poor municipalities were meant to become sustainable enterprises (presumably profitable enough to keep going), while at the same time providing services at concessionary rates to the most disadvantaged. PC3 operators evolved three different approaches to resolving this dilemma: build the business by creating demand among those who could afford to pay, build interdependent relationships with the municipal government, andlor include a separate area for computer games in their busines~.~ See Annex 1 for wntad information for the PC3 telecentes, and Annex 2 for a list of the important events and dates during the proiect 'See Annex 3 for lists of PC3 telecenter services offered dunng their first month In the fall of 2001 and In February 2002. Bulaarian - - clties and laraer towns have a lonastandina tradition of "qames clubs" and "Internet clubs"-rather like video arcades in use- "- ~ -~~~ ~ smoky dens that are a d&w for unemployed youth and young men. ?he PC3 telecenter project policy was to discourage but not forbid such aames. nor could clients oav for them with .. project .. prepaid cards. Adult clients preferred the quiet, professional, nonsmoking smoky dens that are a draw for unemployed youth and young men. . 4 The project design also required that PC3s offer services with a "common good" purpose without defining the phrase in terms that connected USAID's assumed meaning with that which Bulgariaas might assume. When the PC3 operators were asked to list services they themselves considered a "public good," the result ranged fiom public access @aid) services to charitable efforts for orphanages.' It seems, therefore, that whatever USAID meant by "common good" was interpreted differently by Bulgarians interviewed. A recent World Bank-commissioned study (Information Infj.arhucture Indicators 1990-2010)' characterized Bulgaria among the "latecomer" counbies in achieving participation in the "Internet revolution." As the developed world pushes forward through e-commerce to multimedia applications supported by broadband networks, the latecomers still struggle toward basic connectivity. To get there faster, the study recommends that latecomer countries emphasize three priorities: the goal of basic connectivity, hastened by allowing effective competition, using education as the enabler. The Bulgaria PC3 telecenters contributed to all three. Evaluafion Process Planning The PC3 project gathered and produced a great deal of data in the course of its work, and a plethora of telecenter evaluation literature exists. Desk review of this material consisted primarily of selecting relevant issues to examine. The author of the evaluation also drew on the experience and support of AED/SofiaYs project staff, sociological evaluation research conducted by a Sofia-based think-tank Club Economics 2000, and guidance fiom AEDILeamLis evaluation specialist Fi an Evaluation Framework document was prepared to serve as a working tool for ail parties to share and use in collecting and organizing data. The set of indicators established for research did not change much as the work progressed, though a few quantitative indicators were abandoned due to lack of data. Qualitative research concentrated on a set of structured and semi-structured on-site interviews with all PC3 telecenter managers, staff and local board members present during the site visit, and municipal officials (mayors and deputy mayors). The interview instrument for PC3 telecenter operators sought to determine the effects of the project on the telecenter managers and on their customer base.' Indicator Issues Although this was a pilot project, and its design mandated a monitoring and evaluation procedure, no baseline comparative social, economic, or financial data was systematically collected, either on the successful applicants or on their municipalities. Narrative summaries of the applicants' proposals offered basic descriptions. Because of the short time frame of both project and evaluation, we decided not to organize data for examining telecenter impact at the municipal level but to concentrate on the telecenters themselves. 'See Annex 4 for a list of the services that the PC3 teleenter operators consider "public good services.' 'World BanMnfoDev Executive Summary. information inh-astrudum indicatws. 1990-2010, by Pyramid Research. hUp:lhvww.lnfodev.omIprolect~ntemell3f @See Annex 5 for a list of Evaluation Data Sources. Annex 6 for the Summary Evaluation Framework. Annex 7 for the Club Eamanic Smpe of Work. Annex 8 for the Operator In Depth lnte~iew Instrument, and Annex 9 for a Summary of the in-depth intervWI lindvlgs Another indicator issue shaped the evaluation process: user and usage data collected during the project only accounted for prepaid card users; there was little or no data on paying customers￾information which might have given some measure of real sustainability. Therefore the Club Economica was subcontracted to investigate social and economic aspects of the PC3 telecenters in the contexts of their municipalities and report its assessment of the present and likely future sustainability of the telecenters? PROJECT IMPLEMENTATION Strengths Particular strengths of the project design that can be cited as contributing significantly to its successll implementation are as follows: The mix of subsidies and equipment provided each PC3 telecenter was decided upon jointly after negotiation, based on specific needs. Thus, some got more support for Internet connectivity, others got more computers."' PC3 operators assumed up to half the start-up costs, a commitment of spirit as well as money. "The entrepreneurial instinct is a strong force in making a [telelcenter effective."" The systematic operator selection process lasted 5 months," during which time an initial applicant pool of 92 was winnowed down to 25 shortlisted candidates. Finally, after site visits and interviews, 10 PC operators were selected. AED's project team used an iterative procedure with a rank scoring method to arrive at choices all believed to be the best. Though operator training was not explicitly emphasized in the project design, during its implementation the in-country project stafTdeveloped and delivered a targeted sequence of training sessions for the PC3 operators that directly responded to their training needs and proved to be a critical project input.I3 Weaknesses One project design weakness is the inadequate support of the sustainability requirement by the monitoring and evaluation processes. Not only were data on paying clients not specifically required, Particularly useful in conceiving of good indicators to use for this evaluation were the papers by Emberg, Hudson, and Whyte, all cited in the attached bibliosra~hv. - . . 'OThe standard equlpment opttons were a desktop computer with CD-RW, Windows 98G!OOOIXP, and Ms Office Sulte, modem, pnnter. scanner, wpler, fax machlne, and UPS Connectlvlty equlpment ~ncluded hubs sw tches, routers, satellite dlsh, wlreless dev~ces, and cable. One chose a VCR instead of Drinter. scanner. and copier. Another used the USAID suDDort . . for Windows to install on his own kmputers and for wireless connectivity equipment. " Peter Benjamin, in "African Experience with Telecenten." e-OTI: OnThelntemet, Oct. 2000. l2 Fmm the RFP published January 22. 2001 in all regional and national newspapers, to the subcontract signings in June but a decision was made not to spend money on customizing an off-the-shelf tracking software package for PC3s to automate data collection on the prepaid card users. In addition, the Task Order mandated so many prepaid card data items that the manual system created to capture them became inordinately complex. PC3 operators found administering the prepaid cards a real burden that took time away from activities directed toward building their businesses. Consequently the data derived fiom their record keeping can only be considered approximate, although the project staff strove mightily to keep it accurate.'" Secondary weaknesses had to do with misunderstandings occasioned by required signage: "PC3" in the cyrillic alphabet reads as "RSZ," an acronym for a Bulgarian Labor Minktry service. Thus the PC3 operators were forced to do much local explaining at first, although in at least one town the acronym as "PC3" had achieved a valuable cachet due to the value locally accorded the telecenter. The USAID handshake sign, understood everywhere as denoting an American project, when displayed prominently at the PC3 telecenter entrance was often interpreted by PC3 clients to mean that the PC3 telecenter would go out of business when the project ended. For this reason, some PC3 telecenters de-emphasized the USAID relationship andlor retained their own business name and logo in their external signage. Lessons Learned Two missing inputs made the project harder to manage than was necessary. For telecenters, or indeed any information service project, the following ought to be considered critical inputs. The up-front expense of customizing one of the many available software packages for tracking computer use and accounting for users is likely to be well worth it. In fact, IT analysts have '' In fad. the AEDlSofia Dmiect office subcontracted with a Buharian develowr of an linternet dub trackina sheU to Wornhe for PC3 telecentir data capture neeis. He brought ii to the point whereall PC35 can use 1 for computer -nty and use tradwg, and then rvas drafted into the Bulgarian Army. Its repomng fundions remain undeveloped except in hvo cases where the PC3 twnter operator independently deveioped lhe<hough asyslern for tracking users, as tiind from mrnputer use, remains absent demonstrated the cost-effectiveness of this approach to deployment so many times that it should not be neglected.I5 The up-front expense of ensuring quality Internet connectivity using the best technologies suitable for the region should be a critical input of a telecenter project that is expected to deliver Internet￾based services. Project Direction and Management The AEDISofia staff comprised an IT specialist for IT training, help desk support, and online management of data; a business specialist for both managing project funds and delivering business skills training; and a receptionisthterpreter who did everything else. They were headed by AEDISofia's Country Director, Dr. Iliana Nikolova, a specialist of international repute in computer￾based instructional design on the informatics faculty at Sofia University. Her management of the project thus tended to bring to bear the underlying education and training skills and contacts of her specialty, with extremely beneficial results. Her policies and practices in three areas besides the training program were particularly important to the professional growth of the PC3 operators: Communication policies and practices Promoted a culture of sharing, mutual support, and problem solving - a generally unprecedented approach in Bulgaria. As a result the PC3 operators became a cohesive group. By project's end, the concept of an umbrella organization to continue the PC3 association emerged spontaneously &om the PC3 operators. Facilitated intra-project communication in every way possible. Specific tools and techniques included: a PC3 Web site at htt~://~c3.orbitel.bg with a private area for the operators, a group email address, a shared area for each of the training sessions, contact information, and monthly reports. Ensured quick office capture of and response to PC3 operators' requests and needs by keeping an answering - machine on 24 hours a dav and 7 davs a week. circulating all - staff members' mobile telephone numbers, and monitoring email daily. This service-oriented approach was such a new experience for the PC3 operators that they came to depend on it, perhaps to excess. Used all available promotional tools to accord the project participants and premises a visual identity: colors, logos, signage, Web site design, and large splashy opening ceremonies with official guests, priestly sprinkles, and giveaway prepaid cards. An important and valued result was enhanced local prestige for the PC3 telecenter's local prestige. Incentive structures Signing subcontracts was made concurrent with the important first training session. 'SE~pe~ially the Gartner Group, hUp://www.gartnerqroup.com , a leader in the field 8 Reimbursement of used prepaid card coupons was dependent on receiving them correctly completed with narrative monthly reports. Additional program and use data was elicited fiom the operators by handing out questionnaires at the beginning of a training session and collecting them at the end. Training session activities were used to encourage and support PC3 operators in sharing problems and finding ways to solve them, which they would not otherwise have done. In consequence, they learned to rely on each other as well as on the AEDISofia office staff. Future positioning for the PC3 telecenters PC3 operators were required to prepare action plans for their centers as part of their Hungary Telecottage Study Tour activity, followed by formal business plans under the supervision of the project business specialist, Vladimir Ivanov. Most operators later admitted the effort was worth it in terms of knowledge gained about their own businesses. In order to meet the project requirement to develop Bulgarian e-contenf Dr. Nikolova produced bstmctional and reference tools in CD-ROM and online versions that would support senice projection by the PC3 telecenters for some time to come. PC3 Operator Selection AED used a rank scoring method to evaluate he respondents' proposals to the RFP, rating them on the following elements: Ability to provide the human, financial, and material contributions stated in the proposal Demonstrated understanding of a telecenter approach to providing public access to ICT senices, products, and training Viable business plan for the proposed PC3 telecenter Characteristics of the town proposed: population between 5,000 and 30,000, lacking a similar publicly accessible commercial enterprise, and having some USAID or other donor program in place. A first cut of 25 applications were Wer evaluated in site visits during which time the proposal information was confirmed (or not). The respondents furthermore were assessed for such qualities as anticipated level of involvement in PC3 operations, related previous experience, flexibility, pmpsed premises for the PC3, and other pertinent features, including sanitary toilet facilities. The ten finalists selected comprised 5 married couples already in business offering computer services, aged in their 30s, 40s, and 50s; university-educated during the Communist period, usually in computer science, economics, or teaching. The other five were much different: one municipal government professional with experience in USAID's Local Government Initiative @GI) program, one a founder of the first telecenter in Bulgaria, another the headmistress of a secondary school where the PC3 was to be sited, and two extremely quick, ambitious, and hard-working young men in their late 20s. Most town populations are in the 8,000 - 12,000 range, with the smallest at 5,000 and largest at about 26,000. Lessons Learned The painstaking care during the selection process paid off in the high quality of the people selected, their eagerness to learn, and their determination to survive in the new market economy. This is a significant contribution to the project achievements. All the PC3 operators selected had demonstrated intelligence and initiative. Additional common personality characteristics of the Bulgarian PC3 operators include a tendency to be self-directed, optimistic, able to cultivate relationships with local organizations by successfully sailing through small-town political winds, and acutely observant of users' behavior. Some PC3 operators did not choose staff well, however. Some relied on "hiring," without salary, the unemployed who were receiving benefits from the local Ministry of Labor office, or on assigning employees hired for other purposes in another activity to part-time telecenter work. One or two others chose staff with no apparent growth potential or no apparent effort to train and develop them. These circumstances no doubt have a cultural basis and imply no blame. But our on-site evaluation interviews gave the sense that it is the family enterprises that work best because the teamwork comes naturally. Equipment specification, acquisition, and delivery Systems training Since almost none of the PC3 operators had experience with current models of desktop computers and local area networking, the systems administration training session was conducted before equipment delivery occurred. The session agenda illustrates how basic it was.I6 The instructor's report noted that he had to accommodate a very wide range of skills in the students during the instruction, but noted also that the beginners made remarkable progress. The purchase contracts also included vendor delivery to each receiving PC3 and installation there by the PC3 project IT specialist from AED/Sofia to ensure that all went well. Equipment Troubles Brand specifications were worked out in conferring with the PC3 operators, and the specifications were put out for bids locally. U.S. branded equipment could be purchased from subsidiaries in Europe or Bulgaria. Obtaining the waiver for this occasioned a two-month delay, unfortunate given the short project duration. The computers were much appreciated by all. Unfortunately, the U.S. brand photocopy machine (xerox) proved to be a poor choice. The ink cartridge replacements cost over a month's professional wage ($200) with the result that several PC3s had to price photocopying too high for their local markets. Some of the machines were temperamental, troublesome, and finally broke down completely. Most PC3 operators expressed frustration and annoyance over them. Reliable Japanese brand desktop copy machines and ink cartridges are now locally available at prices reasonable for Bulgarian incomes, and high volume copy machines are becoming available in dedicated office supply stores and even bookshops. - - Agendas in Annex 14 Lessons Learned The fluid transitional current state of the Bulgarian economy underlies the unanticipated switch to wireless Internet co~ections from dial-up, and the rapid changes in the photocopying business in Bulgaria. Though both negatively affected the PC3s in ways that might seem minor in the overall scheme of things, they affected them directly and individually. For businesses struggling to survive in a poor town, every penny, every stutinki, counts. In this project, the photocopiers might better have been an item for the PC3 operators to contribute, if they wished. Finally, the equipment mix established for the PC3 telecenters at project start-up assumed that the dial up Internet connection available to most at the time would continue. As it happened, the initial project training session introduced the PC3 operators to wireless technologies - less expensive and more reliable than dial-up. Within months, most had switched to wireless connectivity, making the project-supplied modems superfluous. It soon was apparent that a fast, reliable Internet connection was a significant comparative advantage for PC3s competing against other providers. Some wished that more attention had been given to ensuring a quality Internet connection for all at start-up. It might be argued that this circumstance simply reflected rapid changes in Bulgaria during the project period. According to a recent CDT study, "The group of technologies that have now emerged under the generic umbrella of '%less local loop" (WLL) may be particularly successful. Fey] offer special promise for llfilling the universal service obligation ... WLL offers a less expensive alternative [to fixed-line installment in first-time user networks] ... that avoids many logistical hurdles, hm municipal permission to ... mountains and rivers."" Prepaid cardcoupon procedures Purposes The PC3 Project provided a limited number of prepaid cards to the PC3 operators for distribution in their communities in order to Dromote the activities of the centers and to heh create a new clientele a quickly, thus reducing the initial business risk for the PC3 operators. As a form of subsidized access to the products and services offered by the PC3 telecenters, the prepaid cards were intended to stimulate public usage of the PC3 centers and to promote information technologies skills enhancement. This they did. The description of the PC3 Prepaid Cards Distribution and Administration scheme illushates the exhaustive manual detail made necessary by the decision not to automate the process.'* The data gathered this way does, however, suggest the scope of use by coupon users. This don summarizes the data collected on card distribution and coupon use, and the lessons learned fiom using this tool. Coupon Users Figures 1 and 2 show students and youth as the principal coupon users and the Internet as the service most heavily used. Times of use peak in the afternoon after school. Student and youth training " Center for Development & Technology. Bridging the Digital Divide: Internet Aasss in C%nbal& Eastern Europe. Match M02 h~p:lhmrw.cdt.oq/intemationaVceeacae&rewrt.shtml '= See Annex 10. PC3Tekcenter Prepaid Cards Distribution and Administration 11 usually consisted of the basic "Introduction to Computers" course, after which they came mainly for online chat and web surfing. "Other" services and users indicated on the graphs, as specified by the operators in Figure 3, illustrate the diversification of both during the project period. The Club Economica's three-day observation survey of users conducted near the end of the project (April 11-13,2002) reveals further diversification of services in Figure 4. Training was mainly for children (probably Comenius Logo classes); youth interest in games is evident; and there is greater use of computer applications. New services have emerged: computer services (i.e. hardware) used by adults, a "minibar" (snacks and sodas) for the kids, and considerable use of IP telephony ("phone" label). Paying Users Since the monitoring and evaluation scheme did not capture data on paying clients, there is no trend data to show growth in this side of the business. However, Figure 5 illustrating Paying Users at PC3s shows the results of two snapshots: one taken in a special data gathering exercise for one month only (15 February - 15 March, 2002), and the other during Club Economica's 3-day observation survey in April. In the first, coupon users still predominate, with only one exception - Aitos, where youth services predominate (i.e. chat, Web surfing, and games). Also, PC3 clients were still encouraged to use up coupons before the project ended. During the 3 days in April, however, most users paid in cash at all PC3s. This would suggest that they all have successllly established a paying customer base. Prepaid card distribution Distribution and management of the prepaid cards has yielded most of the lessons learned from this project - lessons applicable whether the cards are tracked manually or electronically. Project design envisioned that the AED/Sofia ofice would distribute the cards to designated target groups in each town. Project implementation changed this to a more workable process whereby each PC3 operator chose a board of directors fiom among local leaders and heads of target organizations such as schools. Together each developed a list of prospective card recipients for review and approval by AEDISofia. Most of the cards allocated to each PC3 were given to the operators when the PC3 opened, and a second allocation of cards four months later. Distribution Effects Few PC3s still had a board active in decision making at the time of the evaluation. But most had former board members who remained active users interested in the PC3 well-being. The long term effect of board involvement was the strengthening of connections between the PC3 and the important local organizations who became supporters and c~stomers.'~ 0 PC3s who distributed their prepaid card allotment all at once created a demand they couldn't meet: too many coupon users appeared at one time. This was worse for business when a paying customer found all computers occupied by coupon users. 0 An opposite effect could also occur from too-ready distribution: people would take a card just because it was a gift and never return. This meant lost potential income to the PC3 operators, - -- '*See Annex 1 I for the PC3 telecenter operators' lists of their board members, and A12 for their lists of institutional partners. 12 who made considerable efforts to get them back. Children and students, the most enthusiastic early adopters, would tell their parents who would then become intere~ted.~' Parents with children abroad soon learned to stay in touch with them by email. Some prepaid cards were used deliberately as a kind of "loss leader." "We gave them to those we knew could pay after they learned," said one PC3 operator. In some towns this was interpreted as favoritism and occasioned an envious response from businesses who did not get cards, or disapproval from municipal officials who do not condone favoring one business over another. Coupons were used mostly for IT training since most users had little or no experience with a computer, let alone the Intemet. PC3 operators deemed training se~ces the best use of coupons since they contributed directly to building a clientele. The 10 coupons in each card were enough for the introductory courses. Prepaid cards helped overcome Bulgarian small town suspicion of anything new, according to several of the PC3 operators Prepaid card management Cards lost or unused represent lost income to the PC3s. Those, who learned to manage their cards tightly, distribute them in scheduled increments, and specify them for training or only Internet use, did better with them. Annex 7 on PC3 Performances 9/01-3102 compares cards reimbursed (used) and cards lost; i.e. given to the PC3 but not reimbursed and including possibly cards not yet distributed. It shows that the aeatest level of reimbursements was in the three wrest PC3 towns (Biala, Madan, Omurtag) where the prospects of paying customers seemed lessto the operators. The PC3 with the highest number of unused cards (Varshec) is based in a school and run part time by unpaid school staff. One supposes that administrative attention there is inconsistent if not entirely absent. Prepaid card quantities The number of cards to be allocated to each PC3 was based on town population in 10,000 in~rernents.~' This meant that the town with 9,000 could get reimbursed up to $1500 while the town with 11,000 could get reimbursed up to $3000. Also, the largest town (Aitos at 26,000) got the most cards, rather resented by the PC3 operators in the smallest and poorest towns who felt they had to work harder to build their businesses. Another sore point: the least experienced PC3 operator opined that the lack of experience deserved more prepaid cards! On the other hand, an experienced operator, although satisfied with the number of cards he received, opined that too many cards removed the incentive to build the business. Children being as they are, their prepaid cards muld easily be lost, tom, or end up in the washing machine. One mother tore up her son's prepaid card as a punishment for getting into trouble at schwl. The Technical Annex of each submntrad gives a reimbursement level of 51500 for tms under 10.000.53000 for towm over 10.0a). and $6000 for the one town over 20.000. Developed at Comenus Untventfy. BraUslavq m Ule late 19705 Emrn MlTi Logo cnvrmnmcnt for icachtng chsldrcn roR*8c programmmg Comrntus Logo spread Ulmugh the Slovak school system dunng the 19805 In 1994, Dr. ll~ana Nikolova had headed a One of the experienced PC3 operators, who believes that training is the most important prepaid card use, recommends assigning different values to the cards: one to cover a whole training course, and another to cover other services. Two particularly thoughtful and experienced PC3 operators, who both think the 10,000 population increment unfair, suggested different formulae for determining prepaid card quantity based on 5,000 or even 2,000 population increments plus a factor for the local unemployment rate, school age population, and number of small businesses in the region. Lessons Learned Prepaid cards are a powerful tool for creating direction in small businesses in poor environments. Consequently it would be useM to give more careful attention to planning their distribution rules than was the case with the PC3 project. They not only mitigate business financial risk but also build a customer base by encouraging customer loyalty and developing customer skills, not to mention being a good promotional tool. Because they affect business behavior and operator's decision making, they might be used to direct services toward specific beneficiary groups. PC3 OPERATOR TRAINING Program The purpose, logistics, and lessons learned from the training program created during the project in response to PC3 operator training needs is detailed in Annex 13: PC3 Operator Training Program, and the full panoply of session agendas is given in Annex 14: PC3 Operator Training Agendas. The project design assumption that two or three training sessions would suffice was a serious underestimation. Sessions on business and systems management are to be expected, but additional sessions were developed with a view to enhancing not only the operators' skills in several ways but also the types of services they could offer if they chose. A second business and financial management session was added after important changes in Bulgarian tax legislation. All sessions were on immediately practical topics calculated to help the PC3s succeed. Costs stayed down because Bulgarian facilities, lecturers, and trainers were employed. Because all the operators lacked training experience and in order to strengthen their abilities to provide training services, two sessions were added that relied on a train-the-trainer perspective. These two train-the-trainer sessions devoted to learning MS Office applications and IT for children using Comenius Logoz2 had the strongest effect on those PC3 operators already interested in training. Certainly the tool provided them the opportunity to introduce a new range of services to a new user group: young children. In the municipalities where the PC3 developed a Logo program in cooperation with local teachers, it has become very popular with children and well supported by parents. project to produce a Bulgarian version, with sponsorship from IBMIBulgaria, Soros FoundationISofia, and the Bulgarian Ministry of Education. See http://www-iea.fmi.uni-sofia.bqlvaluelour proi.htm The program was piloted in several Bulgarian schools, none in a PC3 telecenter municipality. As word spreads, it is good for Comenius Logo to be associated with the PC3s in peoples' minds. All the trainers who use it expressed a wish for further training themselves in order to better use and teach the program activities. Annex 15 : Summary of training held in all PC3 telecenters through March 2002 14 Participant Training After the required first training session where subcontracts were signed, the PC3 operators were free to select and send the most appropriate staff member to training. The PC3 project staff strongly advised the center operators to select carefully among prospective long-term staff. Out of 45 trainees, only four were subsequently thought inappropriate choices, because they had displayed indifference in the classroom, or were subsequently dismissed ffom the PC3 staff. Two PC3 operators who made two of these unfortunate choices were also the only two who indicated, in the training needs assessment, a need for a preliminary selection of trainees by the PC3 Sofia project staff. All others strongly affirmed that each PC3 should choose trainees themselves. On the one hand, trainee selection is a supervisory responsibility, but on the other hand the less experienced supe~SfJ1~ may have needed more counseling in human resources. Staff development is an area of concern that might be advisable to include explicitly in business management training. All post-training participant evaluations were positive, most stating "more than I expected." From post-session comments and during evaluation interviews, it was reported that the strongest effects were ffom learning how to manage a telecenter business and the valued contacts made with presenters at the training sessions. Also, there has been a large multiplier effect in the provision of PC3 services, and over 4,000 PC3 clients have been trained in a variety of computer uses.u As Bulgarian society values formal certificates and certifications, requests have emerged for a PC3 certification process, which would - accord a higher status and employability to those who had completed such abrocess. This is important to consider in the future. Lessons Learned During evaluation inte~ews, it was noted that few project-funded trainees seemed to be passing on skills learned to other PC3 staff, nor did a sense of teamwork seem apparent unless the PC3 was, in effect, a family business. These circumstances, no doubt a matter of culture and attitude at present in Bulgaria's transitional state, might have minimized the ripple effect that the training sessions could have had. There are delightful examples that illustrate the opposite effect, however. One PC3 operator does supervise effectively a trainer by reviewing written lesson plans before each class. The student then became interested, which resulted in creation of a student manual, based on lesson plans. Nevertheless in the future, training in small business management might incorporate explicit training in team building and staff development concepts and practices, especially in cultures where these skills tend to be ~dimentary or even absent. CONTENT DWELOPMENT AED's initial implementation plan intended to "convert existing information resources and training materials broadly relevant to the economic and social development of under-served communities into Bulgarian language computer-based and Internet accessible formats." In view of what is now known about the development of information idiastmctures and skills, this objective was quite irrelevant. 15 For one thing, poor people in developing areas need and use local information more than they need and use existing information from an alien context.24 For another, information seeking and using skills do not just happen to people suddenly exposed to more information than they ever knew existed. AEDlSofia resolved the dilemma and met the requirement by creating five products for client support and self-instruction for PC3 telecenters to use. Content Development Process A structured process elicited needs from the PC3 operators and exposed them to potential content providers among government, donor, commercial, university, and NGO entities. A preliminary list of potential products was narrowed to a final list of five products. Two already existed: the Logo software translated into Bulgarian, and an online Internet course the AEDISofia Project Director had previously collected from university student projects, edited for the PC3s, developed into a 300-page commercial product for two groups (beginning and advanced), and published as a CD-ROM. Two reference resources were also produced on CD-ROM: a "PC3 How to...?" information resource (a sort of micro-encyclopedia) with components written by various experts, and a "PC3 Internet Catalogue" (a bookmark or favorites list). Finally, the project business specialist, Vladimir Ivanov, contributed a print version of the small business training course he had been giving for years, somewhat edited for PC3 use. There was not enough time to digitize it properly before the project ended.25 Lessons Learned Concurrently with the PC3 project, an Internet information infrastructure in Bulgaria was developing rapidly: Bulgarian government, business, and NGOs were climbing onto the Web with their own information in the Bulgarian language, and several Bulgarian portal sites emerged (examples include httu:l/b~oven.bg ). At least two print directories to Bulgarian Web sites were published in 2001 although only two of the PC3 operators, who discovered this, purchased them for client use; they were not, oddly, given to each one as part of the project package of books and software. One NGO even launched a project to digitize Bulgarian literature (htto:Nslovo.orbitel.brr, ). These developments are following a normal course of Internet penetration as different sectors of activity incorporate Internet tools in their normal operations. Development of national language content parallels For a review of this subject, see Richard Heeks, 'Information and Communication Technologies, Poverty and Development," No. 5 in the Develo~ment Infomatics Working Pacer Series. Institute for Development Policy and Management, University of Manchester. June, ~-~., "Described in Annex 16 : PC3 Telecenter Content Development Formal Process. See also Annex 17 : PC3 telecenter How to ? Reference Topics. Annex 18 . PC3 Teecenter Catalogue topic structure, and Annex 19. Content Valued by PC3 Telecenter Clients development of organizations and institutions with useful information to communicate electronically, especially information relevant to economic activity: "If less developed countries and regions are to implement telecommunication networks and information services that will serve their interests in local and regional economic development, the new communication systems must promote local and regional communication and information networks within the context of the paaicular economic, social, political, and cultural institutions of the region. The new systems must increase the incentive to look first inside the region for economic activity, before going outside.'"" AEDISofia's decision to create Bulgarian language products specifically for the support and training of PC3 operators and their clients was therefore consistent with the approach of project management, relevant and appropriate. During the project period, the appearance of municipal LANs, populated with local content in Bulgarian, could also be considered a project effect that met the requirement for producing digitized Bulgarian content. The information search needs expressed by the PC3 operators both during product development and the evaluation interviews (Annex 19) offer a realistic picture of real-world information needs among - the literate poor struggling to survive in a new market-oriented economy. The important thing to notice here is the immediate life purposes that usually underlie information seeking. OPERATOR INTERVIEWS Most of the PC3 operators, asked what they would most like to learn fiom the evaluation process, replied that they'd like to know how they are doing in comparison to each other. While it is true that information service performance can in theory be evaluated, comparative performance data on the ten PC3s is too sketchy and unreliable to be really useful for this. We cannot compare increases in the proportion of paying customers to coupon users in the absence of that data. Indicators for telecenter service performance can include such negatives as the amount of down time due to equipment or connectivity failures, and such positives as the numbers of people served or a William H. Melody, in The Information Society: The transnational earnomic cantext and its implications' in Tranvlationcll Communications, pp. 3940. numbers of service requests successfully answered by the telecenter staff.27 Figure 7: PC3 Performances 9/01 - 3/02 represents an attempt to pull together available data that might suggest the quality of PC3 management (total service events, total training service class hours, and prepaid cards reimbursed and lost) and explore any possible relationships. Surprisingly, there seems to be no relationship between the amount of time spent training ahd the number of service events. On the other hand, the curves representing percent of cards reimbursed and increases in types (variety) of services offered run roughly pardleiover half the distance across the chart. But thev have no amarent relationship to the number of service events either. Figure 6: a - PC3 Training Services Performance suggest a pretty inefficient ratio of trainees (few) to instruction hours (many). Since most training pertained to the "introduction to computers" course, simple inexperience may lie behind this observation CLUB ECONOMICA 2000 SURVEY DATA Useful studies of information behavior published in the past decade or so have concentrated on the complicated social contexts within which the information behaviors take place. Not only is information exchange a social phenomenon, but so are information seeking, information use, and perceptions of information value, all of which are embedded in a social milieu?' Consequently, measuring impact - whether that of one small town telecenter or the Internet in the entire developing country - is quite imp~ssible?~ It might be necessary to inquire about effects over time. These considerations -the lack of reliable data, the social aspects of information behavior, plus the short time frame of the evaluation - resulted in a decision to have Club Economica 2000 staff conduct two sociological opinion surveys of 100 PC3 center users (1 0 at each) and 200 neighbors (20 in each town), and interview the mayors of the towns. The principal survey findings are: 1. High community awareness of the PC3 telecenters, expressed both by neighbors and Mayors. 27 For a thorough review of telecenter evaluation research indicators, see Heather E. Hudson, 'Designing Research for Telecenter Evaluation" - ~ ~~~ which resulted from IDRC's Acacia P:roaram of telecenter research in Africa. For a common s&e revlew of the issue, see ~ichel J.Menou. 'Impact of the Internet: some conceptual and methodological issues, or how lo hit a moving target behind the smoke screen,' 1999. http:/lwww.1drc.c~telecentre/evaluallonlAW24 lrnp.Ddf " For the sdooortino amuments. see John A. Dalv. 'Measurina ImDacts of the lntemet in the Developing World,' 1999, on the Web at 2. Generally positive expectations. Interviewees believed that the PC3 telecenter would continue in business because "computers are the futuree' (both users and neighbors), there are good quality services (users), and "the town needs the center" (neighbors). Moreover, both users and neighbors expect to continue using PC3 telecenter services even after distribution of free coupons ends and even when many have a computer at home. 3. Predominance of repeat users. 58% had been using the services for up to a year, and 77% came every day or several times a week. The PC3s have loyal clients. 4. Different reasons for coming to the PC3 telecenter and different perceptions of its importance. Users consider information and knowledge the prime reasons for coming, with work, relaxation, gaining skills, and entertainment not far behind. Interviewed neighbors, who were all young, emphasized entertainment and training. AU ages, however, ranked a "link with the world" high. 5. Lack of ideas among users and neighbors about potential new PC3 services, and unifordy high value placed on the quality of the services and professionalism of the staff. The survey investigated the perceived ratio of price to in PC3 services, and found that consultations were rated significantly higher in quality than in price. 6. Positive attitudes of Mayors toward all the PC3s, particularly when their activities are perceived as socially beneficial orimportant to the municipality. One PC3 organized charitable activities; another provided educational services. Mayors were most supportive in municipalities where the PC3 offered customized services such as stafftraining or Web site development. Sustainability Findings Club Economics 2000's three-day sociological survey found indications of sustainability of the PC3 telecenters in their stable client base and the generally positive regard accorded them from both citizens and officials in their municipalities. These are important advantages that the PC3 telecenters - should work to capitalize on in their future strategic planning. One way some PC3s might do this is to establish cooperation agreements with important local organizations. This could transform their present non-market advantage into a future competitive advantage in the local marketplace. In eight of the towns, half or more users and neighbors intend to continue using PC3 services after the prepaid card subsidy ends. It should be understood that this positive flavor might be related to the specific population of respondents, who considered themselves economically average or above average for their town. Therefore, the current pricing structure is affordable, and the PC3 telecenters should keep their prices in this range. "When the delivery of free coupons ends, are you going to use again the center services?" (% of 300 respondents) itos iala 60 ~ ~ 30 10 Effects of Business Conditions Vetren Vatshets Zlatomad - . - - Omurtag Pravets Triavna The PC3 telecenters occupied an open market niche when they were established. This had two major effects. First, the USAID prepaid card subsidy was not seen as favoring one business over a similar business in the same town, and, second, the PC3 telecenters pioneered quality Internet access in their municipalities. This in turn created consumer demand, especially where the PC3 telecenter is an ISP and/or provides hardware troubleshooting services to computer games clubs that might otherwise be seen as their competitors. Competition 70 50 80 . . 50 40 50 Competition is considered the principal business risk for some telecenters," curiously enough most feared by those with strong managers. Understandably, poverty and the low purchasing power of the citizenry are risk factors cited by the PC3s located in the poorest regions. According to the Club Economica 2000, "In the short-term the competitive environment will not threaten the existence of the PC3 centers as a whole. It would, however, force some of them to give up the rapid development of services related to Internet provision. In the long-run the competition regarding the Internet provision will intensify, and pressure especially on small companies like the PC3 centers will increase, so that they will have to look for their specific position on this market." 0 0 0 0 0 10 'See the data table in Figure 11 30 50 20 50 60 40 100 100 100 100 100 100 Business Plans The Club Economica 2000 also evaluated the PC3 telecenter's recent business plans as another task in their scope of work. They found that the business plans were generally not based on a sound, thought-out strategy and were too product-focused, instead of client-focused. These conditions are attributable to inexperience and lack of practice due to the short duration of the project. Nevertheless, the acquisition of business skills gained fiom project training made PC3 operators aware of the needs for continuous monitoring of results and for regular planning in their businesses. Future business skills training is recommended, however, in order to strengthen the PC3 telecenter operators' abiities to plan strategically. Finally, all are urged to develop client-oriented business plans and express them in their business strategies. PC3 Incomes Comparison of fist and last project month incomes for all PC3s shows an overall increase in income of 267%. Additionally, in the last month there is a higher increase in income from non-Internet services, and a 17% drop in Internet services income, suggesting further diversification of services?' PC3 Services Nevertheless, the Club Economica 2000's observations in the PC3 telecenters and interviews noted that Internet use far exceeded any other service, at least during the three-day period, as the table below shows. Also, although users and neighbors could not think of any additional new, useful services when asked, their short exposure to ICTs makes it unlikely that they could. Telecenter research has demonstrated that the poor will seek and even pay for information content they believe is important to them. Therefore, new services should be based on study and understanding of the information needs the client populations experience in the course of daily living. From this knowledge, a client-oriented business plan emerges. Consequently, rather than increasing diversification of services, the Club Economica 2000 recommends the opposite - strategic senice specialization according to the primary information needs of the local customer base?' See Fiaures 7.8. and 9 for PC3 teleoenter inmme data aathered bv the Club Emnomica 2000 club ~&nomica'2000 also cautions that tivenifying PC~ telecenth services can have serious tax implications in Bulgaria, so does the form of ownership. These cautions are detailed in Annex 20. Skill Evolution among Users Club Economica 2000 interviews sought to identify local reasons for using PC3 services. Figure 10 summarizes the percent of each response. "Information", "knowledge", and "I'm working" are at the 80% - 90% level. "Relaxation" and "entertainment" are not far behind at around 75%. This is, in fact, normal and to be expected. Turkish and Pakistani Internet users displayed the same behavior: "For all the differences, [between Turkey and Pakistan] one of the striking similarities is the eagerness of many citizens to begin using the Internet, once senrice is available at reasonable costs. In both countries, hundreds of thousands of people are drawn to the Intemet for socializing and entertainment, in the first place, and information in the second.'"13 The PC3 project design seems to have assumed that PC3 clients who became Intemet users would naturally become information users. This simply does not happen. New information use skills take time to develop, usually in roughly three phases as observed by the evaluator: Phase I: Discovery of a new communication and entertainment medium. Hence, PC3 clients come to use chat, email, entertainment surfing, and computer games. They practice eye-hand coordination and strengthen visual perception and navigation skills. New Bulgarian users are somewhere in Phase I, just as new Turkish and Pakistani users are. Phase 11: Tool-using skills develop by discovering remote sites that are useful, using instructional and content-bearing media, and learning useful applications. With practice, an awareness of different information structures and resources applicable to different purposes begins to develop. PC3 operators seem generally to be somewhere in Phase 11. Phase 111: Deliberately develop content seeking and management skills for downstream and extended purposes, clients, or products. These are the skills of a researcher with experience in any kind of profession or discipline. It is simply too much to Wolcoit 8 Goodman, The Internet in Tudiey and Pakistan: A Comparative Analysis, p. 115. expect that PC3 telecenter clients reach Phase III in less than six months of the project duration. Computer use Computer use in the PC3 telecenters ranges between 3 and 9 hours per day. Clients generally spend one hour, the value of one coupon, or paying one lev (roughly $0.50). Peak use tends to be during after-school hours, as we have seen earlier, and the preponderance of users are students. For this reason, the Club Economica 2000 also recommends that some of the PC3 telecenters not only strengthen training programs in IT skills but consider developing strategic relationships with the national formal education system, especially as the Ministry of Education incorporates IT education in the standard curriculum. Table C Operator Characteristics Indepth interviews at each site revealed that it was the personality and attitudes of the PC3 operator and the quality of local organization relationships with the PC3 that seemed to be an important determinant of PC3 success in sustaining itself. An active, social personality, a positive, upbeat attitude, and setting with a calm, orderly, and professional tone all made a real difference. Finally, the demonstrated ability to accept and exercise personal responsibility for outcomes was a key characteristic among successful PC3 operators. ASSUMPTIONS AND OUTCOMES The project design assumed that the successful PC3 would somehow catalyze economic development in its town. What actually happens is that telecenters support the business communication needs of a broadly developing SME sector; they rise together. This is now happening in some, but not all, of the PC3 municipalities. the PC3 telecenters would build their businesses on public access services to individuals. In fact, the PC3 telecenters that seem to be doing best have learned to strengthen organizational relationships with town offices, libraries, schools, and other institutional sources of potential business. the successful PC3 telecenter would spin off new small businesses, particularly desktop publishing.'4 What actually happened was that putting Intemet connectivity in place drew new ISP competitors to the region as well as new computer games clubs, some also offering Internet access. PC3 operator response to competition differed. Some felt competition was "unfair," while others took it as a challenge to be fearlessly met or a problem to be anxiously solved. appropriate use of the prepaid cards would bring ICT services to the most disadvantaged or marginalized populations. This did not happen without deliberate action by a PC3 operator to reach them. What the cards actually did was to bring basic computer and Internet skills to over 4,000 people who had never before put finger to keyboard. d PC3 operators would quickly become skilled Intemet searchers along the lines of a professional reference librarian, conducting information searches for clients. This has not happened yet, as indicated by the need for the reference tools developed by AEDISofia. FUTUREPROSPECTS Probable future models for individual PC3s Each of the ten PC3 telecenters shows signs of moving toward one or another of the following models, although they are by no means mutually exclusive: %But no DTP software was included in the granted equipment. In Bulgaria at present, MS Word meets document design needs.everywhere beyond Sofia. Model 1: Model 2: Model 3: Model 4: Model 5: Model 5: Multi-service telecenter for a small customer base, like a village telecottage, surviving by meeting a broad variety of customer needs. One of the PC3 operators, inspired by the study tour of Hungarian telecottages, envisioned building a chain of telecottages in Bulgaria to serve towns with populations under 4,000. Only 20% of Hungarian telecottage income, however, comes from local services. The rest is a national government subsidy. Training center for IT skills (adults and children), foreign language learning, vocational retraining. Developing in this direction successfully would take establishing contractual - - relationships with the appropriate national ministries, especially education and labor. In addition, a formal certification process would give the PC3 training center . - - an authoritative standing to attract trainees. Designated partner of the municipal government for electronic delivery of social and educational services to the local population. A strong, competent, and involved mayor is definitely a positive factor here. Two or three PC3s have established strong relationships, installed a municipal LAN, trained municipal workers, and others. After the idimhucture development phase is over, however, it will probably be necessary for the PC3 to stake out a role and responsibility for creating and maintaining local content services. Designated partner of local schools, orphanages, and hospitals for computer installation, maintenance, and user training services. In the case of such s~eciakd institutions, development of curricula andkell local content should occur in parallel with infrastructure development. Cybercafe for email, surfing, - snacks, and games. - PC3s located in a tourist area, or among a large population of understimulated boys and unemployed young men, are likely to move in this direction. Available PC3 user data confirms that the young - are the most kequent users, so the fact that several have added snack bars is not &prismg. Node of a national ISP. One PC3 operator was a local ISP representative before assuming his PC3 position. Another may soon become one. In the short term this may offer some stability of income, but in the long term each PC3 must assess what its own strategic place in the ISP market should be. Building National and Donor relationships PC3 relationships with local government (municipal or national) entities and donor programs do not necessarily - mirror those at the national level. Possible local collaborations may be constrained or even prevented by ignorance or indifference at the national level. This applies particularly to donor organizations, which might achieve beneficial cross-fertilization with better coordination, internal and external. For example, the PC3 operators kquently cited the UNDP-funded labor and ~hitalishte~~ development projects, both establishing services quite like those of the PC3s within information centers. ~~ "A Chitalishte is a locallv owned and managed libraw or reading mom. Invented in the XK oentuw during the Ottoman period to mrve Bulgarian wnure and i&ture, chita~tes bere used during the Communist penod to spread a&plabl~socialst ideologies. ~taflwere generally appomted by and from the local 'nomenklatura.' Hence in the present day some chrtatishtes can still retain the dead air of the Communist Past In another example, a USAID program supporting a private growth enterprise or citizen participation initiative might not seize the opportunity to support another local USAID program, even when the PC3 might be a useful tool. Collaborations with umbrella organizations, associations, and NGOs All the PC3 operators want to sustain their collaboration and mutual support systems. They also perceive many benefits in a coordinated approach to identifying and applying for resources, seeking discounted software licenses and connectivity fees, and providing industry standard certification courses such as for CISCO or MOUS at concessionary rates. The present evaluation sees additional key benefits. In view of the conclusion that PC3 operator training is a continuing process, and that human capacity development and small business development are ongoing, an umbrella organization should consider possible activities directed to the following objectives in addition to those aforementioned specifically by the PC3 operators: 1. Maintaining service quality standards of members 2. Operating an ongoing operator training program in systems and business management 3. Coordinating a program for distance education 4. Developing capacity of operators to develop capacity of their staffs and clientele 5. Setting cost of ownership standards for PC3 equipment 6. Developing incentives for providing concessionary service to the under-served or needy 7. Maintaining operator commitment at the beginning of the project 8. Providing legal and financial reference services for the whole group 9. Pioneering e-commerce and business-to-business applications as PC3 services 10.Establishing contacts and negotiating activities with external telecenter associations, e.g. the European Network of Telecenters. There is already a Bulgarian Association of Telecenters, although its program, membership, and means of support are unclear.36 Bulgaria is too small a country to support two telecenter associations, and all appreciate the advantage of larger numbers. The relationship of the two organizations would have to be rationalized in some agreed-upon way to advance this possibility. An experienced external organization development consultant would no doubt be able to help define a single telecenter association that would offer interested parties the greatest benefits from member~hip.~~ Model Replication Requirements Once the project is conceived of as a human capacity and small business development project with a telecenter label, rather than simply as a telecenter project, its applicability to a wide range of social and economic development objectives becomes clearer. As a telecenter functions as a complex communication center, it can conceivably handle whatever communications its Iocal users and user groups need to send or receive electronically. Therefore the lessons learned from the Bulgaria PC3 project design and implementation can be used to inform the design of future development projects that envision using telecenters to deliver project benefits. Six lessons learned include " See http://www.infotel.bg-+elecenter/englactWhtx 37 Possible relevant donor programs to investigate include the World Bank's InfoDev grant program, and the World Bank Institute's Knowledge for Development Partnership. 1. A bidding and careful selection process to identify the best applicants, since telecenter success depends on high quality managers 2. Clear objectives about balance of business development and social senice components; cleat definition of "common good" and "public good" in culturally appropriate terms understood by all 3. Up-front investment in the best, fastest Internet connection available and in automating data capture for monitoring and evaluation purposes 4. A design for a strong and continuing operator training program using in-country resources that can result in useful in-country contacts and relationships among the trainers and beneficiaries 5. A design explicitly to support, first, user communication efforts and interests, followed by user learning and finally infomation use. 6. A culturally-appropriate startup activities plan that brings local prestige to the telecenter There is one group of planning tools not used in the PC3 telecenters project that might be useful in the future. A number of "e-readiness" assessment tools exist that would be useful for planners and by the communities themselves. Several organizations, such as the Markle Foundation, UNDP, Bridges, and HIID, have devised such tools. A complete reference resource list appears on the World Bank's InfoDev site at h~://www.infodev.org/ereadinessImethodolo.h . USAID and AEDReamLink Projecf Outcomes AED/LeamL,iis Bulgarian PC3 project supported by USAID has demonstrated that a small enterprise/ICT business model can succeed where certain economic and social conditions exist, principally in an emergent SME sector and developing Internet connectivity environment. Also, it has been shown that, with appropriate preparation, a structured use of prepaid service vouchers can help build a small business and reach specific user groups with appropriate preparation. Furthermore, created in 20 months, 10 apparently sustainable businesses trained over 4100 Bulgarians in basic computer and Internet skills. The 10 businesses increased income by over 200% in less than six months. Finally, this telecenter businesdse~ce model, as opposed to a wire-it-and-leave-it model, increased Internet penetration in Bulgaria By supporting a project that treated operators as professional equals, the Bulgarian PC3 project transformed their oppohties in a positive way. BIBLIOGRAPHY Benjamin, Peter. "African Experience with Telecenters" in e-OTI:OnTheInternet, October 2000. h~://www.isoc.ore/oti/vrintversion/1OOObeniarninnhtm . Center for Democracy & Technology. Bridging the Digital Divide: Internet Access in Central and Eastern Europe. March 20,2002. httv://www.cdt.org/intemational/ceeaccess/revo~.shtml. Contreras-Budge, Eduardo. "Telecenter Assessment and Evaluation: Components, Guidelines and Tools" AED Ledink internal draft document in progress, April 2000. Daly, John A. "Measuring Impacts of the Internet in the Developing World" in Information Impacts Magazine, May 1999. hrto://www.cisv.orp/imv/mavh 99/dalv/05 99dalv.htm Emberg, Johan. Towards a Framework for Evalujation of Multipurpose Community Telecentre Pilot Projects Implemented by ITU and its Partners. Paper delivered at University of Guelph conference on Partnerships and Participation in Telecommunications for Rural Development: Exploring What Works and Why. 1998. http:llwww.itu.intlITU-D/univ access/telecentres/va~ers/guel~h.html Heeks, Richard. Information and Communications Technologies, Poverty and Development. Development Informatics Working Paper Seris No. 5. University of Manchester, Institute for Development Policy and Management. June, 1999. http://www.man.ac.uk/idvm/vd~m dv.htm#devinf wp International Development Research Centre (IDRC). Telecentre Evaluation : A Global Perspective. [conference papers edited by] Ricardo Gomez and Patrik Hunt, IDRC. 1999. Online index to all papers at httv://www.idrc.ca/telecentre/evaluatio~1 TOC.htrn1 See especially: Hudson, Heather E. "Designing Research for Telecentre Evaluation", http://www/idrc.ca/telecentre/evaluatio~O Des.htm1 and Whyte, Anne. "Understanding the Role of Community Telecentres in Development - A Proposed Approach to Evaluation " htt~://www/idrc.ca/telecentre/eva~uatio~Oa Und.htm Melody William H. "The Information Society: The Transnational Economic Context and its Implications" in Transnational Communications : Wiring the Third World by Gerald Sussman and John A. Lent. Sage Publications, 1991. Menou, Michel J. "Impact of the Internet: some conceptual and methodological issues, or how to hit a moving target behind the smoke screen" Reprinted from Nicholas, D. & Ian Rowlands (eds), The Internet: its impact and evaluation Proceedings of an international forum,.. Aslib,1999. http://www.idrc.ca/telecentre/evaluation/4 Imv.~df National Telephone Cooperative Association (NTCA). Initial Lessons Learned abut Private Sector Participation in Telecentre Development : A Guide for Policy Makers in Developing Appropriate Regulatory Frameworks. NTCA, 2001. httD:Nwww.ntca.ordintlIF~L4LALDdf Ovcharova, Nora. Development of a Monitoring and Evaluation System for the Public Computer and Communication Centers (PC3 telecenters) in Bulgaria. Sofia University thesis, 2001. USAID Center for Economic G~owth, Office of Emerging Markets. Initiative on Internet for Economic Development: Initial Survey of Bulgaria September 19-30,1999. Internal working document. Wolcott, Peter, and Seymour Goodman. The Internet in Turkey and Pakistan: A Comparative Analysis. Stanford University, Center for International Security and Cooperation, December, 2000. httD://mosaic.unomahaedu/TurkPak 2000.Ddf World BankMoDev. Information Infrastructure Indicators, 1990-2010 : Executive Summary and Data on Eastern Europe. Grantee: Pyramid Research, cambridge, MA. 1999. URL httD://www.infodev.org/Droiects/Internet/375~vramidlPvramidSummm.Ddf World Bank Operations Evaluation Department & IFC Operations Evaluation Group. Infomation Infrastructure: The World Bank Group's Experience. By Alain Barby Rafael Dominguez, and William Melody. 2001. htt~://www.ife.ordoeg or httD://www.worldbank.ore/html/oed ANNEX 1: PC3 OPERATORS' DIRECTORY As of March, 2002 ANNEX 2: PC3 IMPORTANT DATES Openings 1 1 September - Pravets 29 September - Apriltsi 13 September - Tryavna 5 October - Madan 14 September - Byala 19 October - Vetren 27 September - Aitos 9 November - Zlatograd 28 September - Omurtag 26 November - Varshets Training Sessions 1. Initial Training for the PC3 Operators and Managers 6-10 June 2001, Vitosha, Hotel "Kremikovtsi" 2. Specialized Training for Trainers for the PC3 Centers 12-20 August 2001, Hotel "Orlitsa", Pamporovo 3. System Administration Training for the PC3 Operators (Train of Trainers) 15-1 8 November 2001, PC3 Telecenter, Apriltsi 4. "IT in Education" Training 29 November-2 December 2001, Private Language School, Sofia 5. Web Design Training 21-26 January, 200, Center for Information Society Technologies, Sofia 6. PC3 Management Training 20-24 February, 2002, Plovdiv 7. Third Country Training- Hungarian telecottages Study Tour 17-28 March, 2002 Conferences, Workshops (formal project presentations) PC3 Pre-bid Conference, Sofia, February, 2001 PC3 Content Workshop, 20 December 2001, American Center, Sofia First Tuesday, @AEF and USAID event), American Center, Sofia, February, 2002 Informal Meetings Meeting with CRS Country Director Meeting with UNDP/Chitalishte Project Meeting with UNDP/JOBS Project Meeting with representatives fiom the Ministry of Transport and Communications ANNEX 3: SERVICES OFFERED BY THE PC3 TELECENTERS lnfonafion Source: Information Collection Sheets filled in by the PC3 operators at the PC3 Management Issues Training - February 2002 Pricelists, provided by the PC3 operators with their monthly reports PC3 Telecenter Aitos 4priltzi Byala Vetren Varshetz Initially Offered Sewices JDuring the First Month afferopening) - Internet - E-mail - IRC - Internet and e-mail - Desktop publishing, printing - Recording CDs - Scanning pictures and documents, and their processing - Copying, fax - Telephone - Recording TV programs on CDs - VHS games - Creating databases - to ring files and documents (free) - Internet access, e-mail - Access to computer applications - Recording CDs - Training of children and adults -Printing, copying, scanning - Designing Web sites - Software and hardware services - Selling diskettes and recording information onto diskettes - Internet, e-mail - Fax - Telephone - Printing, scanning -Copying - Selling diskettes - Recording information frodonto diskettes - Copying - Electronic data recording - Internet - Individual training - Desktop publishing - Priming Currently Offered Sewices (Feb~ary, 2002) All Initial Services (left column) plus: hinting - Scanning - Recordmg CDs - Desktop publishing - Individual and group trainings - Development of a City Lan Network - Dial-up Internet access for fms and home users - DSL Internet access for fms and home users - Logo courses for children - ~esignin~ web sites All Initial Services (left column) plus: - Maintaining and installing computer systems - Selling computers - Maintenance and repair of computers - Preparation of electronic photoalbums All Initial Services (left column) plus: - Processing of information - Preparation of invitations, business cards, ads - Logo courses for children - Dial-up Internet for local clients All Initial Services (left column) plus: - Introduction to Computers courses - Designing Web sites - Desktop publishing - Recording CDs All Initial Services (left column) plus: - Group trainings - Keparation of advertising materials - Scanning - E-mail - Fax - Internet - Computer processing - Copying - Preparation of business plans - Fax - Public telephone - Selling computer and office supplies - Binding - Help with the fill-in of various application forms - Translations - BulgarianEnglish, BulgarianRussian, BulgarianIGerman - Language courses - English, Spanish - for children and adults -Preparation of business plans - Preparation of various reports and documents using APIS Public Good Services (offered for free): -Access to juridical literature and laws -Filing individual or business correspondence - Computer magazines - Statistical reference books -Information and consultations about programs and projects of NGOs, schools, businesses Preferential services: - For users with VIP cards -For budget organizations and members of the Center for Development of Zlatograd Municipality - For users with prepaid cards - Trainings - Internet - Desktop publishing - Scanning, copying, printing - Fax - Various services to companies -Preparation of business cards - E-mail and Internet - dial-up connection - Training for adults - copying, printing, scanning - Desktop publishing - Selling diskettes - Advertising materials -Designing Web sites, plus monthly support and update - Work with MIS -Preparation of business plans - Internet access for households . . - Advenisements -Desktop publishing - Trainings - Other office services - Designing Web sites - Videotaping - Recording CDs 4U Initial Services (left column) plus: . Download with a satellite . Recording CDs Selling Internet cards, diskettes, CM Taking pictures with a digital camera . Designing Web sites . Evaluation of machines and technical devices . Training in Logo, Eaglish language. Web design Binding 911 Initial Services (left column) plus: . Internet - wireless, constant speed 33.6 Kbs Educational courses for ad* and children, wth individual and group . Rearding CDs Advertising materials, business cards, invitations ANNEX 4: PUBLIC-GOOD SERVICES OF THE PC3 TELECENTERS Sources: Information Collection Sheets, filled in by the PC3 operators at the PC3 Management Issues Training - February 2002, Plovdiv; follow-up communications between PC3 operators and AED Sofia (Iliana). Aitos: Creating a basic infrastructure for Internet transmission from Bourgas to Aitos towns. Working together with the schools in Aitos Teaching people who have never had access to information technologies Free Internet access for the Municipality Apriltzi: Providing Internet to the town, thus giving local residents the opportunity to communicate with people from all over the world. Creating a municipal network Byala: Training children from the Home for Disadvantaged Children and Teenagers. Vetren: Introduction to Internet and e-mail, training; Public telephone. Varshetz: Training; provision of information on different topics, useful to local users Free-of-charge promotions Contribution to the Bulgarian web site for literature "Slovoto" Zlatograd: Offering information and office services; helping representatives of local businesses. Stimulating unemployed and needy people with prepaid cards and other bonuses. Filing individual and business correspondence for free Offering computer magazines, statistical reference hooks, juridical literature and laws for free reading Information and consultations about programs/projects of NGOs, schools, businesses Madan: Training Omurtag: Prepaid cards; Free presentations in the evenings Providing Internet to the local community Contribution to the Bulgarian web site for literature "Slovoto" Pravetz: Training; invitations for chitalishte "Zarja;" the school newspaper Making prints on T-shirts for gifts at celebrations. Tryavna: Training; popularizing the possibilities Internet gives its users Contribution to the Bulgarian web site for literature "Slovoto" Training children from "SOS Children's Village - Tryavna" ANNEX 5: EVALUATION DATA SOURCES AED: Task Order HNE-1-10-96-00018-00 PC3 Project Initial Implementation Plan. December 12,2000 PC3 Project Request for Proposals, issued January 23,2001 Pre-bid conference questions and answers Applicant selection process documentation Finalists' original proposals Technical annexes to PC3 operator subcontracts Sofia Project Office: Usage data on prepaid card distribution and coupon reimbursements PC3 information sheets on clients and services, completed by operators February 2002 Operator training programs, agendas, post-session participant evaluations, trainer reports Presentations prepared for UNESCO [?I and UNDP meetings Monthly Reports to AED Washington [dates] Quarterly Reports to AED Washington [dates] Email traffic between Sofia office and PC3 operators [in Bulgarian, tmnslated excerpts] PC3 Web site at httu://www.~c3.orbitel.bg Nora Ovcharova, Development of a Monitoring and Evaluation Sysrem for the Bulgaria Public Computer and Communication Center project, MA Thesis, Sofia University, 2001 Club Economics, Sofia: Structured survey of PC3 neighbors (n=200) and users (n=100) Three-day observation in PC3s of users and their activities while in the PC3 Interviews of mayors and deputy mayors of PC3 municipalities Analysis of economic, political, and social factors affecting sustainability of the PC3s bibliography] Summative Evaluation Specialist {AED Consultant] Desk research on telecenters and ICT in transitional economies Review of Sofia Project Office files In-depth interviews of Country Director Iliana Nikolova and Sofia project staff Site visits, all 10 PC3s, April, 2002 In-depth semi-structured interviews of all PC3 operators Group interviews of available PC3 trainers, board members, and users ANNEX 6: SUMMATIVE EVALUATION FRAMEWORK March 14,2002 Used as a planning tool by evaluation team Goals of the Evaluation Provide useful learning experiences for all stakeholders, including USAIDIBulgaria, USAID Economic Growth Bureau, AEDILeamlink, Sofiya project office, PC3 operators individually and as a group, and PC3 board members at each site. Use data already collected by project and/or readily available to Club Economica; some LOE into formatting or re-formatting of spreadsheets, and translation of selected reports into English; most new data qualitative, to come from evaluator's interviews during site visits; some quantitative possibly from Club Economica Look for what is actually there and what has actually happened, without assumptions, but ask questions based on what experience has shown in other telecenters elsewhere to work and not work, to be effective and not. Intended results Identify and disseminate any apparent short term trends or signals through analysis of the monitoring data already collected from each PC3 operator, singly and as a group Identify signs of trouble and signals of opportunity for the group of ten through on-site interviews of Sofiya project coordination staff, individual PC3 operators, and their users, partners, and board members, as feasible. Ascertain how sound a footing for the future the PC3s seem to have achieved since they opened with the help of USAID's investment. Research and suggest a typology of information-seeking purposes and needs of such communities as served by the 10 PC3s that might inform future telecenter establishment and growth in Bulgaria and elsewhere. Indicator #1 Trends in actual numbers of users 1. Coupon users formats needed and data sources: For each PC3, the actual number of coupon books distributed over the actual number of months since opening, disaggregated by recipient user group or organization typology. Source: PC3 data collection #15. Maintained in spreadsheet by Sofia project staff. For each PC3, coupon users by month, disaggregated by age range (ie, youth to 18 years, adult, elderly), occupation, user group, and user level, against services actually used according to standard list, with subtotals, in cells with formulae that can yield 36 totals across all PC3s for each disaggregated indicator. Source: spreadsheet of data from coupons submitted. 2. Paying users: For each PC3, by month, paying users, (not disaggregated by demographics unless significant). - against services purchased, if available. Not: This &a was not collected ,- - aspart of the monitoring and evaluation routine. Subsequently Sofa project st@ collected for one month only, and Club Economica collected PC3 income forfirst and last months ofproject operations. 3. Ratio of coupon users topaying users: Trends in services for paying customers versus coupon users. For each PC3, by month, total coupon users and total paying customers for &ch paid service. Note: This data was not directly available, and collecting it was abandoned due to strong efJects of coupon use on operators' business behavior. 4. Intended users versus actual users: Have new and unexpected user groups emerged since PC3 opened? If so, what are they? When there are major user groups organizations, eg, school or government office, ascertain distance from telecenter ( 2 blocks, 1 km, etc.) Note: Distancesproved irrelevant in small towns. < 5. Repeat users? Data probably not readily available. Anecdotal from PC3 operator interviews? Do different users from the same group or organization qualify as "repeats? Note: Club Economics's 3-day user swvey established a general picture. 6. Baselinepossibility? Many of the PC3 operators were already running a computer service of some kind before they got AED support, and some proposals offer user numbers of that existing service. Can more be extracted here? Note: Extant numbers were so small, undrhe increase in client numbers so great, that this indicator became irrelevant. Indicator #2 Performance, role, and function of each P3 in its municipality 1. Institutional partners List for each PC3. Sources: extracted from proposals when info present, and, ffom questionnaires for "PC3 Management Issues Training" andfor message traffic between PC3 and Sofiya project office. 2. Board members and their roles in community and perceptions Extracted fiom Questionnaires as above. Who are they in the community? Does anyone seem to be a spark, an advocate, a facilitator, etc.? Follow up interview question for onsite focus group of board and partners if possible: "Has PC3 changed anything (or, is it changing..?) for anyone besides the PC3 operators in your town/ municipality/village? If so, what was changed for whom?" Transcribdtranslate replies. 37 3. Services actually provided at each PC3 Classified as follows, based on services list compiled at Sofiya project office. Need a format to give comparative totals for each PC3 so as to compare them, and to ascertain whether each PC has added new services since opening. The classification given here is intended to elicit a sense of local perception of services: as a hardware provider, a learning opportunity, and their value to user groups. Listed below general types of services with specific examples. Public IT access and use onsite: Self-service use of PC for application, email, surfing, games, etc Basic IT training in how to use PC, email, or Internet Self-service use of other equipment: copier, scanner, telephone,etc Customized IT hardware, software, telecom services for offsite clients: Installation of LAN, connectivity, software, hardware Sureadsheet or database maintenance * Network, PC, and equipment maintenance Office support services onsite: Resume preparation Photocopy, fax service Word processing Binding, laminating, and document collation Sale of office supplies Business support services for offsite customers, for example Commercial Web pagelsite building Printing and publishing services: brochures, etc. Translation services Training services for vocational skill-building (not just how-to-use-the-PC): Applications such as Word, Excel, Pagemaker, Photoshop Database applications, such as Access, Oracle Server, LAN, or database management skills Web page writing, such as HTML, CGI, ASP, XML, Java, Javascript Foreign language training using digital resources Content development services in Bulgarian language, for example: Digitizing material for issue on CD Preparing and distributing bookmark sets of useful Web sites in Bulg&ian such as newspapers or government service sites Providing a reference library for PC3 users of magazines and newspapers Preparing obituary notices Information content brokering and reference services Using the MIS legal information system 38 Obtaining correct governmental information/contacts for users Researching market opportunities for an agriculture cooperative Identifying donor programs of possible local interest Downloading and disseminating critical weather information Identifying and disseminating regional, national, or international employment opportunities Other services that might be surprising? Specific local adaptation? Create a format to show specific services for each PC3, from proposal and email message traffic, compile, and fill in gaps during onsite PC3 operator interview. Note: Compiled andprovided by Sofa project office. 4. PC3 Performance and Management Opening hours (&om Management Issues Questionnaire #8) have they changed? More? Or less? Since opening, has there been any down time due to facility or connectivity disruption? What was it and did it have to do with the Internet connection? Source: operator reports and email traffic, with interview followup. Note: Depended on the Internet connection-jequent for sites that have not adopted wireless connection and still rely on dialup. Luck of sufficient building heat in winter affected one site. Since opening, what specific audience research, marketing, and promotional activities has each PC3 undertaken? To what audiences through what media with what result? Source: operator monthly reports and email traffic, with interview followup. Cable TV; newspaper coverage of openings. Internal assessment by Sofya project staff. How did PC3 operators perceive use of coupons? As loss leader, as promotional tool, as a charitable donation - what operators say, not these terms! What actual results do PC3 operators believe the coupons achieved? Did they actually reduce their business risk? See interview summary. Appearance and maintenance of premises and equipment; usual staff hours workedlweek Real world usage: Club Economica 3-day user (100 respondents) and neighbor (200 respondents) surveys and onsite observation, mayoral interviews. 5. PC3financial outlook comparison Operating costs/month (actual or average or estimated average, whichever is easier, for each PC3. Note: This was notpossible. Revenueslmonth (actual, coupon reimbursements plus customer payments) First und last month data obtained by Club Economica. PC3 operator's projected break-even monthlyear with reason (brief). Is it credible? Business plans reviewed by Club Economica and deemed acceptable but without an overarching strategy. How to arrive at % or ratio of common-good, preferential services to revenue￾generating services (in common unit of quantity, time, or cost to deliver). Data not possible to gather. Indicator #3 Role and function of each PC3 operator, in PC3 and in community 1. How knowledgeable about content needs of users andpotential users? Source: monthly operator reports, email traffic, with follow-up interviews. Since most operators have computer science or technical backgrounds, their social knowledge about information seeking purposes, behaviors, and content needs, is worth inquiring about. Note: See content workshop, December 2001; interviews. 2. How "plugged in" to municipal and regionalflows of information andpolitical power? Source: Club Economica mayoral interviews; in-depth operator interviews. 3. Does operator seem to be a community leader or livewire of some kind? This attribute can probably be assessed from existing reports and followup personal interview. The following query is not so easy but is worth examining as possible: 4. What characteristics of the town/municipality might be identified as necessary to predict success and sustainability of a telecenter? - Population density? Young people leaving? Ratio of government to private employment? No, because it depends upon who wants PC3 services. 0 Presence or absence of other donor activity? Note: Peace Corps volunteers have a heIpfir1 influence; otherwise it depends on the donor program and local donor representation. 0 Unemployment rate? Note: Ministry of Labor provides ZTtraining but requires a certifiedprovider; PC3s were slow to realize this connection. 0 Education level of population? Note: Bulgaria has very high educational levels; most have at least a secondary degree 0 Social role of information flow? Joining or separating groups? Note: Social roles are very influential. See text. Indicator #4 Role and Function of Sofiya Project Office 1. Facilitate efficiency and effectiveness of coupon system and its management 2. Identify usefklness of project data and reports gathered and prepared 3. Compile list of training sessions for PC3 operators, dates, syllabi, trainers (include Hungary trip as training) Source: reports, staff interviews, PC3 operator perceptions as well in interviews. Queries for PC3 operator interviews: "How has Sofiya office most helped you?" "What would you have lied to be different in coordination, management?" "What advantages/opportunities for your center do you see in collaborating with other telecenters?" 4. Bulgarian content development report (PC3 data collection list item #23) findings 40 Indicator #5 Impact of Project Training 1. PC3 operator training Compile PC3 operator evaluations of their training sessions as listed under indicator #4 above. Query for PC3 operator interview: "What element of the training you have received has made the biggest difference for you?" "How has your vision for or management of the Telecenter changed as a result of the training you have received?" 2. PC3 user training Compile training reports into format as follows: Course titleltopic as reported # hours of instruction to give course Total # sessions given since PC3 opened Total # students for all those sessions of this course Principal user groups of students (if available, exact numbers) What indication of training results might exist? Did anybody get a new job, a good exam grade, new business idea, etc. ANNEX 7: CLUB ECONOMICA SCOPE OF WORK for USAlDlAED BulgaRla PC3 Project Evaluation Support March 3 1,2002 Collect, format, and analyze financial, economic, and social data from each of PC3 telecenter in the ten different towns so as to assess the likelihood of each telecenter achieving sustainability. For the purpose of this evaluation, sustainability of each telecenter is defined as its ability to generate enough revenue not only to support a continuing business but also to maintain the amount and level of services it now offers free of charge or at concessionary rates to disadvantaged local user groups. Research tasks: 1. Review project documentation provided by the AED PC3 Office in Sofia and the Summative Evaluation Framework dated March 14,2002, sufficiently to understand the goals and objectives of the project and of the present evaluation. 2. Gather financial data available from the AED PC3 Office, supplemented as necessary by interviews of the PC3 operators, sufficient to elicit enough data in order to complete a trend analysis of each telecenter's financial and business condition since it opened. Also, assess the business skills, attitudes, and behaviors of the PC3 operators from the quality of their present business plans and from their ability to provide ready business-oriented information to the evaluation. 3. Conduct structured interviews of users and neighbors at each telecenter town designed to elicit perceptions and uses of the telecihter and to identify any changes the telecenter presence has brought since its opening. Gather just enough data to assess the sustainability of the telecenter in the context of its local social and economic environment. For this task, interview no more than 10 users of each telecenter, and no more than 20 community members and neighbors of the telecenter, including the local mayor. The interviews must begin no later than April 5,2002, and be completed in no more than one week. Reporting tasks: 1. Provide essential financial data to the AED PC3 Project Office in electronic format (Excel spreadsheets) no later than close of business April 17,2002, together with any graphs, charts, or written analysis already drafted. 2. Prepare a written analytic report in English and Bulgarian of the structured interviews with demonstrable conclusions about local perceptions and uses of the telecenters evidenced in the interview responses. This report will consist of no more than 5 pages of narrative, excluding any necessary tables, graphs, or charts appended. Submit this report to the AED PC3 Project Director Iliana Nikolova no later than close of business Thursday April 25, 2002. 3. Provide summary narrative report in English and Bulgarian, both electronically and in print, assessing the stability and sustainability of each telecenter, individually and comparatively, to the AED PC3 Project Director Iliana Nikolova, no later than Saturday April 27,2002. The narrative summary will consist of no more than 5 pages, excluding supporting data in annexes for reference from the narrative. 4. Send electronic versions of the reports specified in tasks 4,5, and 6, via email by the same deadlines to Jeanne TiW, PC3 Project Evaluator, at the following email address: jtifftO,us.net. ANNEX 8 : PC3 OPERATOR IN-DEPTH INTERVlEW INSTRUMENT The purpose of the evaluation is simply to investigate what has hqpenedas a result ojUSAID/XED startup support for your telecenter, to find out what you have learned and think important for others m the worldwide telecenter movement to how. Z am going to ask all ten ofyou the same set ojquestions. Also, sometime next week a researcherfrom Club Economica will ark you a few things (no reports will be required, wepromise!) and will interview your users andneighbors. The results andconclusions from our observations and interviews will be shared with all ofyou through Ilicma 1. So please tell me before we begin what you would most like to learn &om this evaluation process? 2. As a project participant, what have you most valued? 3. In the administration and management of the project by the Sofia project office, what did you especially appreciate? What do you wish had been different? 4. In seven training sessions the project provided you, what was the greatest benefit gained? a. Initial training, Vitosha, June 2001 (included subcontract signing) -. b. Specialized tAg for trainers, ~a&orovo, August 2001 c. System administration training, Apriltsi, November 2001 d. IT in education, Sofia, Nov-Dec 2001 e. Web page design, Sofia, January 2002 f. PC3 management training, February 2002 g. Hungarian telecottage study tour, March, 2002 h. Content workshop, Sofia, AmCultCtr, Dec 2001 5. What would you most like to learn next over the coming 1-2 years? How do you think you might gain that learning? 6. What was the real effect on your business of using the prepaid cards? How did they most help or hinder your operations? 7. The original project based the number of cards given to each PC3 on the population of the town. The objective was to enable you to give a gift to the community as well as minimi;re your initial business risk. With your experience, now do you think that basis was suitable, or would you recommend another on which to determine the number of cards given to the PC3? 8. What has been your most difficult or persistent problem (conuectivity, equipment, management, sffig, clientele) during the project period, and how are you solving it? 9. What demand for services have you seen grow; what resistance have you seen occur? 10. What new contacts and relationships have you established since opening? Which do you most value? Why? a. local b. elsewhere in Bulgaria c. abroad 11. What information content and information activities is valued in your community? Cite examples. 12. Who are your principal advocates and supporters in your town? 13. Who do you see as your competitors in your town-competitors in providing training, Internet access, office services, other services you offer? 14. What future do you see for your telecenter alone; and for the group of ten now operating? t'C 15. What might have become of your business without USAID/AEDYs short term investment? What difference has that investment made? ANNEX 9: PC3 OPERATOR IN-DEPTH INTERVIEW FINDINGS A content analysis of the transaipk and notes made during the interviews idenWied dusters of recurrent themes on the summary below is based. 1. What they would like to learn from the evaluation process ... Most were interested in a comparative assessment, wanting a disinterested assessment of how each PC3 is doing in relation to the others. The more thoughtful operators were also interested in ideas and methods for improving services and identifying needs. 2. What have they most valued in participating in the PC3 project ... Content analysis gave the most weight to equipment, hardware, and software, followed by business experience and training, new relationships with each other, with the Sofia project office staff, and with presenters at the various operator training sessions. 3. What aspects of project management were most valued, what might have been different ... Intra-project communication and team building was greatly valued, while the Sofia project office's quick response support for problem solving came a close second. Overall, the project staffprofessionalism and organization most impressed the operators although they also found that enforcing the reporting deadlimes could interfere with business development tasks. Administering prepaid card data collection was very burdensome. 4. Operators' attendance at, and benefits from, the 7 project training sessions and content development seminar ... None went to all sessions, though all operators attended the initial training and the management training sessions, both highly rated, as were the presenters at these sessions. [check evaluations] Several sent staffacting as trainers to the two train-the-trainer sessions. However by April 2002 at least two trainees were no longer on PC3 staffs, thus losing the benefit The staff acting as systemlnetwork administrator (in some cases, the operator; in others, another) was sent to the system administration training session. Its level of instruction worked best for the beginners, (who grew noticeably by the end of the session, according to the instructor). The more advanced participants found that individual support as needed from the Sofia project office IT specialist was more useful. 5. What operators would like to learn over the next 1-2 years, and how they'll gain this knowledge ... Five of the ten specified more on systemlnetworkhternet administration. Five charact& various aspects of NGO finance and management and services management, including services for the disabled and disadvantaged. Other areas mentioned were distance education, children's IT and language training, sources of donor information, and tourism planning. 6. What was the real business effect of using prepaid cards The cards were most effective when consciously used for promotion and popularization, to overcome Bulgarian suspicion of anything new, and for building clientele among businesses and organizations thought to be potential paying customers. Initial distribution all at once proved disadvantageous. For instance, interested recipients created demand beyond what could - - be serviced, and disinterested recipients took cards because they were free but subsequently lost or destroyed them, thus losing potential PC3 income. Scheduled monthly distribution worked better by regulating demand and maintaining popular interest. Cards given to children for training would soon draw their parents' interest. Card users who had used up their coupons in training would often become paying customers. Since reimbursement from project funds was based on receipt of certified used coupons, the PC3s had no income the first month; this constrained use of consumables such as printing paper. Moreover, the detailed paper-based administration of coupon data to get reimbursement drained time and energy that might better have gone into building the business. Bulgarian Internet clubs have converted to plastic cards, like phone cards, readily scanned and accounted for. This project ended up appearing behind as a result. The prepaid cards were not perceived as reducing business risk (as project design assumed) although when their administration was well managed it provided early income stability. It was the training sessions that reduced business risk. 7. Operators' recommendations on the basis for determining quantity of prepaid cards to be distributed to each PC3 ... Most thought that a quantity based only on population in 10,000 increments was not fair (a town with 9,000 got $1500 worth of prepaid cards, while a town with 11,000 got $3000 worth). Two especially thoughtful operators suggest devising a formula based on population increments of 2000 only, weighted against several additional relevant factors: labor force, unemployment rate, school age children, competitors (ISPs, games clubs, office services, etc.), numbers of small businesses, number of computers available in the PC3. Operators particularly interested in training suggested that coupons of different training values would permit giving more advanced training to clients unable to afford continuing after basic training using coupons, thus strengthening the "common good" effect. Two suggested separate cards for training and other services 8. The most difficult or persistent problem during the project period ... Most had difficulty with the Xerox copier provided by the project. Replacement cartridges were so costly that the PC3s priced themselves out of the copying business .~ in some towns. - Many of the copiers broke down anyway. Internet access difficulties, lost prepaid cards, low incomes of the community, and personnel were the four most persistent difficulties cited. 9. What demand has grown; what resistance observed ... Resistance came from older people still in positions of some authority (municipal or school officials) who miss socialist securities and fear new ways of doing things. Demand is growing among sectors that begin to see the advantages of new communication channels. Advantages are perceived for sending data to national government ministries of health and labor, providing municipalities with a desired LAN, and helping businesses'who have discovered Web advertising. Parents have discovered email for staying in touch with children studying abroad, and enthusiasts who download music, software, and ring tones for mobile phone enjoy the access. 46 10. Valued new contacts and relationships established during the project ... Most often mentioned (4-6 times each) were new clients, local organizations, the other PC3 operators, and contacts made on the Hungary study tour (funded outside the PC3 project but applied for through the Sofia PC3 project office). 1 1. Valued information content and activities in your town ... See appended list in Attachment N. Compare with content list worked out at Content Development Workshop; see report in Attachment N. 12. Principal advocates and supporters in your town ... Mayor, municipal officials, school principal, hospital chief, businesses. 13. Competitors ... Computer clubs (everywhere), chitalishte (in one town), other ISPs (3 towns), Partners for Local Development (PLD) and other commercial business support services, Labor Ministry contractors who provide IT training for the unemployed, secondary schools that offer IT curricula. 14. What future do you see for your PC3 a~Id for the group as a whole ... PC3s should be actively involved in IT education for children, establishing school computer centers and IT curricula. PC3s should work with the Ministry of Agriculture to help agriculture cooperatives as well as with "middleman" between the tourist sector and secondary vocational schools. Also, PC3s have the best, fastest, most reliable Internet connection of any other service. PC3s sell computers, software, support, and consultations; establish a professional atmosphere in the PC3 as competitive advantage over smoky computer clubs; and develop distance education programs for business and language training. 15. What difference has USAID's investment made to your business ... PC3 operators who had already been in business referred in various ways to the grant of equipment that gained access to the Internet and opened their towns to the world. Donated software helped them expand their services rapidly, particularly for children in Logo classes. The operator training sessions helped reduce business risk by showing them how to manage a telecenter. A few found that establishing an Internet connection in the town drew competition, thus broadening Internet penetration intithe region. One convinced the local CommIhity Develo~ment Center that the PC3 was a useful addition to the community and motivated its board of directors to help develop it. ANNEX 10: PC3 PREPAID CARDS DISTRIBUTION AND ADMINISTRATION Procedural description based on The PC3 Prepaid Cards System, by lliana Nikolova, Country Director Purpose The PC3 Project provided a limited number of prepaid cards to the PC3 operators for distribution in their communities in order to promote the activities of the centers and to help create a new clientele quickly, thus reducingthe initial business risk for the PC3 operators. As a form of subsidized access to the products and services offered by the PC3 telecenters, the prepaid cards were intended to stimulate public usage of the PC3 centers and to promote information technologies skills enhancement. Value The prepaid card was non-transferable and not exchangeable for its monetary value. A Card Holder could fully or partly cover service cost with one or more coupons (each coupon can only be used to its full value - 1 lev per coupon, even if the service costs less than 1 lev). The coupon could not be used for computer games. Distribution The number of prepaid cards given to each PC3 was based on increments of 10,000 in the town population, at a value of $1500 per 10,000. The value of each card was10 Bulgarian leva, containing 10 coupons, each valued at 1 lev). AED reimbursed the PC3 Operators for the value of certified coupons submitted each month until the end of the Project. Initially the end date was November 2nd, 2001, later extended to May 24,2002.' Originally, AED's Sofia project office was to manage distribution of the prepaid cards to potential "public good" users, such as teachers, health workers, and agricultural agents. Soon the Bulgarian country director decided that the PC3 operators themselves could more effectively distribute the cards than could the Sofia project office. PC3 operators were tutored and encouraged to choose a local Board of Directors to help decide who should receive the prepaid cards in their town and prepare preliminary distribution lists for review by the Sofia project office. In some cases, Board members actually doled them out to individuals. In others, they were given to groups (school class, municipal offices). Administration Administration of the prepaid cards was originally planned as an automated system until AED deemed that the expense of customizing an off-the-shelf software product for installation at all the PC3s was too great.' To implement the PC3 project monitoring and evaluation indicators, however, meant that the prepaid cards had to be used as a data collection instrument as well as a subsidy. Accordingly a paper-based system was devised which collected demographic data on the cards and use data on the coupons within. Though complete and clever, the system and procedures were also complex, time-consuming, and labor intensive both operators and Sofia. ' Nonetheless the Bulnarian - country director found a young ~-~ Bulgarian software developer, who created a product all PC3s can and do use, but who was suddenly drafted into the army before its reporting and transmission functionalities could be develo~ed. Called Kaiowas, the package can secure and track local computer use. A few PC3 operators with programming skills have expanded reporting f"nctio& successfully. 48 For example, after each usage of prepaid senices or products the Card Holder completes and gives the PC3 operator the necessary number of coupons. The operator registers the usage in the PC3 card log sheet - and the customer sim - it. The used couwns are stored at the PC3 Center, at the end of each reporting period are attached to the monthly reimbursement request voucher, and submitted to the PC3 Proiect office in Sofia as stated in the Subcontract. The usage of coupons is registered in a special electronic tabld that also is sent to AED. During the distribution of the cards (from an AED representative with the assistance of the PC3 operator and the PC3 Board members) the holders of the cards should sign in the "Card Holder's Signature" field on the form. The completed and signed forms are sent to AED-Sofia (the PC3 operator keeps a copy). The PC3 operator also inputs the data from these forms in a specially designed spreadsheet (a sample has been sent by e-mail to all operators). Completed spreadsheets must be e-mailed to: pc3office@~c3.orbitel.bq. PC3 Card Log Sheet - a regisbation form, where Card Holder's personal data is completed when the client uses the card for the first time. The Log Sheet is stored at the PC3 Center for further registration of coupon usage. Coupons: Every time the customer uses a coupon, helshe should fill in the necessary information in it, sign it and hand it back to the PC3 operator who keeps it as a receipt The used coupons are attached to the monthly reimbursement request voucher presented to AED. Reimbursement The reimbursement procedure went as follows: each used coupon was attached to the monthly reimbursement request voucher presented to AED. The data from each coupon should have been entered in a spreadsheet to register the PC3 service usage, specially designed with pre￾inserted coupon numbers, lists to choose values fiom for user groups and services, and protection of certain cells, at AED-Sofia and sent via e-mail to all PC3 centers. At the end of each reporting period, all used coupons and the spreadsheet filled in for the respective reportiag period were submitted to the PC3 Project office in Sofia Correctly completed and signed coupons are an important part of the monthly financial report submitted to AED, since incorrectly PC3 prepaid cards may have resulted in delayedlrefused reimbursement by AED? PC3 CARD DATA PREPAID COUPON CARD Unforlunately some card usen assumed the card was no longer valid after the original end date printed on it 49 ANNEX 11: BOARD MEMBERS AT EACH PC3 TELECENTER Information Source: Infomation Collection Sheets, filled in by the PC3 operators at the PC3 Management Issues Training - February 2002, Plovdiv. PC3 Board (quote from "PC3 Prepaid Cards System Description") "Each PC3 Operator sets up a PC3 Board (Group of people willing and able to help with the popularization and promotion of the PC3 activities and the usage of the prepaid cards in the local community; usually each Board member is associated with a specific community group). Information about the Board members must be sent to the AED office in Sofia The members of the Board assist in the preparation of the preliminary lists for the distribution of the prepaid cards. The PC3 operators hand out to each Board member a certain number of blank Distribution forms. The number of forms handed out to each Board member depends on the estimated number of potential clients from the community group(s) the PC3 Board member is associated with and the previously agreed proportional distribution of the prepaid cards amongst different community groups (discussed at previous meetings between AED and the PC3 operators)." Aitos (five members): Hristo Stamatov - Commander of army unit 26200 Ivan Ilarev - Manager DNA Hristina Stoilova - Deputy Mayor Radostin Gugalov - Headmaster Maria Kalinova - Teacher Apriltzi There isn't a currently functioning PC3 board. Byala There isn't a currently functioning PC3 board. Venelina Tzvetanova - Assisting with cards distribution and organization of class visits. Vetren There isn't a currently functioning PC3 board. Varsheb (three active members) Tzveti Konovsk - Manager of "Tintjava" Complex Emil Krustev - Manager of "Electrostart" Vladislav Ivanov - Manager of "Elsner Product" The members of the Varshetz PC3 Board helped with the repair of the PC3 telecenter's building, provided money for a part of the computer equipment. They requested training courses at the PC3 telecenter and support its activity in various ways. Zlatograd (seven membersx) Krassimir Mladenov - Member of the Board of Managers (BM), help with the prepaid cards, evaluation Vaklin Chilingirov - Accounting Borislav Nikolov - Member of BM Anton Ushev - Member of BM Dimitar Alendarov - Member of BM Based on a summary report by PC3 Counhy Director lliana Nikolova and interviews with her and PC3 operators during the evaluation process. 50 Tito Valenzov - Member of BM Dr. Angel Belev - List Madan (four members) Mitko Petarov - To help whenever necessary Shinka Hadjieva - To help whenever necessary Janka Bodukova - To help whenever necessary Nikolai Shtiljanov - To help whenever necessary Omurtag Marijana Tosheva - Popularizing the telecenter Janka Alexandrova - Works among teachers and students Pravetz (five members of which only two were very active) Pzetza Hpistova - Helped very actively with prepaid cards distribution Penka Slavcheva - Helped very actively with prepaid cards distribution Di Nakova - Did not participate actively in the prepaid cards distribution because of lack of time. Trifon Trifonov - Did not participate actively in the prepaid cards distribution because of lack of time. Petar Simeonov - Did not participate actively in the prepaid cards distribution because of lack of time. Tryavna (five members of whom all were active at start-up) Vladimir Knunov -Popularizing the PC3 telecenter (active during the first month) Zdravka Kirilova - Popularizing the PC3 telecenter (active during the first month) Stanimira Gencheva -~o~ularizin~ the PC3 telecenter (active during the first 2 months) Velichka Bratoeva - Popularizing the PC3 telecenter (active during the first 2 months) Dr. Aneta Janeva - ~o~&izin~ the PC3 telecenter (still helping actively) ANNEX 12 : INSTITUTIONAL PARTNERS OF EACH PC3 TELECENTER Information Sources: lnformation Collection Sheets, filled in by the PC3 operators at the PC3 Management Issues Training - February 2002, Plovdiv; PC3 telecenters' presentations, made by Uie PC3 operators at the same training; and "After-PlovdiV communication between PC3 operators and AED Sofia (Iliana). Aitos (five institutional partners) The Municipality of Aitos "Nikola Vaptzarov" High School" "Hristo Botev" High School TSS "Zlatna Niva" "St. St. Cyril and Methodius" Comprehensive School. Aprilbi (two institutional partners) The local library -planning to create a "virtual library" in cooperation with the PC3 telecenter The school in Apriltzi Byala (many partner firms) Private firms and enterprises - for dial-up Internet access. The chitalishte 4 Other fms for office services. Vetren (four institutional partners) The mayor's office The school The post office The Association for Telecommunications. Varsheb (eight institutional partners) Local administration The Police Department in Varshetz "Vasil Levski" Elementary School "St. St. Cyril and Methodius" Elementary School "Ivan Vazov" High School The comprehensive school in the district of "Zanojene" The local health center The Labor Office. Zlatograd (many partner firms and institutions) Private businesses (23) The Municipality of Zlatograd 3 chitalishtes The Police Department 2 schools Sports clubs Church, Muslim and school boards of trustees. Madan (three institutional partners) PMR - a non-governmental organization for local development. The chitalishte. The local Labor Office. Omurtag (three institutional partners) Agricultural cooperatives The Museum of History The Municipality of Omurtag Pravetz (eight institutional partners) "Vasil Levski" Comurehensive School The high school in the town of Pravetz The Municipality of Pravetz The Bulgarian Telecommunications Company A factory in Botevgrad in which disabled people work "Metalik" AIS - Jablanitza "Vasil Levski" Comprehensive School in the village of Jarnka Botevgrad hospital Tryavna (many institutional partners) The hospital The Municipality of Tryama The high school One nongovernmental organization Businesses (approximately 30 so far) - want to advertise on Tryavna's web site. ANNEX 13: PC3 OPERATOR TRAINING PROGRAM' Program Purpose The PC3 project staff believed that training would be crucial support for all PC3 operators to help them achieve the overall project objectivesustainable telecenters. Though some had already launched small businesses, and some were trained as computer scientists during the socialist period, all lacked skills in contemporary operating systems, software, and business methods. Most had no experience in training, and none in training children in IT. Nevertheless, all had demonstrated initiative and learning capacity during the application process, and three of the PC3 project staff members have strong professional backgrounds in ICT and business training. This conjunction proved to be highly beneficial to the project outcomes. The PC3 project's initial scope of work envisaged only two or three operator training sessions. In the end, seven training sessions were conducted, each and all ranked highly by the participants in post-session evaluations. Held soon after the 10 finalists had been chosen, the first training session brought in an extemal expert for instruction in worldwide telecenter experienceZ as well as several Bulgarian experts and officials involved in ICT, IT education, and small business development. Also, this first workshop was designed explicitly to achieve several additional objectives: to obtain the signatures for each operator's subcontract, to lay the ground for creating group cohesion among the PC3 operators, and to elicit their assessments of their staff training needs. Subsequent operator training sessions were created from an interactive feedback process that identified both existing and emerging training needs. Logistics Since the PC3 operators could not leave their businesses for long, each training session was intensive, with a concentrated agenda, held in a quiet and isolated place. The PC3 project team drew primarily on local Bulgarian resources for trainers and venues, with very cost effective re~ults.~ This was possible because Bulgarian professional salaries and fees were dreadfully low during the project period. Also the PC3 project staff members were qualified to present some of the training themselves. ' For some rearon neither the European Union nor national aid agencies other than USAlD participated, although many such entities were found to have touched individual PC3 activities during the evaluation. Heather Hudson, University of San Francisco 'Only the first (for Dr. Hudson) and seventh (Hungary Teleconage S~dy Tour) training sessions used extemal resources. The project staff organized three themselves, and subcontracted three to local training organizafions 54 ANNEX 14: PC3 OPERATOR TRAINING AGENDAS TI. Initial Training for PC3 Managers and Operators 6-10 June 2001, Hotel "Kremikovtsi," Vitosha - 21 participants Overview of global telecenter experience IT use for economic and social development Small enterprise planning and operations elecenter operations and business plan development Technical considerations for PC3s CT training (preparation, materials development, outreach, training techniques) IT in education (national school curricula for IT, TOT) T2. Computer Applications - Training of Trainers for the PC3 Centers 12-20 August 2001, Hotel "Orlitsa," Pamporovo -'I0 participants MS Word, Excel, Powerpoint; Internet browser T3. System Administration Training for the PC3 Operators 15-1 8 November 200 1, PC3 Telecenter, Apriltzi - 10 participants Overview of PC3 computer systems, file systems Operating systems -comparison of Windows 9x, Windows NT12000 and Linux. Windows 2000 administrative tools, user rights and accounts hierarchy. Hardware and software installation and maintenance Common problems -diagnosing and fixing. TCP/IP protocol. Internet morphology - addressing, subnetting and data transporting. Local Area Network (LAN) topologies. Setting up a LAN - planning, cabling, testing and tuning T4. ICT in Education - Training of Trainers for the PC3 Centers 29 November-2 December 2001, "C. and M." Private Language School, Sofia - 19 participants Juridical and ethical aspects. Health issues. National school curricula for ICT for different age groups. Building ICT skills for children (6-10 years old): Methodological arpects. Software, graphics, presentations, internet suitable for children. Internet for teenagers. Ethics in the virtual space. Problem solving using ICT. Applications of ICT in project work. Logo educational philosophy - overview and main principles. Logo microworlds for children (graphics, text, sound, animation). Comenius Logo -Introduction to progpmmhg and project development National school curricula for ICT for different age groups. T5. Web Design Training - Training of Developers for the PC3 Centers 21-26 January, 2002, Center for Information Society Technologies, Sofia - 12 participants 55 Introduction to Web design HTML - structure, tags, basic techniques CSS JavaScript - language basics DHTML, XML, VRML trends Web multimedia Interactive web applications Web-integrated databases Individual project development (course follow-up -online consultancy) While the project Web site includes a description and contact information for each PC3, five participants built Web sites of which the following were "live" on May 5,2002: PC3 project htt~://pc3.orbitel.bg/en/training.html Madan http://free.bo1.be/Dc3madanl Omurtag httv://www/g;eocities.com/erdincrasimov/ Tryavna http://www.travna.bd Vetren http://pc3vetren.hit.bg/ Zlatograd http://www.zlatog;rad-bg.com/ T6. PC3 Management and Financial Issues - Tmining for PC3 Managers 20-24 February, 2002, Plovdiv - 18 participants Common taxation problems ..Social insurance Client and customer relationship management Cost estimation; formulating service price Budgeting and cash flow control Current investments administration T7. Hungarian Telecottages Study Tour - Regional Training for PC3 Managers 17-28 March, 2002, Hungary - 11 participants visiting five Hungarian telecenters History, development and main characteristics of the Hungarian telecenter network The basic set of required telecenter services in Hungary Standardized services and quality assurance Supporting network services Strategic and business planning Telecenter sustainability -the experiences gained in Hungary Organization of the Hungarian telecenter network Cooveration with the - governmental and business sectors Action planning - a required activity for the participants where each participant developed an action plan taking into account the ideas and experiences gained from the training. ANNEX 15: TRAINING SERVICES SUMMARY, ALL PC3S Source: Sofia project office, prepared from PC3 Operators' Training Reports for 2001 and monthly frwn January through March, 2002, submitted on MSWord forms and manually totaled. While figures can give a sense of relative volumes, they should be considered approximate only. Training in the 10 PC3 Teleeenters by the end of March, 2002: 1 User Grouo I Total No. of PeooleTrninedl I State Employees 1 214 Private Businessmen / 232 Students 1 3024 Unemployed 1 380 nth-- --.-.- I dd - TOTAL 1 4157 Group Training: Topic Introduction to Computers Internet, +mail MS Windows I -- . .* Computer Applications MS Paint MS Office Businasand Communications Desktop Publishing English Language Individual Training: 57 No. of Courses 42 39 36 Multimedia for Children Web Design Rognunming in Pascal TOTAL I MS Excel I 12 1 204 1 63 MSWord 19 8 4 17 Study Hours 704 466 4.49 . . 16 1 20.3 Number of Peopk Trained 619 216 la 20 96 20 1 135 474 211 1 175 163 358 56 .. . 187 341 30 331 1 -. 282 7 2290 ANNEX 16: PC3 CONTENT DEVELOPMENT FORMAL PROCESS Excerpted from a Brief Reporf on PC3 Content Developmenf by lliana Nikolova, PC3 Country Director, April 2002 Introduction The initial concept for the content development component of the PC3 project has been the following: "AED will assist selected Bulgarian organizations to convert existing information resources and training materials broadly relevant to the economic and social development of under-served communities into Bulgarian language computer-based and Internet accessible formats." This initial concept was too broad and vague and during the implementation phase of the project, through discussions within the PC31Sofia project team and with AED and USAIDISofia coordinators, it was further shaped and became more focused on PC3, namely content development related to PC3 centers information needs. Approach to PC3 content development: Studying PC3 information needs A survey of the PC3s information needs was organized (via e-mail) by PC31Sofia office. The PC3 operators were asked to do local research on the existing and potential information needs of their existing and potential clients and to summarize the findings in an email to the PC3 office. Then the individual inputs were integrated and a final list of PC3 center information needs was compiled. Identifyingpotentia content providers PC3ISofia team together - with Nora Ovcharova USAIDISofia PC3 coordinator, "scanned" the local organizational environment (USAID partner organizations, other donor organizations, NGOs. educational institutions. etc.) and identified about 55 organizations and individuals - who might have suitable content. They included Bulgarian ministries, associations, and NGOs, Sofia University departments, donor organizations, commercial entities such as CISCO, and the Peace Corp~.~ All were invited to attend a PC3 Content Workshop on December 20 in the American Center, Sofia, and nearly all came. PC3 Content Workshop The workshop objective was to bring together demand (the PC3 operators) and supply (potential content providers). The Sofia project office supplied invitees with supporting data and identified PC3 information needs, agenda, and suggestion forms. Over 50 participants gathered, the potential content providers making short presentations on potentially relevant Web sites, databases, training sites, educational programs, and online courses. The Sofia project office subsequently analyzed the 24 suggestions offered by 11 people and developed a final list based on appropriateness, feasibility, added value, and cost of development. Items already on the Web were added to the "PC3 Internet Catalogue" product. The event had an unplanned effect. It provoked considerable interest in the PC3 project achievements, and some presenters requested additional meetings and partnerships. ANNEX 17: PC3 "HOW TO...?" REFERENCE TOPICS A Bulgarian language reference and self-training tool was organized and prepared by the Sofia Project Office on topics frequently requested by PC3 operators and their clients. The "How to ..." reference was produced in both CD-ROM and Web versions for distribution to the PC3s. [ Education, I Educational system in Bulgaria science, and culture - Preschool education I Schonl education I - University education Professional education Language education Distance learning Encyclopedias, dictionaries, references, training 1 materials 12&< 2208~ >so YEARS 20 50 YEARS YEARS YEARS ........ - - ...................... PC3 Coupon Usr Groups 9lOl-M2 .......... PC3 Coupon Usr Senhas 9ml-3RlZ PC3 Coupon Use Times PC3 Coupon Users by Gender FIGURE 2: PREPAID CARD UTILIZATION DATA Aitos Apriltzi Biala Madan Omurtag Pravec Tryavna Varshec Vetren Zlatograd Total Coupon users: Services: Trainina Teacher 254 312 150 2i9 334 Health worker 32 100 33 176 144 Govtemploy 234 551 588 154 176 Business 333 515 408 635 320 Student 4487 1446 3440 1942 3198 Unemployed 355 391 260 262 860 Other 17 168 68 101 1036 lnteme-web E Mail Chat 1205 798 1061 513 860 Applications 16 41 58 103 68 Faxicopying 215 285 37 17 335 Other 2 22 435 2 53 PC3 Services by Coupon User Groups 9/01-3102 Training InterneWeb E Mail Chat Application use Faxlcopying Other Total Teacher 323 1664 209 123 453 545 116 3433 Health worker 468 565 38 22 146 138 143 1520 State employee 760 2097 148 234 267 499 614 4619 Private business 645 3250 244 357 343 493 1113 6445 Student Unemployed Other' Total See table following for definitions of "other" users and services. Prepaid Card Utilization After Five Months Aitos Apriltzi Biala Madan Omurtag Pravec Tryanva Varshec Vetren Zlatograd Cards received 1400 500 800 500 800 500 800 500 500 500 Cards distributed 1101 490 595 348 590 484 691 438 496 380 % Distributed 78.64% 98.00% 74.38% 69.60% 73.75% 96.80% 86.38% 87.80% 99.20% 76.00% Cards reimbursed 602 373 495 353 607 359 383 364 354 243 % Used' 43.00% 74.60% 61.88% 70.60% 75.88% 71.80% 47.88% 12.80% 70.80% 48.60% %Unused' 45.32% 23.88% 16.81% -1.44% -2.88% 25.83% 44.57% 85.39% 28.63% 36.05% ^ percent of receivedcards used by clients; percent of distributedcards not reimbursed and therefore unused or lost FIGURE 3: "OTHER" COUPON USERS AND SERVICES "Other" Users (as described by PC3 operators) accountant administrator agronomist architect archives worker artist Bank manager bar tender building engineer cashier casino worker concert master cook courier driver economist electric engineer engineer fireman fitness vainer folklore group forklift driver furniture worker "Other" Services scanning printing CD recording Floppy disk recording Windows innallation document preparing foliage binding death note making advenisements WEB advenisements gas smion worker guard HGO historian jurist lawyer library worka manager military miner mining engineer motherhood museum worker musician Peace corpse volunteer pensioner pharmacy worker police officer policeman post office worker priest programmer inviiations menus business cards translation downloading web design ip telephone binding materials brochure preparing VAT reference wnsumnbles repom school worker m seller shift chief singer -Playa social eare worker soldia SOS-kid village store worker tailor tax adminimator technician traffic manager mer typist veterinarian waiter waiaesr worker FIGURE 4: PC3 SERVICES USE PC3 Services by Client Groups Coupon Users 9B1-3B2 State emplow nea~th wrbr 4NHG MB EMAL CHAT BPPUCA FAX6 OTHER TlON COPY PC3 Services Used by Client Age Groups 900 zoo 1 no - C = 0 U 0 Figure 4: PC3 Services Use (contd) PC3s Client Payment Method C OUPOns and money money c oueons mu IS + ma Coupon Users & Paying Users 1 SFe b-1 SMar 2002 5000 T ! municipalities FIGURE 5: PC3 PAYING CUSTOMERS FIGURE 6: PC3 TRAINING SERVICES AND PERFORMANCE PC3 Client Trainees Coupon Users Figure 6: PC3 Training Services and Performance (contd) PC3 Training Services 2000 7- - .- m 0 0 D (D asz a, p 2 Ec c - E?g- 5 - ,3z .- m E -02 3 F -E (D V) a, - a f; r LU V) sg?z .- E 8 c 02 m - a, m C - 0 2 :: - k a, 2 .- C n - 9 0 Y) - C 7--- m' s jmNUMCLlENTS TRAINED. ALL PC3s / - --. -. - -- .. PC3 Training Sewices Performance 6 TOTAL NUMTRAWES (ndrvtclass) rn TOTAL WURS of BJSTRUCIKW\] ' FIGURE 7: PC3 TELECENTER PERFORMANCE PC3 Incomes Compared PC3 Performances 9m1-3/02 - - ALLSVCEVBiE -CMTRAB (indiv~lass) -SVCTYF€NCFIU\SE(%) +- CAFIDS GBwBURSKI (%) -CAI333 LOST (%) FIGURE 8: PC3 INCOMES first and last months of project *includes coupon reimbursements? 1 Total Income (from highest to lowest first month I Total h~ome (from highest to lowest % difference) FIGURE 9: PC3 INCOME TOTALS AND BY TYPE OF USE Services income 1 Income from Internet Use and All Other 1 All PC3s (ranked by % difference, highest to lM) First Last % Month Month Difference I Services I ,411 PC3 between All other flrst Last % S~MC~S: Month Month Dmerence Coupons reimbursed Web site creation Application use Decorative printing Sales Disk recording Other services Scanning Telephone use Training Hardware & LAN services Document prep Photocopying Business support Faxing Document production lntemet use Games per hour Total per month Coupons reimbursed Web site creation Application use Decorative printing Sales Disk recording Other services Scanning Telephone use Training Hardware & LAN services Document prep Photocopying Business support Faxing Document production Games per hour Ail Other se~Cf?S Income lnternet use Income FIGURE 10: INTERVIEWED USERS' REASONS FOR USING PC3 TELECENTER SERVICES Reasons for Using PC3 Setvices lnfonne Knowl- I'm Relaxa. Gam Emer- lmpmva Competi lmpmva lbn edga wolking tbn skills laimnt IWE ng with job here job others charcas Source: Data collected by Club Econornica, April 2002 FIGURE 11: ENVIRONMENTAL FACTORS AFFECTING PC3 GROWTH Legend: Missing Evaluation points: No or insignificant influence of this factor on Me PC3 center 1 =weak influence of the factor External 2 = average influence of theef&or 3 = strong influence of the factor PC3 Site Aitos Aprilki Internal 'Evaluation of the Factors Restricting PC3 Development" by Club Economica. April 2002 Source: Operator and neighborhood interviews, April 2002 Short￾term Competition 3 Total evaluation points 5 12 Poverty, unemployment 1 1 High speed of IT development 1