December 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Davorin Pavelic (Team Leader), Jasmina Mangafic (Business Development Specialist) and Danijel Hopic (Senior Evaluation Specialist). PERFORMANCE EVALUATION OF THE EXCELLENCE IN INNOVATION ACTIVITY: PERFORMANCE EVALUATION OF COMPETITIVENESS IMPROVEMENT THROUGH THE APPLICATION OF INFORMATION AND COMMUNICATION TECHNOLOGIES IN FOUR SECTORS IN BOSNIA AND HERZEGOVINA April, 2014 USAID BiH EXCELLENCE IN INNOVATION COOPERATIVE AGREEMENT AWARD No. 168-A-00-07-00101-00 DISCLAIMER The views expressed by the authors in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Performance Evaluation of the Excellence in Innovation Activity CONTENTS Acronyms 1 Executive Summary 2 Purpose of the Evaluation and Evaluation Questions 11 Project Background 12 Evaluation Methods and Limitations 17 Findings, Conclusions and Recommendations 24 Annexes 55 Annex I: Evaluation Statement of Work 56 Annex II: Data Collection Instruments 66 Annex III: Sources of Information 72 Annex IV: Disclosure of Any Conflict of Interests 74 Annex V: Questions for the EIA Self-Assessment 77 Annex VI: Request for Information to e-BIZ Centers 80 Annex VII: E-BIZ Centers’ Stuff Survey 81 Annex VIII: SME Survey 86 Annex IX: Questions and Concerns Expressed by the EDC 87 ACRONYMS AED The Academy for Educational Development AFIP Agency for Financial Information and Consulting Services AOR Agreement Officer Representative BiH Bosnia and Herzegovina CAD Computer Aided Design CAM Computer Aided Manufacturing CNC Computer Numerical Control COP Chief of Party CSA Cost Sharing Agreement EDC Education Development Center EIA Excellence in Innovation Activity FBiH Federation of Bosnia and Herzegovina FOG Fixed Obligation Grant ICT Information and Communication Technology NMFA Norwegian Ministry of Foreign Affairs PMP Performance Management Plan SME Small and Medium Size Enterprise SOL Solicitation TTC Textile Technology Center USAID United States Agency for International Development 1 EXECUTIVE SUMMARY This report is a performance evaluation of an activity funded by the United States Agency for International Development (USAID) mission in Bosnia and Herzegovina, the Excellence in Innovation Activity (EIA), an economic development activity that commenced in December 2007 and ended in August 2013. The aim of the Activity was to enhance the competitiveness of SMEs within four industries in Bosnia and Herzegovina through the transfer and application of information and communication technologies previously inaccessible to SMEs in BiH. The prime awardee/implementer for the Activity was ‘fhi360’ while the primary implementing partner responsible for the day-to-day implementation was the Education Development Center (EDC). The Evaluation was conducted and completed while the EI Activity was still ongoing. Information and data collection commenced in June 2013 and the first interview was on 24 June 2013 with the EIA, its COP and other staff members. Information and data collection ended in August 2013, with the last interview held on 23 August 2013 with the EIA. The Evaluation Report was prepared on the basis of information and data received within the aforementioned period. PURPOSE OF THE EVALUATION AND EVALUATION QUESTIONS The evaluation served a dual purpose: (1) to provide information for the design of a new USAID project1 within the Economic Growth Portfolio and (2) to learn to what extent the objectives and goals of the Activity were realized. The evaluation reviewed the accomplishments of all four USAID supported e-BIZ centers taking into consideration both the original Cooperative Agreement period and the subsequent extension. The main evaluation questions sought answers in relation to the level of success achieved in terms of capacity building efforts aimed at e-BIZ center staff, improvements in the competitiveness of the served SMEs as well as the targeted industries in their entirety and an analysis of the financial and operational 1According to the new USAID definition, “A project is defined as a set of executed interventions (activities) over an established timeline and budget intended to achieve a specific development purpose in order to solve a defined problem. In general, it is expected that a project purpose will correlate to intermediate results (IR) in the CDCS’ Results Framework”. 2 sustainability of the e-BIZ centers. The Evaluation Team has strong reservations about the information and data provided by EIA, the e-BIZ centers and interviewed SMEs. The EIA and e-BIZ centers interfered with the evaluation process at several points by adjusting documents and providing documents that were produced after the evaluation process had begun. This interference made a realistic evaluation of project performance difficult and challenging. The Evaluation Team is aware of the Activity implementation issues that caused discontinuity in the project management flow and had negative impact even on some e-BIZ Centers, such as Exploring BiH. Here we primarily refer to: (i) AED suspension from receiving new U.S. Government awards pending an ongoing investigation by the USAID Office of Inspector General (OIG); (ii) Dismissal and replacement of COP at the late stage of the project; (iii) issues with involvement of Exploring Macedonia in the Activity and issues related to ownership of Exploring Macedonia and eFondacija over the Exploring BH and attempts of Exploring BH Director, Ms. SV, to resolve these issues by investing her own capital into the project and takeover Exploring BH. PROJECT BACKGROUND The Excellence in Innovation Activity utilized information and communication technology (ICT) as part of the USAID effort to generate employment and promote the expansion and competitiveness of SMEs in BiH. The Activity aligned to the Foreign Assistance Framework Economic Growth Program in the area of Private Sector Competitiveness and the program element Private Sector Capacity. The EIA provided assistance to and collaborated with BiH entrepreneurs and firms to establish and operate sustainable e-BIZ centers/enterprises capable of offering high impact ICT applications and services to SMEs in the targeted and supported sectors. Collaboration with the e-BIZ centers/enterprises was intended to enhance the capacities of SMEs to utilize the ICT applications and services offered fully with a view to improving their competitiveness. The EIA furthered the USAID BiH strategy on economic growth in terms of the priority to increase growth and the competitiveness of SMEs in the local, regional and global marketplace. In cooperation with the Norwegian Ministry of Foreign Affairs (NMFA), during the original period (2007-2010) the EIA established four e-BIZ centers with one in each of the following four sectors: wood processing (USAID funded), tourism (USAID funded), machine tools (NMFA funded) and agriculture (NMFA funded). Two additional USAID funded e-BIZ centers were established during the extension period (2010-2013): one to support BiH SMEs in the textile industry and the other to support various industries related to transport/logistics services. USAID funded the establishment of four e-BIZ centers through four sub-grants or fixed obligation grants (FOG) to the following companies/legal entities: • ‘Artisan’ (a wood processing company) to start and develop ‘Arteco’ (a wood products e-BIZ center); • ‘Fondacija e-turizam’ to start and develop Exploring BiH (the tourism portal e-BIZ center); • ‘Kismet’ to start and develop the Textile and Technology Center (TTC) e-BIZ center to support SMEs in the textile industry; and • ‘Satwork’ to start and develop ‘Transwork’ (the e-BIZ center aimed at supporting SMEs from various industries related to transport/logistics services). 3 EVALUATION QUESTIONS, DESIGN, METHODS AND LIMITATIONS In order to answer the main evaluation questions, relating to areas such as increased capacity amongst e-BIZ center staff, the financial and operational sustainability of the e-BIZ centers and increased competitiveness of SME clients, the Evaluation Team performed a desk review of the existing documentation created prior to and during project implementation. The team also carried out preliminary interviews with management and staff from the e-BIZ centers as well as the EIA staff and conducted other key informant interviews and questionnaire surveys. The Evaluation Team compared the data collected through the surveys and interviews with the data contained in the EIA reports. The team analyzed financial statements in order to determine any changes in the financial performance indicators for the e-BIZ centers. The team used a number of financial analysis techniques to determine the level of financial sustainability of the e-BIZ centers as well as to determine potential changes in the competitiveness of the SME clients. In addition, in order to evaluate the performance of the e-BIZ centers in a comprehensive and objective manner the Evaluation Team used a balanced scorecard tool. The tool uses both non-financial and financial measures to provide a broad overview of their business performance. The team also analyzed the level of competitiveness of the client SMEs by comparing the data gathered through the surveys and interviews with that of the SME managers. The main limitations were the questionable validity and reliability of the data provided by the e-BIZ centers and the EIA, a complete absence of baseline data on the indicators relevant to the evaluation questions, the shortcomings of “post-then-pre” methodology1, the fact that the characteristics of the comparison group could not be taken into consideration and the extremely low response rate for both groups of SMEs (client and non-client). FINDINGS AND CONCLUSIONS Each of the e-BIZ centers had a different impact on the competitiveness of the SMEs within their respective industries. Yet when considering the increased competitiveness of the SMEs the only certain thing is that the EIA significantly raised the competitiveness of e-BIZ centers’ parent companies. While it was not possible to confirm any impact on competitiveness of SMEs in the tourism sector covered by Exploring BiH, all interviewed SMEs from the logistics and transport sector confirmed that the services provided by the e-BIZ center Transwork had improved their control over their operations and thus reduced their costs. Transwork is the only e-BIZ Center that performed as expected when it comes to the number of clients served and its impact on clients’ competitiveness. However, the EIA distorted the market in logistics and transportation industry by supporting one of the numerous companies that provides GPS monitoring and tracking service. The attempt was made to differentiate Transwork from other similar service providers by creating a bundle of products that does not exist as a comprehensive package of services in BiH market. However, Transwork’s sales data show that over 90% of revenues are coming from sale of GPS devices, installation of GPS devices and subscription to GPS tracking service. The E-BIZ center Arteco made quite an important contribution to the wood processing sector by creating an outsourcing opportunity for SMEs in the development of product technical documentation and prototypes. However, since the design of new products is not so frequent in BiH present demand amongst SMEs within the wood processing sector for this service remains low. Greater demand for prototype development came from foreign customers. 4 The e-BIZ center TTC has also brought important services to the market such as fast pattern/prototype development, grading, marker making and cut precision. It is not that the technology for these operations does not exist in BiH SMEs or that they cannot operate without it, because all of these operations can be done manually, but rather that the existing process is slow and the precision of the cut is not high. In addition, when the SMEs receive clothing patterns in digital form they usually do not have the ability to access them and transfer them to material; this is the point where TTC steps in. As one of the interviewed SME owners stated, “Given that we have no ICT capacity required we would never get a job with our present foreign client. However, we were able to convince them that we have the necessary ICT capacity by showing that we use the services of TTC as our ICT resource center”. Almost 90% of BiH textile companies work for foreign customers on basic cutting and sewing operations based on received patterns; however, ICT capacities in the majority of these companies are non-existent. Therefore, if adequately mentored and managed, TTC could become the ICT resource center for many BiH textile companies presently without any ICT capacity. Unfortunately, after being operational for almost two years TTC’s outreach remains low and it continues to use its resources mainly in support of production in its parent company. EIA management of the M&E system was poor. The EIA invested little or no effort to establish baseline data or track changes in the competitiveness of SMEs within the targeted industries. The Evaluation Team effort to reconstruct this data through an SME survey failed due to the extremely low response rate amongst the e-BIZ centers’ clients. In the Start-up Implementation and Annual Work Plan2 the proposed method for tracking increases in the competitiveness of SMEs is the number of customers that report that e-BIZ center services have strengthened their business through increased productivity, increased sales, decreased costs and other improvements. The plan was that the EIA perform annual surveys in order to determine any changes in the above-mentioned indicator. The EIA was expected to perform the first survey of this kind after the first year of the centers’ operations. Yet during preliminary and informal interviews conducted with the EIA, on June 24 2013, EIA staff stated they had not performed the survey through which customers could report whether or not their business had been strengthened by using the services of the e-BIZ centers. Despite the fact that this survey is included in the EIA M&E plan as well as in the Data Quality Assessment as one of the major tools for tracking EIA progress, there is no evidence that the EIA surveyed customers of all four e-BIZ centers after the first year of the Activity. The e-BIZ centers for which the Evaluation Team can confirm that they have performed this survey are Transwork and Arteco. Arteco hired an external consultant to perform this survey. On July 17 the Evaluation Team sent a Request for Information to the EIA (see Annex 5) asking for a report on “the results of the surveys of the e-BIZ centers SME clients“ performed after the first year of center operations (see question 3.4), as specified in the Start-Up Implementation Plan. 2 Start-Up and Implementation Annual Work Plan; DOT-ORG BiH Excellence in Innovation Project; Academy for Educational Development; USAID/BIH Excellence in Innovation Cooperative Agreement Award No. 168-A-00-07- 00101-00 8 February 2008. 5 The Evaluation Team received a folder titled 'E-BIZ Center's First Year Clients' Survey’ on July 26; however, a subsequent review of the submitted documentation revealed that the documents had been created on 25.07.2013. The review of the documents also resulted in the following findings: o three of the surveys were done by the e-BIZ centers and only one by the EIA (although according to the project documentation the surveys were the obligation of EIA); o the questionnaires used were typical customer satisfaction questionnaires with little or no questions related to the impact that the services of the e-BIZ centers had had on their customers’ competitiveness (despite the fact that the impact that the e-BIZ centers had on their clients competitiveness was the main purpose of the survey); and o the only survey conducted by the EIA, in addition to the e-BIZ center clients, also interviewed potential e-BIZ center clients i.e., SMEs interviewed during the market survey. The EIA invested a lot of effort in order to increase the capacity of the e-BIZ centers to attract and retain customers; e-BIZ center staff evaluated the training and direct technical assistance as good and helpful. Sales and marketing were the areas that the EIA gave highest priority. Capacity building activities related to sales and marketing ranged from two organized trainings3 on marketing, the provision of technical assistance for the development of marketing plans and the provision of advisory on-the-job assistance. Only Arteco e-BIZ Center staff expressed reservations about the value of certain marketing methods they had been asked to use (promotion and marketing via e-mail). However, the number, frequency and staff coverage of the group trainings/seminars organized by the EIA could have been higher. A few organized group seminars related to marketing and financial management were organized; however, Transwork4 and TTC staff did not attend any of these marketing and financial management trainings and the EIA did not make any effort to repeat them after the two new e-BIZ centers had been established and included in the Activity. Capacity building efforts to increase e-BIZ center staff knowledge and skills in areas of high impact information and communication technology and services were basic and insufficient. The quality of human resources is the key success factor that will determine an e-BIZ centers ability to become a center of excellence for the dissemination, transfer and application of high impact ICT in the target industries. It is the highly educated employees with top-notch expertise in the application of ICT and the capacity to provide continuous new and innovative services within their respective industries that affords e-BIZ centers their sole competitive advantage and makes them unique and different from other manufacturing shops, in contrast to easily replicable equipment and services. 3 BiH Entrepreneurship and Business Development Seminar (a one-day seminar on marketing strategy, customer/ service protocols, effective networking etc.) held on 31.08.2010 and Power of Sales (a two-day seminar) held 30- 31.03.2011. 4 The Activity in its 2011 Annual Report reported false statement that, beside Satwork’s owner, Transwork staff also attended this two-day seminar. Please see page 3 of the Annual Report for the period January – December 2011. 6 Many factors prevented the buildup of the e-BIZ centers’ top quality teams. The first relates to insufficient attention and effort to provide training for e-BIZ center staff in ICT and ICT applications in specific industries. Having in mind that e-BIZ centers are supposed to be centers of excellence for the dissemination and transfer of new high impact information and communication technologies, the EIA has done too little to build e-BIZ center staff knowledge and skills in the application of high impact ICT. The second one is the remote geographic location of two e-BIZ centers (Arteco and TTC) which limited their ability to attract employees with university level education. A review of the CVs of e-BIZ center staff revealed that, with the exception of Transwork, all e-BIZ center staff had a rather low average level of education attainment. In addition, in the majority of cases their previous working experience had little relevance for the functions they perform within the e-BIZ centers. The third factor relates to the roles and responsibilities parent companies assign to the e-BIZ centers. Transwork is a case in point. Transwork’s primary role in parent company Satwork is marketing and sales. Thus, employees hired for the Transwork e-BIZ center were skilled primarily in marketing and sales, not ICT or logistics, as needed to achieve the goals of this e-BIZ center. EIA’s capacity building efforts to increase the knowledge and skills of e-BIZ center staff should have been stronger and a priority. The TTC and Arteco e-BIZ centers were the only ones to build technology related capacities. These two e-BIZ centers received machines and equipment as well as training on how to operate them from the producers of the machines/equipment. The other two e-BIZ centers and their staff did not participate in any type activity related to technology skills capacity building. Nevertheless, surveys and interviews where e-BIZ center employees self-reported on improvements in their knowledge and skills indicated that the EIA increased the capacity of e-BIZ center staff. Interviews and a questionnaire survey of e-BIZ center employees confirmed that they were confident in their own capacity to perform their jobs at top performance level. The majority of staff had received some form of training and reported positive improvements in certain skills. Yet Exploring BiH did not have any employees except for the Director, while TTC and Transwork staff did not attend marketing and financial management trainings. EIA project staff provided most technical assistance in the form of day-to-day hands on capacity building. The EIA performed exactly as stated in the Start-Up plan, “The Project will undertake focused training and technical assistance (T&TA) most of which will be provided to the e-BIZ centers by EIA Project staff for day-to-day hands-on and in-depth capacity building”.5 The e-BIZ center staff evaluated this technical assistance as useful. Operational management - The Evaluation Team could not confirm any capacity building assistance directly targeted at improving managerial skills or in adopting and integrating technology in order to gain maximum efficiency and productivity. The Evaluation Team was also unable to confirm any capacity building assistance directly targeted at improving knowledge and skills in other management areas i.e., strategic planning and human resource management. 5 Start-Up and Implementation Annual Work Plan; DOT-ORG BiH Excellence in Innovation Project; Academy for Educational Development; USAID BIH Excellence in Innovation Cooperative Agreement Award No. 168-A-00-07- 00101-00 8 February 2008. 7 All four e-BIZ centers are financially sustainable - This conclusion is based on a simplified and narrow definition of financial sustainability as provided in the Start-Up and Implementation Annual Work Plan6. It is also based on an analysis of e-BIZ center financial reports7 that, in the case of TTC and Arteco, were adjusted during the course of this evaluation, while the background caveat shows that a significant part or the majority of the e-BIZ centers’8 revenue came from their parent companies. After a request for documentation was placed by the Evaluation Team we received financial statements for all years of operations of the e-BIZ centers. The Evaluation Team also asked for the semi-annual financial statements prepared up until the end of June 2013. In the case of Arteco, the EIA, on its own initiative, submitted new financial reports the following month that covered the month of July 2013. These two financial statements were fundamentally different and indicated structural changes in business operations. If, for example, we look at the structure of liabilities and more specifically the liabilities towards the related legal entities9 we can see that on June 30 2013 they amounted to 118,000 BAM. Looking back through the previous annual financial statements the Evaluation Team determined that this amount of liability had existed since the beginning of the Activity. The parent company, Artisan, created this liability. Instead of contributing capital to Arteco, as it was obliged to according to the Fixed Obligations Grant, Artisan gave money to Arteco in the form of a loan. After June 30 2013, in an attempt to improve its financial position, Arteco suddenly eliminated this liability from the balance sheet, compensating for it through the accounts receivable that Artisan owed Arteco for the services provided. The result of this accounting exercise was a decrease in Arteco’s total assets to the amount of 109,429 BAM or 47.06%. The amount of 118,000 BAM that was supposed to be part of the agreed contribution (cost-share) of the owner to the Activity was thus transformed into a liability of the e-BIZ center to its founder. Figure 1: Arteco’s balance sheet adjustment for the period June 30 - July 30 2013. 6 “Definition of sustainable: At the end of the project, each e-BIZ Center has: (a) an annual operational and financial plan that is based on past performance, which demonstrates that the Center will reach the break-even point within one year of the end of the project; (b) the e-BIZ owner’s written commitment to continuing operations of the e-BIZ Center operations after the end of the project” 7 The interview with Arteco’s management wherein they stated that they were asked to adjust their financial reports was audio recorded. 8 Also in the case of TTC and Arteco 9 Since Artisan and Arteco should, in line with the Project, be two independent legal entities and not a parent-child business this method of accounting entry is also incorrect. Owner (Artisan) Arteco d.o.o. BIZ C t (2) Loan amounting to 118,000 (1) Accounts payable amounting to 128,250 BAM 8 If the parent companies are taken out from equation then both Arteco and TTC are not financial sustainable. TTC generates between 80% (2012) and 90% (2013) of its total revenue through services provided to its parent company, Kismet d.o.o, and to Vocal Productions d.o.o. Arteco generates about one-third of its total revenue by providing services to its parent company, Artisan d.o.o. All of the e-BIZ centers, with the exception of Transwork, had poor financial stability. Other types of financial analysis, such as Z-Score and the Kralicek Quick Test, indicated that all of the e￾BIZ centers, except for Transwork, had issues with financial stability. The e-BIZ centers did not achieve the financial benchmarks set in their respective business plans. For example, the business plan projections for TTC’s revenue were not achieved. Similarly, Arteco failed to achieve the financial projections and target capital values specified in its business plan. The financial reporting system for the e-BIZ centers was inadequate. The e-BIZ centers were never instructed in how to record the USAID and parent company contribution in their financial statements. The Evaluation Team was not in a position to verify the amounts and structure of the cost￾share contribution from the founders of the e-BIZ centers. When it came to operational sustainability, according to the definition from the Cooperative Agreement, each e-BIZ center must be operationally sustainable10 i.e., each e￾BIZ center must have a business plan. However, each e-BIZ center has different operational capacities to achieve its vision. The level of operational capacity/sustainability depends on the level of dependence on the respective parent company. An e-BIZ centre with a high level of dependence on its parent company equates to less embedded capacity. For example, the e-BIZ center Transwork has no capacity in the area of information and communication technology design, development and implementation. All technological capacity rests with the parent company Satwork, while Transwork acts solely as a sales and marketing department of its parent company. Exploring BiH has no capacity at all within itself or within its legal founder. Transwork is the only e-BIZ center that achieved its targeted number of clients. TTC did not even manage to achieve 50% of its targeted number of clients. According to data received from its accountant, Exploring BiH did not manage to achieve the targeted number of 70 clients. Finally, Arteco served many clients but only 38 SMEs from Bosnia and Herzegovina. The majority of services provided by the e-BIZ centers were not new or non-existent in the market. All of the services provided by TTC were either already present or can be done manually. The main contribution of TTC’s services to some of its clients came in the form of precision tailoring and faster production. Prototype development was the only unique service provided by Arteco. Transwork services can be provided by a dozen other service providers, while Exploring BiH is perceived as just another small travel agency. The results of the SME questionnaire survey also confirmed that although awareness of the services provided by the e-BIZ centers was high the majority of SMEs did not consider these services to be new or previously non-existent in the market. 10 Operational sustainability is defined as a, “Center has a business plan or annual operations plan that defines its offerings and the revenue needed for self-sustainability by the end of the project”. 9 Financial data on changes in the volume of sales of SMEs that received services from the e￾BIZ centers does not constitute the basis for the conclusion that the e-BIZ centers improved the competitiveness of their clients. In all four sectors almost half of the SMEs that received SME services experienced a decrease in their total volume of sales or did not experience any change in total sales volume. The Evaluation Team could not find evidence that the e-BIZ centers had an impact on associations. One of the reasons for this could be the attempt by the e-BIZ centers to use the trade associations as a marketing channel through which to sell their services. In addition, it might be the case that the associations and the public sector alike perceived the entire project as a private company endeavor. The Evaluation Team could not find evidence that the e-BIZ centers had had any impact on the sectors/industries. Financial data on profitability in all four sectors did not indicate any significant improvement in the competitiveness of the target industries. According to existing studies, over the period 2009-2011 there was no increase in total employment in the wood processing sector. In contrast, the entire sector lost 17% of total jobs during the period 2008-2009.11 Part of this could be attributed to the general economic downturn; however, the Evaluation Team could not find any evidence that the EIA had had any impact on increasing employment in the wood processing industry. 11 BiH Industrial Outlook for 2012: Wood and Metal Processing Sectors; Fostering Interventions For Market Advancement (FIRMA) September 2012. 10 PURPOSE OF THE EVALUATION AND EVALUATION QUESTIONS PURPOSE OF THE EVALUATION This evaluation served a dual purpose: (1) to inform the design of a new USAID project in the Economic Growth Portfolio; and (2) to learn to what extent the project’s objectives and goals have been achieved. Additionally, USAID Evaluation Policy prioritizes the building of local evaluation capacity and the use of local expertise in evaluations. USAID BiH procured this evaluation using solely local expertise in order to inform the design of its future M&E contract. The evaluation of the EI Activity reviewed the accomplishments of all four USAID-supported e-BIZ Centers. The evaluation considered both the original contract period and the extension. The primary audience for this evaluation is the USAID Mission in BiH and USAID Washington. Secondary users are the implementing partner(s), local organizations with whom USAID collaborates and other donors. The performance period began on July 18, 2013 and the fieldwork was completed on September 5 2013. EVALUATION QUESTIONS Text box 1: Evaluation Questions Question 1: To what extent has the EI Activity succeeded, in a measurable way, in capacity building for e-BIZ center Staff? Question 2: Are all four USAID supported e-BIZ centers financially and operationally sustainable? Question 3: To what extent has the EI Activity succeeded, in a measurable way, in improving the competitiveness of the SMEs served by the four e-BIZ centers? Question 4: What impacts have USAID‐ supported e‐ BIZ Centers had on associations, sectors, and industries targeted by the interventions? Question 5: What have been the main obstacles in the implementation of the EI Activity? 11 PROJECT BACKGROUND The EIA was designed to utilize ICT as part of the USAID effort to generate employment and promote the expansion and competitiveness of local SMEs in BiH. The Activity aligned to the Foreign Assistance Framework Economic Growth Program in the area of Private Sector Competitiveness and the program element Private Sector Capacity. The specific objectives of the EIA were to strengthen the competitiveness of SMEs in selected sectors by increasing their use of value-added ICT to offer new high value-added products and services in demand in the domestic and/or global markets. The EIA performed an analysis of all four of the target industries in order to determine the situation concerning the use of ICT by SMEs in BiH. The Industry Analysis of the wood industry revealed that most of the sector was made up of companies with less than 25 employees, which were not able to invest in new technology such as CAD/CAM, research and development and marketing. These companies also had trouble enforcing quality control standards and in the design of modern furniture. The inability of SMEs to invest in CAD/CAM and the absence of CAD/CAM outsourcing services led to the conclusion that the competitiveness of the wood processing sector could be improved if access to CAD/CAM, manufacturing capabilities that deal with ‘hard to produce’ specialty pieces of furniture and CAD/CAM and CNC training were to be provided. In the tourism sector it was determined that very few tour operators in the country choose to focus on inbound tourism, since BiH does not have adequate hotel capacity, infrastructure and activity venues that are competitive in the leisure marketplace. The main shortcomings in the use of ICT within the tourism sector were the absence of online booking mechanisms and the online promotion of adventure tourism and recreational activities such as rafting, biking and climbing that are poorly structured and organized. The same situation existed in the online promotion of attractions such as parks, historical sites and cultural festivals i.e., the Sarajevo Film Festival and Sarajevo Jazz Festival. Based on the estimated increase in the importance of Internet reservations and BiH’s focus on niche adventure travelers, free independent travelers and occasional transiting tourists who book their rooms online it was determined that an online reservation and booking system would help promote inbound international and national tourism. This would also provide an effective and efficient mechanism for the distribution of information along the value chain. The main conclusion of the study was that, besides the previously mentioned online booking and reservation system, a well designed online marketing and promotions service and a function that enables tailored online packaging are potential high-impact and sustainable ICT applications for a Tourism e-BIZ center.12 When it came to the logistics and transportation sector the majority of companies had less than 25 employees and less than 10% of them were specialized, the majority dealt with general goods and less than 10% worked solely on the domestic market. A very small number of companies from the 12 Source: USAID BiH, Excellence in Innovation, ‘Identification of High Impact Applications and Opportunities in the Tourism Sector’ April 4 2008 briefing paper. 12 transportation sector had integrated IT solutions and only a few were using GPS to track the location of their trucks. The EIA identified two high impact IT facilitated services that would reduce costs and improve the efficiency of SMEs. The first was Fleet Management Systems (FMS) used to track trucks and other assets used in transportation, including the location, usage and information about them as well as the scheduling of their use. The second was the online supply and demand truck market where both customers (in need for transport) and suppliers (transport and logistics companies) post their requests; this system can also be operated via SMS. Although there is a European portal for the supply and demand of freight (www.timocom.com) there is no local portal in BiH.13 About 90% of output in textile and clothing related to LOHN-based services (sewing and finishing clothing) was for customers based in Germany, Italy and Austria. Production was fully dependent on foreign clients and labor rates were low so the industry operates with low profit margins (circa 5%). The number of BiH companies that adopted a business model for building up their own brand and sales channels remains very small. Most BiH LOHN producers focused on (non-computerized) cutting and sewing services as their EU buyers supplied them with CAD generated paper roles with graded patterns. Several larger firms utilized CAD/CAM technology but few had even earned the trust of their partners to the point where the buyers supplied them with general model sketches and allowed them to complete the computer assisted design and production of EU bound clothing products. On the other hand, almost all small BiH producers did everything manually and used paper cutouts as they usually serve as cooperating partners for bigger BiH firms with orders that exceed their production capacity. The Industry Analysis identified CAD/CAM services (design, pattern making, grading, marker making and material cutting) and CAD/CAM training as factors that could improve the sector’s competitiveness.14 In cooperation with the Norwegian Ministry of Foreign Affairs (NMFA), the EIA established four e-BIZ centers during the original period (2007-2010), one in each of the following four sectors: wood processing (USAID funded), tourism (USAID funded), machine tools (NMFA funded) and agriculture (NMFA funded). USAID funded the establishment of two additional e-BIZ centers during the extension period (2010-2013), one to support BiH SMEs in the textile industry and the other to support various industries related to transport/logistics services. The four USAID funded e-BIZ centers were established through four sub-grants to the following companies/legal entities: • Artisan (a wood processing company) to develop the wood products e-BIZ center; • Fondacija e-turizam to develop the tourism portal e-BIZ center; • Kismet to develop an e-BIZ center to support SMEs from the textile industry; and • Satwork to develop online support for SMEs from various industries related to transport/logistics services. 13 Source: USAID BiH, Excellence in Innovation, ‘Identification of High Impact Applications and Opportunities for Logistics and the Transportation Industry’, May 2010 industry analysis. 14 Source: USAID BiH, Excellence in Innovation, ‘Identification of High Impact Applications and Opportunities Apparel Sector’, May 2010 briefing paper. 13 Objectives and Results Framework The Excellence in Innovation Activity and the e-BIZ centers were expected to achieve a complex network and broad range of objectives and results. These objectives were set within the Start-Up Implementation and Annual Work Plan15 and the Business Plans for all four e-BIZ centers. An overview of all of the objectives and results is given below in Table 1. 15 Start-Up and Implementation Annual Work Plan; DOT-ORG BiH Excellence in Innovation Project; Academy for Educational Development; USAID BIH Excellence in Innovation Cooperative Agreement Award No. 168-A-00-07- 00101-00 8 February 2008. 14 Table 1: Objectives and Results Framework The Excellence in Innovation Activity objectives are to strengthen the competitiveness of SMEs in selected sectors by increasing their use of value-added ICT to offer new, high value-added products and services that are in demand in domestic and/or global markets of the selected sectors. 1 Expected results for the EI Activity are: Result 1 At least two e-BIZ centers that are financially and operationally sustainable by the end of the program providing high impact ICT applications for targeted and/or supporting industries (this result was amended and the number of sectors and e-BIZ centers expanded); and Result 2 Significantly improved competitiveness of at least fifty SMEs in each of the targeted sectors through increased use of value-added ICT (this result was amended and the number of minimum served SMEs was changed). Exploring BiH Arteco TTC Transwork (i) Increase the competitiveness of BiH tourism service providers in international markets. (i) Increase the competitiveness of BiH wood processing SMEs by utilization of modern technology, expansion of the export customer base, application of the best technology solutions in product finishes, etc. (i) Increase the competitiveness of BiH SMEs in the textile industry through the utilization of modern technology, expansion of their export customer base and application of the best technology solutions for product finishes. (i) Increase the competitiveness of the BiH transport/logistics providers in international markets. (ii) Build the capacity of the Exploring BiH e‐BIZ Center employees. (ii) Build capacity for Arteco employees. (ii) Build capacity for TTC employees. (ii) Build the capacity for the e‐BIZ Center employees. (iii) Become self‐sustainable. (iii) Become self‐sustainable by the end of the Project. (iii) Become self‐sustainable by the end of the Project. (iii) Become self‐sustainable by the end of the Project. (iv) Be the most comprehensive source of online information and services for tourists seeking to visit BiH and for tour operators seeking to arrange tours to BiH for their clients. (v) Serve as an online booking gateway for tourism service providers in BiH. (vi) Contribute to an increase in the volume of tourism services in BiH. (vii) Provide services to at least 70 SMEs in the tourism sector during the first two years of operations. (iv) Provide technical services to at least 50 SMEs in the wood processing sector during the years of operation. (v) Contribute towards an increase in the export of processed wood products from BiH. (vi) Contribute to an increase in employment in the wood processing industry. (vii) In addition, Arteco will develop a website that should become an information source for the wood industry and provide the following information: a) database of designers, manufacturers, material suppliers and trainers; b) information on access to finance (banks, microcredit organizations, government grants, etc.); and c) calendar of events (fairs, etc). (iv) Serve at least 40 SMEs from the textile industry by the end of program. (v) Establish linkages between European buyers and BiH textile SMEs. (vi) Highly trained and organized employees. (vii) Continual updating and utilization of state‐ of‐the‐art advanced technology. (viii) The Center will also develop a website that will serve as an information source for the textile industry by providing the following information: (a) database of designers, manufacturers, material suppliers and trainers; (b) information on access to finance (banks, microcredit organizations, government grants, etc.); and (c) calendar of events (fairs etc). (iv) Become the most comprehensive source of online information and services for SMEs seeking and/or providing logistics / transport services in BiH. (v) Serve at least 40 SME from the logistics and other industries to help them cut costs and increase profit. 15 The objectives and results in Table 1 have been somewhat adjusted during the life span of the Activity. The business plans for the e-BIZ centers and the addendum to the Activity Document reflect these adjustments. The addendum to the Activity Document extended EIA interventions to include the transport and textile sectors. 16 EVALUATION METHODS AND LIMITATIONS The evaluation is based on a mixed-method design and is reliant on the benefits gained through triangulation.16 The design of the evaluation, including the methods of data collection and analysis, was determined by the evaluation questions, the availability of baseline data and the M&E reports produced by the EIA. Data was collected using several methods: desk review, semi-structured interviews and surveys. Desk review The Evaluation Team conducted a desk review of the relevant EIA documents, background material and websites. The documents reviewed by the team included the Cooperative Agreement and the subsequent extension, work plans, annual reports and the fixed obligation grants (FOGs) awarded to the four companies that were responsible for starting four new companies (the e-BIZ centers). The teams also reviewed the accompanying business plans for the e-BIZ centers, the market research analyses conducted by the EIA, the Activity M&E Plan, Data Quality Assessments (DQAs) and the financial statements of the e-BIZ centers as well as the CVs of key e-BIZ center personnel. The desk review was the first stage of a triangulation process that helped to independently assess the subject under evaluation from at least three different sources of information or methods of information gathering. Survey For survey purposes, two (2) questionnaires were developed. The first was a Questionnaire for Employees and the second a Questionnaire for SMEs. The Evaluation Team sent the Questionnaire for Employees to all four e-BIZ centers as part of its Request for Information, while the Questionnaire for SMEs was sent to 660 SMEs (see Annex 2) from the targeted industries (both users and non-users of e￾BIZ center services). The questionnaires were designed to collect information on the impact that the e￾BIZ centers had on those SMEs that utilized their services. The team sent the questionnaires via email but the response rate was extremely low for both groups of SMEs (client and non-client). The data gathered through the questionnaire surveys was analyzed using SPSS software (Statistical Package for Social Sciences). Figure 2: The number of responses per industry. 16 Data triangulation requires the use of a variety of data sources, including documents, interviews, surveys and triangulation methods. 11 8 5 8 197 158 112 199 0 50 100 150 200 250 Wood Industry Tourism Transportation Textile industry Response No Response 17 Due to the low response rate, small sample size and non-response bias, the inferences based on the SME survey statistics cannot be considered valid and therefore the survey results are not presented in this report. The response rate was low amongst the SMEs' and the sample limit for the analysis was too small to provide descriptive statistics. The response rate from the wood industry and tourist industry was around 5 percent, while it was around 4 percent for transportation and the textile industry. Semi-structured interviews The Evaluation team conducted interviews with four (4) managers of the e-BIZ centers, two (2) employees of the-BIZ centers (Arteco and TTC) and 24 e-BIZ center clients identified by the EIA. Exploring BiH and Transwork did not have employees that dealt with information and communication technologies and the Evaluation Team was not in a position to conduct these interviews. The EIA provided lists of all organizations and individuals that had received services from the e-BIZ centers; however, these lists contained many individuals, organizations and companies that could not be considered as e-BIZ center clients, as specified by the EIA’s documents.17 We found a large number of individuals, secondary school students, furniture designers and foreign companies in these lists. The Evaluation Team used the following criteria to select SMEs from these lists for interview: balanced territorial distribution (two entities, different parts of BiH), an almost equal number of SMEs from all targeted industries and SMEs of different size (micro, small and medium). The majority of interviews were tape-recorded (with the permission of the interviewees) to enable the Evaluation Team to confirm information if clarification was required. In addition, members of the Evaluation Team also took notes during each interview and later reviewed and synthesized them before preparing the interview minutes. Both team members reviewed the minutes again for the purpose of verification and clarification. To ensure that the data was valid and reliable the team triangulated the interview data between respondents and between sources and identified areas of agreement or disagreement. Financial calculations Financial calculations were used to assess the key financial indicators. For question 1, the Evaluation Team addressed all five areas of EIA capacity building efforts. • Technology skills: to build strong skills in all areas needed for the development/adoption and delivery of high impact ICT application services. • Financial management: to build the ability to track, analyze and allocate finances effectively and to use financial reports to manage a company. • Sales and marketing: for local, regional and international markets (as appropriate) to identify 17 All Activity documents specify that the beneficiaries of the Activity must be SMEs from Bosnia and Herzegovina. Furthermore, as defined in the Annual Report for the period January 2011 – December 2011, the unit of measure for the number of e-BIZ center customers is “the number of customers paying for e-BIZ center services”. 18 and enter attractive markets, respond to market demand, interact effectively with existing and prospective customers, develop and administer effective promotional programs, use proven sales techniques and presentation skills. • Operational management: build managerial skills in adopting and integrating technology in order to gain maximum efficiency and productivity. • Other areas of management: build strategic planning and human resource management skills. The indicator(s) in the EIA M&E plan were not used to track capacity-building tasks within these five areas. The Evaluation Team assessed the EIA reports (and other documents as identified by EIA staff) that addressed capacity building efforts. In addition, the Evaluation Team surveyed all e-BIZ center employees, interviewed e-BIZ center management and key personnel and compared the data obtained with the EIA reports. The nonexistence or incomplete baseline data necessitated the use of an individual recall/retrospective interviewing technique through which respondents were asked to recall the situation at around the time the Project began18. Therefore, in an effort to reconstruct the baseline data, the survey questionnaire for e-BIZ center employees contained ‘before and after’ questions. For question 2, the Evaluation Team evaluated both the financial and operational sustainability of all four e-BIZ centers. As defined by the Cooperative Agreement, financial sustainability is achieved when an e-BIZ center is able to deliver products and services to the market at a price that covers its expenses and generates a profit. In order to determine financial sustainability based on the above-mentioned definition the Evaluation Team analyzed the financial statements of the e-BIZ centers. However, since the above￾mentioned definition of sustainability is extremely narrow and is used primarily as a cost-accounting tool that only covers a point in time it would not provide information on the ability of the e-BIZ centers to sustain the provision of the Activity’s benefits in the long run. Therefore, the Evaluation Team proposed a different approach and introduced a new balanced score card method along with new financial indicators to measure financial sustainability. The latter were accepted by USAID as part of the approved Evaluation Work Plan. The Evaluation Team started with an analysis of the business plans of each e-BIZ center and extracted key performance indicators for the center in question. The Evaluation Team calculated indicators’ actual values and compared them with the planned ones. Taking into account the purpose of this evaluation as well as the main activities of the e-BIZ centers, the key questions that were addressed related to short-term liquidity, capital structure and solvency, profitability analysis and break￾even analysis. The Evaluation Team analyzed the four main aspects of the financial sustainability of the e￾BIZ centers and the respective financial ratios as predictors of financial distress. The evidence suggested Altman's Z-score and Kralicek Quick Test, probably the most well known bankruptcy prediction models, as useful screening, monitoring and attention-directing devices for this purpose. Within this analysis of financial sustainability, the Evaluation Team determined the percentage of total services provided by the e-BIZ centers to those companies under the same ownership as the respective e-BIZ centers. Operational sustainability was determined through onsite visits and a review of, amongst other things, 18 According to research, one of the limitations of this technique is that about 20% of critical details of an event are irretrievable after one year from their occurrence and 50% are irretrievable after 5 years. 19 the staffing, equipment, organizational structure and procedures of the e-BIZ centers. To answer question 3, the Evaluation Team attempted to collect information on the EIA results as well as evidence on the performance of the targeted SMEs. Additionally, the team conducted a questionnaire survey and key informant interviews (selected SME owners/general managers). As for questions 3 a-c, the team compared market analyses prepared for each of the four industries and the data from those surveys with the current situation within the targeted SMEs served by the four e-BIZ centers. To answer this question, and due to the survey failure, the Evaluation Team analyzed changes in the competitiveness of the targeted SME clients. The team did this by tracking changes in certain indicators, such as profitability and sales. In order to determine whether the e-BIZ centers’ SME clients had experienced an increase in the volume of sales or profitability the Evaluation Team used available financial data on the performance of companies and sectors collected by the Agency for Financial Information and Consulting Services (AFIP). The Evaluation Team only managed to collect data for companies and sectors in the Federation of Bosnia and Herzegovina. Through the preliminary and informal interviews, the Evaluation Team learned that no information of that kind had been collected. The EIA explained this fact by citing the M&E plan in which they are expected to measure SME competitiveness not through quantitative financial indicators but rather through “surveys in which customers can report whether their business has been strengthened by the use of e￾BIZ center services”.19 In explanation of why they had failed to collected financial data on the sales and profitability of SME clients served by the e-BIZ centers the EIA stated that by asking for sensitive financial information the e-BIZ centers would lose clients. However, such financial reports for the targeted SME clients are commercially available at an affordable price (15 BAM per company) through AFIP FBiH.20 Although there is no evidence that the e-BIZ centers had surveyed their clients on a regular (annual) basis the EIA reported results for this indicator each year in the work plan and M&E plan. More importantly, USAID data quality assessment (DQA) of this indicator verified the existence of the survey. The Evaluation Team tried to reconstruct the missing data through a questionnaire survey. The questionnaire asked selected SME clients whether the technical assistance and services provided by the e-BIZ centers had contributed to an improvement in their business operations. The team also conducted interviews, based on a structured interview guide, with selected SME owners in order to determine to what extent the activities of the e-BIZ centers had contributed towards changes in the selected indicators. The EIA provided the Evaluation Team with market analyses prepared for each target industry. Through these studies, the EIA gained in-depth understanding of the market opportunities for the e-BIZ centers and which high impact ICT applications they could offer SMEs. However, these studies did not analyze the existing competiveness of the SMEs from the target industries. The Evaluation Team 19 Start-Up and Implementation Annual Work Plan; DOT-ORG BiH Excellence in Innovation Project; Academy for Educational Development; USAID BIH Excellence in Innovation Cooperative Agreement Award No. 168-A-00-07- 00101-00 8 February 2008. 20 http://www.afip.ba 20 had to use the data from the market surveys to make a comparison between the SMEs surveyed for the Market Analysis and those SMEs served by the e-BIZ centers to determine any changes in competitiveness. Yet this comparison was not possible because most of the companies surveyed for the Market Analysis and those served by the e-BIZ centers were not the same. In addition, as previously mentioned, the Market Analysis did not look into the competitiveness of the surveyed SMEs. Question 4: What impacts have USAID-supported e-BIZ Centers had on associations, sectors and industries targeted by the interventions? In order to determine the impact that the USAID supported e-BIZ centers had had on associations, sectors and industries, the Evaluation Team conducted interviews with the e-BIZ centers managers and directors of SMEs served by e-BIZ Centers. Question 4 was also included in the SMEs Questionnaire that the team distributed to all SMEs. Question 5: What have been the main obstacles in the implementation of the EI Activity? This included the identification and analysis of both solved and unsolved impediments that represented an obstacle to the smooth implementation of the activities of the USAID supported e-BIZ centers, as identified through the interviews with key informants. Limitations The EIA did not collect baseline data on the majority of indicators relevant to the evaluation questions. Therefore, the Evaluation Team was forced to use retrospective evaluation techniques wherein respondents, such as e-BIZ center employees and SME clients, were asked to describe their pre-project status in terms of knowledge and skills or competitiveness. However, the post-then-pre method has several weaknesses in terms of the validity21 of data:22 • recall - the inability to accurately recall past attitudes and behavior; • social desirability bias - a person’s need to report change or improvement that fits program expectations or to inflate the perceived improvement in those areas that are most important to them personally; • effort justification bias - occurs when participants subconsciously report improvements many times in order to justify the time and energy they have invested in program attendance; and • cognitive dissonance: occurs when participants report an improvement even if it did not occur in order to meet their own expectation that it ‘should have changed’. The evaluation was limited to the approved M&E indicators for the Activity. Some of these indicators, such as the financial sustainability indicator, were inadequate to measure the accomplishments of the EIA. 21 Validity: covers the strengths of our conclusions, inferences and or propositions. 22 Laura Colosi and Rachel Dunifon, Cornell University, “What’s the Difference: Post-then-Pre or Pre-then-Post?” 21 The Monitoring and Evaluation Plan of the EIA did not set indicators to track the performance of activities, including the following: • build the capacities of e-BIZ center employees; • become the most comprehensive source of online information for tourists seeking to visit BiH and for tour operators seeking to arrange tours to BiH;23 • become the most comprehensive source of online information and services for SMEs seeking and/or providing transport/logistics services in BiH24; • increase the volume of tourist services in BiH;25 • increase the competitiveness of BiH tourism service providers on the international market;26 • contribute to an increase in the export of processed products from BiH27; • contribute to an increase in employment within the wood processing industry;28 • expand the export customer base;29 • establish linkages between buyers in the EU and BiH textile SMEs;30 and continuously update and use state-of-the-art advanced technology31. The reliability and validity of the data represented the most important limitation for this evaluation. Several spot checks made by the Evaluation Team proved that there was an issue with the reliability and validity of the data provided by the EIA and the e-BIZ centers. This primarily relates to information on e-BIZ centers’ staff attendance at trainings, financial reports submitted by e-BIZ centers and surveys on SME satisfaction with received services performed by the e￾BIZ Centers and the EIA. The Evaluation Team took the subjectivity/objectivity of key informants in account during the evaluation process. The Evaluation Team expected that key informants would either overstate or understate the impact of project activities and therefore in order to ensure reliable data the team triangulated the data sources. There was no control over comparison group characteristics in the survey performed by the Evaluation Team. The comparison group represented a randomly selected sample of the sector; however, the selected SME clients of the e-BIZ centers were usually the most competitive export oriented companies. 23 Tourism e-BIZ Center Business plan; Exploring BH d.o.o. , Sarajevo; March 30, 2009. 24 Tourism e-BIZ Center Business plan; Exploring BH d.o.o. , Sarajevo; March 30, 2009. 25 Tourism e-BIZ Center Business plan; Exploring BH d.o.o. , Sarajevo; March 30, 2009 26 Tourism e-BIZ Center Business plan; Exploring BH d.o.o. , Sarajevo; March 30, 2009 27 Business Plan, Arteco – Wood Technology Center; Ismar Alagic; January 9, 2009 28 Business Plan, Arteco – Wood Technology Center; Ismar Alagic; January 9, 2009 29 Business Plan, Arteco – Wood Technology Center; Ismar Alagic; January 9, 2009 30 Textile Technology Center Business Plan; March 2011. 31 Textile Technology Center Business Plan; March 2011 22 The Evaluation Team was expected to use data from the market surveys and compare it with the existing situation within the targeted SMEs served by the four e-BIZ centers. However, due to difference in the characteristics of the market survey sample and the group of SMEs served by the respective e-BIZ centers this was not feasible. In addition, the market survey did not look into the competitiveness of the surveyed SMEs. The inferences based on the SME survey statistics cannot be considered valid due to the low response rate, the small sample size and non-response bias. The response rate was low amongst the SMEs' and the sample limit for the analysis was too small to provide descriptive statistics. The response rate remained extremely low in spite of follow up e-mails and telephone calls to a number of SMEs. One of the examples of interference with the evaluation process is the provision of contradictory data on the same topic. The Evaluation Team received documents from the EIA and the e-BIZ centers yet in the majority of cases the list of SME clients received from the e-BIZ centers and the EIA were different. Likert scales within the questionnaire. SMEs were asked to estimate whether their business had improved due to the services provided by the e-BIZ centers. Any changes were evaluated according to the Likert Scale, wherein the quality of the evaluation was dependent on the individual understanding of categories such as “applicable to the full extent” and “not applicable at all”: the same change occurring within two companies might be assessed differently by different employees. Other Issues that affected the Findings The Evaluation Team has strong reservations about the information and data provided by the EIA, the e-BIZ centers and the interviewed SMEs. The EIA and the e-BIZ centers interfered with the evaluation process at several points by adjusting documents and providing documents produced after the evaluation process had begun. This interference made the evaluation process difficult and makes the evaluation findings suspect. More specifically the issues listed below. • Information provided by the e-BIZ centers to the Evaluation Team on attendance at the trainings did not reflect what happened in reality. The e-BIZ centers were instructed by EIA staff to report more capacity building activities.32 • The financial reports of the Arteco e-BIZ center were adjusted significantly during the one￾month period after the evaluation process had begun. • As previously mentioned, the lists provided by the EIA and e-BIZ centers of SMEs served by the e-BIZ centers were different; even lists of clients that came from one source were not identical. • During the initial meeting with the EIA, the EIA staff stated that no formal survey on the competitiveness of SME clients had been performed; this was also confirmed by the EDC at the 32Transcripts of audio-recorded interviews. 23 end of the process. The nearest to it were the telephone inquiries from the e-BIZ centers to their SME clients asking them about their level of satisfaction with the services provided. Nevertheless, the EIA submitted list of SMEs that had participated in the questionnaire survey. Several companies stated during field interviews that they had never participated in any kind of survey: neither through the questionnaire nor via the telephone. The majority of issues with the validity and reliability of data on the performed surveys related to the TTC and Exploring BiH e￾BIZ centers. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS In contrast to the challenging activity design, which promised the achievement of a complex system of objectives and results, the monitoring and evaluation plan only followed Activity performance through two indicators. The Start-up and Implementation Annual Work Plan specified list of indicators are shown below. • The number of e-BIZ centers opened and sustainable. • The number of responses to the customer survey that reported businesses strengthened by using the e-BIZ services. Table 2: Outputs and Indicators Document Title Output Indicator Annual Performance Report 2008 Project Output 1 Two sustainable e-BIZ centers that provide high impact ICT applications for specified sectors; each e-BIZ center has served at least 50 SMEs. Two market analyses, each determining whether prospective high impact applications could become the core offering of a sustainable e-BIZ center. Bids received from local entrepreneurs interested in investing in e-BIZ centers. Annual Performance Report 2008 Project Outputs Four sustainable e-BIZ centers that provide high impact ICT applications for specified sectors; each e-BIZ center has served at least 50 SMEs. Significantly improved competitiveness of at least 50 SMEs in each targeted sector and/or supporting sectors through increased use of value-added ICT. FOGs signed for the development of the business plans and implementation of the business plans. Business plans for the wood and tourism e-BIZ centers completed. Business plans for wood and tourism reviewed and approved by USAID. Financial reports submitted by all e-BIZ centers on a monthly basis and hands-on financial management training provided. EIA and Arteco have visited over 30 wood manufacturing companies. Exploring BiH has held various presentations on accommodation facilities. Three experts for the wood e-BIZ Center and two experts for the tourism e- BIZ Center hired by the EIA. In addition, EIA arranged for four journalists to 24 visit BiH to promote the tourism portal. Annual Performance Report 2010 Project Outputs Four sustainable e-BIZ centers that provide high impact ICT applications for specified sectors; each e-BIZ center has served at least 50 SMEs. Project Outputs: Two sustainable e-BIZ centers that provide high impact ICT applications for specified sectors; each e-BIZ center has served at least 40 SMEs. • Four market analyses, each determining whether prospective high impact applications could become the core offering of a sustainable e-BIZ center. • Bids from local entrepreneurs interested in investing in the e-BIZ centers received. • Two market analyses conducted, each determining whether prospective high impact applications could become the core offering of a sustainable e-BIZ center. • Bids from local entrepreneurs interested in investing in the e-BIZ centers received. 1) Target: 2 out of 2 centers have a business plan that shows that they will become self-sustainable within the life of the Project. 2a) Target: each center services at least 20 customers. 1) Definition and Unit of Measurement Unit of measurement: the number of e-BIZ centers. Definition of operational: a center has a business plan or annual operations plan that defines its offerings and the revenue needed to achieve self-sustainability by the end of the Project. Definition of sustainable: (a) An e-BIZ center with an annual operations and financial plan, based on past performance and demonstrating that the center will reach the break-even point within one year of the end of the Project. (b) The e-BIZ center owner’s written commitment to continue operation of the e￾BIZ center after the end of the project. 2) Unit of measurement Number of customers Definition: a) number of customers paying for e-BIZ center services; and b) number of e-BIZ center customers reporting that the e-BIZ center services have strengthened their business. Annual Performance Plan 2012 Target: 2 out of 2 Centers have operations plans that show that they will become self￾sustainable within the life of the project. Target: 40 customers per center. b) 40 customers from each e￾BIZ center report that e-BIZ center services have strengthened their businesses. 1) Definition and Unit of Measurement: Unit of measurement: the number of e-BIZ centers. Definition of operational: An e-BIZ center has a business plan or annual operations plan that defines its offerings and the revenue needed for achieving self-sustainability by the end of the project. Definition of sustainable: (a) An e-BIZ center has an annual operational and financial plan based on past performance demonstrating that the center will reach the break-even point within one year of the end of the project. (b) A written commitment by the owner of the e-BIZ centre to continue operations of the e￾BIZ center after the end of the project. 2) Unit of measurement 25 Number of customers Definition: a) number of customers paying for e-BIZ center services; and b) number of e-BIZ center customers reporting that e-BIZ center services have strengthened their business. The Activity’s M&E Plan specifies the way in which sustainability and competitiveness are to be measured. Financial sustainability is achieved when an e-BIZ center reaches the break-even point, while SME competitiveness is not measured using quantitative data but through a “survey in which customers can report whether their business has been strengthened by the use of e-BIZ center services”.33 The Annual Performance Plan provides a definition of operational sustainability. Operational sustainability is defined as a, “Center has a business plan or annual operations plan that defines its offerings and the revenue needed for self-sustainability by the end of the project”. E-BIZ center clients are BiH SMEs receiving high impact information and communication technology applications and services and paying for them. More specifically, the unit of measure for the number of customers served is provided in the Annual Report for the Period January–December 2011. It is defined as, “a) The number of customers paying for e-BIZ center services and b) the number of e-BIZ centers’ customers reporting that e-BIZ center services strengthened their business”.34 The above definition was of great importance for the Evaluation Team because EIA staff provided lists of clients served by the e-BIZ centers that contained a large number of individuals, NGOs and foreign companies. Through the interviews that were conducted, it became obvious that EIA staff as well as e-BIZ center management were not clear which clients they were supposed to serve despite the fact that this is clearly defined in the Annual Performance Report. During the interviews the definition of ‘client’ ranged from those individuals, organizations and companies that visited the e-BIZ center website to those that appeared on the e-BIZ center websites to those that visited the e-BIZ centers during the Open Day event. It is therefore of great help that the Annual Performance Plan defines the number of e-BIZ center clients as the, “Number of customers paying for e-BIZ center services”. In addition, from the Cooperative Agreement we have learned that “the EIA Project aims to utilize information and communication technologies (ICT) as part of USAID’s efforts to generate employment and promote the expansion and competitiveness of BiH small and medium enterprises (SMEs). The proposed project will work within the Economic Growth Program area of Private Sector Competitiveness, focusing on the Private Sector Productivity element by providing assistance to and collaborating with: (a) BiH entrepreneurs and firms to establish and operate sustainable ‘e-BIZ Centers/Enterprises’ that offer high impact information and communications technologies (ICT) applications and services to SMEs in targeted and supporting sectors and (b) SMEs, in order to improve their capacity to fully utilize the high impact ICT applications and services offered by collaborating e-BIZ Centers/Enterprises to improve their competitiveness“. 33 Start-Up and Implementation Annual Work Plan; DOT-ORG BiH Excellence in Innovation Project; Academy for Educational Development; USAID BIH, Excellence in Innovation Cooperative Agreement Award No. 168-A-00-07- 00101-00 8 February 2008. 34 See page 32 (Table2: Project Monitoring Plan) of the Annual Report for the period January–December 2011, BiH Excellence in Innovation Project February 8, 2012. 26 As opposed to the rather simplified way in which the project M&E Plan measured activity performance, the evaluation questions required more sophisticated and measurable evidence of successful performance. Therefore, in its proposal and work plan, which were accepted by USAID, the Evaluation Team proposed the use of additional methods for the analysis of financial sustainability. EVALUATION QUESTIONS: FINDINGS AND CONCLUSIONS QUESTION 1: To what extent, in a measurable way, has the EI Activity succeeded in capacity building for e-biz center staff? Excerpt from the Start Up and Annual Implementation Plan The type and scope of capacity building effort were set within the Start-Up and Implementation Annual Work Plan: “The EI Project will also provide in-depth capacity-building to strengthen the e-BIZ centers’ ability to offer high quality products and services to SMEs. The Project will undertake focused training and technical assistance (T&TA) most of which will be provided to the e-BIZ centers by the EI Project staff for day-to-day, hands￾on and in-depth. International and national short-term experts will be engaged to address particular needs. A plan will be developed for each short-term expert to maximize the impact of his/her involvement, but will also be sufficiently flexible to take advantage of new insights the expert may provide during a given assignment. It is anticipated that the bulk of the short-term expertise will focus on sales, marketing and development of high impact ICT applications”. FINDINGS The surveys and interviews where e-BIZ center employees self-reported on improvements in their knowledge and skills indicated that the EIA has increased their capacity. Interviews and a questionnaire survey of e-BIZ center employees confirmed their confidence that they possessed the capacities necessary to perform their jobs at top performance level. The majority of staff received some form of training and reported a positive improvement in certain skills. Exploring BiH did not have any employees except for the Director himself, while TTC and Transwork staff did not receive marketing and financial management training previously organized for Arteco and Exploring BiH. The Evaluation Team assessed the EIA reports on capacity building efforts and conducted an analysis of the EIA documents and reports on the training and technical assistance provided. In accordance with our Request for Information sent to the EIA, the Evaluation Team received two sets of documents: • trainings provided to e-BIZ centers; and • technical assistance provided to e-BIZ centers. Throughout the life of the activity, the EIA organized three group trainings: (i) a two-day Power of Sales seminar, (ii) a two-day Financial Management training and (iii) a one-day BiH Entrepreneurship and Business Development training. According to the attendance lists provided by EIA, the trainings delivered and staff trained were as shown below. 27 • Between March 30 and 31 2011 a two-day Power of Sales seminar was attended by 20 persons in total, out of which only 3 were e-BIZ center employees (two Arteco employees - one of whom no longer works for Arteco - and one Exploring BiH employee). Additionally, the owner of Exploring BiH35 and the owners of the Satwork and Kismet (companies which at that point in time were, in cooperation with the EIA, expected to establish two new e-BIZ centers in the textile and transport/logistics sectors) attended36. In sum, out of 22 current employees (from all four centers) only two employees still with their respective centers received this training. • On August 31 2010 a one-day BiH Entrepreneurship and Business Development training was attended by 17 persons in total, out of which only two were e-BIZ center employees (both from Arteco) and one person37 linked with Exploring BiH. Out of the 22 current employees in all four e￾BIZ centers only two employees attended this training. • On January 29 2009 a one-day financial training was attended by three Arteco/Artisan employees and two persons38 linked with Exploring BiH. Out of the 22 current employees in all four e-BIZ centers only two employees that are still with these centers received this training. Financial management capacity building efforts were focused mainly on increasing the capacity of e-BIZ center staff to track and record cost-sharing arrangements based on the EIA internal cost-share tracking system. The e-BIZ centers were never properly instructed or trained on how to track and record cost-share arrangements in their own financial statements and in accordance with the approved e-BIZ centers’ business plans. Because of this the e-BIZ centers financial reports do not give an accurate picture of the e-BIZ centers financial position. Capacity building related to the EIA internal cost-share tracking system was done primarily through direct on-the-job technical assistance. The focus of the assistance was on how to track and record finance related to cost-sharing arrangements. Analysis of the financial reports of the e-BIZ centers could not confirm that the cost sharing structure and amount, as specified in all four Fixed Obligation Grants (FOGs) and accompanying business plans, were fulfilled. Technology skills: The training provided by the equipment producers increased the capacity of Arteco and TTC e-BIZ center staff to operate software and the received equipment. The other two e-BIZ centers did not have technical staff that dealt in any way with ITC. Training on how to operate and use the software and equipment were only delivered to the two e-BIZ centers that received machines and software (TTC and Arteco). The producers of the delivered equipment provided these trainings and were evaluated by the two e-BIZ centers staff as very useful. Without any doubt, these trainings increased the capacity of the staff of these two e-BIZ centers to operate the equipment provided and to use the design and manufacturing software. This is a straightforward conclusion since in the majority of cases the e-BIZ center staff that operated the software and machines had no previous relevant knowledge or experience. According to the information received from the e-BIZ centers, seven (7) TTC 35 Mr. VŠ is not linked to the company Exploring BiH in a formal way. Mr. VŠ did not make himself available to the Evaluation Team for interview. 36 The Activity in its 2011 Annual Report reported false statement that beside the Satwork’s owner, Transwork staff also attended this two-day seminar. Please see page 3 of the Annual Report for the period January – December 2011. 37 Mr. VŠ is not linked to the company Exploring BiH in a formal way. Mr. VŠ did not make himself available to the Evaluation Team for interview. 38 Mr. VŠ and Mr. MI are not linked to the company Exploring BiH in a formal way. Mr. VŠ did not make himself available to the Evaluation Team for interview, while Mr. MI’s status in the project remains rather unclear. 28 employees and three (3) Arteco employees received this training. Out of the seven TTC employees listed as participants, according to the organizational chart, of the CAD/CAM trainings three of them operated the CAD/CAM system. Out of total 14 Arteco’s and TTC’s employees who received trainings on how to operate received software and equipment, three (3) were women. All tree women were TTC’s employees. Table 3: Relevance of the software and equipment training Employee initials e-BIZ center Attained level of formal education Presently occupied post Relevant previous experience39 EM Arteco Mechanical engineer Director 12 years AB Arteco Carpenter – vocational school CNC operator None EH Arteco Carpenter – vocational school CNC operator 3 years MB Arteco Machinist – vocational school Carpenter/ modeling Not applicable40 MH Arteco Economist – secondary school Administration Not applicable41 AH TTC Secondary school Textile technician 7 years42 EB TTC Transport technician – secondary school Unknown None43 EM TTC Vocational trade school Tailor None44 EK TTC IT engineer Marketing assistant None45 IP TTC Mechanical technician – secondary school Administration 3 years46 AO TTC Dental technician – secondary school Tailor None47 ZM TTC Tailor – vocational school Foreman None48 39 Previous working experience relevant for the training received. 40 Employee did not use the software or donated equipment. 41 Employee did not use the software or donated equipment 42 TTC enlisted Mr. AH as a participant of the CAD training; however, his present position in TTC is as a Textile Technician and not a CAD operator. 43 TTC submitted the CV of Ms. AB but she was not on the list of TTC employees. 44 TTC enlisted Mr. EM as a participant of the CAD training; however, his present position in TTC is as a Tailor and not a CAD operator. 45 TTC enlisted Mr. EK as a participant in the CAD training yet his present position in TTC is as a Marketing Assistant and not as a CAD operator. 46 TTC enlisted Mr. IP as a participant in the CAD training yet his present position in TTC is as an Administrative Assistant and not as a CAD operator. 47 TTC enlisted Ms. AO as a participant in the CAD training yet her present position in TTC is as a Tailor and not as a CAD operator. 48 TTC enlisted Mr. ZM as a participant in the CAD training yet his present position in TTC is as a Foreman and not as a CAD operator. 29 AJ TTC Textile Engineer CAD Operator Unknown49 EM TTC Chemistry Teacher CAD Operator – CAD Unit Head None The Evaluation Team held reservations about the data obtained through the survey of e-BIZ center staff. The Evaluation Team received information on the delivered training and technical assistance from the e-BIZ centers. However, information provided by one of the e-BIZ centers about the training turned out to be different from the attendance list provided by the EIA. When asked why they claimed that their staff had attended trainings organized before the e-BIZ center in question had been established, the Director of this e-BIZ center answered that they had been instructed to do so by the EIA because the EIA did not have a sufficient number of capacity building activities.50 Of the 22 surveyed employees (out of which seven (7) women) of the four e-BIZ centers 18 reported participation in some form of capacity building activity. Two TTC employees stated that they had not participated in any capacity building activity, while another two TTC employees failed to answer the question. The number of training days received by staff while working for the e-BIZ centers varied from one (1) to nine (9) training days.51 The largest number of training days provided were for the Director of Arteco (9) and the Director of Sales at Transwork (6). An average positive increase is shown for all targeted skills at the Arteco e-BIZ Center. The highest increase was in the perception that now they better understand the main priorities of the industry they work in. There was a high increase in the following targeted skills (difference of 2 or above): • introduction to the laws governing financial practice; • efficiency of managing the company; • ability to manage company finances independently; and • ability to see the developmental vision of the business. The lowest increase was found in better understanding the roles and responsibilities within the organizational structure. Introduction with the laws, independently managing company finances and understanding roles and responsibilities within the organizational structure were the skills with the lowest baseline score, which indicates the lowest presence prior to employment at the Arteco e-BIZ Center. There was no increase in technical skills at the Exploring BiH e-BIZ Center: computer skills, understanding software, hardware, troubleshooting, teamwork and the organizational structure.52 All of these technical skills had a baseline estimate of maximum. The highest increase reported was in skills related to the efficient management of the company and understanding the main priorities of the tourist industry (400%). All skills showed an average positive increase at the Transwork e-BIZ Center. The highest 49 Mr. AJ had 3 years of work experience but it was not possible to conclude from his CV whether his previous experience was relevant to the type of training received. 50 The interview conducted with the Director of the e-BIZ center was audio recorded. 51 For more information please see Annex VII. 52 For more information please see Annex VII. 30 increase was in understanding software applications currently in use at the e-BIZ center.53 Having in mind that work at the Transwork e-BIZ Center is based on software applications this skill is extremely, if not the most, important. Additionally, there were large increases in the following skills: • understanding hardware application currently in use at the e-BIZ center; • understanding the main priorities of the industry in which they worked; and • troubleshooting minor glitches. The remaining skills showed an increase of one (1) or below. The lowest increase was in computer skills that had the highest baseline estimate. All targeted skills showed a positive average increase compared to the baseline level the at the TTC e-BIZ Center. The highest increase was in understanding hardware and software applications used by the company and troubleshooting minor glitches, while the remaining skills showed less than a one (1) increase. The Evaluation Team interviewed just two employees since only Arteco and TTC had technical staff dealing directly with ICT. Both interviewees expressed their satisfaction with the training and technical assistance received and they were confident in their level of skill and knowledge required to perform the tasks within their job description. The Evaluation Team did not have any meaningful method to evaluate the impact of day-to￾day and hands on type technical assistance in a measurable way. Most of the technical assistance provided to the e-BIZ centers by EIA project staff came in the form of day-to-day and hands on capacity building provided by EIA project staff. In terms of operational management, the Evaluation Team could not confirm any capacity building assistance directly targeted at improving managerial skills to adopt and integrate technology in order to gain maximum efficiency and productivity. The Evaluation Team was also unable to confirm any capacity building assistance directly targeted at improving knowledge and skills in other areas of management i.e., strategic planning and human resource management. CONCLUSIONS The quality of human resources is the key success factor that determines the ability of e-BIZ centers to become centers of excellence for the dissemination, transfer and application of high impact ICT in the targeted industries. In contrast to easily replicable equipment and services, highly educated employees with top-notch expertise in the application of ICT and the capacity to provide continuous new innovative services in their respective industries is the only competitive advantage that makes an e-BIZ center unique and different from other manufacturing shops. 53 Increase in skills was evaluated on the basis of employees’ personal perceptions by using recall interviewing technique where e-BIZ center employees were asked to describe their pre-project status in terms of knowledge and skills or competitiveness. 31 Capacity building efforts related to increasing e-BIZ center staff knowledge and skills in areas of high impact information and communication technologies and services were basic and insufficient. Employees of the e-BIZ centers received only a 3-day generic financial and marketing management training and technical trainings on how to operate the received equipment (machines) provided by the equipment suppliers. Having in mind that e-BIZ centers are supposed to be centers of excellence for the dissemination and transfer of new high impact information and communication technologies, the EIA has done too little to build e-BIZ center staff knowledge and skills in the application of high impact ICT. A review of the CVs of the e-BIZ center staff also showed that, with the exception of Transwork, the average level of education attainment amongst all e-BIZ center staff was rather low. In addition, in the majority of cases their previous working experience had little relevance for the jobs they performed at the e-BIZ centers. For example, not a single Transwork employee, although well educated, had any education or work experience background in the areas of ICT or logistics. Therefore, capacity building efforts to increase the knowledge and skills of e-BIZ center staff was even more important. Basic technology related capacities were only built at the TTC and Arteco e-BIZ centers. These two e-BIZ centers received machines and equipment as well as training on how to operate them from the producers of the machines/equipment. The other two e-BIZ centers did not participate in any form of activities related to the capacity building of technology skills. Sales and marketing were the areas that EIA gave the highest priority. The e-BIZ center staff evaluated these trainings and direct technical assistance as good and helpful. The EIA invested a lot of effort in order to increase the capacities of the e-BIZ centers to attract and retain customers. Capacity building activities related to sales and marketing ranged from two organized trainings54 in marketing to the provision of technical assistance for the development of marketing plans and the provision of advisory on-the-job assistance. Only Arteco e-BIZ Center staff expressed reservations about the value of certain marketing methods they were asked to use (promotion and marketing via e-mail). QUESTION 2: Are all four USAID supported e‐BIZ centers financially and operationally sustainable? As defined by the Cooperative Agreement, financial sustainability is achieved when an e-BIZ center is able to deliver products and services to the market at a price that covers its expenses and generates profit. In order to determine the financial sustainability of the e-BIZ centers, based on the above-mentioned definition, the Evaluation Team analyzed their financial statements. However, as the aforementioned definition of sustainability is extremely narrow and only covers a point in time without providing information on the ability of the e-BIZ centers to sustain the provision of the benefits of the EI Activity in the long-run, the Evaluation Team proposed a different approach. USAID accepted the latter suggestion as part of the approved Evaluation Work Plan. 54 The BiH Entrepreneurship and Business Development Seminar was a one-day seminar on marketing strategy, customer/service protocols, effective networking etc. held on 31.08.2010. The Power of Sales seminar held between the 30 and 31 March 2011 was a two-day seminar. 32 Analysis of financial and operational sustainability was performed based on the e-BIZ centers’ financial reports and balanced score card analysis. The Evaluation Team also analyzed the business plans of all the e-BIZ centers with special attention paid to the financial projections within these business plans. The team extracted the main financial indicators and compared the planned ones from the business plans with the actual financial position of the e-BIZ centers at the end of the project. Arteco e-BIZ Center FINDINGS If considered from the perspective of the definition of sustainability as set in the Cooperative Agreement, the Arteco e-BIZ Center is financially sustainable. Arteco achieved the break-even point for the first time in 2011 and on 31.07.2013 revenue exceeded expenses by 4.15%. The majority of indicators signaled significant structural changes in Arteco's business operations over the period 31.12.2012 to 31.07.2013 yet they were not the result of changes in the center’s business operations but rather the result of an exercise in financial engineering. Liquidity improved significantly primarily due to cash transfers from the parent company Artisan to Arteco’s account. Liabilities stemming from loans provided by Artisan were eliminated and a significant amount of accounts receivables was collected (again from Artisan). This was actually a case of offsetting accounts receivables because the short-term debt and accounts receivables were associated with the same legal entity (Artisan). While liquidity was improved through the injection of cash an analysis of business operations showed a decrease in the average size of customer orders, an increased period of payments to suppliers, a longer accounts receivables collection period and, in comparison with 2010, a decrease in revenue per employee by 300%. Artisan registered its cost-share contribution to Arteco in the form of a short-term loan as opposed to a contribution in the form of capital. This practice resulted in a high level of indebtedness of Arteco. Arteco maintained this short-term debt, to the amount of 118,000 BAM, throughout the duration of the EIA but eliminated it in one month (30.06.2013 – 31.07.2013) through repayment of the loan to the parent company Artisan. Arteco generated the money for this loan repayment through the collection of accounts receivables from its parent company Artisan to the amount of 128,250 BAM. Other types of financial indicators analyzed by the Evaluation Team indicate that Arteco is under threat of potential bankruptcy. The value of Altman’s Z-Score test55 on June 30 2013 was 1.3, which falls within the range of a company at risk of bankruptcy; however, following a financial engineering exercise that occurred between June and July 2013 Arteco’s financial health improved dramatically to the point where on July 31 2013 its Z-Score value had increased to 2.22.56 55 Z-Score definitions: X1 = Working Capital / Total Assets; X2 = Retained Earnings / Total Assets; X3 = Earnings before Interest and Taxes / Total Assets; X4 = Market Value of Equity / Total Liabilities; X5 = Sales/ Total Assets. Z-Score bankruptcy model: Z = 1.2X1 + 1.4X2 + 3.3X3 + 0.6X4 + .999X5 Explanation of Z-Score values: (i) above 2.6 – financial health, (ii) between 1.1 and 2.6 - possible bankruptcy within one year and (iii) less than 1.1 – extreme bankruptcy risk. 56 More information about Altman’s Z-Score Test and its validity can be found at http://pages.stern.nyu.edu/~ealtman/PredFnclDistr.pdf and http://ecedweb.unomaha.edu/EBJI%202010HayesHodgeHughes.pdf 33 Arteco’s financial stability is poor. The Kralicek Quick Test indicated that Arteco’s financial stability was poor, even after the aforementioned financial engineering exercise. On July 31 2013, the value of the Kralicek Quick Test was 0.32.57 A large part of Arteco’s revenue comes from services provided to its parent company, Artisan. It generated 83.56% of total revenue through the provision of services to BiH SMEs in 2010 with the remainder generated through services to individuals and foreign companies. Artisan participation in total revenue received from BiH SMEs in 2010 was 35.96%. Arteco’s revenue coming from services provided to BiH SMEs was only 36.59% in 2011 and only 33.79% in 2012. Artisan’s share of total revenue received from BiH SMEs in 2012 was 66.3%. In the first six months of 2013, Artisan’s share of total revenue received from all types of customers (SMEs, individuals and foreign companies) was 42.05%. The share of income generated by Arteco through the provision of services to foreign companies is becoming predominant. The share of revenue from foreign companies in 2012 reached 62.99% of Arteco’s total revenue. Arteco’s business plan specified that the value of long-term assets should be much higher than the current value. The existing value of current assets was much higher than the value of long￾term assets. The value of total assets on July 31 2013 was at 19.6% of the value of total assets planned for 2010. Arteco applied two models in an attempt to increase the export of processed products from BiH The first was to establish contact with a foreign company/customer that had its own product and find a local producer for the foreign company, for which it took a 10 % commission (an example is the company Scano from Norway), (2) Arteco develops product and technical documentation, hires local wood processors and then exports products. Arteco provided these services from 2009 to 2011. They stopped after USAID asked them to do so and advised them that this was not the purpose of the project. Based on an analysis of Arteco’s financial reports, matching foreign buyers and domestic producers was a lucrative business line. During the period 2009-2011 this type of service generated the largest share of Arteco’s total revenue. Through its commission for providing services (10%) Arteco generated around 150,000 BAM annually. CONCLUSIONS It was difficult to assess Arteco’s financial position due to the strong business connection with its parent company Artisan. In addition, much of its revenue stems from the provision of services to foreign companies, individuals and other types of organizations that are not BiH SMEs. 57 Explanation of Kralicek Quick Test values: > 3.0 excellent, > 2.2 very good, > 1.5 good, > 1.0 average, > 0.3 poor, ≤ 0.3 insolvency beginning, ≤ 0.0 moderate insolvency and ≤ -1.0 extreme insolvency. More information about Kralicek Quick Test can be found at http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=13&ved=0CDkQFjACOAo&url=http%3A%2F%2Fwww.thinkmind.org%2Fdow nload.php%3Farticleid%3Dicsea_2012_6_40_10455&ei=yHZIU9_uJoaG0AWFloCgBw&usg=AFQjCNEseE6ayLm9HmhcWxJO2VGKhk0HQw&sig2=JyP uWG0x7_1Sc9dY6Ux6ng&bvm=bv.64542518,d.d2k 34 The Arteco e-BIZ Center has achieved the break-even point, as the measure of financial sustainability; however, other financial indicators indicate poor financial stability and the threat of potential bankruptcy. Arteco has not achieved the financial projections and capital target values as specified in the business plan. Financial projections set the value of Arteco’s capital in 2009 at 711,720 BAM; however, on June 30 2013 the total value of Arteco’s capital was 2,000 BAM. Arteco has not achieved the target values for current and long-term assets set in the business plan. Arteco worked to increase the export of processed wood products from BiH by providing the service of matching foreign buyers and domestic producers. This type of service generated up to 2 million BAM during the period 2009-2011 through exports by BiH SMEs in the wood processing industry. Yet the Evaluation Team could not confirm any significant contribution towards an increase in the export of processed products. Total exports of the BiH wood processing industry in 2011 was 747.5 million BAM and therefore, based on the volume of export facilitated through services provided by Arteco (around 2 million BAM annually), it is fair to say that its contribution was not significant. Exploring BiH e-BIZ Center FINDINGS According to the definition set by the Cooperative Agreement, the Exploring BiH e-BIZ Center is financially sustainable. The Center achieved the break-even point on June 30 2013 and revenue exceeded expenses by 1%. During the four years prior to 2013 Exploring BiH only achieved the break-even point during 2010. Exploring BiH does not have stable revenues. Exploring BiH’s actual revenue exceeded the planned amount by 37.5 % in 2010 yet despite the higher than planned level of revenue Exploring BiH ended the year with a loss. Total revenue dropped by 55% in 2012 compared to 2011. Profitability was low and insufficient to cover the accumulated loss from 2010. It is necessary to stress the fact that for the last two and a half years the amount of equity held by this company has amounted to zero (0) BAM while the company's accumulated losses exceed the initial capital. Other types of financial indicators analyzed by the Evaluation Team indicate that Exploring BiH is under extreme threat of bankruptcy. The value of Altman’s Z-Score test has been negative throughout the entire life of the Project. Exploring BiH’s financial stability is poor, as indicated by the Kralicek Quick Test58. The value of the Kralicek Quick Test was -1.05 on December 31 2012, while on May 31 2013 the Kralicek Quick Test value was 0.5. 58 This test has four partial results and one overall result. Equity Ratio = Total Owner's Equity / Total Assets Debt Settlement Period from Cash Flow = (Liabilities - Cash) / Cash Flow) Cash Flow = P/L Acc. Period + Assets Depreciation + Reserves and Deferred Income 35 The ownership structure remains unclear. The ownership and management structures of Exploring BiH are not transparent and remain unclear to the Evaluation Team.59 The USAID approved FOG and the business plan for Exploring BiH did not include a balance sheet. Therefore, it was impossible to compare the planned assets, liabilities and capital values with those realized. As stated during the interview with the Director of Exploring BiH, in the early stages of the project it was already clear that the website/portal was a failure. The explanation given by EIA staff for the reasons behind this was that at the time the project commenced there was no booking website in BiH and nobody was able to foresee what would happen in the area of ICT within the tourism sector (booking.com etc). However, within the Market Analysis Report, the EIA recognized that, “A large number of booking agents are active in BiH but three - www.booking.com, www.venere.com and www.all￾hotels.com - are dominant. The EIA also learned that several sites, including www.ljeto-zima.com and www.sobe.ba, offered detailed accommodation listings with information, photographs, prices and e-mail links for reservations through the hotel’s own websites”.60 The Market Analysis stated that www.bhtourism.com has a different purpose than Exploring BiH and that it promotes Bosnia and Herzegovina in general as well as BiH tourism as opposed to merely promoting BiH tourism service providers. Although recognizing the fact that www.bhtourism.com has 2 million hits monthly61 the main criticism of the webpage was that it is was not updated on a regular basis. Exploring BiH has not improved or updated its website for more than two years. Exploring Macedonia built the Exploring BiH portal at the very beginning but since then made no effort to update or improve it. According to the Director of Exploring BiH, an internal evaluation that took place in 2010 confirmed the inadequacy of the website. At that point, the evaluators recommended that a complete redesign of the website be undertaken yet no action was taken for almost two years after the recommendation was made. The website was changed just recently with additional support provided by USAID. According to the Director of Exploring BiH, the main reason for such a situation was the lack of interest and commitment by the website/portal ‘owners’, namely Exploring Macedonia and Fondacija e￾turizam. The majority of interviewed clients found remembering any cooperation with Exploring BiH very difficult. Out of seven (7) interviewed hotels, six (6) experienced difficulty in recalling cooperation Operating Cash Flow / Sales = Operating Cash Flow / Net Sales (Revenue) - This ratio, which is expressed as a percentage, compares a company's operating cash flow to its net sales or revenues, which gives investors an idea of the company's ability to turn sales into cash. ROA (Return on Assets) = Net Income / Total Assets - An indicator of how profitable a company is relative to its total assets. ROA gives an idea as to how efficient management is at using its assets to generate earnings. Calculated by dividing a company's annual earnings by its total assets, ROA is displayed as a percentage. Sometimes this is referred to as "return on investment". 59 Exploring Macedonia had 49 % of ownership of Exploring BiH despite it not participating in the tender process. USAID only found out about Exploring Macedonia’s ownership stake in Exploring BiH by chance and requested them to get out from the ownership structure. After this, Ms. Sasa Vasic bought Exploring Macedonia and eFondacija’s shares of ownership in Exploring BiH through the newly established company Sasa d.o.o. However, it seems that Sasa d.o.o. is still not a formally registered company (at the time the interview was conducted – 29.07.2013). 60 Market Analysis of the High Impact ICTs as e-BIZ Center Core Offerings in the Tourism Sector 61 Market Analysis of the High Impact ICTs as e-BIZ Center Core Offerings in the Tourism Sector 36 with Exploring BiH and considered Exploring BiH as one of many small tourist agencies with insignificant contribution to their business operations. The Director of one hotel in Tuzla could not remember if he had worked with Exploring BiH, while the Director of another hotel in Mostar was aware of Exploring BiH only as one of many small privately owned tourist agencies. The Director of a Hotel in Zenica could not remember any cooperation with Exploring BiH, and a hotel manager in Banja Luka considered Exploring BiH to be just one of many privately owned tourist agencies that sporadically brought guests to his hotel. One rafting company owner claimed that contact with Exploring BiH brought more harm than good to his company, while the sales managers of two major hotels in Sarajevo considered Exploring BiH to be one of many travel agencies not worthy of mention business wise. The interviewed clients could not confirm any significant benefit from their cooperation with or promotion done by Exploring BiH. The Evaluation Team conducted interviews at seven hotels and one rafting agency (please see Annex III to the Evaluation Report) included on the list of clients provided by Exploring BiH and the EIA. None of them could confirm any significant benefit through promotional services provided by Exploring BiH or through their contractual relationship with the e-BIZ center. The Evaluation Team could not find any evidence that the EIA measured whether Exploring BiH was indeed the most comprehensive source of online information for tourists. Many websites/portals provide tourism related information and promote Bosnia and Herzegovina.62 Even another USAID activity that deals with tourism stated that it did not consider Exploring BiH to the most comprehensive source of information in Bosnia and Herzegovina and it continues to support the establishment of a new tourism website. The Director of Exploring BiH stated that other USAID activities created competition to Exploring BiH. CONCLUSIONS Exploring BiH achieved financial sustainability by reaching the break-even point and having revenue that exceeded expenses by 1%. However, other financial analyses indicate poor financial stability and extreme bankruptcy risk. From the very beginning up until the present, the Exploring BiH portal has not been sufficiently technologically advanced to allow for direct booking via the portal. Bookings were made via e-mail communication with the Director of Exploring BiH and very soon, it became clear that Exploring BiH could not compete with ‘Booking.com’.63 There is a link to the booking.com website at the Exploring BiH web site, However, that cannot be considered as a provision of possibility for direct booking. 62http://www.bhtourism.ba/eng/, http://www.tourism-in-bosnia.com, http://www.visitmycountry.net/bosnia_herzegovina/en/ Foreign Investment Promotion Agency of Bosnia and Herzegovina promotes this website as the one that will “introduce BiH to millions of Internet users in an innovative way by providing visibility of historic, cultural facilities and monuments and tourist attractions, museums and many other resources of the country. According to the Director of Innovativo d.o.o., Mr. Mario Gerussi, all this is presented in one place through the use of multimedia elements such as videos, photos, virtual 360 tours, audio and gmap. During the interview, they presented the Tourism Exchange, which is located on the website, and its role to facilitate supply and demand between travel agencies and service providers. http://fipa.gov.ba/novosti/vijesti/default.aspx?id=2505&langTag=en-US 63 http://www.booking.com 37 Exploring BiH has failed to achieve any significant impact on the businesses of its clients. During the interviews with hotels served by Exploring BiH, hotel managers could not tell the difference between Exploring BiH and other privately owned tourist agencies. In addition, its clients could not confirm any significant benefit from their cooperation with Exploring BiH or if they had benefited from its promotional activities. Exploring BiH is not operationally sustainable. Exploring BiH has only one employee (the Director) and it has no office space or long-term assets. It has no capital and its accumulated losses have exceeded the initial capital value for the last two and a half years. The structure of ownership remains unclear. Transwork e-BIZ Center FINDINGS According to the definition set by the Cooperative Agreement, the Transwork e-BIZ Center achieved financial sustainability on 30.06.2013. Transwork did not reach the break-even point at the end of 2012 because its total expenses (171,752 BAM) were more than twice as high as its total revenue (72,981 BAM). The USAID approved balance sheet in the Transwork business plan (part of FOG) does not balance. Therefore, the Evaluation Team could not compare the planned and realized financial indicators that are calculated from the balance sheet. The majority of indicators signaled significant positive changes in the business results and improved business operations created better financial results in 2013. Sales were higher by 49.53% during the first six months of 2013 when compared to total sales in 2012. Revenue increased with almost no increase in expenses. Liquidity was high and short-term assets were several times higher than liabilities. Satwork injected 152,914 BAM in the form of initial paid in capital at the end of 2012. Transwork used this injection of capital to cover its accumulated losses from 2012. The majority of expenses came in 2012 generated through the procurement of goods (37,578 BAM) and services (68,038 BAM) from the parent company, Satwork. Other types of financial indicators analyzed by the Evaluation Team indicate that the financial stability and health of the Transwork e-Biz Center is excellent. The value of Altman’s Z-Score test on 30.06.2013 was 3.75, which indicates excellent financial health (at the end of 2012 it was - 3.84 indicating extreme risk of bankruptcy). The Kralicek Quick Test indicated that Transwork’s financial stability was excellent: the value of Kralicek Quick Test on 31.07.2013 was 3.29. The majority of Transwork’s revenue came from the sale of GPS devices and subscriptions to GPS tracking and monitoring services. In 2012, 95% of total revenue was generated through the sales of GPS devices and the GPS tracking and monitoring services. In the first six months of 2013 revenue from sale of GPS devices and the GPS tracking and monitoring services contributed 88.9% of total revenue. All operational capacity rests with the parent company Satwork. The only operational capacity of Transwork is the marketing, promotion and sale of products and services procured from or developed by its parent company Satwork. Transwork employees did not have any technical/technology expertise for the development and production of services and products related to information and communication 38 technology. The only written procedures that define the business process at Transwork are those related to the promotion and sale of services. According to the Director of Transwork, the company did not succeed in becoming the most comprehensive source of online information and services for SMEs seeking and/or providing transport/logistics services in BiH. He stated that the reason behind this is Timocom.com, a German based freight-exchange web portal established in 1997 that covers the whole of Europe. Timocom’s main partner in Bosnia and Herzegovina is UMT from Banja Luka (www.logistica.ba). The Director was of opinion that Transwork would only have a chance of competing with Timocom if all similar service providers from the region were to join their efforts and resources and if much larger amounts of money were to be invested in the development and promotion of the portal. During interviews, all of Transwork’s clients confirmed that they used Timocom. CONCLUSION Transwork is financially sustainable and is a financially sound company. However, operationally, Transwork is just a marketing and sales department of its parent company Satwork. The majority of Transwork revenue comes from the sale of Satwork services that were standard prior to the establishment of Transwork; many similar companies provide such services. In 2012, Transwork generated 95% of its total revenue through the provision or sale of GPS devices and GPS tracking and monitoring services. Revenue generated through the sale of GPS devices and GPS tracking and monitoring services contributed 88.9% of total revenue in the first six months of 2013. Transwork failed to become the main web-based freight-exchange web portal, which was the main idea behind the center. Its clients do not pay for its services and, as stated by the EIA staff, the web portal is used as a “marketing platform”. Transwork cannot exist/operate without its parent company Satwork. Transwork employees do not have any technical/technology expertise for the development and production of services and products related to information and communication technology. TTC e-BIZ Center FINDINGS According to the definition set by the Cooperative Agreement, the TTC e-BIZ Center achieved financial sustainability on 30.06.2013. TTC managed to cover its expenses and generate a profit to the amount of 1,661 BAM on 30.06.2013. However, TTC did not reach the break-even point at the end of 2012 and it was only able to cover 57.66% of its expenses. The majority of indicators signaled that in the first six months of 2013 TTC’s business operations stabilized in comparison with 2012. TTC’s liquidity was satisfactory in the first six 39 months of 2013 with total revenue exceeding the 2012 figure by 28.38%; however, TTC’s net profit was only 1,661 BAM or 1.7% of total revenue. The number of supplier payment days (220) and the number of days for collection of accounts receivables (465) was too long64. TTC should adjust its existing difference of dynamics of payments made to suppliers and dynamics of accounts receivables collection in order to avoid potential problems with liquidity. This harmonization of payment days and accounts receivables collection days should not be an issue since over 90% of its financial transactions are between TTC and its parent company Kismet d.o.o. Financial projections in 2011 set the value of TTC’s capital at 325,000 BAM and in 2012 at 390,000 BAM. However, at the end of 2012 the paid in capital was at 2,000 BAM. Only six months later, on June 30 2013, the paid in capital reached the value of 140,000 BAM. Yet there was no court decision to prove that the capital had increased. The business plan set the projections for TTC’s revenue. The planned revenue for 2011 and 2012 was 491,130 BAM. TTC’s total revenue in 2012 was 73,976.79 BAM, while its losses in 2012 were 51,888 BAM. Other types of financial indicators analyzed by the Evaluation Team indicate that the financial stability and health of the TTC e-Biz Center is moderate. The value of Altman’s Z-Score test on the 30.06.2013 was 2.89, which indicates sound financial health (at the end of 2012 it was -2.52 indicating extreme risk of bankruptcy). Yet the Kralicek Quick Test indicated that TTC’s financial stability was poor and that the company is becoming insolvent. The value of the Kralicek Quick Test on 31.07.2013 was 0.24. Total revenue generated through the provision of services to its parent company, Kismet d.o.o, and to Vocal Production d.o.o., which is owned by the owner of TTC, in 2012 was 79.34%. Revenue from the sale of services to these two companies in the first six months of 2013 contributed 90.73% of total TTC revenue. CONCLUSIONS TTC is financially sustainable but its net profit was rather low. However, since the majority of TTC’s revenue comes from the sale of services to its parent company, Kismet d.o.o, and to Vocal Productions d.o.o., which is also owned by the owner of Kismet, TTC would have not achieved financial sustainability if solely dependent on the provision of services to unrelated companies in the open market. Other types of financial analysis indicate that TTC’s financial health is good. However, it is difficult to evaluate TTC’s financial heath individually due to its strong linkages to its parent company. TTC failed to achieve the financial targets set by the business plan. TTC did not achieve the financial projections and target values for its capital, as specified in the TTC business plan. In addition, TTC 64 The average 465 days for collection of accounts receivables was the result of non-payment for TTC services by its parent company Kismet d.o.o.. Standard invoice payment practice is Net 30, Net 60 to maximum Net 90. 40 failed to meet the business plan projections for its revenue. GENERAL CONCLUSIONS All four e-BIZ centers are financially sustainable. Yet this conclusion is based on a simplified and narrow definition of financial sustainability provided by the Cooperative Agreement. Analysis of their financial reports has shown that they have been significantly adjusted since the end of June 2013 and there is the fact that a significant part or the majority (in the case of TTC and Arteco) of their revenue comes from the parent companies. If the parent companies are taken out of the equation then Arteco and TTC are not financially sustainable. TTC generates between 80% (2012) and 90% (2013) of its total revenue from services it provides to its parent company, Kismet d.o.o, and to Vocal Productions d.o.o. More specifically, out of total 73,976.74 KM of the TTC’s revenues in 2012, 51,988.51 KM was received for services provided to the parent company, Kismet doo, and additional 6,703.6 KM for services provided to the Vocal Production doo, which is owned by the owner of TTC - Kismet doo. Only 20.66% of total TTC revenues in 2012 came from “outside clients”. In 2013 the situation even worsened since only 9.28% of the total TTC’s revenues came from “outside clients”. Arteco generates about one-third of its total revenue by providing services to its parent company, Artisan d.o.o. (Table 4), Table 4: Share of Revenues from Outside Clients in Total Arteco’s Revenues Arteco's Revenues 2009 2010 2011 2012 7 months 2013 Artisan – Parent Company (KM) 9,273.00 108,193.00 116,064.00 37,496.00 35,128.00 Third Party SMEs Sales (KM) 220,853.00 165,971.00 54,968.00 23,888.00 9,900.00 Total (KM) 230,126.00 274,164.00 171,032.00 61,384.00 45,028.00 Third Party SMEs Sales (%) 95.97 60.54 32.14 38.92 21.99 All of the e-BIZ centers, with the exception of Transwork, have poor financial stability. Other types of financial analysis, such as Altman’s Z-Score and the Kralicek Quick Test, indicate that all of the e￾BIZ centers, with the exception of Transwork, have issues with financial stability. The e-BIZ centers did not achieve the financial benchmarks set in their respective business plans. For example, the business plan projections for TTC’s revenue were not achieved and the financial projections and target values for Arteco’s capital, as specified within the Arteco business plan, were not achieved. The e-BIZ centers' financial reporting systems were inadequate; the e-BIZ centers were never instructed in how to record the USAID and parents company contributions in financial statements. The Evaluation Team was not in a position to verify the amounts and structure of the cost￾share contribution of the founders of the e-BIZ centers. 41 When it comes to operational sustainability, according to the definition from the Cooperative Agreement, each e-BIZ center is operationally sustainable65 i.e., each e-BIZ center has a business plan. However, each e-BIZ center requires different operational capacities to achieve its vision. The level of operational capacity/sustainability depends on the level of dependence on the respective parent company. Exploring BiH Operational Sustainability The Evaluation Team was not given access to any information related to the operational capacity of Fondacija e-turizam, which is the founder/owner of Exploring BiH. Exploring BiH does not have any significant operational capacities (no office space, no assets and no employees, except for the Director) and therefore the Evaluation Team estimated the operational capacity of this e-BIZ center as non-existent. The operational procedures and operational capacity of its founder remain unknown. Transwork’s Operational Sustainability Transwork’s operational capacity is at the level of a marketing and sales department or as it was defined by the EIA staff during the interview, “Transwork is a marketing platform”. Transwork is an extension of its parent company, Satwork. Transwork operates from rented premises owned by Satwork, Satwork provides its administrative capacity (accounting, procurement and maintenance) and its ICT services and research and development capacity stems from Satwork. Transwork only has a sales force and operational procedures related to marketing and the sale of services and acquiring new clients. Arteco’s Operational Sustainability Arteco is a subsidiary of its parent company Artisan d.o.o. Arteco’s main operational capacity stems from its parent company Artisan doo. Out of five (5) employees one is the Director, three (3) are production floor workers and one is an administrative assistant. The main operational capacity of Arteco is the organization of production processes. The types of written procedures that govern the operations of Arteco also indicate that its main operational capacity lies in the organization of production. Arteco has the following written procedures: (i) fully developed Solid Works training, (ii) written procedures for prototype development and (iii) a written guide on how to operate the CNC machine. TTC’s Operational Sustainability In comparison with the other e-BIZ centers TTC has the strongest organizational and operational capacity. It has largest number of employees and sound operational capacities through its parent company Kismet d.o.o. TTC’s strongest operational capacities are in production organization and marketing. QUESTION 3: To what extent has the EI Activity succeeded, in a measurable way, in 65 Operational sustainability is defined as, “Center has a business plan or annual operations plan that defines its offerings and the revenue needed for self-sustainability by the end of the project”. 42 improving the competitiveness of the SMEs served by the four e-BIZ centers? a. What was the situation of the targeted SME population before the EI Activity began? b. What is the situation of the targeted SME population now? c. Did EI respond to their identified needs? Excerpt from the Start-Up and Annual Implementation Plan “However, the project will not gather quantitative data on sales, production, customers etc. for e-BIZ Center customers as these are proprietary data that SMEs are very reluctant to provide. The EI project must be careful to conduct the evaluation in such a way as to avoid alienating the e-BIZ Centers’ customers, who are paying for the services. Thus, the EI project plans to conduct a survey in which customers can report whether their business has been strengthened by the use of e-BIZ Center services and, if so, what aspect has been strengthened. It is worth repeating that the e-BIZ project is different from most development projects in that the SMEs are paying market prices for services, in contrast to most development projects that provide free or subsidized services. Thus, for the e￾BIZ Center services, one can justifiably infer that customers are gaining value because they are paying for the service, whereas when other development projects give free or subsidized services one must find more intrusive ways to determine whether the services have actually benefited the SMEs”.66 To what extent has the EI Activity succeeded, in a measurable way, in improving the competitiveness of the SMEs served by the four e-BIZ centers? How many SMEs did the e-BIZ centers serve? As stated in the Cooperative Agreement, one of the three anticipated project outputs/outcomes was a significant improvement in the competitiveness of at least forty (40) SMEs in the textile sector, at least forty (40) SMEs in the transport and logistics sector, at least fifty (50) SMEs in the wood processing sector and at least 70 SMEs in tourism sector. In total 200 SMEs and/or supporting sectors should have improved their competitiveness through the increased usage of value-added ICT. There are several criteria to determine who can be counted as a client served by the e-BIZ centers but we will keep it down to two: (i) BiH SMEs and (ii) SMEs that have received services and paid for such services. Arteco provided services to 38 BiH SMEs over a period of 4.5 years. Out of this number, 28 companies received services only once over the 4.5 year period. Out of the 38 SMSs 13 received services of a value of less than 500 BAM (USD 356.65). TTC provided services to 19 SMEs over a period of 1.5 years. Two of these SMEs were companies owned by the owner of TTC, Kismet d.o.o., and Vocal Productions d.o.o. The services provided to these two companies generated 79.34% of TTC’s total revenue in 2012 and 90.73% in 2013. Out of 19 SMEs five (5) received a one-time service of a value of less than 500 BAM (USD 356.65) each. Transwork provided services to 48 SMEs over a period of 1.5 years. More than 80% of Transwork’s total revenue related to the sale of GPS devices and GPS tracking and monitoring services. Table 5: Percentage participation of individual services in Transwork’s total revenue 66 Start-Up and Implementation Annual Work Plan; DOT-ORG BiH Excellence in Innovation Project; Academy for Educational Development; USAID BIH Excellence in Innovation Cooperative Agreement Award No. 168-A-00-07- 00101-00 8 February 2008. 43 Unit Price (BAM) % Participation in Total Revenue in 2012 % Participation in Total Revenue in 2013 GPS device 205 57.2 36.6 GPS subscription 20 37.8 52.3 GPS device instillation 20 3.3 3.5 Fuel consumption control 3 0 0.05 Devices dismantling 20 0.04 0.02 Devices transfer 20 0.06 0.02 Installation of fuel tank lid with sensor 70 1.4 7.47 Other services 0.2 0.04 The Evaluation Team was not able to determine the exact number of clients of Exploring BiH. The Evaluation Team received several diverse lists of clients served by Exploring BiH. The first list listed 48 clients, the second listed 71 clients, the third was called the “frequent clients list” also with 48 clients and, finally, the fourth list received from the Exploring BiH accountant had 45 clients in total. The Evaluation Team knows for a fact that even if a client had a contract with Exploring BiH that does not mean that a business transaction happened between them. In the Start-up Implementation and Annual Work Plan the proposed method for tracking the increased competitiveness of SMEs was the number of customers that reported that e-BIZ center services had strengthened their business through increased productivity, increased sales, decreased costs or other improvements. In addition, as previously mentioned, the EIA reported to USAID every year during the life of the Activity. The EIA was supposed to perform annual surveys in order to determine changes to the above-mentioned indicator. The first survey of this kind was expected after the first year of the centers’ operations. However, during preliminary informal interviews with the EIA that took place on June 24 2013 the EIA staff stated they had not performed any surveys in which customers could report whether their business were strengthened through use of e-BIZ center services. Despite the fact that this survey was scheduled in the EIA M&E Plan and its completion confirmed through the Data Quality Assessment conducted by USAID COR there is no evidence that the e-BIZ centers’ customers were surveyed by the EIA, even after the first year of the Activity67. The Evaluation Team sent a Request for Information (see Annex 5) to the EIA on July 17 asking them to provide a report on the “results of the surveys of the e-BIZ centers’ SME clients“ performed after the first year of the centers’ operations (see question 3.4), as specified in the Start-Up Implementation Plan. The Evaluation Team received a folder titled “E-BIZ Center's First Year Clients Survey” on July 26. A review of the properties of the submitted documents revealed that, for example, the Exploring BiH survey report was created on July 25 2013, despite the fact that the survey was supposed to have taken place in 2010. However, the Evaluation Team did not investigate further in order to confirm any potential file manipulation. The review of these documents also resulted in the following findings: 67 Please see “Response to Evaluators for BiH Excellence in Innovation (EI) Project” (Annex X) 44 o three of the surveys were done by the e-BIZ centers and only one by EIA (according to the project documentation, these surveys were the obligation of the EIA); o questionnaires used were typical customer satisfaction questionnaires, with little or no questions concerning the impact that e-BIZ center services had on their customers’ competitiveness (even though the impact that the e-BIZ centers had on the competitiveness of their clients was the main aim of the survey); and o the survey performed by the EIA, in addition to e-BIZ center clients, also interviewed potential e-BIZ center clients i.e., SMEs interviewed during the market survey (in the case of Arteco). Lastly, following an interview with the EIA, conducted on August 23 2013, where the team again raised the question of the validity of the survey data the Evaluation Team received a formal response from EDC (quote): “We acknowledge that formal Customer Surveys were not conducted following the initial Year 1 surveys for ARTECO and Exploring BiH. It is important to note however that the Centers have continued to survey clients on an informal basis and they have informed EI about responses from the field so the feedback was incorporated in work plans. Moreover, EI Project staff regularly met with SME clients and prospective clients. TTC and Transwork also conducted customer surveys on time (i.e., following their first full year of operation). During Friday’s conversation, the evaluators mentioned that EDC should have conducted the customer surveys. This may be a matter of debate; however, in order to prepare the e-BIZ Centers for self-sustainability, we provided capacity-building support to the Centers to obtain customer feedback, and believe it was preferable for the Centers to conduct the activities required to obtain this feedback, as the Centers are the primary interface with the SME clients”.68 The EIA conducted an Industry Analysis 69 and a Market Analysis of the four target sectors but none of them established data and indicators on the competitiveness of SMEs or the target industries. The industry studies provided information related to SMEs utilization of ICT. The studies provided a basis for identifying potential high impact ICT applications that could become the core offerings of the e-BIZ centers for the four (4) target sectors. The Market Analysis70 that followed the Industry Analysis aimed to ascertain the potential demand amongst SMEs for these identified high impact ICT applications. A more in-depth understanding of market opportunities for e-BIZ centers to offer services based on high impact ICT applications was gained through the Market Analysis. None of these studies (industry and market analyses) provided baseline data on the competitiveness of SMEs in the targeted industries. In order to find out whether the e-BIZ centers increased the competitiveness of their SME clients the Evaluation Team performed a questionnaire survey. The EIA provided a list of SME clients from the four target industries. The Evaluation Team then compiled a list of SMEs that were not e￾BIZ center clients and sent a questionnaire via e-mail to 666 SMEs from the four target industries. The response rate was extremely low: 5% for the wood processing and tourism industries and 4% for the transport and textile industries. 68 Document titled “Questions and Concerns” received from the EDC on 29.08.2013 (see Annex IX) 69 Briefing Paper: Identification of High Impact Applications and opportunities 70 Briefing Paper: Identification of High Impact ICTs as e-BIZ Center Core Offerings 11 8 5 8 197 158 112 199 0 50 100 150 200 250 Wood Industry Tourism Transportation Textile industry Response No Response 45 Figure 3: The number of responses per target industry. Only a few out of the total number of companies that responded to the questionnaire had established business collaboration with the e-BIZ centers. Therefore, the only characteristic we could use for comparison and where we had almost the same number of companies in two groups to compare was whether a company had or had not ‘established contact with an e-BIZ center’. In total, 13 companies (40.6%) were in contact with the e-BIZ centers. Replacing the main evaluation variable (established business cooperation with an e-BIZ center) with ‘established contact’ did not provide an answer to the main evaluation question: did the competitiveness of companies collaborating with the e-BIZ centers increase at a higher rate than of those companies without established collaboration with the e-BIZ centers? Difference in the competitiveness between companies ‘in contact’ with the e-BIZ centers and companies that have not had ‘contact’ with them is irrelevant to the evaluation and therefore results of the survey are not presented in this report. Due to the survey failure, the Evaluation Team analyzed changes in the volume of sales and the profitability of SME clients. In order to determine whether the e-BIZ centers’ SME clients had experienced an increase in the volume of sales or profitability the Evaluation Team used available financial data on the performance of companies and sectors collected by the Agency for Financial Information and Consulting Services (AFIP). The Evaluation Team only managed to collect data for companies and sectors in the Federation of Bosnia and Herzegovina. Of the 25 SME clients from FBiH, served by the Arteco e-BIZ Center in 2011, 13 experienced a decrease in their total volume of sales in comparison with 2010. In addition, out of 25 Arteco e￾BIZ Centre SME clients 14 experienced a decrease in their total volume of sales in 2012 compared to 2011. 46 Figure 4: Annual percentage changes in the sales of Arteco’s clients during the period 2010-2012. Out of 14 SME clients of the TTC e-BIZ Center in 2012 six (6) experienced a decrease and another two (2) experienced only a 1% increase in the total volume of sales in comparison with 2011. Figure 5: Annual percentage change in the sales of TTC’s clients during the period 2011-2012. Out of 17 SME clients of the Transwork e-BIZ Center in 2012 five (5) experienced a decrease and an additional two (2) recorded an increase of less than 1% in the total volume of sales in comparison with 2011. -4 7 20 -64 11 -14 12 290 -67 -13 14 9 -41 -5 -24 -37 -17 -89 -13 8 110 -35 54 51 4 17 49 0 -7 -97 -99 -18 10 -4 -46 -9 -3 -19 -11 -61 -23 -2 7 0 2011 2012 -2 44 19 24 10 -5 22 1 55 -30 3 -26 -18 1 47 Figure 6: Annual percentage change in the sales of Transwork’s SME clients in the period 2011-2012. Ten (10) out of 19 SME clients of the Exploring BiH e-BIZ Center experienced a decrease and another two (2) experienced less than a 1% increase in the total volume of sales in 2011 in comparison with 2010. Furthermore, in 2012 nine (9) out of 19 Exploring BiH e-BIZ Center SME clients experienced a decrease and two (2) others experienced an increase of less than 1% in the total volume of sales compared with 2011. As we can see from the Graph below, almost none of Exploring BiH’s SME clients avoided experiencing dramatic and negative oscillations in sales volume during the last four years. Figure 7: Annual percentage changes in the sales of Exploring BiH clients for the period 2009-2012. CONCLUSIONS Transwork is the only e-BIZ center that achieved its target number of clients. TTC did not even manage to achieve 50% of the targeted number of clients. Exploring BiH, according to data received from its accountant, did not manage to achieve the target number of 70 clients. Finally, Arteco served many clients but only 38 were SMEs from Bosnia and Herzegovina. -3.31 13.09 17.23 28.84 12.44 -44.88 0.10 11.14 -8.59 -5.81 4.49 21.54 26.41 0.96 -12.49 15.0011.69 46 42 14 35 10 9 -10 -8 -21 145 -74 144 -2 -10 7 43 33 0 2 -66 -8 1 10 -5 -3 18 3 -48 -70 44 -5 -11 8 24 -12 -4 -1 -21 4 1 4 -11 -3 -10 69 9 -99 18 1 30 7 -5 -21 -11 2010 2011 2012 48 EIA management of the M&E system and the client-base system management were poor. The EIA invested little or no effort to establish baseline data and track changes in the competitiveness of SMEs from the target industries. The Evaluation Team effort to reconstruct data through an SME survey failed due to the extremely low response rate of the e-BIZ centers’ clients. The majority of services provided by the e-BIZ centers were not new or non-existent in the market. All TTC’s services either already existed or could be performed manually. The main contributions made by services provided by TTC to some of its clients were precision tailoring and faster production. Prototype development was the only unique service provided by Arteco. The services Transwork provided were also available through a dozen other service providers, while the perception of Exploring BiH was that of just another small travel agency. Transwork’s services were relevant to the improvement of the business operations of its clients. The GPS monitoring and tracking service contributed to better cost control and cost reduction. However, one should bear in mind that this service came from Satwork and that Transwork only acted as a marketing and sales department for its parent company. All interviewed Exploring BiH clients considered the received services insignificant in terms of their business operations. As previously mentioned, Exploring BiH clients had a hard time remembering if they had cooperated with Exploring BiH. To them, Exploring BiH was just another small private travel agency that sporadically brought in a negligible number of guests. TTC’s clients gave a mixed response concerning the importance of TTC services in relation to their business operations. Some SMEs used TTC services when overwhelmed with work and had a bottleneck at the production shop. Others used TTC services when they needed a prototype developed quickly. TTC was of most importance to textile companies that only had basic technological production capacities and applied for ‘LOHN’ jobs with foreign clients. They used TTC as a referent point and presented its technological capacity as their own. However, one has to bear in mind that TTC uses almost all of its capacities to support its parent company. Arteco has two basic groups of clients. The first are small one-time clients that hire Arteco just as a production shop, without Arteco having an impact on their business operations. The second group, of about 20 clients, comprises small production shops that Arteco/Artisan use to increase their production capacity for larger orders for Artisans products. The second group of companies, together with Artisan, forms a small wood processing cluster and Arteco/Artisan increased the capacities and competitiveness of these companies significantly. However, these companies remain completely dependent on Arteco’s parent company Artisan. The financial data concerning changes in the volume of sales of SMEs that received services from the e-BIZ centers does not constitute the basis for the conclusion that the e-BIZ centers improved the competitiveness of their clients. Almost half of the SMEs in all four sectors that received SME services experienced a decrease in the total volume of sales or did not experience any change in the total sales volume. 49 QUESTION 4: What impacts have USAID-supported e-BIZ Centers had on associations, sectors, and industries targeted by the interventions? Excerpt from the Start-Up and Annual Implementation Plan “The EI Project team has also initiated contacts with various business associations and public institutions, including regional development agencies as well as universities. Based on these interchanges, the EI Project team anticipates cooperating with the Regional Development Agency for Central BiH Region (REZ), the Sarajevo Regional Development Agency (SERDA), the Foreign Trade Chamber and the Federal Investment Promotion Agency (FIPA), the BiH Tourism Association, as well as various associations of farmers, among other BiH organizations”.71 FINDINGS Field interviews conducted with the management of the four e-BIZ centers confirmed that there had been no success in establishing cooperation between the e-BIZ centers and trade associations. According to the key informants, there was no understanding and interest on the part of the trade associations to play a significant role in the Project. Transwork made an effort to establish cooperation with different associations and Chamber of Commerce. They managed only to organize presentation to seven companies in cooperation with ZE-DO Canton Chamber of Commerce. However, that presentation did not result with continuation of cooperation. The Activity had several meetings with the Tourist Association of BiH but no agreement was reached to establish cooperation. Artisan’s owner stated that they tried to establish cooperation with trade associations but with no success. Management of the four e-BIZ centers and management from the client SME interviewed by the Evaluation Team stated that it was not realistic to expect a project of this size to increase the competitiveness of the entire sector/industry. The wood processing sector has around 18,000 companies and it is not realistic to expect that a project providing a one-time service to about 50 SMEs could improve the competitiveness of the entire wood processing sector. The Evaluation Team also analyzed changes in the average profitability of companies in all four sectors in the Federation of Bosnia and Herzegovina in order to determine whether, regardless of the cause, any positive changes in profitability took place during the life span of the Project. Average profitability in all four sectors was negative (loss). Average profitability in the wood industry changed from -6.62% in 2009 to -3.87% in 2011 and -5.1% in 2012. Average profitability in the tourism sector changed from -7.75% in 2009 to -7.19% in 2010 and -7.6% in 2012. Average profitability in the textile industry changed from -5.36% in 2009 to -1.88% in 2011 and -1.03% in 2012. Average profitability in the transport industry changed from -5.36% in 2009 to -1.88% in 2011 and -1.03% in 2012. The Evaluation Team could not find any evidence of the EIA effort to determine the status and track/monitor changes in the “volume of tourism services in BiH”. The Industry Analysis and 71 Start-Up and Implementation Annual Work Plan; DOT-ORG BiH Excellence in Innovation Project; Academy for Educational Development; USAID BIH Excellence in Innovation Cooperative Agreement Award No. 168-A-00-07- 00101-00 8 February 2008. 50 Market Analysis reports did not provide baseline data on the volume of tourist services in BiH. The same is valid for the objective to, “increase the competitiveness of BiH tourism service providers in international markets”. Total employment in the wood processing sector was 20,01172 in 2009, a reduction of 17% compared to 2008. Furniture production accounted for 7,942 of total employment in the wood processing sector (20,133)73 in 2011. CONCLUSIONS The Evaluation Team could not find evidence that the e-BIZ centers had had any impact on associations. One of the reasons could be the attempt by the e-BIZ centers to use the trade associations as a marketing channel for the sale of their services. In addition, it might also be the case that the associations and the public sector alike perceived the entire project as a private venture endeavor. For example, management of Transwork informed the Evaluation Team that, when they attempted to establish cooperation with the association and present Transwork to membership of the association, they were asked to pay if they wanted to present their business to association’s membership. They refused to do so. The association asked them to pay for presentation of Transwork’s products and services to its membership because there was no prior cooperation with the association and because presenting commercial products and services to association membership is a private company’s private endeavor. The Evaluation Team could not find evidence that the e-BIZ centers had any impact on the target sectors/industries. Financial data on profitability in all four sectors did not indicate any significant improvement in the competitiveness of the target industries. According to existing studies, in the period 2009-2011 there was no increase in total employment in the wood processing sector. This can be partially attributed to the general economic slowdown; however, the Evaluation Team could not find any evidence to support the conclusion that the EIA had any impact on increased employment at the level of the wood processing industry. QUESTION 5: What were the main obstacles to the implementation of the EI activity? Did unforeseen developments in information and communication technology constitute an obstacle? The EIA claims that one of the main obstacles was that ICT progressed dramatically from the project’s inception phase and that this substantially changed the relevance of the Project. EIA cite the entrance of ‘booking.com’ as an example and game changer in the booking aspect of the tourism sector. However, changes in ICT technology do not constitute a justifiable reason for the issues that Exploring BiH and Transwork faced when developing their portals. EIA had a sufficiently long research phase in which to 72 Source: Perspektive razvoja drvne industrije u Bosni i Hercegovini i njen utjecaj na ukupnu vanjskotrgovinsku bilancu; Lordan Ilicic; 2011 73 Izgledi BH industrije u 2012. godini; Drvni i metaloprerađivački sektori; Fostering Interventions For Market Advancement (FIRMA); September 2012 51 anticipate potential changes in technological development and to adjust the project design accordingly. Instead, the EIA worked with the e-BIZ centers to develop and place a set of services that already existed within the market. The primary interest of the EIA was to achieve the financial sustainability of the e-BIZ centers. There was a lack of involvement and support from the public sector. The EIA and e-BIZ centers failed to gain interest and support from public sector organizations and institutions such as ministries of economy, research institutes and faculties, business association, etc. Therefore, the entire effort and burden of advancing all four target industries was placed with small companies struggling to achieve the brake-even point. There is a low level of awareness amongst SMEs concerning the importance of ICT in terms of their competitiveness. The management of the e-BIZ centers believed that SMEs in Bosnia and Herzegovina were still not aware of the benefits they could gain through the introduction of ICT applications to the production process. Therefore, they did not manage to expand their client base and disseminate ICT applications throughout the target industries. As subsidiaries of their parent companies, the e-BIZ centers remain highly dependent. Although formally registered as limited liability companies the ownership of the e-BIZ centers rests with their parent companies. They are thus heavily dependent on their parent companies' management decisions, finances and technological and administrative capacities. The exception is Exploring BiH; however, it does not actually have an ownership structure. The three remaining e-BIZ centers are primarily service providers for their parent companies. The geographic location of the e-BIZ centers makes it difficult for them to attract high quality employees and experts. The only competitive advantage the e-BIZ centers might have to give them an edge in their respective sectors is the quality and expertise of their teams and their ability to research, develop, adopt and apply high value ICT to the production process. Due to their geographical location, TTC and Arteco had trouble hiring and keeping employees with university level qualifications. 52 RECOMMENDATIONS 1. USAID BiH should pay far more attention to the process of approving the initial work plan, M&E plan, and sub-grants. The initial EIA work plan set the indicators for the life of Activity. The Evaluation Team strongly believes that the two performance indicators proposed by the EIA and approved by USAID AOR were inadequate. Furthermore, three of the four business plans that accompanied the four FOGs contained obvious errors that USAID did not recognize when it evaluated and approved them. 2. USAID BiH should focus much more on regular and rigorous monitoring of the implementation of its activities/projects. The fact that the EIA reported results based on non￾existent data for several years indicates a serious problem in the way USAID BiH managed and oversaw the Activity. The lack of an adequate USAID monitoring system to track progress towards the defined objectives and goals on a regular basis resulted in some surprising findings in the evaluation. 3. Baseline data that is comparable with the expected results should be collected by the implementing agency without exception. The EIA invested little or no effort to establish baseline data and track changes in the competitiveness of SMEs from the target industries. The absence of adequate baseline data makes monitoring the progress of an activity difficult. Furthermore, if baseline data is missing the evaluation of achieved results is difficult. If an activity is to be evaluated at the end then it necessitates a process of data reconstruction and additional surveys and analysis. 4. The designated USAID staff should conduct thorough Data Quality Assessment (DQA). The EIA case clearly demonstrates that the reviewer should insist upon the submission of evidence in support of the reported achievement of results by implementers. The last DQA performed in the case of the EIA was in April 2013 and it failed to clarify the reliability of the submitted information. 5. USAID BiH should consider requesting site visit reports from its CORs as no formal site visit reports were written during the life of the EIA. 6. The choice of the procurement mechanism raises the question of cooperative agreement or contract. The Evaluation Team believes that a contract rather than a cooperative agreement would have been a much better choice in the case of the Excellence in Innovation Activity. 7. There is no project design that by default guarantees success. Although the Activity was designed and relatively successful in one country does not by default mean that it will be successful if replicated in another. The main argument behind the EIA’s claim that the Activity in BiH was a success was that the same project with the same design was successfully implemented in Macedonia. The Evaluation Team has not entered into a discussion and elaboration on the arguments against such a thesis as the person responsible for reviewing the initial project design should have challenged such a position and asked for an adjustment of the project to meet the environment and specificities of BiH. 8. The recipients of the USAID assistance should have recorded the cost-sharing and other financial agreements in their financial reports in accordance with the country specific accounting standards. This should have been clearly specified as an obligation of the assistance recipients and regularly monitored by the implementing organization and USAID. The e-BIZ centers were not instructed on how to record the USAID and parent company contributions within their financial 53 statements. The Evaluation Team was not in a position to verify the amounts and structure of the cost￾sharing contributions from the founders of the e-BIZ centers. 9. The number, frequency and staff coverage of the training/seminars organized by the EIA should have been higher if the aim was to achieve a significant increase in the capacities of e￾BIZ center staff. The Evaluation Team is of opinion that highly educated employees with top-notch expertise in the application of ICT and the capacity to continuously provide new innovative services in their respective industries are the only competitive advantage that makes an e-BIZ center different from other manufacturing shops. Too much attention was given to sales and marketing capacities and too little to the e-BIZ centers’ capacities to perform their core business, i.e. application of new ICT. Only three group seminars related to marketing and financial management were organized and they have not covered all e-BIZ centers. Transwork and TTC staff did not attend marketing and financial management trainings and the EIA did not make any effort to repeat the trainings after these two new e-BIZ centers were established and included in the Project. In order to disseminate and transfer ICT to SMEs in the selected industries the e-BIZ centers must have employees with strong expertise in application of new ICT. Having in mind that the employees of the e-BIZ centers had a rather low average level of attained education as well as little or no working experience relevant to the application of ICT, much more effort should have been given to ICT related trainings and technical assistance. Capacity building efforts related to ICT were limited to trainings provided by producers of equipment that was handed over to the e-BIZ centers. Furthermore, capacity building related to ICT was done only in cases of ARTECO and TTC, which were the recipients of the equipment. No ICT training or technical assistance was provided to the other two e-BIZ centers. 10. The e-BIZ centers should have built in capacity to research and apply ICT in their production processes, not to reduce it to the marketing departments of their parent companies. Presently all ICT research and development and ICT applications in production capacities rest with the parent companies. 11. More effort should have been invested in attracting and retaining highly qualified staff with relevant knowledge, skills and experience. The e-BIZ centers, especially ARTECO and TTC, should have done much more in terms of creating more attractive packages to attract highly qualified employees, due to their remote location. Another option would be significant investment in education and training of the existing staff. 12. More should have been done in raising awareness amongst SMEs about the benefits of introducing ICT into the production processes. Although the EIA and e-BIZ centers made a lot of effort to market their products they failed to raise awareness amongst SMEs concerning the importance of applying ICT in production processes. This is partially a consequence of the failure to establish meaningful cooperation with the public sector and business associations. Management of the e-BIZ centers believed that SMEs in Bosnia and Herzegovina were still not aware of the benefits they could gain through the introduction of ICT applications to the production process. Therefore, they did not manage to expand their client base and disseminate ICT applications throughout the target industries. 13. The involvement of the public sector is of great importance. It is not possible to implement this type of project, with ambitious goals such as increasing the competiveness of entire industries, successfully without the involvement of the public sector. The involvement of government, business associations and R&D institutions would have multiplied the project impact several times. 14. More should have been done to establish long term cooperation between the e-BIZ centers and SMEs. The majority of SMEs were one-time clients and the e-BIZ centers did little to adequately manage their relations with them and turn this into long-term cooperation. 54 15. Arteco and its owners should be required to meet the financial target values set by the business plan. Financial projections set the value of Arteco’s capital in 2009 at 711,720 BAM; however, on June 30 2013 the total value of Arteco’s capital was 2,000 BAM. Values for current and long-term assets set in the business plan were also not achieved, as well as ratio between long-term and current assets set in the business plan. 16. Financial position, structure of ownership and capital of Exploring BiH should be further investigated, clarified and brought in line with its business plan. Exploring BiH has no long-term assets, no capital and its accumulated losses have exceeded the initial capital value. The structure of ownership remains unclear. 17. TTC and its owners should be required to achieve the financial target values set by the business plan. TTC did not achieve the financial projections and target values for its capital as specified in the TTC business plan. In addition, TTC failed to meet the business plan projections for its revenue. 18. USAID should not support interventions supporting “one of many” similar service providers and this way distort the market. The Evaluation Team does not see any added-value in a USAID intervention that supports ‘one of many’ similar service providers. The intervention could do more harm than good by supporting one company with non-commercial financial resources in a reasonably well served market where existing company/companies provide a more comprehensive and technologically advanced service. The Evaluation Team is of opinion that the failure was caused by a desire to achieve a ’quick win’ and a poor understanding of the transport/logistics sector and market as well as an inability to foresee future developments in the area of the provision of services to freight forwarders. 19. The Evaluation Team could not find any reasonable justification as to why the promotion of tourism for the entire country was entrusted to one person without any institutional capacity. The entire effort remained a private endeavor of one person with no organizational or institutional capacity and without a transparent ownership structure. Exploring BiH failed to establish any meaningful working relations with associations and public institutions relevant for the development of tourism in Bosnia and Herzegovina. 20. The Evaluation Team does not see any added-value of a USAID intervention that supports ‘one of the many tourism websites’ that, despite the grants and technical assistance provided, failed to become the most technologically advanced and most comprehensive source of information. Other similar initiatives by private and public sector organizations promote Bosnia and Herzegovina and provide support for the establishment of linkages between tourism service providers, tourists and tour operators. Based on the evidence and conclusions drawn out, the failure was caused by a poor understanding of the tourism sector and market and an inability to foresee future developments in the area of the provision of services to service providers, tour operators and tourists. 21. TTC, if adequately mentored and managed, could become the ICT resource center for many BiH textile companies without any ICT capacity. Almost 90% of BiH textile companies work for foreign customers on basic cutting and sewing operations based on received patterns; however, ICT capacities in the majority of these companies are non-existent. Therefore, if adequately mentored and managed, TTC could become the ICT resource center for many BiH textile companies presently without any ICT capacity. Unfortunately, after being operational for almost two years, TTC’s outreach remains low and it continues using its resources mainly in support of production in its parent company. 55 ANNEXES 56 ANNEX I: EVALUATION STATEMENT OF WORK PERFORMANCE EVALUATION Statement of Work (SoW) ACTIVITY INFORMATION IDENTIFICATION Program/Project Title: Excellence in Innovation Activity Cooperative Agreement: LWA GDG‐ A‐ 00‐ 01‐ 00014‐ 00 Award Number: 168‐ A‐ 00‐ 07‐ 00101‐ 00 Original Start‐ End Dates: December 3, 2007‐ December 2, 2010 Original Budget: $3,164,622 Modification #1: Two‐ year extension, December 3, 2010‐ December 2, 2012 Modified Budget: $5,164,622 Modification #2: Nine‐ month extension, December 3, 2012‐ August 31, 2013 Modified (Final) Budget: $4,598,126 Implementing Organization: FHI 360/EDC AOR: Dobrila Vukmanovic DESCRIPTION Activity Purpose The main purpose of the Excellence in Innovation (EI) Activity is to enhance the competitiveness of BiH small and medium‐ sized enterprises (SMEs) in selected sectors by increasing their use of value‐ added information and communications technology (ICT) so that they can offer new, high value‐ added products and services that are in demand in domestic and/or global markets of the selected sectors. Development Challenge USAID’s current Country Development Cooperation Strategy for Bosnia and Herzegovina 2012‐ 2016 (CDCS) identifies the continued need to improve private sector growth as part of a larger overall strategy to help Bosnia and Herzegovina (BiH) meet the challenges of economic growth. As such, increasing competitiveness and growth of SMEs is a top priority for BiH economic development and is highlighted in the Mission’s CDCS. As part of the State Department’s Transformational Diplomacy efforts to tie foreign assistance more closely to foreign policy goals, the State Department identified five policy goals to be pursued in BiH. “Building a Self‐ Sustaining Economy” is included as one of these five goals, and encouraging private sector expansion is highlighted as one of the means to achieve this goal. To achieve economic growth on a sustainable basis, the BiH economy must improve its competitiveness on both a macro and a micro level. For BiH SMEs to compete successfully on a regional level and sustain future pressures of EU accession, they must (improve their ability to) absorb new technologies. One aspect of the required technological upgrade is their ability to effectively utilize relevant, value‐ added ICT applications and services. Effective 57 use of ICT will enable BiH businesses to attract and communicate with customers, employ efficient supply and distribution channels and improve business processes. BiH SMEs do not fully understand how ICT can help them compete, and they have limited capacity to use ICT effectively to operate in export and domestic markets, manage supply and distribution chains, or better manage their business overall. Although a few large companies utilize ICT for management processes, the notion of using modern ICT systems for business functions – management, production, distribution, etc. – is alien to most BiH firms. To the extent that companies use ICT, it tends to be focused on accounting. One of the greatest challenges to building the competitiveness of BiH companies in international markets involves lack of information among senior managers. This is not to say that businesses are resistant to using ICT. Most companies spend about $70/month on Internet connectivity. Unfortunately, SMEs do not appear to be using their computers effectively. Businesses rarely update software after their initial computer purchase, and they often use pirated programs. Few seem to use ICT to improve production processes, marketing and sales, sourcing or management decision‐ making. Information‐ intensive areas are suffering from lack of ICT (e.g., inventory management, marketing, and quality control). A large part of the problem may be simple ignorance of how to use ICT to become more competitive. Use of open source software is limited. Businesses themselves recognize the problem. The companies reported that the major obstacle to effective use of ICT relates to human resources. The Excellence in Innovation Activity in USAID/BiH Economic Growth portfolio addresses the above challenges. Background The EI Activity utilizes ICT as a part of USAID’s efforts to generate employment and promote expansion and competitiveness of BiH SMEs. The Activity is aligned with the Foreign Assistance Framework Economic Growth Program Area of Private Sector Competitiveness and Program Element Private Sector Capacity. It provides assistance to and collaborates with: (a) BiH entrepreneurs and firms to establish and operate sustainable “e‐ BIZ Centers/Enterprises” that offer high‐ impact ICT applications and services to SMEs in targeted and supported sectors; and (b) SMEs to improve their capacity to fully utilize the high‐ impact ICT applications and services offered by collaborating e‐ BIZ Centers/Enterprises to improve their competitiveness. The EI Activity furthers USAID/BiH strategy, for which an economic growth priority is increasing the competitiveness and growth of SMEs in the local, regional, and global marketplace. In cooperation with the Norwegian Ministry of Foreign Affairs (NMFA), the EI Activity established four e‐ BIZ Centers during the original period (2007‐ 2010), one in each of the following four sectors: wood processing (USAID funded), tourism (USAID funded), machine tools (NMFA funded) and agriculture (NMFA funded). During the extension periods (2010‐ 2013), two additional USAID‐ funded e‐ BIZ Centers were established, one to support BiH SMEs from the textile industry, and the other to support various industries related to transport/logistics services. The four USAID‐ funded e‐ BIZ Centers were established through four sub‐ grants to the following companies/legal entities:  Artisan, a wood processing company to develop the wood products e‐ BIZ Center;  Fondacija e‐ turizam, to develop the tourism portal e‐ BIZ Center;  Kismet, to develop an e‐ BIZ Center to support SMEs from the textile industry; and  Transwork, to develop online support to SMEs from various industries related to transport/logistics service. a) Wood Technology e‐ BIZ Center – Arteco (http://www.arteco.ba/) 58 The EI Activity supported the establishment of the Arteco e‐ BIZ Center, which provides technical services to BiH SMEs in the wood processing sector in to increase their competitiveness. Arteco, which began operations in October 2009, offers critical and complex services that are not easily accessible and serves as a driving force of economic development in the BiH wood processing industry through application of new technologies – prototype development, design, CNC machining, and vacuum drying – as well as education and consulting services. The primary objectives of Arteco are to:  Provide technical services to at least 50 SMEs in the wood processing sector during the years of operations;  Increase the competitiveness of BiH wood processing SMEs by utilization of modern technologies, expansion of export customer base, application of the best technology solutions in product finishes, etc.;  Contribute to an increase in the export of processed wood products from BiH;  Contribute to an increase in employment in the wood processing industry;  Become self‐ sustainable by the end of project; and  Build capacity for Arteco employees In addition, Arteco developed a website that should become an information source for the wood industry and provide the following information:  Database of designers, manufacturers, material suppliers and trainers.  Information on access to finance (banks, microcredit organizations, government grants, etc.)  Calendar of events (fairs, etc.) b) Tourism e‐ BIZ Center – Exploring Bosnia (http://www.exploringbosnia.com/) The EI Activity also supported the establishment of Exploring Bosnia, which began operations in August 2009. Its mission is to become a one‐ stop shop that generates in‐ bound tourism, improves the competitiveness of BiH tourism companies in international markets, and serves tourists, especially international tourists, business travelers, and others visiting BiH. In addition to the online web portal, Exploring Bosnia provides booking services for fee‐ paying clients and sells memberships for services to SMEs in the tourism sector. The primary objectives of Exploring Bosnia are to:  Be the most comprehensive source of online information and services for tourists seeking to visit BiH and for tour operators seeking to arrange tours to BiH for their clients;  Serve as an online booking gateway for tourism service providers in BiH;  Contribute to an increase in the volume of tourism services in BiH;  Increase the competitiveness of BiH tourism service providers in international markets;  Provide services to at least 70 SMEs in the tourism sector during the first two years of operations;  Become self‐ sustainable; and  Build capacity for the Exploring Bosnia e‐ BIZ Center employees. c) Textile Technology Center (TTC) (http://www.ttcbh.ba/index.php/bs/) For the apparel sector, the EI Activity assisted in establishment of the Textile Technology Center (TTC), which was opened in early 2012. Its mission is to strengthen the competitiveness of SMEs from the textile/apparel industry through improved use of Computer‐ Aided Design/Computer‐ Aided 59 Manufacturing (CAD/CAM) technology and other IT‐ related services. Using the most modern technology, the Center provides services to national clothing manufacturers so they can better compete in domestic and international markets. As a result, textile industry SMEs should see an increase in manufacturing output and sales and an improvement in capacity utilization, thus achieving the ultimate objective of the TTC. A provision for good quality and speed in service should be based on experienced and trained workers of the TTC. Such center is expected to become a source of experienced and technologically educated textile labor for manufacturers of clothing throughout BiH. The primary objectives of the TTC are the following:  Serve at least 40 SMEs from the textile industry by the end of program  Establish linkages between European buyers and BiH textile SMEs;  Increase the competitiveness of BiH SMEs in the textile industry through utilization of modern technologies, expansion of their export customer base, and application of the best technology solutions in product finishes;  Highly trained and organized employees  Continual update and utilization of state‐ of‐ the‐ art advanced technology  Become self‐ sustainable by the end of program; and  Build capacity for TTC employees The Center will also develop a website that will serve as an information source for the textile industry by providing the following information:  Database of designers, manufacturers, material suppliers and trainers • Information on access to finance (banks, microcredit organizations, government grants, etc.); and  Calendar of events (fairs, etc). d) Logistics/Transport Technology e‐ BIZ Center – Transwork (http://www.transwork.ba/) The Transwork e‐ BIZ Center for the logistics/transport industry was founded in 2011 whilst it started operating in early 2012. Transwork’s mission is to increase competitiveness of BiH transport and logistics companies and SMEs from other industries using transport/logistics services. Applying IT technologies will help these SMES cut costs and increase profits. Specifically, the goals of the Transwork e‐ BIZ Center are:  Become the most comprehensive source of online information and services for SMEs seeking and/or providing logistics/transport services in BiH  Serve at least 40 SME from the logistics and other industries to help them cut costs and increase profits  Increase the competitiveness of the BiH transport/logistics providers in international markets  Become self‐ sustainable; and  Build capacity for the e‐ BIZ Center employees EXISTING DATA USAID/BiH will provide the Contractor with the following documents: Excellence in Innovation Activity work plans; Monitoring and Evaluation (M&E) plans; quarterly and annual progress reports; EI‐ conducted market analyses; business plans for each USAID‐ supported e‐ BIZ Center; and other key activity reports and documents. EVALUATION RATIONALE 60 Evaluation Purpose This evaluation is meant to serve a dual purpose: (1) to inform the design of a new USAID project 1 in the Economic Growth Portfolio and (2) to learn to what extent the project’s objectives and goals have been achieved. Additionally, USAID Evaluation Policy prioritizes building local evaluation capacity and using local expertise in evaluations. USAID/BiH intends to procure this evaluation using solely local expertise to inform the design of its future M&E contract. Evaluation Scope The evaluation of the EI Activity will review the accomplishments of all four USAID‐ supported e‐ BIZ Centers. The evaluation will consider both the original contract period and the extension. Audience and Intended Uses The primary audience for this evaluation is USAID Mission in BiH, and USAID/Washington. Secondary users are the implementing partner(s), local organizations with whom USAID collaborates, and other donors. Evaluation Questions (and their relative priority) 1. To what extent has the EI Activity succeeded in a measurable way in capacity building for e‐ BIZ Center Staff? 2. Are all four USAID‐ supported e‐ BIZ Centers financially and operationally sustainable? 3. To what extent has the EI Activity succeeded in a measurable way in improving the competitiveness of the SMEs served by the four e‐ BIZ Centers? 1 According to the new USAID definition, a “project is defined as a set of executed interventions (activities) over an established timeline and budget, intended to achieve a specific development purpose in order to solve a defined problem. In general, it is expected that a project purpose will correlate to an intermediate results (IR) in the CDCS’ Results Framework. a. What was the situation of the targeted SME population before the EI Activity began? b. What is the situation of the targeted SME population now? c. Did EI respond to their identified needs? 4. What impacts have USAID‐ supported e‐ BIZ Centers had on associations, sectors, and industries targeted by the interventions? 5. What have been the main obstacles in the implementation of the EI Activity? Request for Recommendations Based on a review of the EI Activity’s implementation and results, what recommendations does the evaluation team have for possible future USAID programming to improve the competitiveness of BiH SMEs? Performance Period The performance period will start on/about July 8, 2013. The final evaluation report will be completed 61 and submitted to USAID/BiH two weeks from the receipt of USAID/BiH comments (approximately in Sep 2013). TECHNICAL REQUIREMENTS BASIC REQUIREMENTS Design Based on the information provided within this SoW, the Contractor will submit a written description of the proposed methodology that it would apply to carry out the evaluation. The Contractor is asked to use the design and methodology that will generate the highest quality and most credible evidence that corresponds to the question being asked. Credible evidence depends on sound social science methods and tools used in a manner that minimizes the need for evaluation‐ specific judgments and ensures, to the maximum extent possible, that if different qualified evaluators were to undertake the same evaluation, they would arrive at the same or similar findings and conclusions. After the award, the Contractor will submit to USAID a detailed design and evaluation plan as a first deliverable. To carry out this evaluation, it is anticipated that the evaluation team will need to perform the following tasks:  Initial orientation meetings with USAID/BiH and the project implementer;  Review of key documents and background material on competitiveness of BiH SMEs  Review of project documentation provided by the Mission. The existing data section of this SoW provides a list of the project documents that the Mission or project implementer will provide to the Contractor implementing the evaluation after the contract is signed  Identification of any other relevant performance information sources;  Meet with USAID to review the evaluation objective  Collect data from a broad range of stakeholders, including but not limited to participating SMEs and  Report preparation and briefing METHODS Data collection method The Contractor will provide a detailed data collection method for each evaluation question in the evaluation design. It is anticipated that data will be collected through desk reviews, surveys and key informant interviews: For question 1, the Contractor will address all five areas of EI capacity‐ building efforts:  Technology skills – to build strong skills in all areas needed to develop/adopt and deliver high‐ impact ICT applications services  Financial management – to build ability to track, analyze and allocate finances effectively and to use financial reports to manage the company  Sales and marketing – for local, regional and international markets as appropriate to identify and enter attractive markets, respond to market demands, interact effectively with existing and prospective customers, develop and administer effective promotional programs, use proven sales techniques and presentation skills;  Operations management – managerial skills in adopting and integrating technology to gain maximum efficiencies and productivity; and 62  Other management areas – i.e., strategic planning, human resource management The EI Activity conducted an array of capacity‐ building tasks in these five areas, including hands‐ on training with e‐ BIZ Centers and formal workshops. Those efforts, however, were not tracked with an indicator(s) in the Activity’s M&E plan. Thus, the Contractor will assess the Activity’s reports (and other documents as identified by EI staff) that address those capacity‐ building efforts. In addition, the Contractor will survey e‐ BIZ Center employees and interview e‐ BIZ Center owners and key personnel, and compare the data obtained with the EI reports. The Contractor should consider reconstructing baseline data. In such a case, the Contractor is required to expand on any limitations on the proposed methodology. For question 2, the Contractor will evaluate both financial and operational sustainability of all four e‐ BIZ Centers. For the purpose of this evaluation, financial sustainability is achieved when an e‐ BIZ Center is able to deliver products and services to the market at a price that covers its expenses and generates a profit. In determining whether each of the four e‐ BIZ Centers has achieved financial sustainability, the Contractor will examine both projected and actual financial statements for each Center (projected financial statements were prepared as part of each e‐ BIZ Center business plan and will be provided by USAID/BiH, whereas actual financial statements will be provided by the EI implementing partner). Operational sustainability will be determined through the on‐ site inspections and review of e‐ BIZ Center staffing, equipment, organizational structure, procedures, etc. To answer question 3, the Contractor will need information on the EI Activity results as well as evidence of the targeted SMEs’ financial performance (e.g., increased sales/profits). Additionally, the Contractor will conduct key informant interviews (selected SME owners or general managers). As for questions 3a‐ c, the EI implementing partner will provide the Contractor with market analyses prepared for each of the four industries. The analyses include baseline market surveys for each of the four sectors the e‐ BIZ Centers serve. The survey instruments were designed to gather information on company size, product portfolio, technology utilization, export sales, and overall business outlook. The Contractor is expected to use the data from those surveys and compare them with the current situation within targeted SMEs served by the four e‐ BIZ Centers. It is important to note, however, that some of the e‐ BIZ Center customers were not surveyed and the Contractor is expected to address this data gap in the technical proposal. To answer questions 4 and 5, the Contractor is mainly expected to conduct key informant interviews. Data disaggregation When applicable, the data collected must be disaggregated by gender (e.g., percentage of trainees that were female). Data analysis The Contractor will provide a detailed data analysis plan in the evaluation design. The plan must directly address each evaluation question with specific methods for collecting and for analyzing the data that will be used to answer it. Methodological strengths and limitations The Contractor will disclose all methodological strengths and limitations in the evaluation design. STAFFING 63 Team size and qualifications The evaluation team shall demonstrate understanding of the BiH economic and business environments and show familiarity with USAID’s Evaluation Policy. The evaluation team of two to three persons – one Senior Business Development Expert, one Senior Evaluation Specialist, and possibly one Business Development Expert – will work under the guidance of the COR from Program Office or his/her designee. The Contractor is at liberty to propose expatriate advisors for the position of the Senior Evaluation Specialist. The technical proposal shall include an overall staffing plan for the evaluation (included in the page limitation). The Contractor must describe the roles and responsibilities of the evaluation team and identify key personnel. The Contractor shall demonstrate its technical staffing expertise, capacity in a staffing plan and understanding of the U.S. Government’s requirements. The Contractor shall clearly describe the professional qualifications of proposed key personnel. Key Personnel The Senior Business Development Expert must have at least 10 years of experience in business development. Experience in at least two of the EI Activity’s four industries is required. Exceptional knowledge of finance and accounting is required. Hands‐ on experience in specifying and using ICT business applications is highly desirable. Experience in managing successful start‐ up ICT companies or implementing business plans that focus on using ICT support business strategy is also desirable. MBA degree is required. The Senior Evaluation Specialist must have a deep understanding of social science research and statistical methods. A master’s degree from a U.S. accredited university in statistics, business administration, economics, public policy, program evaluation, or other relevant field is highly desirable. The Specialist must have at least five years of direct experience in conducting evaluations and at least 10 years in international development. The Evaluation Specialist must have experience in developing evaluation designs, including collecting and analyzing data and writing evaluation reports. Experience in business development sector is highly desirable. In addition to detailed curriculum vitae for the proposed key personnel, the proposal shall include a summary of proposed individuals’ experience in: 1) business development; 2) industries/sectors targeted by the EI Activity; 3) BiH politics and economics; and 4) evaluation. The Contractor will also submit examples of past evaluation reports produced under the direction of a proposed team leader and/or evaluation specialist. Conflict of interest All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest. MANAGEMENT INFORMATION DELIVERABLES Detailed Draft and Final Evaluation Plan (two hard copies and one soft copy) A draft and final Evaluation Plan shall be completed by the lead evaluator within one week of the award of the contract and presented to the COR or his/her designee. The plan will contain a work plan and evaluation design. The evaluation design will include a detailed evaluation design matrix (including the key questions, methods and data sources to be used to address each question and the data analysis plan for each question), draft questionnaires and other data collection instruments or their main features, known 64 limitations to the evaluation design, and a dissemination plan. The final design requires COR approval. Unless exempted from doing so by the COR, the design will be shared with country‐ level stakeholders and implementing partners for comment before being finalized. The work plan will include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation team. Draft Report (five hard copies and a soft copy) Subject to USAID/BiH review. Due two weeks after fieldwork is completed. Final Evaluation Report (5 hard copies and a soft copy) The evaluation final report should include an executive summary, introduction, background of the local context and the activity being evaluated, main evaluation questions, methodology or methodologies, limitations to the evaluation, findings, conclusions, and recommendations and lessons learned (if applicable). The report shall not exceed 30 pages excluding appendices. The executive summary should be 3‐ 5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include:  The Evaluation Statement of Work;  Any “statements of differences” regarding significant unresolved difference of opinion by funders, implementers, and/or members of the evaluation team;  All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides;  Sources of information, properly identified and listed; and  Disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. The Contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within 30 calendar days of final approval of the formatted report. In accordance with USAID evaluation policy, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report:  The evaluation report should represent a thoughtful, well‐ researched and well organized effort to objectively evaluate what worked in the activity, what did not and why?  Evaluation reports shall address all evaluation questions included in the statement of work.  The evaluation report should include the statement of work as an annex. All modifications to the statement of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline, need to be agreed upon in writing by the technical officer.  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation, such as questionnaires, checklists and discussion guides, will be included in an annex in the final report.  Evaluation findings will assess outcomes and impact on males and females (when applicable).  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. 65  Sources of information need to be properly identified and listed in an annex.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action‐ oriented, practical and specific, with defined responsibility for the action. The Contractor should refer to the USAID How‐ to Note on Preparing Evaluation Reports (Annex A). The evaluation report must follow the USAID Evaluation Report Template (Annex B) and USAID Sample Evaluation Report Cover (Annex C). Flash drive including all instruments and data in formats suitable for re‐ analysis All records from the evaluation (e.g., interview transcripts or summaries) must be provided to the COR. All quantitative data collected by the evaluation team must be provided in an electronic file in easily readable format agreed upon with the COR. The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. Oral briefings The evaluation team will conduct an exit briefing the fieldwork completion. LOGISTICS USAID/BiH will provide facilities for oral briefings to staff and other stakeholders on evaluation findings. LOE AND BUDGET Estimated level of effort The Bidder is at liberty to develop the team’s own prospective allocation of level of effort and accompanying budget. A two‐ to three‐ person team consisting of one Senior Business Development Expert, one Senior Evaluation Expert, and one Business Development Expert would work for approximately eight weeks (Monday to Saturday)  One week for planning and preparation;  Four weeks in the field to collect and analyze data and present preliminary evaluation findings; and  Three weeks in total to work on both the draft and final report ESTIMATED BUDGET TBD 66 ANNEX II: DATA COLLECTION INSTRUMENTS SME QUESTIONAIRE USAID Mission in BiH conduct evaluation of the Project "Excellence in Innovation Activity" implemented with the objective to improve competitiveness of small and medium size companies in selected sector in BiH by increasing values using information and communication technologies to offer new products and services at demand in domestic and international market. Selected sectors are: textile industry, tourism, wood processing industry and transport industry . Within each sector establishment of single e-BIZ Center is supported with the task to enable access to new products and services that are based on new information and communication technologies. In order to assess effectiveness of the Project, we kindly ask you to sincerely answer following questions. Answers will be aggregated, anonymous, and used only for evaluation purposes. 1. Name of company 2.Post title of the person filling the questionare 3. Gender of the owner (" Male ("Female 4.Gender of the General Manager (" Male ("Female 5. Sector r textile r tourism r wood processing r transport 6. Are you aware of products and services provided by-BIZ Center from your Industry l'Yes l'No 7.1f the answer to previous question was YES,please describe how did you find out about products and services of e-BIZ Center D We received promotional material D We received an e-mail D Through TV and radio D Over the internet (browsing) D Through partners D Something else, describe 8. Are you familiar with the offer of e-BIZ Center l'Yes l'No 9.1fyou responded YES to question 8, did new offer (product and/or services) emerged on the market after establishing e￾BIZ Center l'Yes l'No Desribe product I services 10. Are you in contact with e-BIZ Center l'Yes l'No Describe motives 11. Did you establish business collaboration with e-BIZ Center 67 l'Yes l'No Describe motives 12. Did your company participated in the process for identification of key priorities within your industry that is conducted by EDS (Project implementer) l'Da l'Ne 68 13. Could you please estimate to what extent you are satisfied with specific aspects of your cooperation with your suppliers? If you cooperate with e-BIZ Center, please estimate collaboration with e-BIZ center on the left side, while your collaboration with other suppliers should be estimated to the right! If you do not collaborate with e-BIZ Center, fill the right side only! Please estimate by selecting appropriate circle that best describes your level of satisfaction! 1 -not satisfied at all; 5- fully satisfied e-BIZ CENTAR OTHER SUPPLIERS r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 respond time to request frequency of contact frequency of contact after the contracted business Ways of communication Creativity in responding the requests Their promotion Clarity of the offer Offered educations Prices Other non-financial gains r 1 r 2 (' 3 l' 4 l' 5 r 1 r 2 (' 3 l' 4 l' 5 r 1 r 2 (' 3 l' 4 l' 5 r 1 r 2 (' 3 l' 4 l' 5 r 1 r 2 (' 3 l' 4 l' 5 r 1 r 2 (' 3 l' 4 l' 5 r 1 r 2 (' 3 l' 4 l' 5 r 1 r 2 (' 3 l' 4 l' 5 r 1 r 2 (' 3 l' 4 l' 5 r 1 r 2 (' 3 l' 4 l' 5 14. Could you please estimate to what extent you have undergone change in the last two years in following areas (on a scale from -5 to 5)!-5- change is substantial and negative; 0- there is no change;5- change is substantial and positive Number of employees {' -5 {'-4 (' -3 (' -2 l'-1 (' O ('1 (' 2 (' 3 (' 4 (' 5 Own competitiveness Number of employees two years ago CJ r -5 r -4 r -3 r -2 r -1 r o Current number of employees D r 1 r 2 (' 3 l' 4 l'5 Number of contracted works (' -5 (' -4 (' -3 (' -2 (' -1 r o r 1 r 2 (' 3 l' 4 l'5 Sales Profit Productivity Export r -5 r -4 r -3 r -2 r -1 r o r -5 r -4 r -3 r -2 r -1 r o r -5 r -4 r -3 r -2 r -1 r o r -5 r -4 r -3 r -2 r -1 r o r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 ratio of export two years ago D current ratio of export Product portfolio Technology utilization r -5 r -4 r -3 r -2 r -1 r o r -5 r -4 r -3 r -2 r -1 r o r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 15. If you collaborate with e-BIZ Centers estimate (on a scale 1 to 5) to what extent your change can be attribute toe-BIZ centre in the following characteristics: -change is not attribute toe-BIZ Centre;5- change is to full extent attributed to collaboration with e-BIZ Centre Own competitiveness r 1 r 2 (' 3 l' 4 l'5 Number of contracted works (' 1 (' 2 (' 3 (' 4 (' 5 Sales Profit Productivity Export 69 Product portfolio Technology utilization r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 r 1 r 2 (' 3 l' 4 l'5 16.1n your opinion, what is the impact of USAID supported e-BIZ Centers on Associations, Sector and Industries? I Thank you for filing the questionnaire 70 STAFF QUESTIONAIRE USAID Mission in BiH conduct evaluation of the Project "Excellence in Innovation Activity” that has been conducted in collaboration with thee-BIZ Center, in order to improve competitiveness of the small and medium size companies in selected sectors in BiH through increasing values using the information and communication technologies in order to offer new products and services that are in demand in domestic and international market. In order to assess effectiveness of the project we kindly ask you to sincerely answer following questions. Answers are anonymous and there is no need to sign the questionnaire. Name of e-BIZ Center 1. How long are you working for e-BIZ Center (months) 2. What position 3. Gender (" Male (" Female 4. Does company pay retirement fund for you, health insurance and taxes l'Yes l'No 5. During your employment,have you participated in educations with the objectives to increase your business skills l'Yes l'No 5.1 If you answered YES on question 5, how much education you participated in (No of days) 5.2 If you answered YES on question 5, state education themes and name of main company providing education 6. Estimate to what extent you have had developed below listed skills and knowledge prior to working for e-BIZ center and to what extent these skills and knowledge are developed today? Please make you estimate by selecting appropriate circle that best describes your skill/knowledge level! 1 -not developed at all; 5- developed to full extent PRIOR WORKING WITHE-BIZ CENTER TODAY l'1 l'1 l'2 (" 3 l'2 (" 3 l'4 l'4 l'5 l'5 Computer skills l'1 Understanding hardware of application currently using in (" 1 l'2 l'2 (" 3 (" 3 l'4 l'4 l'5 l'5 e-BIZ center l'1 l'2 (" 3 l'4 l'5 Understanding software of application currently using in e-BIZ center l'1 l'2 (" 3 l'4 l'5 l'1 l'2 (" 3 l'4 l'5 Capable to troubleshoot minor glitches in the application l'1 l'2 (" 3 l'4 l'5 system in order to continue with the work l'1 l'2 (" 3 l'4 l'5 Work in a team on business assignments l'1 l'2 (" 3 l'4 l'5 l'1 l'2 (" 3 l'4 l'5 To see developmental vision of business l'1 l'2 (" 3 l'4 l'5 l'1 l'2 (" 3 l'4 l'5 Understand the role of General Manager l'1 l'2 (" 3 l'4 l'5 l'1 l'2 (" 3 l'4 l'5 Understand roles and responsibilities within the organizational structure l'1 l'2 (" 3 l'4 l'5 l'1 l'2 (" 3 l'4 l'5 Understand main priorities of industries working in l'1 l'2 (" 3 l'4 l'5 l'1 l'2 (" 3 l'4 l'5 Independently manager company finances l'1 l'2 (" 3 l'4 l'5 l'1 l'2 (" 3 l'4 l'5 Efficiently managing company l'1 l'2 (" 3 l'4 l'5 ("1 ("2 ("3 ("4 l'5 Introduced with the laws governing financial practice ("1 ("2 (" 3 ("4 l'5 71 7. Estimate to what extent you are satisfied with the following characteristics of your working environment Give your estimate by selecting appropriate circle next to the option best describe your level of satisfaction! 1 - not satisfied at all; 5 -fully satisfied Your own skills need to run business application (' 1('2 ('3 ('4 rs Quality of education provided by EDC (' 1('2 ('3 ('4 rs Quality of education provided by e-BIZ Center (' 1('2 ('3 ('4 rs Quality of work assignment demonstration on a working post r 1 ('2 ('3 ('4 rs Team effort (' 1('2 ('3 ('4 rs Clarity of roles (' 1('2 ('3 ('4 rs Role of Manager in e-BIZ Center (' 1('2 ('3 ('4 rs Working time (' 1('2 ('3 ('4 rs Salaries (' 1('2 ('3 ('4 rs Work volume (' 1('2 ('3 ('4 rs Work safety (' 1('2 ('3 ('4 rs Business sustainability ('1('2 ('3 ('4 rs Your own employment expected timeframe (' 1('2 ('3 ('4 rs Rewarding system in place (' 1('2 ('3 ('4 rs Manager's communication with you (' 1('2 ('3 ('4 rs Manager's communication with customers (' 1 ('2 ('3 ('4 rs Alignment of internal policies with Labor Law (' 1('2 ('3 ('4 rs Possibilities for promotion ('1('2 ('3 ('4 rs Possibilities for in-service education (' 1 ('2 ('3 ('4 rs Role of EDC in the business of e-BIZ Center (' 1('2 ('3 ('4 rs Role of USAID in the business of e-BIZ Center (' 1('2 ('3 ('4 rs Thank you for your cooperation 72 ANNEX III: SOURCES OF INFORMATION BIBLIOGRAPHY OF DOCUMENTS REVIEWED 1. Statement of Work for EI Evaluation Project 2. Evaluation team proposal 3. Industry Analysis 4. Market Analysis 5. Cooperative Agreement and its First & Second Extension 6. Start-Up Implementation Plan 7. Business plans and Milestone tables 8. FOGs 9. EI Annual reports 10. EI Quarterly Reports 11. EI Work plans 12. EI Capacity Building Consultants SOW 13. EI Training for Capacity Building 14. E-BIZ Centers Client Lists 15. Current organizational charts 16. Current employees CVs 17. E -BIZ Centers’ First Year Clients Survey 18. Data quality Assessment sheets for EI Project 19. E-BIZ Center's Financial Statements LIST OF PERSONS INTERVIEWS Key informants: • USAID Project Manager • EDC Chief of party • e-BIZ Centers’ Owners (key manager) • e-BIZ Centers’ Managers (key manager) if different from owner • All employees available in the e-BIZ Centers on the evaluation visit day • Key managers from SMEs using e-BIZ Centers’ services • Key managers from SMEs not using e-BIZ Centers’ services • EDC Headquarters staff • Representatives of Business Associations • Representatives of International Organizations 73 Date of Interview Company Industry Town BIZ Center 24.06.2013 The EI Activity Sarajevo The EI Activity 13.07.2013 Arteco Management – introductory meeting Wood Tesanj - Medakovo Arteco 13.07.2013 Mr. Sasa Micic – introductory meeting Transport Banja Luka Transwork 24.07.2013 Zenko doo Textile Zenica TTC 24.07.2013 Pero doo Wood Zenica Arteco 24.07.2013 TTC Management - introductory interview Textile Mala Brijesnica TTC 25.07.2013 Entasis Wood Zenica Arteco 25,07.2013 Elchy Tex Textile Žepče TTC 26.07.2013 Jadrina doo Wood Doboj Istok / Gracanica Arteco 26.07.2013. Premier doo Textile Doboj TTC 26.07.2013 Zeka doo Trade / Transport Gracanica Transwork 29.07.2013 Ms. Sasa Vasic￾introductory interview Tourism Sarajevo Exploring BH 30.07.2013 Hotel Tuzla Tourism Tuzla Exploring BH 30.07.2013 Olimp Textile Gracanica TTC 30.07.2013 Heroks Transport Tuzla Transwork 31.07.2013 Standard Wood Sarajevo Arteco 01.08.2013 Granof Textile Sarajevo TTC 02.08.2013 Hotel Bosna Tourism Banja Luka Exploring BH 02.08.2013 Krajinalijek doo Transport Banja Luka Transwork 02.08.2013 Hotel Sarajevo Tourism Sarajevo Exploring BH 06.08.2013 Goran Pelja Wood Sarajevo Arteco 06.08.2013 Hotel Evropa Tourism Sarajevo Exploring BH 06.08.2013 Hotel Heco Tourism Sarajevo Exploring BH 06.08.2013 Mr. Renato Zrnic Tourism Sarajevo USAID Firma 15.08.2013 IM Transport Transport Žepče Transwork 15.08.2013 Hotel Dubrovnik Tourism Zenica Exploring BH 16.08.2013 Ms. Sasa Kotlica Wood Sarajevo USAID FIRMA 16.08.2013 Sarajevska pivara Transport Sarajevo Transwork 16.08.2013 Adana Co doo Wood Hadzici Arteco 19.08.2013 Transwork Transport Banja Luka Transwork 19.08.2013 Arteco Wood Medakovo Arteco 22.08.2013 Neretva rafting Tourism Konjic - Djajici Exploring BH 22.08.2013 Hotel Bristol Tourism Mostar Exploring BH 23.08.2013 TTC Textile Brijesnica Mala TTC 23.08.2013 The EI Activity Sarajevo The EI Activity 74 ANNEX IV: DISCLOSURE OF ANY CONFLICTS OF INTEREST Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Davorin Pavelic Title Senior Business Development Expert Organization Team of Individual Experts Evaluation Position Team Leader Team Member Evaluation Award Number USAID Project/ Activity(ies) Evaluated USAID/Bosnia Excellence in Innovation Activity I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: Close family member who is an employee of the U operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 12 June 2013 75 Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Jasmina Mangafic Title Business Development Expert Organization Team of Individual Experts Evaluation Position Team Leader Team Member Evaluation Award Number USAID Project/ Activity(ies) Evaluated USAID/Bosnia Excellence in Innovation Activity I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: Close family member who is an employee of the U operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 76 Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Danijel Hopic Title Senior Evaluation Specialist Organization Team of Individual Experts Evaluation Position Team Leader Team Member Evaluation Award Number USAID Project/ Activity(ies) Evaluated USAID/Bosnia Excellence in Innovation Activity I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: Close family member who is an employee of the U operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 77 ANNEX V: QUESTIONS FOR THE EIA SELF-ASSESSMENT 17.07.2013 General Remarks: Requested reports are enlisted in the Cooperative Agreement and the Start-Up Implementation Plan as an obligation of the EI project Other information / documents / reports are needed for answering 5 (five) evaluation questions defining the evaluation frame and scope Parts of the text in quotations „“are extracted from the Cooperative Agreement and the Start-up Implementation Plan • Please provide definition of the term “targeted SMEs” • Please provide monitoring plan of the Project • Please provide monitoring reports • Please provide lessons learned report Question 1 - To what extent has the EI Activity succeeded in capacity building for e‐BIZ Center Staff in a measurable way? 1.1. Please provide detailed CVs of e-BIZ Centers employees 1.2. Please explain how the project will utilize PPP. In addition, please document all activates based on / related to PPP? 1.3. Please provide documentation containing evidence of sustainability -„activities providing products and services to SMEs are sustainable“ 1.4. Please provide a comprehensive table related to capacity building activities that will contain: • Title of training / technical assistance provided • Dates of trainings / technical assistance • Duration • List of visits of international and national experts and type of technical assistance provided by each expert • Number of people trained / provided with hands-on technical assistance (please attach list with names of trainees / people receiving technical assistance) • Names of trainers / consultants (please attach TORs and reports submitted by consultants) 78 1.5. Please explain how training needs were identified or / and on what basis the training program was designed 1.6. Please provide documents (training plan / agenda / PPP / teaching notes / trainers bio / training evaluation ) related to „strong hands-on training, especially on sales / marketing and financial management“ 1.7. Please provide documents (technical assistance plan / notes / consultant reports / consultant bio / evaluation of quality of technical assistance) related to „strong hands-on training, especially on sales / marketing and financial management“ 1.8. Please explain (document) what has been done in order to „Assist e-BIZ Centers to develop and implement effective promotional programs to attract SME customers (training sessions for SMEs on understanding cost-effectiveness of the high impact application / provision of individualized customer training)“ 1.9. Please explain (document) what has been done on „building associations' leaders understanding of the value of high impact applications and communicate the value of high impact applications“ 1.10. Please explain (document) what has been done and how in order to „work jointly with e-BIZ Centers to help potential SME customers understand and utilize e-BIZ Center services“ 1.11. Please provide description / documents related to capacity building activities aimed at e-BIZ SME customers 1.12. Please explain (document) assistance the EI Project provided to e-BIZ Centers in development of detailed business plans. How this assistance was provided, by whom, in what areas and with which level of effort Question 2 – Are all four USAID‐supported e‐BIZ Centers financially and operationally sustainable? 2.1. Please provide report / analysis on e-BIZ Centers sustainability 2.2. Please explain what „financial management process of the Center” is and what was done by the EI project in order to establish the above mentioned process 2.3. Please provide Annual Financial Reports of e-BIZ Centers (Balance Sheet, Income Statement, Cash Flow and other legally obligatory financial reports) 2.4. For the year 2013, please provide preliminary semi-annual financial report (if not yet officially submitted to BiH tax authorities). If the semi-annual financial report was already submitted to BiH Tax authorities, please provide a copy 2.5. Please provide information on e-BIZ Centers products/ services portfolio and financial contribution (share) of each product / service in total revenues. These data are needed on an annual basis from the beginning of the Project until today. 2.6. Please provide breakdown of revenues of the e-BIZ Center by each SME customer, for each year of e-BIZ Center operation 79 2.7. Please ask e-BIZ Centers to fill in required input data into the Balanced Scorecards provided in a form of an excel file Question 3 - To what extent has the EI Activity succeeded in a measurable way in improving the competitiveness of the SMEs served by the four e‐BIZ Centers? 3a - What was the situation of the targeted SME population before the EI Activity began? 3b - What is the situation of the targeted SME population now? 3c- Did the EI respond to their identified needs? 3.1. Please provide breakdown e-BIZ Center SME clients by size (micro-small, medium, large – please use criteria such as number of employees, turnover...) 3.2. Please provide breakdown of e-BIZ Centers SME clients by the following categories: • Signed long-term (more than a year and multiple services) contracts about provision of services; • Signed one-service contracts; • List of prospect clients (in process of negotiation) 3.3. Please provide a report tracking the following USAID standard level indicators: • Number of firms that received USAID assistance to improve their management practices (financial management, strategic planning, marketing, etc.) • Number of firms receiving USAID assistance to invest in improved technologies 3.4. Please provide a report on „survey on Centers’ services value” performed after the first year of Center operations, as specified in the Start-Up Implementation Plan 3.5. Please provide a report on „in-depth customer survey” that should be performed at the end of the project • Please provide “a survey of e-BIZ Center SME customers at an appropriate point sufficiently near the end of the project to capture e-BIZ Centers’ impacts; gender considerations will include identification of the number of companies with a woman owner or General Manager” 3.6. Please provide explanation why there are no baseline indicators such as sales, profitability and productivity for SME clients of e-BIZ Centers 3.7. Please acquire / provide indicators of sales, profitability and productivity for SME clients of e-BIZ centers for the period of the life span of the Project. These indicators are commercially available at AFIP Agencies in FBiH and RS Question 4 - What impacts have the USAID‐supported e‐BIZ Centers had on associations, sectors, and industries targeted by the interventions? Question 5 - What have been the main obstacles in the implementation of the EI Activity? 80 5.1. Please explain the main obstacles (solved as well as unsolved) in the implementation of the EI Activity 5.2. Please provide a document explaining „a gender strategy to support women's equitable involvement in the EI activities including entrepreneur partnership, SME use of e-BIZ Center services and project use of local experts“ ANNEX VI: REQUEST FOR INFORMATION TO E-BIZ CENTERS Dear Sir/ Madam, In order to evaluate financial and operational sustainability of your organization, we kindly ask you to submit the following documentation: 1. Set of annual financial reports for each year of e-BIZ Center operations. Set of financial reports should contain: • Balance Sheet • Income Statement • Statement of Cash Flows • Report on changes in ownership capital, accounting policy and financial reports notes 2. Financial reports for the first six months of 2013 3. Review of types of products and services, prices of products and services and their individual shares in total revenues for each year of e-BIZ Center operations 4. Review of amounts and structure of financial contributions of e-BIZ Center founders 5. Review of e-BIZ Center clients and share of revenues coming from each client in total e-BIZ Center revenues (annually). Arrange list of clients per following criteria: a) signed long-term contract; b) one￾time services; and c) in process of negotiation In addition to the above-mentioned reports, it is also necessary to submit CVs of every e-BIZ Center employee to the Evaluation Team. In addition, all e-BIZ Center employees are required to fill in the Employee Questionnaire attached to this message. We would appreciate if you could fill in and return back to us excel file “Balanced Scorecard Input Data” (see attachment). Due to very tight evaluation schedule, we kindly ask you to submit requested documents not longer than seven (7) days from today (until 12.08.2013) Thank you for your cooperation and understanding. With kind regards, 81 Davorin Pavelic ANNEX VII: E-BIZ CENTERS’ STAFF SURVEY RESULTS Main objective of Staff Survey was to collect quantitative evidence of increased staff capacity building as a result of the EIA Project. Evaluation question 1. To what extent has the EI Activity succeeded in a measurable way in capacity building for e‐BIZ Center Staff? Staff Questionnaire was filled by 22 staff member including all Directors, Administrators and technical personnel being at their post during the interviews taking place in their e-BIZ Center. Number of employees in e-BIZ Centers varies from one (Exploring) to 10 (TTC). Two employees in Arteco (out of 7) do not have a benefits package (retirement fond and health insurance) covered by the employer. Six (6) out of 22 people employed in e-BIZ Centers are female (27%), with all male employees in Arteco (Exploring is having only one staff member that is female). Gender * Does company pays retirement fund for you, health insurance and taxes Cross tabulation e-BIZ Center Gender Yes No Total Arteco Male Count 5 2 7 % within Gender 71.4% 28.6% 100.0% Total Count 5 2 7 % within Gender 71.4% 28.6% 100.0% Exploring Female Count 1 1 % within Gender 100.0% 100.0% Total Count 1 1 % within Gender 100.0% 100.0% Transwork Male Count 1 1 % within Gender 100.0% 100.0% Female Count 3 3 % within Gender 100.0% 100.0% Total Count 4 4 % within Gender 100.0% 100.0% TTC Male Count 7 7 % within Gender 100.0% 100.0% 82 Female Count 2 2 % within Gender 100.0% 100.0% Total Count 9 9 % within Gender 100.0% 100.0% * TTC has one additional staff member who filled the questionnaire however, not filling data on benefit package and therefore missing in a table Out of 22 staff members from all four e-BIZ Centers only two members from TTC are reporting not participating at any capacity development event. These two staff members are at TTC for about 2 months only. Two members from TTC have failed to respond to the question and these are staff member being in TTC for about 11 and 15 months. During your employment, have you participated in educations with the objectives to increase your business skills: Frequency Percent Valid Percent Arteco Yes 7 100.0 100.0 Exploring Yes 1 100.0 100.0 Transwork Yes 4 100.0 100.0 TTC Yes 6 60.0 75.0 No 2 20.0 25.0 Sub Total 8 80.0 100.0 Missing 2 20.0 Total 10 100.0 Number of training days personnel received while working for e-BIZ Centers vary from 1 to 9 training days. Largest numbers of training days have received director of Arteco (9) and Director and Director of Sales in Transwork (6). In Arteco most of personnel received only 1 training day (4). Exploring has only one staff member (Director), and she received three (3) training days while leading the company. TTC company staff have all responded equally to the question related to number of training days, which is referring to the attached list of trainings. Only one person responded to have participated in one (1) training day, while it is assumed that remaining nine (9) staff members have all participated in the trainings listed. If you answered YES on question 5, how much education you participated in (No of days) Number of training days Frequency Percent Valid Percent Arteco 1 4 57.1 57.1 2 2 28.6 28.6 9 1 14.3 14.3 Total 7 100.0 100.0 Exploring 3 1 100.0 100.0 Transwork 4 1 25.0 25.0 5 1 25.0 25.0 83 6 2 50.0 50.0 Total 4 100.0 100.0 TTC 1 1 10.0 100.0 Missing 9 90.0 Total 10 100.0 All personnel have been asked to assess to what level they have adopted new skills while working for e￾BIZ Centers. Results are presented as an average score of difference between baseline estimate and current estimate. Difference is a measure of progress in related skill. Difference span results from 0 to 4, where 4 relates to skill that was not existent at all prior to employment in e-BIZ Center and advancing to maximum during employment. Difference zero indicates no change at all. Theoretically, difference could be negative if employees feel deterioration in skill. If base and increase combined form the result of 5, it indicates development of skills to full extent. In e-BIZ Center Arteco, all targeted skills have an average positive increase. Highest increase is in the perception that they now had better understand main priorities of industry they are working in. High increase is in following targeted skills (difference 2 or above): • Introduced with the laws governing financial practice • Efficiently managing company • Independently manage company finances • See developmental vision of the business Lowest increase is in better understanding roles and responsibilities within the organizational structure. Introduction with the laws, independently managing company finances and understanding roles and responsibilities within the organizational structure are the skills with lowest baseline score, indicating lowest presence prior employment in Arteco e-BIZ Center. Only Director has answered questions related to managing skills, while technical staff responded only to questions related to computer skills, software, hardware, troubleshooting and team work. While increase in Director’s technical skills are minimal (1), higher increase is visible in managing skills where Director reports the increase for about 100% compared to baseline score. In e-BIZ Center Exploring, where results are sole result of company Director, it is visible that there is no increase in technical skills (computer skills, understanding software, hardware, troubleshooting, teamwork and organizational structure). All these technical skills have baseline estimate at maximum. Understanding roles and responsibilities in organizational structure is not at highest level with no change in final estimate compared to baseline. Director however report better understanding the role of General Manager. Highest increase is in skills related to efficiently managing company and understanding main priorities of the industry working in (from minimum to maximum) In e-BIZ Center Transwork, all skills have an average positive increase. Highest increase is in understanding software application currently using in e-BIZ center, having in mind that the work of e￾BIZ center Transwork is based on software application it comes as the very important skill, if not the most important. Additionally, large increases are in following skills: • Understanding hardware of application currently using in e-BIZ center • Understanding main priorities of the industry currently working in • Troubleshooting minor glitches Remaining skills have an increase of 1 or below. Lowest increase is in computer skills that have highest baseline estimate. 84 In e-BIZ Center TTC all targeted skills have positive average increase compared to baseline level. Highest increase is in understanding hardware and software of applications used in the company and troubleshooting minor glitches, while remaining skills have less than 1 increase. e-BIZ Center Arteco e-BIZ Center Exploring e-BIZ Center Transwork e-BIZ Center TTC Perceived average baseline level of skills and knowledge prior working for e-BIZ center and increase attributed to capacity developed while working for e-BIZ Center Staff questionnaire requested staff members to select level of work satisfaction related to 21 different characteristics. Staff members of e-BIZ Center Arteco and TTC have selected an average score on all 21 different characteristics with over 4 (with most of them being 5) indicating full satisfaction in all selected areas. Employees of e-BIZ Center Exploring (Director) is least satisfied with own skills needed to run business application, quality of education provided by EDC and the role of USAID in the business. Somewhat higher but still meaning medium satisfaction is with work volume and team effort. As there is only one staff member, there was no education being organized by Exploring for their staff (thus the scoring is 3.29 2.86 2.71 2.43 3.57 2.50 3.00 2.00 2.50 2.00 3.00 2.00 1.14 1.29 1.43 1.57 1.43 2.00 1.50 1.00 2.50 2.00 2.00 2.00 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Computer skills Understanding hardware of application currently using in e-BIZ center Understanding software of application currently using in e-BIZ center Capable to troubleshoot miner glitches in the application system in order to continue with the work Work in a team on business assignments To see developmental vision of business Understand the role of General Manager Understand roles and responsibilities within the organizational structure Understand main priorities of industries working in Independently manage company finances Efficiently managing company Introduced with the laws governing financial practice BASE INCREASE 5.00 5.00 5.00 5.00 5.00 3.00 4.00 4.00 1.00 3.00 1.00 3.00 0.00 0.00 0.00 0.00 0.00 2.00 1.00 0.00 4.00 2.00 4.00 2.00 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Computer skills Understanding hardware of application currently using in e-BIZ center Understanding software of application currently using in e-BIZ center Capable to troubleshoot miner glitches in the application system in order to continue with the work Work in a team on business assignments To see developmental vision of business Understand the role of General Manager Understand roles and responsibilities within the organizational structure Understand main priorities of industries working in Independently manager company finances Efficiently managing company Introduced with the laws governing financial practice BASE INCREASE 4.00 1.75 1.50 1.25 3.75 3.75 3.75 3.50 2.25 2.00 2.50 2.50 0.75 2.75 3.00 1.75 1.00 1.00 1.00 1.00 2.25 1.00 1.00 1.00 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Computer skills Understanding hardware of application currently using in e-BIZ center Understanding software of application currently using in e-BIZ center Capable to troubleshoot miner glitches in the application system in order to continue with the work Work in a team on business assignments To see developmental vision of business Understand the role of General Manager Understand roles and responsibilities within the organizational structure Understand main priorities of industries working in Independently manager company finances Efficiently managing company Introduced with the laws governing financial practice BASE INCREASE 3.40 1.90 1.56 2.40 4.50 3.90 4.20 4.10 4.00 2.50 2.40 2.40 0.20 2.60 2.56 2.00 0.50 0.70 0.70 0.50 0.78 0.20 0.80 0.40 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Computer skills Understanding hardware of application currently using in e-BIZ … Understanding software of application currently using in e-BIZ … Capable to troubleshoot miner glitches in the application system … Work in a team on business assignments To see developmental vision of business Understand the role of General Manager Understand roles and responsibilities within the organizational … Understand main priorities of industries working in Independently manager company finances Efficiently managing company Introduced with the laws governing financial practice BASE INCREASE 85 missing). While most of average satisfaction levels are over 4 in e-BIZ Center Transwork in some characteristics they are in between 3 and 4. These are: • Possibilities for in-service trainings • Possibilities for promotion • Rewarding system in place • Employment timeframe (sustainability) • Salaries e-BIZ Center Arteco e-BIZ Center Exploring e-BIZ Center Transwork e-BIZ Center TTC Average satisfaction level per industry 4.86 5.00 5.00 5.00 4.86 4.86 5.00 5.00 5.00 5.00 5.00 4.86 5.00 5.00 4.86 5.00 5.00 5.00 5.00 5.00 5.00 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Your own skills need to run business application Quality of education provided by EDC Quality of education provided by e-BIZ Center Quality of work assignment demonstration on a working post Team effort Clarity of roles Role of Manager in e-BIZ Center Working time Salaries Work volume Work safety Business stainability Your own employment expected timeframe Rewarding system in place Manager's communication with you Manager's communication with custemors Alignment of internal policies with Labour Law Possibilities for promotion Possibilities for in-service education Role of EDC in the business of e-BIZ Center Role of USAID in the business of e-BIZ Center 1.00 2.00 5.00 3.00 5.00 5.00 5.00 4.00 3.00 5.00 5.00 5.00 5.00 5.00 5.00 5.00 5.00 5.00 4.00 2.00 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Your own skills need to run business application Quality of education provided by EDC Quality of education provided by e-BIZ Center Quality of work assignment demonstration on a working post Team effort Clarity of roles Role of Manager in e-BIZ Center Working time Salaries Work volume Work safety Business stainability Your own employment expected timeframe Rewarding system in place Manager's communication with you Manager's communication with custemors Alignment of internal policies with Labour Law Possibilities for promotion Possibilities for in-service education Role of EDC in the business of e-BIZ Center Role of USAID in the business of e-BIZ Center 4.50 4.50 4.50 4.50 4.75 4.75 4.75 5.00 3.25 4.25 5.00 4.75 3.75 3.50 4.75 4.75 5.00 3.75 3.50 4.75 4.75 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Your own skills need to run business application Quality of education provided by EDC Quality of education provided by e-BIZ Center Quality of work assignment demonstration on a working post Team effort Clarity of roles Role of Manager in e-BIZ Center Working time Salaries Work volume Work safety Business stainability Your own employment expected timeframe Rewarding system in place Manager's communication with you Manager's communication with custemors Alignment of internal policies with Labour Law Possibilities for promotion Possibilities for in-service education Role of EDC in the business of e-BIZ Center Role of USAID in the business of e-BIZ Center 4.40 4.88 4.90 5.00 4.90 4.90 5.00 4.60 4.20 4.60 4.80 4.80 4.80 4.60 4.90 5.00 4.60 4.40 4.60 5.00 5.00 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Your own skills need to run business application Quality of education provided by EDC Quality of education provided by e-BIZ Center Quality of work assignment demonstration on a working post Team effort Clarity of roles Role of Manager in e-BIZ Center Working time Salaries Work volume Work safety Business stainability Your own employment expected timeframe Rewarding system in place Manager's communication with you Manager's communication with custemors Alignment of internal policies with Labour Law Possibilities for promotion Possibilities for in-service education Role of EDC in the business of e-BIZ Center Role of USAID in the business of e-BIZ Center 86 ANNEX VIII: SME SURVEY Main objective of SMEs Survey was to collect quantitative evidence of improved competitiveness of targeted SME (served by the four e-BIZ Centers) Evaluation question 3. To what extent has the EI Activity succeeded in a measurable way in improving the competitiveness of the SMEs served by the four e‐BIZ Centers? a. What was the situation of the targeted SME population before the EI Activity began? b. What is the situation of the targeted SME population now? c. Did EI respond to their identified needs? In order to measure increase in competitiveness of SMEs served by the four e-BIZ Centers, evaluation team has developed a questionnaire measuring cooperation with e-BIZ Centers and other suppliers and potential influence on their competitiveness, productivity, sales, products diversification and profit. EDC has supplied evaluation team with the list of SMEs from four (4) targeted industries, served by four (4) e-BIZ Centers. USAID Mission in BiH has sent an email to all companies requesting them to participate in evaluation of e-BIZ Centers and respond in filling the questionnaire. Response rate for all four (4) industries was scarce and extremely low as presented in a chart below. Response rate in Wood Industry and Tourism is about 5% while for Transportation and Textile Industry about 4%. Number of responds per Industry The response rate from the wood industry and tourist industry was around 5 percent, while it was around 4 percent for transportation and the textile industry. Of the 199 questionnaires that were sent out to companies in the textile industry only eight (8) were filled in and sent back to the Evaluation Team. Of the 112 questionnaires that were sent out to companies in the transport sector only five (5) were filled in and sent back to the Evaluation Team. 11 8 5 8 197 158 112 199 0 50 100 150 200 250 Wood Industry Tourism Transportation Textile industry Responds No Responds 87 Out of 158 questionnaires sent to companies in the tourism sector only eight (8) were filled in and sent back to the Evaluation Team. Of the 197 questionnaires that were sent out to companies in the wood industry only 11 were filled in and sent back to the Evaluation Team. However, due to the low response rate, small sample size and non-response bias, the inferences based on the SME survey statistics cannot be considered valid and therefore the survey results are not presented in this report. ANNEX IX: QUESTIONS AND CONCERNS – EXPRESSED BY THE EDC August 27, 2013 Response to Evaluators for BiH Excellence in Innovation (EI) Project Questions and Concerns The EI Project Evaluation Team Dear Davorin – Thank you for the opportunity to meet with you and Jasmina, as well as with Elma on Friday, August 23, 2013, to discuss the EI External Evaluation. The EI Project would like to emphasize the following points to address the questions you asked us to consider as well as concerns we have about the focus of this Evaluation, based on Friday’s conversation. In short, we believe that the Evaluation as conducted so far is likely not accurately taking into account the feasibility of the EI project design, the significant impact and results to date, or the sustainability of the e-BIZ Centers now and in the future. As of now, we have seven primary issues of concern relating to the Evaluation: 1) The EI Project Design is based on a proven model and this is not being taken into account 2) The definition being used by the Evaluators related to sustainability does not align with the definition of sustainability that the EI Project has used since its inception 3) The accounting method that the Evaluators are using to measure sustainability is not realistic 4) The financial statements provided by the e-BIZ Centers indicate they are on a path to sustainability and these statements appear not to be adequately regarded by the Evaluators 5) The Evaluators’ definition and application of the term ‘Client’ is not appropriate for the EI’s developmental context 6) The focus on the Centers’ stand-alone capability paints a misleading picture of the EI Project’s success and the prospects for long-term sustainability of the Centers 88 7) Exploring BH needs to be recognized for its successes as well as the challenges We believe is it extremely important for you and USAID to take these issues into account so as to develop a report that provides a realistic picture of the EI Project’s impact and that assists USAID in making effective program decisions going forward for similar types of projects (which we understand per the 2011 USAID Evaluation to be a key goal for conducting this type of evaluation). Below please find explanations and details related to these issues. Issue 1: EI Design is Based on a Proven Model The Evaluation does not appear to recognize the proven success of the model on which the EI project design is based and the evidence supporting the prospects for long-term sustainability of the e-BIZ Centers based on this model. Model does not have anything to do with project management and management practice exercised in Bosnia. Even when it is proven that interference of actors from Macedonia into the EI project in Bosnia and Herzegovina jeopardized project implementation? Even when the entire team that was implementing project was different? Even when half of that team was fired? Even if clients interviewed were not able to confirm that competitiveness of their businesses was increased due to assistance provided by the project? T Recommendation: We respectfully ask the Evaluators and USAID to consider that the EI design was based on the same design as that of the USAID e-BIZ Macedonia project, which developed e-BIZ Centers that continue to provide value-added services to SMEs long after project end (2007) and has been cited by USAID and World Bank for its success (for example, please see “Information Communication Technology as a Catalyst to Enterprise Competitiveness,” Business Growth Initiative, February 2010.) We have reliable evidence to show that at least four of the six e-BIZ Centers in Macedonia are continuing to operate and work effectively with their SME clients (two of the Centers are independent companies; two Centers, owned by universities, have joined other service centers to provide broader sets of services). Please refer to the USAID e-BIZ project on the USAID/Macedonia website at http://macedonia.usaid.gov/en/sectors/economic/ebiz.html The Centers developed by the e-BIZ Project, with the same design, approach and implementer as the EI Project, provides the Evaluators and USAID a reliable picture of where the e-BIZ Centers in Bosnia will likely be in six years, and the underlying quality of the EI design. We therefore respectfully suggest that before the draft Final Report is issued, USAID/BiH and/or the Evaluators contact USAID/Macedonia to find out more about these e-BIZ Centers. EDC would also be happy to help in any way possible to put the Evaluators and USAID in touch with the e-BIZ Centers. We would also be happy to provide project documents that demonstrate the similarities in design between the two projects, e.g., please take note of the Program Description (Attachment B) / Project Summary of the EI Cooperative Agreement which specifically mentions the Macedonia e-BIZ Model as the foundation for the EI Project Design. 89 Issue 2: Definition used for Sustainability Does Not Align with the Definition used by the EI Project since Project Inception We are unclear why the Evaluation’s approach in terms of measuring the e-BIZ Centers along the definition of financial sustainability as used in the Evaluation Scope of Work departs from the definition of sustainability used by the EI Project since its inception. Recommendation: We respectfully suggest that the Evaluators and USAID use the standard of sustainability contained in the EI Life of Project Implementation Plan / M&E section and the Annual Work Plans or at least explain more fully why a different definition of sustainability is being used. This is the standard of sustainability that the EI project has been using for the past five years. It is also a reliable indicator in that it is the same as the one that was used for the e-BIZ Macedonia project, which established Centers that have proven to be sustainable. We would be happy to provide the PMP for the e-BIZ Macedonia project to substantiate this. As we show below, the e-BIZ Centers have been successful not only in achieving this sustainability target, but they have met or exceeded all of their objectives in terms of generating in-kind investment, serving SME clients, and perhaps most importantly, catalyzing innovation in their respective sectors. To suddenly adopt a new definition of sustainability in 2013 is both to unfairly “move the goal posts” at project end and to obscure the developmental impact that the e-BIZ Centers have made. Furthermore, it misses entirely the fact that the sizeable in-kind investment made by the parent companies alone is a strong indicator that most of the e-BIZ Centers will continue to operate successfully going forward in their own right. Issue 3: Accounting Method Used to Determine e-BIZ Center Sustainability is not Realistic The Evaluation is not judging the e-BIZ Centers financial health and statements in a realistic manner. In particular, the Evaluators appear to argue that the financial statements and accounting methods used by the Centers are not valid because the Centers have not recorded owners’ in-kind contributions as expenses, and because the Centers did not book equipment contributed by the project. The reliance on this type of accounting method obscures and inaccurately portrays the reality of how start-up firms in many parts of the world and firms in Bosnia generally function, and as we understand it, what is permissible under Bosnian law. Reliance on this type of accounting methodology also paints an inaccurate picture of the Centers’ prospects for long-term sustainability, particularly based on the definition of sustainability in the LPIP that the EI Project has been using for the past five years. Recommendation: The Evaluators should take note of the fact that it is common business practice in Bosnia to not record owner in-kind contributions of labor as expenses, and in fact that as we understand it, this practice complies with Bosnian law. It might be booked as an expense if the owner expects to be paid for the in-kind labor, but this is demonstrably not the case in the context of the e￾BIZ Centers. Another scenario is that if the owner of the firm contributes in-kind labor, then it should be registered as an investment into the company, not as an expense. If the in-kind contribution is registered as an investment, then the owner’s work would be entered as an investment into the company (as intangible fixed assets) and to be registered as capital. However, this would require registration with the court, which defines the amount of the investment. Apparently the court procedure in Bosnia last several months and it is expensive to register the investment (minimum 500 KM). As such, there is generally no business rationale for supporting the registration of the owner’s in￾90 kind contribution as an investment, and not registering this is from our understanding within the bounds of Bosnian law. Second, the Evaluators should note that, for rational business reasons, the Centers included investments and liabilities in their financial statements based on the types of contributions received from the EI Project. For example, Transwork included in its financial statement the contribution of equipment made by its parent company Satwork. They did not include provision of office space and in-kind work by the owner when not connected to service development. On the other hand, ARTECO and TTC are not including in-kind labor, rent, and in-kind equipment in their financial statements. Services rendered by TTC to Kismet and ARTECO to Artisan were included in the financial statements. There is a very important reason for this difference of recording contributions made by the EI Project that relates to the time value of money. For each milestone that Transwork completed, Satwork received the funds from EI soon thereafter and was able to pass them onto Transwork without delay. On the other hand, the bulk of EI’s contributions to ARTECO and TTC focused on the initial provision of equipment. Title to that equipment resides with EDC until the project is over. That equipment can only be transferred to the Centers after the project is over. As such, the parent company (Artisan and Kismet) would logically decide not to book this equipment as assets in their financial statements. It is our understanding that the USAID-funded Business Finance Project utilized a similar method of accounting and this was deemed satisfactory. Issue 4: Financial Statements Indicate the Centers are on a Path to Sustainability If the Evaluators and USAID review the income statements provided by the e-BIZ Centers, they will note that the Centers are showing satisfactory financial health especially in light of their business plan projections. In short, the income statements for Transwork and TTC show the following (in KM): Name of Firm Total Planned for the first five quarters in terms of profit/loss after taxes Actual for the first five quarters in terms of profit/loss after taxes (2012 and 6 months of 2013) Actual 2013 in terms of profit/loss after taxes (first six months) Transwork -143,915 KM -89,878 KM 9,544 KM TTC -230,752 KM -50,227 KM 1,661 KM The income statements for Exploring BH and ARTECO show the following (in KM): Name of Firm Total Planned for the first two years in terms of profit/loss after taxes Actual for the first two years in terms of profit/loss after taxes (2010 and 2011) Actual 2012 in terms of profit/loss after taxes ARTECO -93,830 KM 12,069 KM 20,821 KM Exploring BH 20,596 KM -12,894 KM 302 KM 91 Against the backdrop of these financial figures, the Evaluators are urged to take note that it takes time for a start-up to achieve profitability, especially given the recession and the difficult business environment of BiH. Based on our experience in Macedonia, and as statistics demonstrate in other countries (e.g. India, Philippines, US), it takes at least 4-5 years for most start-ups to break even. As an aside, we suggest that the Evaluators consider whether the Centers’ ability to meet their business plan financial targets is the most important determination of whether the Centers are on a path to sustainability. As the 2013 Harvard Business Review article “Why the Lean Startup Changes Everything” notes, “business plans rarely survive first contact with customers.” The article confirms what experienced business people know: startups are successful when they respond to market demand, not when they adhere to an initial business plan. Issue 5: Definition and Use of the Term ‘Client’ is Inaccurate The Evaluators’ focus on SME ‘clients’ as paying customers in terms of cash payments misses the point and business realities both of the start-up context and of the economic picture in Bosnia, and more broadly it is an incorrect use of the term client as defined in standard development context. Recommendation: We recommend that the Evaluators adopt a definition of the word ‘client’ that accords with standard and correct usage of the term. For example, the Oxford Dictionary defines client as a “person or organization using the services of a lawyer or other professional person or company.” Payment in cash is not one of the characteristics of being a client. And it should be noted that nowhere in the EI Project documents vis a vis USAID does it insist that a client has to be one that pays cash for services. As such, there is ample evidence that the EI Project has achieved its goals in terms of the number of SME clients served – and that this level of service, when combined with other factors such as owner in￾kind investment, strongly demonstrates the prospects for sustainability and improved competitiveness in the respective sectors. For example: • Exploring BH: A client would include any SME that got sales via Exploring BH and also those SMEs that have signed contracts and have received assistance from Exploring BH to create or improve their online presentation. All of these SMEs have received marketing service via their presence at the Exploring BH web portal even if few of them hadn't had a sale in the process. • Transwork: A client would include any SME that signed a contract with Transwork and paid monthly subscription for services; it also would include SMEs that use Transwork's free online freight exchange daily and get the benefit of finding the most suitable / affordable transport option for their business. • TTC: A client would include every SME that received a service from TTC and was paid or invoiced for that service. SME clients also include those that received a service but agreed on a compensation (barter) for this service with TTC or its parent firm. Considering the nascent stage of this Center, the EI accepted that we would also recognize as clients those (few) SMEs that have received manufacturing support services as part of promotional activities i.e. free promotions. The rationale is clear – it requires time and effort for TTC to educate SMEs about the advantages of using the new technology but more importantly even the provision of the free services in limited instances is helping the Project achieve our developmental objectives - dissemination of technology and increasing SME competitiveness. 92 • ARTECO’s situation was similar to that of TTC, with however a larger percentage of clients paying for services, as the Center has been in existence longer. Issue 6: Focus on Stand-Alone Capability of the Centers is Misleading The Evaluators ask whether the e-BIZ Centers should be measured based on the level of technical and management capacity within their own (affiliate-type) stand-alone entities as opposed to that of their parent firms. Recommendation: Upon further consideration, EI requests that the Evaluators rethink whether it is relevant to measure the e-BIZ Centers solely or primarily in terms of their stand-alone capability. The Evaluators and USAID are respectfully reminded that the objective of this project was to create service centers that provide value-added services to SMEs during and after the project. That is the only issue of concern. If someone (owner, manager, whatever) in the parent company also provides management guidance to the e-BIZ Center, it has absolutely no bearing on the project objectives or achievements unless the manager/owner will not be working with the e-BIZ Center in the future and continue the commitment to support it. For example, it makes no difference if Transwork works closely with Satwork as it provides a set of comprehensive services – or indeed whether Transwork takes advantage of building on Satworks’ customer base. The only issue is whether Transwork will continue to provide value-added services to SMEs after the project ends and whether Satwork will continue to provide collaborative support as the two companies serve clients. We have ample evidence that the owners are committed to supporting the Centers after project end. Moreover, we urge the Evaluators and USAID to consider each e-BIZ Center in its own context in order to determine the specific approach of each Center and its parent company in achieving this stand￾alone capability. In brief, we believe that TTC has achieved this stand-alone capability, as has ARTECO, in addition to the fact that the owners have expressed commitment to supporting these Centers in the future. We also believe that Transwork achieved this capability as well, while collaborating with the parent firm Satwork. Please note that their business plan is stating that ”Satwork will be technology development arm of Transwork”. Following that plan, Transwork should be viewed as the vehicle for technology dissemination while it should be acceptable that most product R&D is outsourced to the closely integrated parent firm at present. An alternative scenario would create an unnecessary burden to this startup with large overhead cost for keeping full time engineers on staff. Related to this point, we are concerned that the Evaluators seem to be skeptical of the fact that many of the e-BIZ Centers remain tied to their parent companies in terms of services offered. The Evaluators and USAID should consider when it makes good developmental impact as well as business sense for each Center to remain connected with the parent company in terms of services offered. We kindly suggest that, for example, in cases of manufacturing sector Centers (ARTECO & TTC) evaluators consider an “industry or cluster leader” potential of the parent firm within BiH context. In summary, under the evaluators’ concept that the Centers should be completely independent, the entire idea of businesses collaborating to strengthen both companies’ competitiveness would be lost. In fact, the evaluators are reflecting a failed business perspective that EI has been trying to change since the beginning of the project, i.e., BiH companies must learn to collaborate and rely on one another or they will never be competitive in the global market. 93 Issue 7: Exploring BH Needs to be Recognized for their Successes and Potential for Sustainability as Well as Challenges As noted in Friday’s conversation, we agree Exploring BH does not appear currently to have the capacity to likely sustain itself as a successful entity in the long term. We also discussed, as a lesson learned, that the outcome could have been different by choosing a strong private sector partner as opposed to an NGO, and EDC takes responsibility for the partnership between Fondacija and Exploring Macedonia that has not turned out well. However, we believe there is credible evidence that it is premature to dismiss Exploring BH as a failed entity. Furthermore, the Evaluation would do the project and USAID’s intervention a disservice by focusing only on mistakes or the challenges in terms of capacity. Recommendation: At a minimum, to evaluate Exploring BH’s prospects for long-term sustainability, the Evaluation Report should take detailed note of the business strategies that Exploring BH intends to implement and is already implementing. The reduction in Exploring BH’s expenses is a perfectly reasonable and viable business response for a company that is re-focusing on inbound tourism with niche clients. Moreover, the likelihood that Exploring BH will achieve sustainability is reinforced by their redesigned website, recent strong marketing efforts made by the Exploring BH Executive Director to new potential markets (e.g. Russia), and her commitment to invest a significant amount of personal funds in the business. In addition, the Evaluators and USAID should note the success that Exploring BH has had in promoting SME clients on its web portal, and in differentiating itself from other tourism sales intermediaries for its reliance on in-bound tours. In 2008 most BH hotels had weak or non-existent online presence. Exploring BH developed presentation materials for a considerable number of SMEs and posted them in the virtual “storefront” for the first time ever, and managed to sell their offerings using limited marketing resources and EI project support in linking with foreign tour operators. We believe that to move any firm from a complete lack of online presence to first time online presence represents a clear-cut case of improved competitiveness. Furthermore, Exploring BH bank statements from 2009-12 period show that Exploring BH collected commissions for marketing and sales services from more than 160 BiH tourism SMEs. Please note the additional response below related to SME Customer Surveys. SME Customer Surveys We acknowledge that formal Customer Surveys were not conducted following the initial Year 1 surveys for ARTECO and Exploring BH. It is important to note however that the Centers have continued to survey clients on an informal basis and they have informed EI about responses from the field so the feedback was incorporated in work plans. Moreover, EI Project staff regularly met with SME clients and prospective clients. TTC and Transwork also conducted customer surveys on time (i.e. following their first full year of operation). During Friday’s conversation, the evaluators mentioned that EDC should have conducted the customer surveys. This may be a matter of debate; however, in order to prepare the e-BIZ Centers for self-sustainability, we provided capacity-building support to the Centers to obtain customer feedback, and believe it was preferable for the Centers to conduct the activities required to obtain this feedback, as the Centers are the primary interface with the SME clients. 94 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 95 ANNEX 1 EBIZ CENTER CLIENTS - TYPE OF CLIENTS & SERVICES 5.3. Pregledi korisnika Centra - UGOVOR/Jednokratne usluge rbr. 1 2 3 4 5 6 7 Grace Travel d.o.o. Međugorije 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 M&S Wood d.o.o. Sarajevo 39 ZR Sportex d.o.o. SHOPS AND NGOs / ASSOCIATIONS ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE Pobjeda d.o.o. Wood panels d.o.o. Rez d.o.o. Rukotvorine d.o.o. Konjic Rattan Sedia d.o.o. Brčko ONE-TIME SERVICE A R T E C O d.o.o. Medakovo, Tešanj TYPE OF BUSINESS RELATION CONTRACT ONE-TIME SERVICE ONE-TIME SERVICE Artisan d.o.o. Index d.o.o. Gračanica Krivaja -Tvornica mont.kuća Bosnian Beech Line d.o.o. Elgrad d.o.o. Teslić Neimari d.o.o. Termag d.o.o. Winter d.o.o. ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE Pero d.o.o. Zenica Modum d.o.o. Tešanj Economik d.o.o. Vitez Hercing d.o.o. Sarajevo Osnap d.o.o. Sarajevo Sigma-prom d.o.o. Tešanj Zovko oil d.o.o. Žepče Selming d.o.o. Sarajevo Krivaja 1884 d.o.o. Zavidovići Jadrina d.o.o. Gračanica MTTC d.o.o. Gračanica Hifa oil d.o.o. Tešanj Maxy d.o.o. Orašje Wischt d.o.o. Doboj Kraning d.o.o. Puračić Koteks d.o.o. Tešanj Firma d.o.o. Sarajevo Ćemanenergoprom d.o.o. Laki laktaši Sejdićkomerc d.o.o. Swis Design Team d.o.o. Standard Furniture Factory Bosna Commerce d.o.o. CLINET 40 41 42 43 44 Kovan SZR vl.Karalić Said INDIVIDUALS 45 Muharem Ahmić Tešanj 46 Aleksandar Lazarević Pale 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Ibrahimović Junuz 61 Rakanović Momir 62 Arslanović Muhedin 63 Zukić Esad FOREIGN COMPANIES / CLIENTS 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 Scaba APS Danska 81 Streng Plastic AG 82 Bieler Trading Ltd 83 Hari Ćevabdžinica N.Travnik Studio zec Sarajevo Vrhbosanska Nadbiskupija Muris Hasić Mutapčić Edin Samostan Karmel Ćostović A st.radnja Medakovo Joacim SCHidt Muhamed ČauševićTešanj Bulajić Vanja Zenica Muratović salko Sarajevo Brkić Salih Tešanj Stanišavljević Milovan Kovačić Neven Zagreb Thomas Reugg Sarajevo Arslanagić Vedad Sarajevo Himzić Izudin Tešanj Frank Ziprich Miliša Valerija Zagreb Stipić grupa d.o.o. Hrvatska SY.GA.KO. GmbH Njemačka Nika Group d.o.o. Beograd Lavina BGD d.o.o. Beograd Ćorić Nedžad Njemačka Himzić Nusret Julie Fjeldstad Danska Skanno Oy Finska Redesign d.o.o. Hrvatska Lush d.o.o. Zagreb ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE Hrvoje Poljak zagreb Decorus Furniture Indonesia Wood Point Milintijević Branimir Mijo Jeličić Hrvatska Vojnić Mirela Zagreb ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE 84 Casa Furniture ltd 85 Lorca Design Interiors 86 Dedukić Muharem 87 88 Hierve Muebleria Ltd Anel Ćerimagić ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ONE-TIME SERVICE ANNEX 2 - PREGLED PO KORISNICA I UČEŠĆE U UKUPNOM PRIH KORISNIK USLUGA VRSTA UGOVORA REALIZOVANO - 2012. GOD. Procent REALIZOVANO - 2013. GOD. Procent Vokal Productions doo Dugorocni 6703.6 9.06% 21128.2 22.17% Koza Komerc doo Dugorocni 4747 6.42% 320 0.34% Olimp dd Dugorocni 2423 3.28% 0 0.00% Dub-Tex doo Dugorocni 2426.6 3.28% 1550 1.63% Elite-Fashion doo Dugorocni 572.65 0.77% 0 0.00% Deluxe-Fashion doo U procesu 1554.3 2.10% 386.1 0.41% Manico I doo U procesu 0.00% 2500 2.62% Rentex-COM doo Dugorocni 1092.13 1.48% 1863.93 1.96% Ema-Company doo Jednokratni 687.7 0.93% 0 0.00% Premier doo Dugorocni 603 0.82% 788.8 0.83% Kismet doo Dugorocni 51988.51 70.28% 65345.25 68.56% Konfekcija FBM doo Jednokratni 857.1 1.16% 0 0.00% Contessa doo U procesu 21.2 0.03% 0 0.00% Bontex doo U procesu 300 0.41% 0 0.00% Goncin doo U procesu 0 0.00% 240 0.25% Granoff doo Dugorocni 0 0.00% 629.41 0.66% Elchy-Tex doo Dugorocni 0 0.00% 213.25 0.22% Sana Linea ad U procesu 0 0.00% 260.6 0.27% Zenko doo Dugorocni 0 0.00% 90.44 0.09% LENA TEX doo U procesu 0 0.00% 0 0.00% NBM doo U procesu 0 0.00% 0 0.00% SENDY JEANS doo U procesu 0 0.00% 0 0.00% SLOVENKA doo U procesu 0 0.00% 0 0.00% HERTEX doo U procesu 0 0.00% 0 0.00% UNIKOM PROMET doo U procesu 0 0.00% 0 0.00% KOMBAX doo U procesu 0 0.00% 0 0.00% MRAZ doo U procesu 0 0.00% 0 0.00% MILE TEX doo U procesu 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% 0 0.00% UKUPNO 73,976.79 100% 95,315.98 100% HODU (LIST OF CLIENTS and THEIR SHARE IN TOTAL REVENUES one-time one-time one-time one-time 51988.51 21128.2 6703.6 65345.25 one-time 58692.11 86473.45 one-time one-time 0.793385 0.907229 one-time one-time one-time one-time one-time S) 5.2. Pregledi korisnika Centra - Učešće u realizaciji 100% 452522 100% 531210 100% 522350 100% 1 4.00 9273 29.96 108193 24.8 116064 22.41 2 0.31 720 3 0.31 720 4 1.77 4090 5 0.62 1445 6 24.02 55649 0.37 1326 7 Grace Travel d.o.o. Međugorije 2.88 6681 8 2.93 10585 0.73 9 65.42 151548 16.16 58361 10 0.06 200 11 0.11 400 12 4.58 16530 0.64 3000 13 0.09 334 0.27 1263 14 0.09 334 15 0.03 120 16 13.65 49299 17 0.37 1338 18 2.05 7406 19 0.31 1120 0.22 1037 0.37 20 0.12 418 21 0.04 131 22 1.26 4539 23 1.55 5592 24 0.82 2958 25 1.33 4800 26 1.6 7488 5.06 27 0.31 1453 28 0.09 405 29 0.98 4561 0.46 30 0.5 2340 Firma d.o.o. Sarajevo Ćemanenergoprom d.o.o. Laki laktaši Sejdićkomerc d.o.o. MTTC d.o.o. Gračanica Koteks d.o.o. Tešanj Modum d.o.o. Tešanj Economik d.o.o. Vitez Hercing d.o.o. Sarajevo Osnap d.o.o. Sarajevo Rukotvorine d.o.o. Konjic Rattan Sedia d.o.o. Brčko Sigma-prom d.o.o. Tešanj Zovko oil d.o.o. Žepče Selming d.o.o. Sarajevo Krivaja 1884 d.o.o. Zavidovići Krivaja -Tvornica mont.kuća Bosnian Beech Line d.o.o. Elgrad d.o.o. Teslić Jadrina d.o.o. Gračanica Pero d.o.o. Zenica A R T E C O d.o.o. Medakovo, Tešanj rbr. 2009 2010 ANNEX 3 ARTECO CLIENTS SHARE IN T Naziv korisnika Artisan d.o.o. Index d.o.o. Gračanica Termag d.o.o. Neimari d.o.o. Winter d.o.o. Pobjeda d.o.o. Wood panels d.o.o. Rez d.o.o. 2011 2 31 2.73 12776 32 4.42 20645 33 0.40 34 0.05 35 6.32 36 0.67 37 0.21 38 M&S Wood d.o.o. Sarajevo 39 ZR Sportex d.o.o. 220853 165791 54968 SHOPS / ASSOCIATIONS 40 0.99 3568 41 0.02 70 42 6.36 29743 43 5.29 24746 44 2.19 7908 Kovan SZR vl.Karalić Said INDIVIDUALS 45 Muharem Ahmić Tešanj 0.07 152 46 Aleksandar Lazarević Pale 0.35 800 47 0.06 200 48 0.03 100 49 0.03 100 50 0.20 705 51 0.71 2547 0.36 1688 52 4.15 19417 53 0.03 140 54 0.47 2213 55 0.09 410 56 5.9 27574 57 3.98 18620 58 0.01 53 59 0.06 265 60 Ibrahimović Junuz 0.33 61 Rakanović Momir 62 Arslanović Muhedin Muratović salko Sarajevo Brkić Salih Tešanj Himzić Nusret Arslanagić Vedad Sarajevo Himzić Izudin Tešanj Joacim SCHidt Muhamed ČauševićTešanj Bulajić Vanja Zenica Mutapčić Edin Stanišavljević Milovan Kovačić Neven Zagreb Thomas Reugg Sarajevo Hifa oil d.o.o. Tešanj Maxy d.o.o. Orašje Wischt d.o.o. Doboj Samostan Karmel Ćostović A st.radnja Medakovo Vrhbosanska Nadbiskupija Muris Hasić Hari Ćevabdžinica N.Travnik Kraning d.o.o. Puračić Swis Design Team d.o.o. Standard Furniture Factory Bosna Commerce d.o.o. Studio zec Sarajevo 63 Zukić Esad FOREIGN COMPANIES / CLIENTS 64 0.26 591 65 5.73 20704 66 0.18 645 67 9.82 35466 68 5.38 19422 69 1.16 5428 70 5.9 27565 71 7.16 33459 72 0.38 1797 73 5.72 26716 74 4.63 21626 75 7.56 35347 76 2.03 9479 77 1.16 5409 78 0.51 2407 79 0.48 2248 80 Scaba APS Danska 0.69 81 Streng Plastic AG 15.14 82 Bieler Trading Ltd 46.75 83 0.41 84 Casa Furniture ltd 85 Lorca Design Interiors 86 Dedukić Muharem 87 88 Miliša Valerija Zagreb Hrvoje Poljak zagreb Decorus Furniture Indonesia Julie Fjeldstad Danska Mijo Jeličić Hrvatska Vojnić Mirela Zagreb Frank Ziprich SY.GA.KO. GmbH Njemačka Nika Group d.o.o. Beograd Lavina BGD d.o.o. Beograd Ćorić Nedžad Njemačka Milintijević Branimir Stipić grupa d.o.o. Hrvatska Skanno Oy Finska Redesign d.o.o. Hrvatska Lush d.o.o. Zagreb Wood Point Hierve Muebleria Ltd Anel Ćerimagić 191235 100% 93443 100% 1790760 37496 42.05 35128 23.28 306154 Arteco's Revenues 2009 2010 0.05 720 one-time Artisan 9273 108193 0.05 720 one-time Third Party SMEs Sales 220853 165971 0.31 4090 one-time Total (KM) 230126 274164 0.11 1445 one-time Third Party SMEs Sales (%) 95.97 60.54 4.33 56975 0.51 6681 one-time 1228 8.07 6739 1.41 18552 15.96 209909 0.02 200 one-time 1 0.03 400 one-time 2 1.48 19530 0.12 1597 0.03 334 one-time 3 0.01 120 one-time 4 3.75 49299 one-time 0.10 1338 one-time 0.56 7406 one-time 614 0.21 2771 0.03 418 one-time 5 0.01 131 one-time 6 0.35 4539 one-time 0.43 5592 one-time 0.22 2958 one-time 0.36 4800 one-time 8471 1.21 15959 0.11 1453 one-time 0.03 405 one-time 7 776 0.41 5337 0.18 2340 one-time TOTAL REVENUES 2012 I-VII/2013 Ukupno 0.97 12776 one-time 1.57 20645 one-time 667 0.20 164 0.06 831 88 0.01 88 one-time 8 10584 0.80 10584 one-time 1116 0.08 1116 one-time 344 0.03 344 one-time 9 3.43 2867 0.22 2867 one-time 0.16 130 0.01 130 one-time 10 23888 9900 36.14 475400 0.00 0.00 0 0.00 0.27 3568 0.04 35 0.01 105 2.26 29743 1.88 24746 0.60 7908 7.85 6558 0.50 6558 0.00 0 0.01 152 0.06 800 0.02 200 0.01 100 0.01 100 0.05 705 0.32 4235 1.48 19417 0.01 140 0.17 2213 0.03 410 2.10 27574 1.42 18620 0.00 53 0.02 265 546 0.04 546 0.07 60 0.00 60 0.07 55 0.00 55 6.10 5100 0.39 5100 0.00 0 0.04 591 1.57 20704 0.05 645 2.70 35466 1.48 19422 0.41 5428 2.10 27565 2.54 33459 0.14 1797 2.03 26716 1.64 21626 2.69 35347 0.72 9479 0.41 5409 0.18 2407 0.17 2248 1154 0.09 1154 25342 1.93 25342 78233 5.95 78233 688 0.05 688 16.64 13904 1.06 13904 8.28 6916 0.53 6916 0.48 400 0.03 400 6.06 5065 0.39 5065 0.51 422 0.03 422 2011 2012 7 months 2013 116064 37496 35128 54968 23888 9900 171032 61384 45028 32.14 38.92 21.99 ANNEX 4 – ARTECO LIST OF CLIENTS and TYPE OF SERVICES 5.1. PREGLED KORISNIKA CENTRA Red br. Naziv firme Vrsta usluge 1. Muharem Ahmić Jelah bb Usluga sušenja drveta 2. „Artisan „ d.o.o Medakovo Tešanj Usluga sušenja drveta 3. „INDEX“ d.o.o Gračanica edukacija 4. Krivaja-Tvornica montažnih kuća d.o.o Zavidovići Edukacija 5. Bosnian Beech line d.o.o Sarajevo Usluga sušenja drveta 6. Aleksandar Lazarević Pale bb. Usluga sušenja drveta 7. Elgrad d.o.o Teslić Usluga sušenja drveta 8. Simon Irgens Danska Izrada teh.dokumentacije 9. „Grace travel“d.o.o Međugorje Kooperacija i proizvodnja 10. REZ RDA Zenica Edukacija 10a „NEIMARI“ d.o.o Ilidža Sarajevo Kooperacija i proizvodnja 10b „WINTER“ d.o.o Travnik Kooperacija i proizvodnja 11. Admir Ibrahimkadić Tešanj Edukacija 12. Admir Ireiz Tešanj Edukacija 13. Ajdin Ahmetović Tešanj Edukacija 14. Asmir Džinalić Tešanj Edukacija 15. Eldin Hojkurić Tešanj Edukacija 16. Elvedin Šabanović Tešanj Edukacija 17. Harun Halilbegović Tešanj Edukacija 18. Maid Jabandžić Tešanj Edukacija 19. Mensur Krdžalić Tešanj Edukacija 20. Midhat Jareb Bugojno Edukacija 21. Midhat Škapur Tešanj Edukacija 22. Muhamed Hasičić Tešanj Edukacija 23. Nenad Čeki Usora Edukacija 24. Nermin Alkaz Tešanj Edukacija 25. Salih Dubravac Tešanj Edukacija 26. Salih Šabanović Tešanj Edukacija 27. Seid Dervišić Zavidovići Edukacija 28. Semir Aljić Tešanj Edukacija 29. Senad Pilav Tešanj edukacija 30. Fahrija Tihak Bugojno edukacija 31. Vlado Dlakić Usora edukacija 34. POBJEDA d.o.o Bukva Tešanj edukacija 35. „WOOD PANELS“ d.o.o Gračanica Edukacija CAM 36. Muris Hasić Edukacija CAM 37. Edin Mutapčić Sarajevo Edukacija CAM 38. Ado Avdagić Tuzla Edukacija CAM 39. REZ d.o.o Zenica edukacija 40. Rukotvorine d.o.o Konjic Usluga na CNC 41. RATTAN SEDIA d.o.o Brčko Usluga na CNC 42. TERMAG d.o.o Rogatica Usluga sušenja drveta 43. SIGMA-PROM d.o.o Jelah Tešanj Izrada teh.dokum.i usluga na CNC 44. „SAMOSTAN Karmel Sv.Ilije“Prisoje Usluga na CNC 45. REDESING d.o.o Zagreb Izrada prototipa 46. Skanno Oy Finland Izrada prototipa 47. SELMING d.o.o Sarajevo Kooperacija i proizvodnja 48. Krivaja 1884 zavidovići Kooperacija i proizvodnja 49. JADRINA d.o.o Gračanica Usluga na CNC 50. ZOVKO OIL d.o.o Žepče Kooperacija i proizvodnja 51. SZR ĆOSTOVIĆ A.Medakovo Tešanj Usluga na CNC 52. PERO d.o.o Zenica Usluga na CNC 53. MODUM d.o.o Jelah Tešanj Usluga na CNC 54. Milovan Stanišljević Kotor Varoš Usluga na CNC 55. Neven Kovačić Zagreb Izrada prototipa 56. ECONOMIC d.o.o Vitez Kooperacija i proizvodnja 57. HERCING d.o.o Sarajevo Kooperacija i proizvodnja 58. „LUSH“ d.o.o Zagreb Hrvatska Kooperacija i proizvodnja 59. OSNAP d.o.o Sarajevo Kooperacija i proizvodnja 60. KOTEKS d.o.o Tešanj Kooperacija i proizvodnja 61. Svećenički dom Vrhbosanske nadbiskupije Sarajevo Kooperacija i proizvodnja 62. STIPIĆ GRUPA d.o.o Lučko,Hrvatska Kooperacija i proizvodnja 63. THOMAS RUEGG Pirusa 1 Sarajevo Kooperacija i proizvodnja 64. Arslanagić Vedad Sarajevo Kooperacija i proizvodnja 65. SY.GA.KO Njemačka Kooperacija i proizvodnja 66. „FIRMA“ Sarajevo Izrada teh.dokumentacije 67. Himzić Izudin Tešanj Usluga sušenja drveta 68. Ćevapčinica „HARI“ Travnik Kooperacija i proizvodnja 69. „Ćeman Energoprom „ Jelah Usluga sušenja drveta 70. Muhamed Čaušević Tešanj Usluga sušenja drveta 71. Joachim Smidht Sarajevo Kooperacija i proizvodnja 72. „LAVINA BGD“d.o.o Beograd Kooperacija i proizvodnja 73. Ćorić Nedžad Rheinfelden Kooperacija i Njemačka proizvodnja 74. Studio ZEC sarajevo Kooperacija i proizv. 75. Mijo Jeličić Split Hrvatska Kooperacija i proizvodnja 76. NIKA GROUP d.o.o Beograd Kooperacija i proizvodnja 77. Vojnić Mirela Zagreb Hrvatska Kooperacija i proizvodnja 78. LAKI d.o.o Laktaši Izrada teh.dokumentacije 79. Salih Brkić Tešanj Usluga sušenja drveta 80. Nusret Himzić Jelah Usluga sušenja drva 81. SEJDIĆ COMMERCE d.o.o Odžak Izrada teh.dokumentacije 82. HIFA-OIL d.o.o Tešanj Kooperacija i proizv. 83. Branimir Melentijević Beograd Kooperacija i proizv. 84. MTTC doo Gračanica Izrada teh.dokumentacije 85. Salko Muratović Sarajevo Kooperacija i proizv. 86. Frank Zipprich Njemačka Kooperacija i proizv. 87. Maxi d.o.o Orašje Kooperacija i proizv. 88. Hrvoje Poljak Zagreb Kooperacija i proizv. 88. Valerija Miliša Zagreb Kooperacija i proizv. 90. Bulajić Vanja Zenica Kooperacija i proizv. 91. Decorus furniture INDONESIA Izrada teh.doumentacije 92. Bieler Trading Ltd Cyprus Kooperacija i proizv. i izrada teh.dok. 93. SCABA ApS Danska Izrada teh.dokumentacije 94. WISCHT Doboj Usluga na CNC 95. HIERVE-DISENERIA s.a Mexico Izrada prototipa 96. Streng Plastic AG Švicarska Kooperacija i proizv. 97. KRANING d.o.o.Gračanica Usluga na CNC 98. Swiss Desing Team d.o.o Laktaši Usluga na CNC 99. Junuz Ibrahimović Kosova Usluga sušenja drveta 100. STANDARD FURNITURE FACTORY Sarajevo Izrada prototipa 101. WOODPOINT Švedska Izrada prototipa 102. Bosna commerce doo Jelah Usluga sušenja drveta 103. CASE FURNITURE Ltd London Kooperacija i proizv. 104. „KOVAN“Z.R.Gračanica Usluga na CNC 105. M&S WOOD dd Sarajevo Usluga sušenja drveta 106. LORCA DESING INTERIORS Beograd Kooperacija i proizvodnja 107. Rakanović Momir Doboj Usluga na CNC 108. ZR SPORTEX Ključ Izrada prototipa 109. Muharem Dedukić Švicarska Kooperacija i proizv. 110. Arslanović Muhedin Jelah Usluga na CNC 111. Hierve-Muebleria Ltd London Kooperacija i proizv. 112. Anel Ćerimagić Austrija Izrada teh.dokum. 113. 114. 115. 116. 117. 118. 119. Transwork doo Banja Luka ANNEX 5 Pregled vrsta proizvoda i usluga (TYPE OF SERVICES & THEIR SHARE IN TOTAL REVENUES) RB Vrsta proizvoda/usluge jedinična cijena u KM % učešća u ukupnom prihodu 2012 % učešća u ukupnom prihodu 2013 1 GPS uređaj 205 57.2 36.6 2 Usluga pretplata 20 37.8 52.3 3 Usluga ugradnja 20 3.3 3.5 4 Usluga kontrole potrošnje goriva 3 0 0.05 5 Usluga demontaža uređaja 20 0.04 0.02 6 Usluga premještanje uređaja 20 0.06 0.02 7 Usluga ugradnja senzora na poklopac goriva 70 1.4 7.47 8 Ostale usluge 0.2 0.04 100 100 ANNEX 6 PERFORMANCE EVALUATION OF THE EXCELENCE IN INNOVATION ACTIVITY INTERVIEW MINUTES Interview No. 2 – eBIZ Center Transwork Location: Hotel Bosna , Banja Luka, Republika Srpska, Bosnia and Herzegovina Date: 13.07.2013 Interviewer: Davorin Pavelic Interviewee: Mr. S. M., Director Disclaimer: The Minutes are notes of the Interviewee’s statements and opinions given as replies to the Interviewers’ questions. Recorded statements and opinions do not represent interviewers’ opinion on work and operations of the eBIZ Center. Introduction The Interviewer made a brief introduction about the objective of his visit and purpose and process of evaluation. It was stressed that this visit was informal, aiming at introducing the evaluation team and getting better insight into the activities of the Transwork eBIZ Center. Interviewee was informed that, upon analysis of the existing documentation and interviews with Transwork’s clients, the Evaluation team would arrange for the formal interview the Transwork eBIZ Center. Description of the establishment and functioning of the eBIZ Center Satwork (mother company and owner of Transwork) was established in November 2005. Transwork is oriented on SMEs from transport sector that are international freight forwarders. Their goal is to increase competitiveness of freight forwarding companies when they work in EU. Transwork has four employees (Director, Director of marketing and two sales persons). The company is financially sustainable and it registers continuous growth of revenues. Satwork is developing software on the basis of orders received from Transwork. Due to its work for Transwork company, Satwork company increased the number of IT programmers. T-Com was very expensive provider for GPS tracking services in Europe. Transwork made contract with an Austrian provider MToM Telecom which was cheaper. Transwork has about 600 cards with this Austrian provider. Capacity building There was nobody from whom they could learn what to do when establishing Transwork. There is no expert on that. They trained themselves and they learned from internet and by doing it. Neither of the four employees had any experience from transport sector. Majority of technical assistance was provided to them by Ismir from the EI project using the brainstorming method. Dominique (name to be checked!), a logistic officer from Coca Cola, provided them with the list of companies and basic information on what transport companies need. Boba Ilic (name to be checked!) from Belgrade, who owns electronic freight exchange, showed them how that was functioning. Training provided by Transwork to external clients None Transwork’s Clients In the attempt to acquire clients, they tried with presentations and round tables but it did not prove to be a good approach. The biggest challenge was to sell the first ten contracts. Clients are asking for all services to be paid in installments, which burdens Transwork’s cash flow. They charge 10% for sale of equipment (GPS, PDA devices) and majority of their revenues come from subscriptions. Services They started with development of electronic market place for the transport sector. Transwork electronic freight exchange was officially launched on 9 March 2012. They do not charge their clients for this service. They are using it as a marketing tool. After a detailed market analysis they realized they cannot compete with TimoCom. Wherever they showed up trying to establish business relations, TimoCom was already there. In order to make Transwork Fright Exchange competitive, much more money should be invested in marketing. They could have some chance in the market if markets and fright exchange providers from all countries from former Yugoslavia would join and work together. Main service which provides 60% of all Transwork’s revenues is GPS vehicle monitoring and tracking service. Monthly stability of revenues is the most important thing for their business operations. Now they sell PDA devices with built in GPS which, through internet application, enable communication between the operator and driver. Drivers complained it was complicated to drive and use PDA device (touch screen). Transwork reacted by building in predetermined messages that enable driver to give an answer to the operator by one touch on the screen. Last year they equipped 300 trucks with the PDA devices. They tested device with the most demanding customers. Users of GPS device can upgrade received service by installing fuel consumption control system. However, there are some shortcomings in provision of FCS service. They are mainly related to accuracy issues (3% error) and inability to install them on some types of trucks (i.e. Volvo trucks). Other option they provide to their clients for fuel consumption control (more accurate than FSC) is to physically install sensor into the fuel tank. They also provide service (sensor) that registers each opening of the fuel tank. They developed good tracking service but there is no interest in the market for it. Route optimization is something what they miss in their package of services. That is a complex service to develop and require large investment. In that area they are entering into negotiations about partnership with the Roadnet company. They reduced cost and preformed optimization of transport operations of each client. Their clients are more efficient and more organized. In order to ensure growth of Transwork, they have to expand beyond BH borders - BH market is too small. In future they could approach clients and offer them the following deal: provision of optimization of clients’ expenses and, instead of subscription, payment in the amount of 20% of the total cost reduction generated by Transwork services. Competitive advantages of Transwork They are reducing costs of operations of their clients thus increasing their competitiveness. USAID contribution / added value Maybe everything would be done the same way but not this fast. One of the most important things was the reputation they acquired by having USAID backing up the project. ANNEX 7 PERFORMANCE EVALUATION OF THE EXCELENCE IN INNOVATION ACTIVITY INTERVIEW MINUTES Interview No. 11 – eBIZ Center Transwork Location: Transwork, Banja Luka, Republika Srpska, Bosnia and Herzegovina Date: 19.08.2013 Interviewers: Davorin Pavelic Jasmina Mangafic Interviewees: Mr. S. M., Director Ms. V. R., Marketing Director Disclaimer: The Minutes are notes of the Interviewees’ statements and opinions given as replies to the Interviewers’ questions. Recorded statements and opinions do not represent interviewers’ opinion on work and operations of the eBIZ Center. Introduction The Interviewer made a brief introduction about the objective of this visit and purpose of the evaluation. Description of the establishment and functioning of the eBIZ Center Mr. S. M. stated that Transwork has no IT experts and technical expertise. All technical expertise is with the mother company Satwork. All procurement of equipment (GPS device, PDA device, sensors, etc.) needed for provision of Transwork services is done by Satwork. Satwork is charging 10% commission of value of all sold devices. Satwork share in every subscription to Transwork services is 25%. All maintenance and repairs are done by Satwork. Satwork (Mother Company and owner of Transwork) was established in November 2005, working on software development for large companies. Satwork company had GPS tracking services and its own GPS software system IRS before they established Transwork e-BIZ Center. On Satwork web site Satwork is still offering that service. Transwork is oriented on sale of services to SMEs from transport sector, which are international freight forwarders. Transwork has four employees (Director, Director of marketing and two sales persons). Capacity building There was nobody they could turn to and learn what to do when establishing Transwork. There are no experts in that particular area. They trained themselves and they learned from internet and by doing it. None of the four employees had any experience in transport sector. Majority of technical assistance was provided to them by Ismir from the EI project, using brainstorming method. Dominique (name to be checked!), a logistic officer from Coca Cola, provided them with the list of companies and basic information on what transport companies need. Boba Ilic (name to be checked!) from Belgrade, who owns electronic freight exchange, showed them how was functioning. Training provided by Transwork to external clients None Transwork’s Clients In the attempt to acquire clients, they tried with presentations and round tables but it did not prove to be a good approach. The biggest challenge was to sell the first ten contracts. Clients are asking for all services to be paid in installments, which burdens Transwork’s cash flow. They charge 10% for sale of equipment (GPS, PDA devices) and majority of their revenues come from subscriptions. Services Mr. S.M. is of opinion that Transwork’s Business plan is very optimistic and written by technical personnel, not by economists. The GPS tracking system was already developed by Satwork company. Transwork is using the existing Satwork’s IRS GPS tracking system and they just created front Transwork’s web page, which makes possible registration of clients also trough Transwork page. Based on the assessment of clients’ needs, Transwork worked with Satwork on development of communication system between a driver and logistic operator from a company HQ. In cooperation with Satwork, they initiated development of a Fleet management system (FMS) that is capable of reading and transferring wide range of information about driving parameters (vehicle speed, fuel consumption, engine coolant temperature, etc.). However, not one FMS service was sold to clients. Mr. Micic claims that reason is poor education of clients about information technologies. However, there are some shortcomings in provision of FMS service. They are mainly related to its accuracy (3% error) and inability to install it on some types of trucks (i.e. Volvo trucks). They also developed goods tracking service but so far there has been no interest on the market for it. Route optimization service is an extremely complex service and requires a lot of resources for development. They cannot develop it by themselves. They are trying to negotiate with the Roadnet company from USA that has had its own route optimization system for the last 30 years. When they started working on development of electronic market place for the transport sector, they knew very little about it. Transwork electronic freight exchange was officially launched on 9 March 2012. They do not charge their clients for this service. They are using it as a marketing tool. After a detailed market analysis, they realized they could not compete with TimoCom. Wherever they showed up trying to establish business relations, TimoCom was already there. In order to make Transwork Fright Exchange competitive, much more money should be invested in marketing. They tried to expend their business in the region but everybody uses TimoCom that has thousands and thousands offers daily. Main service which provides 60% of all Transwork’s revenues is GPS vehicle monitoring and tracking service. Monthly stability of revenues is the most important thing for their business operations. After this Joint introductory session, the Evaluation team continues interviews separately. Ms. Jasmina Mangafic interviewed Mr. S. M. and Mr. Davorin Pavelic interviewed Ms. V. R. Interview with Ms. V. R. (Marketing Director) Competitive advantages of Transwork Interviewer: Davorin Pavelic Question 1. To what extent has the EI Activity succeeded in a measurable way in capacity building for e‐BIZ Center Staff? Marketing and sales training organized by the EI activity was attended by Mr. S. P. (Satwork owner). Other than that, she cannot recall any other organized training delivered by the EI Activity. Capacity building was mainly done through regular meetings, phone calls and mail communication with Mr. Ismir and Mr. Jusuf Tanovic. She received one-to-one training from Ismir about tracking cost sharing in excel sheet. Customer relationship management (CRM) software is still not completed. Ms. Rujevic stated that quality of provided technical assistance was excellent. Additional training on how to improve sales is needed. That field is continuously developing and any training is more than welcome. When it comes to client surveys, they preformed phone survey on a limited sample of clients. They recently sent out questionnaires to clients and 14 questionnaires were filled by clients and returned to Transwork whilst 4 additional questionnaires were filled by Transwork on the basis of phone conversation with clients. Questionnaire form was provided by the EI Activity. Ms. Dobrila Vukmanovic advised them to take testimonials from clients and put them on their web site. They still have not done that. Sales and Marketing They respond immediately on clients requests. She does not know exactly how often they contact management of their clients. Maybe it is once in six months and the main reason for it is expansion of services provided by Transwork. Follow up with clients is done on need basis. They will try to maintain contact with clients at least once annually. They do not use social media for communication with clients. Their clients do not use computers and internet on regular basis. They do not have written procedures that describe important processes within the company. Satwork has ISO certificate and Satwork follows that kind of procedures. The most important procedure for Transwork are the ones related to sales of services. Sales process is defined with the following steps: • Services offered to client in writing (standardized letter); • Follow up phone call a few days after and arrangement of a meeting (standardized scenarios of communication with client); • Direct presentation of services to client (standardized PPP); • Deal closing; • Contracting; • Equipment installation. These steps describe main processes and operations of Transwork. Transwork Future Transwork has a contract with Satwork and will continue to operate in that way. Transwork is a consulting company which is offering cost optimization services to any company with more than 5 vehicles. No competitor has this package of services like Transwork. Competition is cutting prices of their services. Market of GPS tracking (about 60% of Transwork revenues) is very price sensitive and Transwork is trying to keep pace with them when it comes to prices. Employee Reward System The salary is only reward system that presently exists. There is no defined system of salary rise, bonuses, rewards or promotions. Until now, beside regular salary, they had a spot award but there is no systematic approach to employee rewarding. There is no turnover of employees. Only one employee left and that happened even before official start of Transwork work. Business plan Ms. V.R. is of opinion that business plan goals are completely achieved. However, things changed since then. Prices of services defined by the business plan are much higher that today’s market prices for these services. For example, they planned a 100 BAM subscription for the electronic freight exchange service and they did not manage to charge even one client for that service. The most important service for Transwork is GPS monitoring and tracking system which contributes with about 50% to total revenues. They have 45 clients and until now only one canceled contract with Transwork. To what extent has Transwork succeeded in improving the competitiveness of their clients? Ms. V.R. is not able to confirm whether competitiveness of their clients is improved. They were analyzing only their satisfaction with Transwork’s services. She does not know whether their competitiveness was increased in terms of their sales or profitability. They did not track these indicators. Cooperation with associations They have not managed to establish any significant cooperation with transport sector associations. The most successful thing they did with associations was with the Chamber of Commerce from ZE-DO Canton - they managed to organize a presentation of Transwork through them. However, only ten companies came. They tried the same thing with Una-Sana Canton Chamber of Commerce but they asked for 5,000 BAM of sponsorship for the fair they were organizing. There are several associations. They established contact with Association of domestic and international freight forwarders. They signed a contract with them and they introduced Transwork through SMS to all 400 members. However, nothing happened and their members are small companies with in average two vehicles. At the level of BiH, they tried to establish contact with B&H Foreign Trade Chamber. They talked to Mr. Zijad Sinanovic, asking him to participate at the meeting of freight forwarders. They conditioned their participation at the meeting with 5,000 BAM in sponsorship. They also tried to establish contact with Mr. Velibor Peulic who is the main person in transport sector in BiH but they could not reach him. Transwork’s impact on sector / industry It would make sense to talk about Transwork’s impact on the entire sector / industry only if Transwork would have large enough client base. There is no support from public sector. The entire project was huge challenge for all of them since not a single employee had any experience or knowledge about transport sector. Today they are technologically two steps ahead from others. USAID contribution / added value USAID contributed financially, through advisory services (especially when it comes to marketing which is very important project element for USAID) It is hard to say what would happen if there was no USAID support. It would be more difficult to find money for financing these activities. Maybe Satwork would develop software and services without USAID support, however, Satwork is small company and this financial injection was more than welcomed. Presently they do not have time to deal with it since their main objectives are sales increase and improvement of financial results. It is important they reached brake –even point. USAID support and background helped when talking with potential clients since USAID support is a kind of guaranty that they are serious company. What makes Transwork different from others is package of services they are offering. Software they use was developed in-house. Therefore they are capable of adjusting their services to local market demands. Other companies that are buying licensed software cannot that easily adjust that software and service without high cost. Obstacles They have not get support from transport sector associations. ANNEX 8 PERFORMANCE EVALUATION OF THE EXCELENCE IN INNOVATION ACTIVITY INTERVIEW MINUTES Interview No. 12 – eBIZ Center Arteco Location: Arteco , Medakovo, Federation of Bosnia and Herzegvoina, Bosnia and Herzegovina Date: 19.08.2013 Interviewers: Davorin Pavelic Jasmina Mangafic Interviewees: Mr. F. M. (Artisan owner) Mr. E. M. (General Manager Arteco) Mr. M. H. Disclaimer: The Minutes are notes of the Interviewees’ statements and opinions given as replies to the Interviewers’ questions. Recorded statements and opinions do not represent Interviewers’ opinion on work and operations of the eBIZ Center. Introduction The Interviewer made a brief introduction about the objective of this visit and purpose of the evaluation. Description of the establishment and functioning of the eBIZ Center Vision – mission - goals “Within five years, ARTECO will become the leading company of applied technologies for the wood processing industry in BiH, a central location of knowledge for the wood industry and an organization that will link international buyers with domestic producers”1 “Its mission is to provide technical services to Small and Medium Enterprises (SMEs) in the wood processing sector in Bosnia-Herzegovina (BiH) to increase the competitiveness of these SMEs. ARTECO will offer critical and complex services that are not easily accessible and will be a driving force of 1 Arteco’s Business Plan economic development in the BiH wood processing industry through application of new technologies.”2 “The overall objectives of ARTECO are: • Provide technical services to at least 50 SMEs in the wood processing sector during the first two years of operations; • Increase the competitiveness of the BiH wood processing SMEs by utilization of modern technologies, expansion of export customer base, application of the best technology solutions in product finishes, etc; • Find international buyers for the BiH wood processing SMEs; • Contribute to an increase in the export of processed wood products from BiH; • Contribute to an increase in the employment in the wood processing industr; • Become self-sustainable by December 2010. To its clients, ARTECO will provide: • Large scope of universally accessible services, supported by a unique price/quality ratio to be provided by superior experts and sub-contractors. • Access to modern ICT applications, • Permanent improvement of business processes to improve the competitiveness of the BiH wood processing industry. “3 Capacity building They received the following trainings: • CAD training. • CNC operators training • Training in surfaces polishing Training provided by Arteco to external clients They trained a lot of students but they do not know from which faculties exactly. Arteco’s Clients Arteco and the EI Activity submitted several lists of clients that received Arteco’s services. On the first list there were 88 clients, out of which 38 were companies. On the second list there were 112 clients. They have small (one time) and large clients (large orders). Under large orders they consider, for example, an order to produce 500 chairs. They had about 15 large clients. Some of them are Jadrina from Doboj Istok / Gracanica and Nautica. They have about 15 companies to whom they provide services on regular basis. 2 Arteco’s Business Plan 3 Arteco’s Business Plan Services 1) Finding international buyers for the BiH wood processing SMEs and contributing to an increase in the export of processed wood products from BiH They applied two models in delivering this service: (1) They establish contact with foreign company / customer who has its own product. Arteco finds local producer for them and take 10% commission. Example is company Scano from Norway. (2) Arteco develops product and technical documentation, hires local wood processors and than exports products. Arteco was providing these services from 2009 to 2011. They stoped with it because Ms. Dobrila Vukmanovic asked them to do so. She claimed that this was not the purpose of the project. 2) Furniture design They cooperate with about about 30 designers. Arteco provides service of technical documentation development for designers. Designers come with sketch. Arteco transfers sketch into SolidWorks. After that they jointly adjust model and than though CAD/CAM system Arteco produces prototype. After Joint introductory session, the Evaluation team continues interviews separately. Ms. Jasmina Mangafic interviewed Mr. F. C. (Artisan owner – self invited) and Mr. M. H.c (Administration) whilst Mr. Davorin Pavelic interviewed Mr. E. M. Interview with Mr. E. M. (General Manager) Competitive advantages of Transwork Interviewer: Davorin Pavelic Question 1. To what extent has the EI Activity succeeded in a measurable way in capacity building for e‐BIZ Center Staff? Marketing and sales training organized by the EI Activity was attended only by Mr. Elvedin Madzukic. He stated that the training was excellent. They received (Mr. Muhidin Hasicic and Mr. Mandzukic) direct one-to-one, on job, one￾day training in use of the Customer Relations Management (CRM) system by Mr. Ismar from the EI Activity. The CRM system is one Excel table. SolidWorks for design is paid by Artisan. CNC three-day training is provided by Hamag as a part of CNC machine deal. All other trainings were organized and paid by Artisan. Any further professional development in CAD/CAM systems is needed and welcome. For a range of services they are providing right now they have no need for outsourcing or outside assistance. They received a few requests for prototype production they could not handle since that required 8-axes CNC machine (their machine has 5-axes). Financial and Operational Sustainability Their response time to client’s requests is within 24 hours. There are a lot of clients they do not contact at all after one-time service. They call after they receive service just to check whether everything is ok. They have between 15-20 companies that work for them with whom they have regular daily communication. That communication is mainly of technical nature. With a lot of designers they have frequent technical communication and they are mainly interested in technology issues, i.e. what can be done / made. They do not use social networks for communication with existing clients or for acquiring new ones. Written Operational Procedures: • They have SolidWorks training fully developed • They have written procedures for prototype development • They have written guide on how to operate CNC machine Majority of income was coming from exporting service. After that, CNC machining is contributing the most to the total revenues. They will reinitiate their service of connecting SMEs from B&H with customers from abroad. He believes that out of 100 clients served till now, they have about 40 stabile ones. Arteco Future Arteco in 5 years from now? • Center which makes it possible for people with ideas to transform them into reality. • Expanded service of prototype development with focus on foreign customers, introducing service of wood bending Employee Reward System • The salary plus 10% reward if planed production was achieved. It is clear how and when one is going to be awarded this 10%. • One-time gifts for holidays. Presently they have 5 employees. Until now, 5 employees left the company. Business plan Mr. Mandzukic is of opinion that all elements of Business Plan and its objectives are achieved. To what extent has Arteco succeeded in improving the competitiveness of their clients? Did they contributed to higher sales / profitability of their clients? If yes, how? Mr. Mandzukic claims they increased profitability of their clients. He can say that with confidence for about 20 companies. He knows these people. He knows what they were doing before and what jobs they did thanks to Arteco. They surveyed their clients. One part of survey was done by phone and the otehr by questionnaire. Questionnaire form was received from the EI project. People from USAID continuously insisted on marketing and different ways of contacting clients. They were sending letters to potential clients. This proved to be not very successful. One example is trade fair organized in Tesanj. Arteco also participated. They sent letters and after that e-mails to 80 companies from CRM base to come and see Arteco. Only one client came - that much about marketing through letters and e-mail. They do not have any specific client group they are targeting. All companies from wood processing industry are their target group. They respond to all their requests for design, production improvement, market connections. He knows there is demand for their services. He cannot answer what is the wood processing industry companies’ capacity to absorb and use ICT. He believes Arteco increased competitiveness of their clients. Cooperation with associations No information. Arteco’s impact on sector / industry He believes Arteco increased competitiveness of their clients. He cannot back this statement up with exact data but he knows these companies. USAID contribution / added value USAID brought knowledge, another approach and method for doing business which is very systematic. They allowed designers to create products. Arteco brought a new approach on how wood processing industry should work on furniture design. What would be different if there was no USAID assistance? With USAID, Artisan technologically improved for three levels and their 20 regular clients advanced technologically at least for one level. Each new person trained how to read technical drawings is a progress. Artisan has between 50 and 60 employees. Without Arteco, they would have 30 employees. They are the only company in B&H which uses CAD/CAM system integrated with 5- axes CNC machine. Prevent company from Visoko has a more advanced machine and equipment but they do not use it in a way how Arteco does. In Italy and Germany companies have the same software and machines but Arteco is much cheaper. What you would pay 100 EUR in Germany, you pay 100 BAM to Arteco. They included a lot of young people and designers. A lot of them are coming from Croatia in order to develop their products. A designer from Belgrade brought his students to show them how Arteco works. Can you imagine students from Belgrade coming to a small village Medakovo near Tesanj to se how things should be done?. Obstacle The biggest obstacles are local people and lack of awareness about benefits of new technologies and CAD/CAM systems. Cooperative Agreement Number: 168-A-00-07-00101-00 Modification No. 11 2 1. COVER LETTER 1.1. Delete the  Recipient’s  name  “FHI Development 360 LLC” and  replace it with “Family Health International (FHI  360)”,  wherever it appears in the award. 1.2. In the second paragraph of the cover letter,  delete  “December  2,  2012”  as  the  closing  date  of the award and replace  it  with  “August 31,  2013”.   1.3. In Section  “A. GENERAL”,  article  1  “Total  Estimated  USAID  Amount”, delete the Total Estimated USAID  Amount  of  “$5,164,622”  and  replace  it  with  “$4,598,126”  in  lieu  thereof.   1.4. In  Section  “A. GENERAL”,  article  3  “Cost-Sharing Amount (Non-Federal)”,  delete  “Please refer  to  the Schedule”  and  replace  it  with  “$286,000”  in  lieu  thereof.   1.5. In  Section  “A. GENERAL”,  article  6  “Tax I.D.  Number”,  delete  “452-459-015”  and  replace  it  with  “23- 741-3005”  in  lieu  thereof.   1.6. In  Section  “A. GENERAL”,  article  7  “DUNS  Number”,  delete  “96-845-6694”  and  replace  it  with  “06- 718-0786”  in  lieu  thereof.   2. ATTACHMENT A - SCHEDULE 2.1. In  Section  “A.2 PERIOD OF AGREEMENT”, paragraph 1, delete the award closing date of “December 2, 2012” and replace it with “August 31, 2013” in lieu thereof. 2.2. In  Section  “A.3 AMOUNT OF AWARD AND PAYMENT”, paragraph 1, delete the total estimated amount  of  “$5,164,622” and  replace  it  with “$4,598,126”  in  lieu  thereof.   2.3. In  Section  “A.4 BUDGET”,  delete  cost  elements in their entirety and replace them with the following: “Description Total Budget Total Direct Costs $403,491 Indirect Costs $290,365 Contractual/Sub-Awards $3,904,270 Total USAID Contribution $4,598,126 Cost Share/Leveraged Funds $286,000 Total Program Cost $4,884,126” 2.4. Delete Section  “A.5 REPORTING OF FOREIGN TAXES (MARCH 2006)”, in its entirety and replace it with the following in lieu thereof: “A.5 REPORTING HOST GOVERNMENT TAXES (JUNE 2012) a. By April 16 of each year, the recipient must submit a report containing: Cooperative Agreement Number: 168-A-00-07-00101-00 Modification No. 11 3 (1) Contractor/recipient name. (2) Contact name with phone, fax and e-mail. (3) Agreement number(s). (4) The total amount of value-added taxes and customs duties (but not sales taxes) assessed by the host government (or any entity thereof) on purchases in excess of $500 per transaction of supplies, materials, goods or equipment, during the 12 months ending on the preceding September 30, using funds provided under this contract/agreement. (5) Any reimbursements received by April 1 of the current year on value-added taxes and customs duties reported in (iv). (6) Reports are required even if the recipient did not pay any taxes or receive any reimbursements during the reporting period. (7) Cumulative reports may be provided if the recipient is implementing more than one program in a foreign country. b. Submit the reports to: U.S. Agency for International Development Controller's Office Bank Center, Granite Tower, 3rd floor Budapest 1944, Szabadsag Ter 7-9, Hungary And USAID/Bosnia and Herzegovina Roberta C. Frasurea 1 71000 Sarajevo Bosnia Hercegovina c. Host government taxes are not allowable where the Agreement Officer provides the necessary means to the recipient to obtain an exemption or refund of such taxes, and the recipient fails to take reasonable steps to obtain such exemption or refund. Otherwise, taxes are allowable in accordance with the Standard  Provision,  “Allowable  Costs,”  and  must  be  reported  as  required  in  this  provision.   d. The recipient must include this reporting requirement in all applicable subagreements, including subawards and contracts.” 2.5. Delete  Section  “A.6 INDIRECT COST RATE”  in  its  entirety  and  replace with the following in lieu thereof: “A.6 INDIRECT COST RATE Pending establishment of revised provisional or final indirect cost rates, allowable indirect costs shall be reimbursed on the basis of the following negotiated provisional or predetermined rates and the appropriate bases: Description Rate Base Type Period Fringe Benefits 36.91% (a) Final 01-01-07 – 12-31-07 Fringe Benefits 34.98% (a) Final 01-01-08 – 12-31-08 Fringe Benefits 35.89% (a) Final 01-01-09 – 12-31-09 Fringe Benefits 32.56% (a) Provisional 01-01-10 – 12-31-10 Cooperative Agreement Number: 168-A-00-07-00101-00 Modification No. 11 4 Fringe Benefits 34.57% (a) Provisional 01-01-11 – 06-30-11 Fringe Benefits* 32.10% (a) Provisional 07-01-11 – 09-30-11 Fringe Benefits* 26.31% (a) Provisional 10-01-11 – Until Amended Overhead 37.47% (b) Final 01-01-07 – 12-31-07 Overhead 35.56% (b) Final 01-01-08 – 12-31-08 Overhead 35.19% (b) Final 01-01-09 – 12-31-09 Overhead 35.19% (b) Provisional 01-01-10 – 12-31-10 Overhead 39.11% (b) Provisional 01-01-11 – 06-30-11 G&A* 39.78% (b) Provisional 07-01-11 – 09-30-11 G&A* 36.63% (b) Provisional 10-01-11 – Until Amended Subcontract/ Subrecipient Management 4.51% (c) Final 01-01-07 – 12-31-07 Subcontract/ Subrecipient Management 3.86% (c) Final 01-01-08 – 12-31-08 Subcontract/ Subrecipient Management 3.25% (c) Final 01-01-09 – 12-31-09 Subcontract/ Subrecipient Management 3.25% (c) Provisional 01-01-10 – 12-31-10 Subcontract/ Subrecipient Management 4.26% (c) Provisional 01-01-11 – 06-30-11 Subcontract/ Subrecipient Management* 4.63% (c) Provisional 07-01-11 – 09-30-11 Subcontract/ Subrecipient Management* 3.72% (c) Provisional 10-01-11 – Until Amended *For fiscal years ending September 30, 2011, and 2012, the contractor agrees to ceiling rates of 26.31% for Fringe Benefits, 36.63% for G&A and 3.72% for Subcontract/Subrecipient Management. Base of Application: (a) Total salaries excluding overseas employees paid in local currencies and Third Country Nationals working in a country other than the United States (b) Total direct costs excluding pass-through grant, subcontract/subrecipient, operational expert (OPEXER; host and third country nationals who are primarily employees of host country governments) and other administrated funds, overseas allowances, participant support, non-expandable equipment and in-kind contribution costs (c) Total subcontract/subrecipient expenses including operational experts (OPEXER) and other administrated funds and excluding subcontractor pass-through costs related to pass-through grants, participant support and non-expendable equipment" Cooperative Agreement Number: 168-A-00-07-00101-00 Modification No. 11 5 2.6. Delete  Section  “A.12 EXECUTIVE ORDER ON TERRORISM FINANCING”    in  its  entirety and   replace it with the following in lieu thereof: “A.12 PREVENTING TERRORIST FINANCING -- IMPLEMENTATION OF E.O. 13224 (JUNE 2012) a. The recipient is reminded that U.S. Executive Orders and U.S. law prohibits transactions with, and the provision of resources and support to, individuals and organizations associated with terrorism. The recipient must not engage in transactions with, or provide resources or support to, individuals and organizations associated with terrorism. In addition, the recipient must verify that no support or resources are provided to individuals or entities that appear on the Specially Designated Nationals and Blocked Persons List maintained by the U.S. Treasury (online at: http://www.treasury.gov/resource- center/sanctions/SDN-List/Pages/default.aspx) or the United Nations Security designation list (online at: http://www.un.org/sc/committees/1267/aq_sanctions_list.shtml). b. This provision must be included in all subagreements, including contracts and subawards, issued under this  award.” 2.7. Insert  Special  Provision  entitled  “Nondiscrimination  (June  2012)” as Section A.13 and renumber all subsequent sections accordingly: “A.13 NONDISCRIMINATION (JUNE 2012) No U.S. citizen or legal resident shall be excluded from participation in, be denied the benefits of, or be otherwise subjected to discrimination on the basis of race, color, national origin, age, disability, or sex under any program or activity funded by this award when work under the grant is performed in the U.S. or when employees are recruited from the U.S. Additionally, USAID is committed to achieving and maintaining a diverse and representative workforce and a workplace free of discrimination. Based on law, Executive Order, and Agency policy, USAID prohibits discrimination, including harassment, in its own workplace on the basis of race, color, religion, sex (including pregnancy  and  gender  identity),  national  origin,  disability,  age,  veteran’s  status,  sexual   orientation, genetic information, marital status, parental status, political affiliation, and any other conduct that does not adversely affect the performance of the employee. In addition, the Agency strongly encourages its recipients and their subrecipients and vendors (at all tiers), performing both in the U.S. and overseas, to develop and enforce comprehensive nondiscrimination policies for their workplaces that include protection for all their employees on these expanded bases, subject to applicable law.” 3. ATTACHMENT B – PROGRAM DESCRIPTION 3.1. Insert the following Program Description for the extension period (December 3, 2012 – August 31, 2013): “PROGRAM DESCRIPTION – EXTENSION PERIOD (DECEMBER 3, 2012 – AUGUST 31, 2013) Cooperative Agreement Number: 168-A-00-07-00101-00 Modification No. 11 6 1. BACKGROUND Launched on December 3, 2007, the EI Project utilizes information and communication technology (ICT) as part  of  USAID’s  efforts  to  generate  employment  and promote the expansion and competitiveness of BiH small and medium enterprises (SMEs). The Project is aligned with the Foreign Assistance Framework Economic Growth Program Area of Private Sector Competitiveness and Program Element Private Sector Capacity. It provides assistance to and collaborates with: (a) BiH entrepreneurs and firms to establish and operate sustainable  “e-BIZ Centers” that offer high-impact information and communication technology (ICT) applications and services to SMEs in targeted and supported sectors; and (b) SMEs in order to improve their capacity to fully utilize the high-impact ICT applications and services offered by collaborating e-BIZ Centers/Enterprises to improve their competitiveness. The EI Project furthers  USAID/BiH’s  strategy, for which a top economic growth priority is increasing the competitiveness and growth of SMEs in the local, regional, and global marketplace. The EI Project has established four e-BIZ Centers to date, in four different sectors: wood processing; tourism; transportation and logistics; and textile/apparel. The Project component is being implemented under the dot￾ORG ORG Leader-with-Associates Cooperative Agreement within the framework of the Digital Opportunity through Technology and Communication Alliance (DOT-COM). FHI 360 (FHI) is the prime implementer of the EI activity and the Education Development Center (EDC) is the primary implementing partner/sub-grantee. EI Project Objectives The EI Project objectives are to strengthen the competitiveness of SMEs in selected sectors by increasing their use of value-added ICT so that they can offer new, high value-added products and services that are in demand in domestic and/or global markets of the selected sectors. Through this extension, the EI Project will continue to support the existing EI Project e-BIZ Centers as described in more detail below. 2. DEVELOPMENT CHALLENGES AND HOW EI ADDRESSES THEM There are many development challenges facing SMEs in BiH, including low technical readiness, overly focusing on low skill processes, and few profitable contracts. The ability to absorb new technology is weak across sectors. The majority of managers in SMEs come from technical and production backgrounds and they lack entrepreneurial vision and marketing skills. Generally these managers put their firms into the passive position waiting for European Union (EU) clients to bring them production orders for predetermined products. As such, the challenges do not revolve only around technical issues and staff education but also around the ability to raise awareness and change mindsets in many cases. Furthermore, the BiH governments are not seriously implementing sector development strategies in any of the targeted sectors. The EI Project addresses a crucial area that requires immediate attention: improvement and expansion of BiH SMEs’  effective  use  of  ICT  so  that  they  can  improve  their  competitiveness  in  local,  regional  and  international   markets and continue successfully on the path to EU accession. Currently, BiH SMEs face two major constraints to their use of ICT to improve competitiveness: Lack of access to high-impact ICTs. Although many BiH SMEs have generic ICT (computers, websites, spreadsheet  applications,  etc.),  they  lack  access  to  “high  impact”  ICTs,  i.e., sector-specific ICTs that will directly improve their competitiveness. Cooperative Agreement Number: 168-A-00-07-00101-00 Modification No. 11 7 Lack of ability to use ICT to compete. Business survey data suggests that BiH companies recognize the need to utilize ICT effectively, but lack the knowledge and skills needed to do so. The EI Project activities during the extension period, described below, address these two crucial constraints. 3. EXTENSION ACTIVITIES The activities of the EI Project extension will be focused on providing support to the attainment of full self￾sustainability and strong market position of the existing four e-BIZ Centers1 . The activities of the extension period, described in more detail by sector below, may include providing continued marketing and outreach support to the Centers to help them optimize their engagement with SME clients or potential clients, in both the online web marketing space (e.g. search engine optimization, improving the look, content and ease of navigation of web sites and portals) and in terms of offline marketing (e.g. assisting with database development to track customers and sales, market analysis of key potential clients and industry trends). Other potential support may include specific technical assistance related to the e-BIZ Centers focused on textile/apparel and on logistics/supply chain management, in areas such as improving production processes using new technologies and equipment (for the textile/apparel center), and improving the effectiveness of technology applications developed by the e-BIZ Center for logistics and supply chain management. Anticipated Support – Tourism Sector (Exploring Bosnia) Background Exploring BH is currently struggling. Most of its revenues come from commissions on conference organization and hotel accommodation for long-standing Projects and international organizational clients. Exploring BH currently has no sales from online operations due to their poor web positioning and web promotion. Exploring BH's online portal has not been updated regularly and the firm did not have funds available for investment in online marketing during the past twelve months. The company was, however, able to successfully promote tours in Bosnia and Herzegovina (BiH) and develop business relationships with US and EU-based tour operators in 2010. This resulted in solid revenues from international tourists in 2011. Exploring BH was on its way to expand that market in 2012 when it suffered a major setback earlier this year. For example, the firm invested heavily into a partnership with a German-based tour operator who decided to cease their entire Bosnia operation prior to a launch of a major promotional campaign in the German market. The overall situation is alleviated somewhat by the fact that Exploring BH managed to significantly reduce its operations cost in recent months so they are presently managing to stay afloat in terms of cash flow and covering of operational costs. Furthermore, the Exploring BH's annual statement for 2011 showed that the firm had a small profit (900 KM) on revenues exceeding 300,000 KM per year. 1 The ongoing EI support to the exisiting e-BIZ Centers was delayed due to AED's suspension. Hence, the e-BIZ Centers' planned progression towards self-sustainability was negatively affected. Cooperative Agreement Number: 168-A-00-07-00101-00 Modification No. 11 8 The  issue  with  Exploring  BH’s  online  portal  and  its  weak  performance  is  completely  linked  to  the  issue  of  the   firm ownership -- i.e. the partnership between Fondacija e-turizam (Fondacija -- the majority owner of Exploring Bosnia) and Skopje-based Exploring Macedonia (EM), the minority owner of Exploring Bosnia. Under the arrangement between the two parties, EM provided the Exploring Bosnia domain name, the online booking software and all technical backstopping including domain hosting and site maintenance for Exploring BH. This partnership has not been successful; Exploring Macedonia did not provide the expected increase for Exploring BH in terms of international tourism contacts, improved web presence and site maintenance. Both parties are essentially faulting the other for the breakdown in relations and why the anticipated level of online support has not materialized. For that reason, Fondacija and the director of Exploring Bosnia has been trying, for the past several months, to assume the  full  ownership  of  the  firm  and  to  take over  the  management  of  the  Exploring  BH’s  online  portal.   These  parties  (referred  to  as  “The BiH-based  parties”)  have  also  expressed  readiness  to  provide  additional   investment into the firm in the event they assume full ownership of Exploring BH. According to the latest developments in end of October 2012, the two sides appear close to a workable solution that would allow for the above-mentioned ownership restructuring. Exploring BH's online presence, overall marketing and sales performance and support to local tourism SMEs could be improved significantly in the upcoming period through concerted effort and investment from the firm's owners with support from the EI program if the ownership transfer is accomplished. Activities during the Extension Period The following activities will be carried out to upgrade Exploring BH's online presence and significantly improve its overall business performance: The BiH-based parties assume full ownership of the Exploring BH, including the full rights to the domain name. Exploring BH would then be 100% owned by Bosnian partners. Exploring BH web portal hosting moves to BiH. Redesign of Exploring BH's web page and migration of all content. Development and/or integration of booking software and other key functionalities into the Exploring BH site. Perform Search Engine Optimization (SEO) and regular monthly follow up SEO activities. Continuous update of site content including creation of daily and weekly promotions. Conduct broad online marketing campaign in international markets. Conduct targeted offline and online campaign targeting local and regional corporate and organizational clients. Conduct outreach to international tour operators including follow up marketing campaign aimed at international tour operators that have participated in Bosnia promotional tours organized by Exploring BH in 2010-11. The total period for the successful completion of all listed activities is estimated to be six months. A strong and concerted internet-based tourism marketing effort requires further funding and, in this respect, the additional support outlined in this document could make an important difference at this stage of the development of Exploring BH. Cooperative Agreement Number: 168-A-00-07-00101-00 Modification No. 11 9 Anticipated Support – Textile Sector Background Since its establishment in April 2012, the TTC has provided manufacturing support services (CAD design, pattern making, grading and cutting) to thirteen local SMEs. The Center is in its nascent phase and had some early problems with malfunctioning of the new high tech equipment (solved by distributor fairly quickly) and errors by young operators (solved by additional training paid by the owner). Overall, the Center is now positioned to help BiH SMEs improve their manufacturing efficiency and increase EU orders both in overall volumes and added value category. At present, most EU orders involve low profitable, labor only services from BiH manufacturers. Activities during the Extension Period Execute wide ranging marketing activities, aimed at reaching the targeted number of client SMEs (40) set in the e-BIZ Centers business plans; Development of Client Relationship Management, the ICT based system that will help the e-BIZ Center improve its marketing and client relationship capability; Finalization of all internal processes (organizational, staffing, cost saving etc.) aimed at achieving full operational and financial sustainability; Completion of international outreach to EU based clients aimed at securing new value-added manufacturing orders  for  BiH  SMEs  that  would  utilize  TTC’s  ICT  based  manufacturing  support  services; and Resolving anticipated  technology based production  bottlenecks  identified  in  the  international   consultant’s   report. A detailed operational plan for achieving listed activities during the Project extension period will be developed following approval of the Project extension. Anticipated Support – Transportation and Logistics Sector Background Since its establishment in March 2012, Transwork (the transportation and logistics sector e-BIZ Center) has provided GPS based services, such as vehicle and goods tracking, fleet management, as well as the first domestic on-line freight exchange. To date, Transwork has served fifteen local transportation SME clients, providing various GPS based packages of services (vehicle locators, goods tracking, fleet management system, vehicle communication & navigation). There are approximately 120 active freight exchange members offering on average 30 freights and 80 of various transporting vessels daily.    Transwork’s  founder, Satwork, is continuously developing new services that are distributed through Transwork. In its full capacity, Transwork will be able to provide a wide range of services that are crucial for domestic L&T SMEs in their pursuit of global competitiveness. Activities during the Extension Period Execute series of focused marketing activities aimed at reaching the targeted number of client SMEs (40) set in the e-BIZ Centers business plans; Cooperative Agreement Number: 168-A-00-07-00101-00 Modification No. 11 10 Development of Client Relationship Management, the ICT-based system that will help the e-BIZ Centers improve their marketing and client relationship capability; Completion of all internal steps (organizational, staffing, cost savings, etc.) aimed at achieving full operational and financial sustainability; Complete the new services development (COM/NAV - communication & navigation service and CANBUS - vehicle diagnostics service) per timeline set in the Transwork business plan; and Meet technical conditions for Transwork to become  a  part  of  the  “Freight  In” -- the international freight exchange network. A detailed operational plan for achieving listed activities during the Project extension period will be developed following approval of the Project extension. Anticipated Support – Wood Processing Sector Background Since its establishment in October 2009, ARTECO (the wood processing sector e-BIZ Center) has provided product development, prototype development, technical documentation development, manufacturing services, CAD&CAM training and education (for engineers and machine operators), market research and marketing services to local wood processing SMEs. To date, ARTECO has served over 85 clients, out of which more than 70 were SMEs, while the rest were individual entrepreneurs, architects, craft shops etc. Several ARTECO clients have developed solid wood furniture pieces have won major international design awards in EU in 2011 and 2012. At the same time, ARTECO has had challenges in attracting customers. There are several reasons for this. First, many companies in the wood sector rely on agents to find them Projects; these companies are not proactive in new product or service development. Second, some of these companies do not have the level of technological sophistication to utilize ARTECO's CNC machinery for processing. Third, many potental client companies evidently prefer to reject a new contract than to outsource their processes (many of these companies want to own the process and are not used to cooperation). ARTECO has reacted to these challenges by reducing the price for its services, especially the CNC machinery usage, and doing promotional campaigns to educate potential clients on the beneifts. ARTECO has also begun the process of certificaiton for its partners; certified companies are included into a newtwork, where they obtain work from ARTECO. Activities during the Extension Period During the extension period, EI will provide limited support to ARTECO that will focus on executing marketing activities aimed at further expanding the Center's customer base and selling more CNC-based services. The emphasis will be on marketing ARTECO to small and medium enterprises (SMEs) that have not been contacted before and/or have not used the services of ARTECO but potentially have a demand for those services. In addition, EI is currently assisting ARTECO in developing a Customer Relationship Management (CRM)  database  with  information  about  visited  and  contacted  SME’s. Specific activities that will be completed related to ARTECO are as follows: Cooperative Agreement Number: 168-A-00-07-00101-00 Modification No. 11 11 Gathering and updating the list of existing and potential customers using ARTECO’s  Customer Relationship Management (CRM) database; The activities described above will be conducted on an ongoing and continuous  basis.    ARTECO’s  staff  will   carry out the bulk of these activities with support from EI Project. 4. PROJECT MONITORING AND EVALUATION Project Results Result 1: Four self-sustaining e-Biz Centers, providing high impact ICT services to small and medium enterprises in the target sectors. Result 2: Significantly improved competitiveness of at least forty SMEs in the textile sector, at least forty SMEs in the transportation and logistics sector, at least fifty SMEs in the wood processing sector, and at least 70 SMEs in the tourism sector (200 total SMEs) and/or supporting sectors through increased use of value-added ICT. Result 3: Lessons learned and guidelines for future efforts drawn from the four e-Biz Centers. Establish Outputs and Benchmarks for the Project Extension The EI Project has defined specific, measurable Project outputs and benchmarks for e-BIZ Center sustainability and SME competitiveness, as part of its Performance Monitoring Plan (PMP). In addition, the EI Project will follow USAID guidance and track appropriate Foreign Assistance Framework standard indicators, such as the number of firms that receive USG assistance to improve their management practices and the number of firms receiving USG assistance to invest in improved technologies. To achieve the Project results, the expected outputs and benchmarks during the extension period for each of the e-BIZ Centers are as follows: Exploring BH Fully functional on line portal with daily updates, online booking and online payment functions; Generated significant online visits to Exploring BH portal focused on specific promotions and sales of BiH tourism offerings; Improved outreach to local and regional business clients for conference and seminars organization; Development of new partnerships with international tour operators and further expansion of previously executed initial tours with US and EU based operators; Additional 20 SMEs from the tourism industry offering their services. Cooperative Agreement Number: 168-A-00-07-00101-00 Modification No. 11 12 ARTECO Completed Customer Relationship Management Database. TTC Completion of all activities listed in Anticipated Support section of this document resulting in the successful achievement  of  the  TTC’s  Business  Plan. TRANSWORK Completion of all activities listed in Anticipated Support section of this document resulting in the successful achievement  of  Transwork’s  Business  Plan. 5. PROJECT MANAGEMENT, ADMINISTRATION, AND REPORTING Project Reports During the Project extension, the EI Project will submit work plans and reports described below, as well as any other reports that may be required under USAID regulations. Work Plans Annual Work Plans. The EI Project will submit an Annual Work Plan continuing the same scheduling currently used. The Annual Work Plan will set milestones for the upcoming year (per Performance Monitoring Plan and other performance reports). It will also indicate where adjustments are being taken to address obstructing factors and support achievement of targets and milestones in the upcoming year. Performance Reports During the Project extension period, the EI Project will submit a schedule of regular performance reports that will provide information on progress toward milestones and objectives. At a minimum, these reports will include: Quarterly reports, Annual Reports, and a Final Report (submitted in lieu of the final annual report). All reports will be submitted utilizing the same scheduling and format as is currently utilized by the Project. Project Management Under this LWA, FHI is the prime recipient/implementer and is responsible for overall management and technical oversight, liaising with USAID/BiH, overseeing monitoring and evaluation, submitting Project deliverables, etc. To help ensure that the Project meets its goals and objectives, all important communications with USAID/BiH regarding this Project, including those relating to contractual, budget, technical, and programmatic issues, will be handled by FHI,  cc’ing  the  primary implementing partner EDC as appropriate. FHI will also coordinate with USAID/W as appropriate. EDC is the primary implementing partner responsible for the day-to-day implementation of the Project activities as described above. EDC and FHI will also work in close partnership on the strategic planning/design and implementation of the activities described above. As Cooperative Agreement Number: 168-A-00-07-00101-00 Modification No. 11 13 such, EDC is responsible for the Project office in Sarajevo that manages and conducts the Project activities on a day-to-day basis. The Project office’s  major  responsibilities  for  the  extension  will  be  the  same  as  they are currently: Establish strong working relationships with other USAID-funded Projects that aim to support SME growth, and coordinate activities with those Projects; Conduct the e-BIZ competitive process; Provide training and technical assistance to the Centers, through Project staff and local or international experts; Work with the e-BIZ Centers to promote their services; Build capacity among e-BIZ Centers; Support the e-BIZ  Centers’  sharing  of  experience and  collaboration  for  mutual  advantage;; Suggest revisions in Project approach as appropriate; Work and communicate closely with the USAID Mission; fulfill all reporting requirements. However, USAID policies and regulations concerning privity of communications between the prime recipient, USAID, and the sub-grantee will be observed; and Manage Project evaluation activities. [END OF ATTACHMENT B – PROGRAM DESCRIPTION (EXTENSION PERIOD – DECEMBER 3, 2012 – AUGUST 31, 2013)]” EXCEPT AS SPECIALLY HEREIN AMENDED, ALL TERMS AND CONDITIONS OF THE AGREEMENT REFERENCED IN BLOCK # 3 OF THE COVER PAGE, AS IT MAY HAVE HERETOFORE BEEN AMENDED, REMAIN UNCHANGED AND IN FULL FORCE AND EFFECT. [END OF MODIFICATION NO. 11] ANNEX 10 PERFORMANCE EVALUATION OF THE EXCELENCE IN INNOVATION ACTIVITY INTERVIEW MINUTES Interview No. 0 – The Excellence in Innovation Activity Location: Premises of The EI Activity , Sarajevo, Federation of Bosnia and Herzegovina, Bosnia and Herzegovina Date: 24.06.2013 Interviewers: Davorin Pavelic. The Evaluation Team Jasmina Mangafic, The Evaluation Team Danijel Hopic, The Evaluation Team Elma Bukvic Jusic, USAID Interviewees: Mr. Jusuf Tanovic, Chief of Party Disclaimer: The Minutes are notes of the Interviewees’ statements and opinions given as replies to the Interviewers’ questions. Recorded statements and opinions do not represent interviewers’ opinion on work and operations of the EI Activity. Introduction Interviewer made a brief introduction about the objective of this visit and purpose of the evaluation process. It was stressed that this visit was informal, aimed at introducing the evaluation team and getting better insight into activities of the EI Project. Interviewees were informed that, upon the analysis of the existing documentation and interviews of the EI Activity clients, the Evaluation team would arrange for the formal interview the EI Activity Project. Description of the establishment and functioning of the EI Activity The EI Activity started in 2008 with the market research phase that lasted for 9 months. It was based on the model developed in Macedonia. That research defined services that would be provided by e-BIZ Centers to Tourism and Wood Processing Sectors. As per the EI Activity Project’s understanding, in ordered for something to be considered as a new technology, it should not be available in Bosnia and Herzegovina during the 2008 market research. Since 2008 ICT sector developed significantly and that had a negative impact on the EI Activity initial design. Example of it is entrance of Booking.com which made Exploring B&H booking function obsolete. However, Exploring B&H should not become yet another tourism agency. Owners of the four e-BIZ Centers are successful business people. In three cases e-BIZ Centers were established by the privately owned mother companies and, in a case of tourism sector, e-BIZ Center was established by the NGO Fondacija e-Turizam1 . All e￾BIZ Centers are established as limited liability companies owned by their respective mother companies (or NGO in case of Exploring B&H). Wood e-BIZ Center is established by Artisan doo, located in the village Medakovo near Tesanj. Textile e-BIZ Center is established by Kismet doo and located in the village Mala Brijesnica in Municipality Doboj Istok. Transport and Logistics e-BIZ Center is established by IT company Statwork from Banja Luka. Tourism e-BIZ Center is established by Fondacija e￾turizam in Sarajevo. These centers are not competitors to the companies from their respective industries. The EI Activity used a cost-sharing mechanism in order to join efforts with private sector in establishing e-BIZ Centers. The EI Activity contributed equipment, software and cash. Equipment and other contributions of the EI Activity are transferred to e-BIZ Centers once they fulfill all their obligations and meet milestones. One of the most important things is that e-BIZ Centers do not become donor hunters. Each e-BIZ Center has Business Plan and Performance Monitoring Plan. In two e-BIZ Centers they completed all cost-sharing arrangements. E-BIZ Centers were doing phone surveys on client satisfaction. The EI Activity performed spot-check of clients satisfaction. Services E-BIZ Centers are offering: production support, capacity building and connection to markets. 1 Fondacija e-turizam does not have employees and does not have a web site. According to information available on internet, the only project that Fondacija e￾turizam ever implemented was Exploring B&H project. http://ngo.ba/moj￾profil/userprofile/minja It is important to stress that all e-BIZ Centers provide services to companies from both entities. For example, Transwork e-BIZ Center signed a contract with Sarajevska Pivara. When Exploring B&H started with tourism web site, not a single one of that kind existed and promoted B&H as tourism destination. There was a similar project financed by the EU, but project failed completely. The EI Activity engaged Karim Rashid, famous industrial designer, to work for Arteco e￾BIZ Center and design a chair for them. Two and a half years after beginning of the project, USAID decided to expand it and open two new e-BIZ Centers. Two new e-BIZ Centers (Transwork and TTC) started with offering services to companies exactly one year ago. Transwork doo paid its mother company Satwork doo to develop software for their services. That was a better option than to buy a licensed software since they used local engineers and they can do in-house updates. USAID Director and US Ambassador in B&H visited Transwork e-BIZ Center in Banja Luka and they were very satisfied with what they saw. Challenges One of the biggest challenges was bringing experts into small villages were e-BIZ Centers are located (Arteco and TTC). ANNEX 11 – TRANSCRIPT - Part of the Interview with Arteco Female voice 1: What does Ismir have to say? Male voice 1: You should have brought another headset… Well, they are carrying this on their books, ledger accounts, but it should not be done that way … [unintelligible] Female voice 1: Them? Who are they? Male voice 1: I can tell you the same about the machine, they are not doing valuation… Male voice 2: In my opinion, it could have been carried off-balance sheet. Female voice 1: No, no, I am asking you then, will you do this… [unintelligible] Two male voices: …According to instructions… Female voice 1: But, why did they ask for this? If this is not carried on the books… do you understand what I am talking about here? Why did they ask for this, if all of this was OK? Male voice 2: Will you have some kind of a… will you be meeting them or something, and discussing things with them, about what your plan is? Female voice 1: You have nothing to be concerned about. Male voice 1: I feel bad about it. Female voice 1: So… Male voice 2: Listen to what the woman is telling you! Female voice 1: No, I’m asking you. So, I will only… [unintelligible]… Since you were the only ones to have received this instruction as of 31 July; it was only you who have recapitalized the story. So, you understand me… [unintelligible] … Why did you receive this instruction, if this was OK? Why? I mean, it’s only logical, there’s no need to tell stories. Why did the center give you such an instruction at all, to remove 120,000 Marks from liabilities overnight and to make a capital increase by some 20 [thousand]? Male voice 2: Look, we weren’t hiding anything … [several persons speaking at the same time, unintelligible] Male voice 1: …but we sent it to you as of 30 June for you to see. Male voice 2: To improve the picture, the condition of Arteco, the coefficients, gearing ratio, profitability ratio, you understand? They were asking for this as of 31 July… So, we were simply acting on instructions and we did that… Female voice 1: OK, it is what it is. If you could now, please, just write down an answer on the side of this: like ‘this is so-and-so’. This was signed at home … These are your assets, this is what you invested … equipment and software? Equipment, 69,000 Marks; this software which should have been around 84 [thousand], OK … the equipment should have been 72,000 Marks, and here I only have some car of 30,000 Marks. Is this so? Male voice 2: Yes. Male voice 1: This dry kiln was not there either, it was not in Arteco’s assets; instead, it was used as a CNC machine, having the right to use it. The manager mentioned that this was a contribution … OK, we can talk about all of this, but it was not put through the books from the beginning. ANNEX 12 – TRANSCRIPT Part of the Interview with Transwork Female voice: OK, now we’ve come to capacity building, and I need your help here. One of the questions is, in which way... and then we’ll come to official questions... this is just for me so that I can have these documents... to have it better documented, because that’s my point. You have filled in 4 questionnaires, right? Since there were some centers that have filled in more forms than they have employees... That will be interesting. The first thing, capacity building. This is you, no, this is Sales and Marketing Director, is that you? Male voice: No, I’m the Company Manager. Female voice: OK, Sales and Marketing Director... the Manager. Now, which trainings have you listed here? I’d like to know whether you’ve attended training sessions provided by IE. This is very important to me now. Male voice: That was this Power of Sale. Female voice: Power of Sale was on 31 March 2011. Male voice: 2011, yes. Female voice: You have not been identified on the list of those who attended the training. Male voice: You know how it is... We went for this option… The seminar was attended by S. P., and then he transferred the knowledge to us, that’s how it was... Female voice: This cost sharing report, IE, that’s that? Male voice: It’s about how to fill in cost share. Female voice: Was that some official training? Male voice: Well, Ismir did that for us... Female voice: So you haven’t attended any training there? Male voice: Well, yes, in the sense that we were sitting there in Sarajevo and he was showing us those things... Female voice: It was an Ismir… Male voice: Ismir Korjenić Female voice: OK... Sales Pitch and Sales Strategy Development... Male voice: This was also from EDC, so we had... Female voice: That training was on 30 August 2010. So, it was not possible, theoretically or practically... Male voice: This was something that Ismir was transferring to us when he was coming to us... Female voice: You could not have attended this in either theory or practice. Male voice: Yes, yes... Female voice: So, this is something that is very interesting. I will now ask you to tell me how this came about. Because if someone from EDC helped you with this, sent these lists... Male voice: We didn’t make it up on our own... Female voice: You know what I’m talking about, then. So, they are just getting you into a mess, so to speak. This is something that EDC was supposed to do. Male voice: Yes, this was their doing and they... Did they attend this, S. P.? I don’t know the details, did he?... Female voice: S. P. attended only once, and [unintelligible] two years ago. And then he sends me a list of people whose signatures are there and they let you fill it in in your office and are helping you with this, you know? Male voice: They have attended this, and then they trained us. I don’t know how this was envisaged. Female voice: Who gave you these lists in the first place? Male voice: Probably the people from EDC... Yes, EDC. Female voice: Because you have the exact titles of trainings that you didn’t attend, and you indicated as if you have attended those trainings. Male voice: I don’t know why this is so. Probably someone made it for this particular topic because it was supposed to be like this... Female voice: You know what, this is very important, because now I cannot take these survey questionnaires to be valid. I have to eliminate them automatically from any analysis, because this is, as we have seen now... Especially because… look, you have been in existence as a center for a particular time. Power of Sale has been indicated as attended by everyone, at the time when the center was not in existence at all. Male voice: As I told you, S.P. attended the trainings then and he transferred the knowledge to us... Female voice: This says how many different trainings you attended, and which trainings those were. The answer is crystal clear and one thing leads to another. The only thing that is important to me is for you to say you have never... Male voice: Well, it is so, I cannot say that we did... I don’t know how it is... Female voice: Here, look at this... Male voice: Well, no... This was Valentina, I talked to EDC; they told us to put this, that we attended both trainings, and that Ismir transferred this knowledge to us whenever he came, and he came often. There, that’s the story... This was done based on what EDC told us to do. I don’t know whether it’s correct or false, right or wrong… Male voice: Because, I’m telling you, he attended those trainings and then transferred those things to us... He trained us separately. Female voice: That is great, but that was Ismir, who have been writing the projects so far. And you didn’t attend trainings, because they sent us documents showing who was in attendance. You understand me? And then I identified your owner as a man who attended this three months earlier... Male voice: And he was supposed to transfer to us what he heard there, and we weren’t there in the first place. Female voice: So, this was 31 March, and the company was founded 6 months after that. And all of you have signed as if you attended the training... you could not have, physically, mentally or theoretically... That’s the problem… I mean, it’s my problem, not your problem... Male voice: It is between EDC and their conscience, because they said that we should include it all, because this was, I don’t know... like, they attended it, and then transferred the knowledge to us... ANNEX 13 – EMAIL EXCHANGE ON SUSTAINABILITY DEFINITION From: Vukmanovic, Dobrila Date: Tue, Mar 26, 2013 at 6:04 PM Subject: Re: Sustainability definition To: Rachel Whitmore Cc: Elma Bukvic Jusic Thanks a lot Rachel Have a nice day. Boba On Tue, Mar 26, 2013 at 5:14 PM, Rachel Whitmore wrote: Hi Boba, In reference to your earlier e-mail regarding the definition of Centers self￾sustainability and the point you raised about determining that Centers are sustainable by the end of the program, I would like to inform you that, in general, we agree that the current definition is limiting and would likely make it difficult to assess project sustainability during the anticipated timeframe of EI evaluation. Hence, we agree that the determination of financial self-sustainability for each e-BIZ Center should include the full review of their year- end financial records, the 2011 and 2012 annual income statements and 2013 semi-annual income statements so the evaluators could be in position to eventually confirm sustainability at the end of program as envisioned in CA and e-BIZ Centers business plans. We assume that the operational sustainability would be determined through the on-site inspection and review of Centers staffing, equipment, organizational structure, procedures etc. and that no further clarification from our end is necessary. We appreciate the partnership with USAID throughout this whole process. Please do not hesitate to contact us with any additional questions. Regards, Rachel Rachel Whitmore |Associate Director, Tech @ FHI 360 O: 202.884.8188 | M: 571.594.1754 | Skype: rachel.whitmore | rwhitmore@fhi360.org | www.fhi360.org | ANNEX 14 – ARTECO TRANSCRIPT CAPACITY BUILDING 1 Female voice 1: They mention some Simon Gilbert and Audrich Hüberts, as two international locals... Male voice 1: This guy, I don’t know, they were paying him... they were paying him... But they say he was very expensive, but didn’t do anything. It was some Englishman who came there, he was strolling there, something... But this was requested by these from Boston, it was by reference from this, what was the name, James... That’s what I know. ANNEX 15 – TRANSCRIPT – Part of the Interview with ARTECO Capacity Building Female voice 1: Now we have an ElvirGazić... Male voice1: Emir Galić, this was the dude that they sent to do the same things we’re doing, but for a longer period, right? He was working on... like a little bit of marketing, sale of services, and such things... It was an attempt;he was the ace up the sleeve of that... lady from USAID... Female voice 1: Oh, this, what’s her name... Male voice1: Boba, it’s difficult to pronounce... [unintelligible] Female voice 1: No problem. ANNEX 16 PERFORMANCE EVALUATION OF THE EXCELENCE IN INNOVATION ACTIVITY INTERVIEW MINUTES Interview No. 3 – eBIZ Center Exploring B&H Location: BBI Center , Sarajevo, Federation of Bosnia and Herzegovina, Bosnia and Herzegovina Date: 29.07.2013 Interviewers: Jasmina Mangafic & Davorin Pavelic Interviewee: Ms. S. V., Director Disclaimer: The Minutes are notes of the Interviewee’s statements and opinions given as replies to the Interviewers’ questions. Recorded statements and opinions do not represent interviewers’ opinion on work and operations of the eBIZ Center. Introduction The Interviewers made a brief introduction about the objective of their visit and purpose and process of evaluation. It was stressed that this visit was the informal, aiming at introducing the evaluation team and getting a better insight into the activities of the Exploring B&H eBIZ Center. Interviewee was informed that, upon the analysis of the existing documentation and interviews with Exploring B&H clients, the Evaluation team would arrange for the formal interview with the Exploring B&H eBIZ Center. Description of the establishment and functioning of the eBIZ Center USAID contributed software, equipment and cash. Grant received for establishment of Exploring B&H was given to eFondacija from Bosnia and Herzegovina and Exploring Macedonia. With received USAID money, eFondacija hired Exploring Macedonia to develop software for Exploring Bosnia and Herzegovina. Exploring Macedonia’s in-kind contribution was software and it was registered as short-term loan to Exploring B&H. Macedonians (Exploring Macedonia) had 49% of ownership and than someone found about it. Ms. S.V. said that she bought theirs and eFondacija’s share of ownership in Exploring B&H through the newly established company Sasa d.o.o., which is still not a formally registered company???. All the arrangements were done by Mr. V.S. and Ms. S.V. has no knowledge about details of transactions. A lot of things went wrong and the entire setup was wrong and Exploring Macedonia and eFondacija Bosna i Herzegovina were not committed to the project. Exploring B&H web site was created in 2009 and since then Exploring Macedonia has not done anything to change / improve / upgrade it. There are three local people involved: Ms. S. V. (Director, Exploring B&H), Mr. V. S. (eFondacija founder), Mr. M. I. (employee). USAID grants are transferred to eFondacija and not to the Exploring B&H d.o.o. Contribution of eFondacija and Exploring Macedonia was in in-kind form and it was labor of Mr. V. S., one of founders of eFondacija. That was agreed between Mr. V.S. and Mr. Sancho (Team Leader). Mr. V. S. is the employee of the B&H Regulatory Agency for Communications – RAC (check his status – employee / civil servant? Check USAID rules about engagement of state agency employees to work on projects?). Mr. V.S. now has 33% of ownership share in Sasa d.o.o. Sasa d.o.o. buy out Contribution (cost share) of founders (eFondacija, Exploring B&H) was registered as a short-term loan to Exploring BH. The EI Activity provided logistic support in the form of car and drivers during the entire duration of project, even when they acted as the tourism agency that organizes seminars for some external business client, Exploring B&H had a problem with collection of accounts receivables so they entered into barter deals. They have no information about the number of tourist / overnights they provided for hotels but they will find out and send that info to the evaluation team. USAID contracted two consultants who held training and performed mini evaluation of the project. They concluded that the web portal was a complete failure. Capacity building Two-day seminar “Power of sale” organized in Sarajevo. They also received assistance from Mr. Richard Benks. He was with them for a month, advising on how to build their web site. Training provided by Exploring B&H to external clients None Exploring’s Clients No information provided. Services Ms. S.V. is of opinion that internet booking is a complete failure. Hotels are not paying any more. The only thing left from the original project are customers’ inquiries but even that does not go through the web site. Customers inquiries are received via e-mail exchange. After situation analysis, they realized they could not compete with Boking.com. They organized 8 fem trips whereby EI Activity provided support. These fem trips were financed with USAID grants. They were also bringing foreign journalist who then wrote articles about Bosnia and Herzegovina. Their visits were paid by USAID grants. She organizes visits and tours for individuals / groups of tourists. USAID FIRMA project is financing the same web site (Navigator – Linden), creating a competition to Exploring B&H. Competitive advantages of Exploring BH No information provided USAID contribution / added value No information provided ANNEX 17 PERFORMANCE EVALUATION OF THE EXCELENCE IN INNOVATION ACTIVITY INTERVIEW MINUTES Interview No. 1 – Arteco doo Location: Premises of Arteco d.o.o. , Medakovo, Federation of Bosnia and Herzegovina, Bosnia and Herzegovina Date: 13.07.2013 Interviewer: Davorin Pavelic Interviewees: Mr. F. C., owner Mr. E. M., Director Mr. M. C. Disclaimer: The Minutes are notes of the Interviewee’s statements and opinions given as replies to the Interviewers’ questions. Recorded statements and opinions do not represent interviewers’ opinion on work and operations of the eBIZ Center. Introduction Interviewer made a brief introduction about the objective of his visit and purpose and the evaluation process. It was stressed that this visit was informal, aimed at introducing the evaluation team and getting better insight into activities of the Arteco eBIZ Center. Interviewees were informed that, upon the analysis of the existing documentation and interviews of the Arteco’s clients, the Evaluation team would arrange for the formal interview the Arteco eBIZ Center. Description of the establishment and functioning of the eBIZ Center Artisan (mother company and owner of Arteco) was established in 2007. Artisan joined the project because its development plan was in line with the EI Activity plan. Artisan is growing thanks to Arteco. At the beginning Arteco represented a huge challenge for Artisan since only big companies can have R&D1 . At the same time, when they signed contract with the EI Activity, they also signed a contract for development of a new furniture collection with a local designer. That contract resulted with the three new products winning prize from the German Institute of Design. After that they established cooperation with two more designers and acquired two additional awards for their furniture (Interior Innovation Award). Presently their furniture is distributed through 160 furniture shops around the world. In the beginning, Arteco started with two employees, presently they employ five (5) people: (1) Director and CNC programmer, (2) CNC operator, (3) administrator and wood draying facility operator, (4) technical documentation developer, (5) prototype developer. All of them have secondary school level of education. When asked for their CVs, they stated they did not have them but they would prepare them for the evaluation team. USAID donated 5-axes CNC machine for wood processing. In addition to that, they acquired 5 computers and licensed SolidWorks 3D modeling software and SolidCAM software. These two make integrated CAD/CAM software. Software licenses and training for Arteco staff that was held in Slovenia (Ljubljana) were paid by Arteco and its cost was about 80,000 BAM. The owner stated they paid for a lot of staff that were useless (ubleha2 ). When asked to be more specific, he did not wanted to provide details. Marketing campaign organized through the project was useless (ubleha) and the owner was ashamed to go and beg companies to use their services. Capacity building • Training for SolidWorks and SolidCAM – Ljubljana – paid by Arteco • Training for SolidWorks and SolidCAM – Zagreb – paid by Arteco • Marketing and sale – two-day training – Sarajevo – organized by the EI Activity • Arteco staff did not participate in financial skills training organized by the EI Activity. It is possible that Mr. Ismar who left soon after establishment of Arteco participated at that training. At this moment they could not recall any direct technical assistance provided by the EI Activity. Training provided by Arteco to external clinets Between 30-40 people received training in use of SolidWorks and SolidCAM. Not all of them are from wood processing industry. 1 Remark: Look into the issue of selection of the EI Activity partners 2 Local word for something presented like it is of great value but actually it does note have any substance Arteco’s Clients They can work only with the ones who value quality and responsibility the same way Arteco does. They have had bad experience with a company from Bratunac (no further elaboration was provided). Designer furniture must be of top quality. They have a lot of companies that are attached to them as their suppliers (about 17 companies). These companies and shops exist thanks to them. For example, there is a small firm that produces metal frames for them and they could close their business if there are no orders from Artisan. They have one carpenter who worked as a village handyman and now, thanks to Artisan, has 5 employees. He is developing prototypes for which they do not need CNC machine. They cooperate with Studio Grupa, Rudjer Novar, Nikola Radeljkovic, (check names) They expected larger interest of BiH companies and designers for the prototype development service. Nevertheless, they have more clients of this type (designers) from Croatia. There are SMEs that do not want to procure services from Arteco because they would rather buy their own machine than to procure service from Fadil (owner). Termag is an example of a good client. They trained their staff, developed technical documentation for them, and they prepared prototype of the hotel room for them (doors, shelves, . cupboard). After that, Termag company furnished 60 rooms hotel by themselves. Services They provide the following services: • CNC machining – they use only 10%-15% of the CNC machine machining capacity for external clients. The rest of CNC machine machining capacity is used by Artisan company. This is the most secure source of revenues. Price of one hour of CNC machining is 90 BAM and it is determined by the EI Activity • Prototype development – sporadic services - the service with highest profitability and price is three times higher than production of mass production pieces of furniture. However, this service does not provide steady revenue inflow • Development of technical documentation – sporadic service • Vacuum lumber draying – service planned for small carpenter shops • Training in SolidWorks and SolidCAM software – sporadic service - every time they manage to get group of five people (number of computers they own). The most important service for the Arteco is prototype development. Competitive advantages of Artisan • They are located close to the EU. • They are developing a prototype for a client from Mexico and he is asking them to produce that product for him. • They are close to harbors, which is important for the export to Asia. They export to China. • They have strong technological advantage over other companies in BiH since their CNC machine is integrated with software. • They are planning to expand and buy one more CNC and keep pace with technological development in wood processing industry. DRAFT – 2/27/14 (v2) EXCELLENCE IN INNOVATION ACTIVITY PERFORMANCE EVALUATION – RESPONSES February 28, 2014 Overview FHI360 and EIA (“Implementing Partners”) welcome this opportunity to provide responses (“Response”) to the USAID-funded Excellence in Innovation Activity (EIA) Performance Evaluation (“Evaluation”). The Response below is divided into two categories: 1) responses to the Evaluation from a technical standpoint, including the EIA’s impact on the economic sectors the project served, sustainability of the e￾BIZ Centers, and capacity building of the e-BIZ Centers and client SMEs; and 2) responses to the Evaluation in terms of its methodology and approach. As part of the Response, the Implementing Partners would like to make the following statements related to the Evaluation and to the EIA: 1) We have been and continue to be focused first and foremost on helping support USAID’s goal of fostering growth in the economy of Bosnia and Herzegovina (BiH) and stimulating innovation in the economic sectors that EIA served; 2) We appreciate the thoroughness of the Evaluation, its lessons learned raised by the Evaluation, and look forward to applying these lessons learned to other similar efforts; 3) We note with concern that the Evaluation makes a significant number of assertions both about the EIA’s technical performance and conduct during the Evaluation process that we believe are inaccurate. We lay out these concerns in detail below. 4) The Evaluators have not provided an analysis of the impact of capacity building provided by the numerous local and international consultants that EIA procured during the course of the project. 5) The Evaluators followed the terms of the EIA Cooperative Agreement (CA) and Life of Project Implementation Plan (LPIP) when they were determining the number of Center clients (insisting on BiH SMEs only); however, the Evaluators seemed to disregard the CA and LPIP elements when defining criteria for assessing Center sustainability. In this respect, the Evaluators used measurements for Center sustainability that were not related to and used through the life of the project as approved by USAID. And, problematically, the Evaluators have never shared the changed definition of sustainability with EIA even after such definition was accepted by USAID and used during evaluation of EIA. Echoing the point above, the Evaluation used measurements of sustainability that were never applied to the EIA from the outset of its design, including the Performance Monitoring Plan / Life of Project 1 Implementation Plan through the life of its implementation. As such, the “goalposts were moved” in terms of measuring a crucial element of the EIA’s success and impact. 6) As it relates to the question of cost share, the requirements for cost share were included in the fixed obligation grants (FOG) issued to each of the organizations in order for them to support the E-biz centers. Cost share was documented and reported in accordance with 22 CFR 226.23.We were perplexed by the evaluator’s remarks. 7) Similar to most if not all development activities carried out over a period of several years with multiple stakeholders across several economic sectors, EIA should have done some things differently. We acknowledge this below and appreciate where the Evaluation raises this. However, the overall tenor of the Evaluation unfairly discounts the role of EIA in generating strong impact in the economic sectors, achieving project goals in terms of driving innovations in the sectors, nor strengthening SME capacity – all core factors of the approved cooperative agreement with USAID. We believe there is ample evidence to demonstrate that EIA achieved virtually all of the project goals and objectives as set forth in the Cooperative Agreement and Implementation Plan, and USAID/BiH’s objectives as pertains to the EIA. In this respect we would like to confirm whether the Evaluators have reviewed all relevant documentation that was provided to them or interviewed all key stakeholders in this process. 8) We are concerned and disappointed that the Evaluation states that EIA interfered with the Evaluation process. We do not believe that EIA interfered in this manner and document below the numerous instances where EIA provided copious support to the Evaluation and also show our difference of opinion about specific claims e.g. that documents were amended or that e-BIZ Center staff was instructed to provide inflated training attendance data. 9) While we believe that the Evaluation approached this evaluation with a set methodology, there are many instances in the Evaluation where the underlying assumptions and approaches used led to inaccurate or misleading conclusions or statements. We document these below. 10) During the course of the evaluation, EIA diligently provided supporting materials, data, and actively engaged with USAID throughout this process. However, we feel that there are several areas where the Evaluation missed opportunities to fully engage the knowledge of the project and the beneficiaries. Additionally, EIA feels that a more fully articulated evaluation should have been provided per the USAID Evaluation Policy. 11) The Evaluators have never shared their overall methodology with EIA regardless of the fact that disclosing evaluation methodology to the subject of evaluation is one of the pillars of the evaluation process. 12) The Evaluators did not interview key project personnel (EIA Business Development Manager and the two Chiefs of Party. We believe that failing to do this led to an incomplete understanding of EIA. 13) According to the verbal feedback and written statements from e-BIZ Centers, the project beneficiaries felt that the Evaluation process was more like an audit than an Evaluation interview, and that statements by the Evaluators appeared to indicate a bias to portray the EIA and its staff in a negative light. The beneficiaries reported being extremely uncomfortable with the aggressive manner in which questions were apparently posed to them, and with apparently sensitive information about EIA staff that was also discussed with them. This approach appears to contradict the principles of USAID’s Evaluation Policy that local beneficiaries should be treated with respect. Please refer to the Appendix for statements testifying to this from the e-BIZ Centers. 2 Responses – Technical and Performance Components Overview of Evaluation Comments – Technical and Performance The Evaluation took strong issue with the inherent design of the e-BIZ Project, particularly related to the relationship between parent company and the e-BIZ Center, and the level of outreach of the e-BIZ Centers to the SMEs. It is implied as well that through this design and relatively low levels of capacity building by EIA, the SMEs did not benefit significantly from this project, nor the sectors themselves, including in terms of innovation introduced into the sectors, The Evaluation implies that the design and way in which the EIA was carried out also did not enable the e-BIZ Centers to be sustainable in the long run. We strongly disagree with this assessment. Nevertheless, before providing a point-by-point response to the Evaluation along these lines, the implementing partners would first like to rebut these claims by drawing on an assessment of the e-BIZ Centers that was carried out by David T. King of Cardno Emerging Markets of EIA in December 2013, and which was funded by EIA. The implementing partners believe that the Evaluators should examine closely the findings from this assessment and refer to them when making revisions to the draft Evaluation. The full assessment is included as an annex. An excerpt of these assessment findings are as follows: Key Findings and Conclusions i. Grantees benefitted substantially from the receipt of the systems. There was one potential risk in the basic design of the Project, which was clear from the beginning. The proposals would necessarily need to come from existing companies, since only they would have the experience in their sectors to understand and employ the proposed ICT-based systems effectively. This meant that there was always a certain probability that the systems could be used primarily for the benefit of the grantee company, rather than prioritizing services to other SMEs in the sector. To try to prevent this, the grantee companies were obligated under their contracts with EI to serve other SMEs – with a target of 40-50 for each center – and they understood this commitment. ii. Three of the four e-BIZ centers have performed in the manner expected, with positive development impact on their respective sectors. This means that that these centers – Arteco, TTC, and Transwork – have successfully marketed their services to SMEs in their sectors, that each has built a clientele, that this clientele is expanding, and that their revenues are significant, and for TTC and Transwork strongly growing. iii. The impact of the e-BIZ centers on their sectors is likely to continue to grow. The sales, client numbers, and third party revenues trends for Arteco, TTC, and Transwork are generally positive. The evidence of repeat customer business indicates that services provided are of ongoing value to many outside clients. For both Arteco and TTC, the services that appear to be most useful to third party SMEs involve prototyping – using IT systems to create production designs and specifications for manufacturers that fulfill buyer needs, maximize efficiency in material use, and optimize labor and production equipment use time… 3 iv. At least three of the four e-BIZ centers are likely to be sustainable. The question of sustainability overlaps to an extent with the immediately preceding discussion as to the likelihood of future demand growth for the centers’ services. Sustainability in addition requires that the e-BIZ centers are well￾managed and profitable. It is important to emphasize that the managers of Arteco, TTC, Transwork, and even Exploring all consider their centers to be attractive businesses in their own right and are committed to their expansion, separately from their parent companies v. The fourth center has failed mainly due to global developments, but may still have a future. Exploring is clearly the most problematic center. The basic cause is the very rapid growth of globally￾provided internet-based services, whose underestimation has been a hallmark of the information age. Exploring was seen as a critical need for the development of the tourism sector at a time when there was no booking service for hotel rooms in BiH. The grant was provided to deliver a service that did not exist. However, very soon after this center was launched, it was overtaken by very large booking services – Expedia, Travelocity, TripAdvisor, and especially Booking.com. ************** Before proceeding to the point-by-point response, the EIA must stress that it believes that differing views on several core project postulates are the cause for disagreements on number of evaluation’s findings and conclusions. They are as follows: • EIA believes that at least two, if not all four Centers should have been treated as bona fide startups during this evaluation. It is apparent that Evaluators have failed to treat them as startups during their analysis of Centers’ financial and operational sustainability. • EIA believes that owners have been and still are the pivotal staff members of respective Centers regardless of the fact that they have not formally registered as Center employees. This is an important point since this is the evidence provided in the report of training attendance misreporting. • As it relates to the question of cost share, the requirements for cost share were included in the fixed obligation grants (FOG) issued to each of the organizations in order for them to support the E-biz centers. Cost share was documented and reported in accordance with 22 CFR 226.23.We were perplexed by the evaluator’s remarks. • Transwork’s close integration with Satwork and reliance on its technological support is logical and desirable from a startup business development standpoint and certainly expected from the planning perspective. The Evaluators assert that they do not believe that such integration is desirable but does not provide a substantiated reason for this assertion. • ARTECO’s prototyping services for foreign clients represents a pivotal indirect support to the BiH wood processing sector as these services complete the crucial precondition before those foreign clients could proceed to give manufacturing orders to local SMEs. The Evaluators view the recent dominance of such foreign clients in ARTECO’s revenue structure as a negative development. 4 • The late launch of Transwork and TTC that was caused by external factors, including AED’s suspension that resulted in a delay in issuing the FOGs for the textile and logistics e-BIZ centers. This should have been noted by Evaluators. The fact that these two Centers were operational for less than 16 months at the time of evaluation does not appear to be considered by Evaluators when they were assessing the development progress and achievements of these two Centers. Point – by – Point Response to the Evaluation Comments - Evaluation (p.4): “Each of the e-BIZ centers had a different impact on the competitiveness of theSMEs within their respective industries. Yet when considering the increased competitiveness of the SMEs the only certain thing is that the EIA significantly raised the competitiveness of e-BIZ centers’ parent companies. The EIA distorted the market in some of the industries; for example, in logistics and transportation by supporting one of the numerous companies that provide a GPS monitoring and tracking service.” EIA Response: The suggestion that EIA distorted the market in the BiH logistics and transportation sector needs to be reconsidered, and it is inaccurate to imply that that the primary focus in this sector was to support only the provision GPS monitoring and tracking service. In fact, the primary sector development assistance that EIA helped introduce to the BiH market was creation of a comprehensive transportation & logistics support platform with an array of ICT based services including the first domestic online freight exchange; in the case of the e-BIZ Center (Transwork), GPS services was one of many services. More broadly, the Evaluators are urged to be more specific about the instances of “market distortion” in the case of EIA and clear about how “market distortion” ran counter to the project design (approved by USAID) to introduce new innovations into economic sectors by working with specific businesses as catalysts for driving these new innovations (the e-BIZ Centers). ************** - Evaluation (p.4): “The E-BIZ center Arteco made quite an important contribution to the wood processing sector by creating an outsourcing opportunity for SMEs in the development of product technical documentation and prototypes. However, since the design of new products is not so frequent in BiH present demand amongst SMEs within the wood processing sector for this service remains low. Greater demand for prototype development came from foreign customers.” EIA Response: EIA regrets that Evaluators have failed to recognize Arteco’s strong efforts to educate and train local SMEs on importance of the product design for their overall competitiveness. Although, the demand from local SMEs is still low, it is steadily improving in recent years. More importantly, the Evaluation does not consider that “foreign customers” that have developed prototypes at ARTECO are, in vast majority of cases, customers that were about to place significant manufacturing orders from a variety of BiH producers, including SMEs other than Artisan/ARTECO. The high technology product prototyping at ARTECO was necessary before these orders could proceed. Hence, ARTECO’s servicing of the “foreign customer” for prototyping actually represents indirect servicing of local SMEs i.e. by providing a critical, previously non-existent R&D function. ************** - Evaluation (p.4): “Almost 90% of BiH textile companies work for foreign customers on basic cutting and sewing operations based on received patterns; however, ICT capacities in the majority of these companies are non-existent. Therefore, if adequately mentored and managed, 5 TTC could become the ICT resource center for many BiH textile companies presently without any ICT capacity. Unfortunately, after being operational for almost two years TTC’s outreach remains low and it continues to use its resources mainly in support of production in its parent company.” EIA Response: The Evaluation’s statement about TTC’s length of operation is incorrect; therefore the ET’s conclusion about low outreach should be reconsidered. At the time of evaluation, TTC was in operation for only 15 months as their official opening day was April 20, 2012. Furthermore, the Evaluators were informed that TTC slowed their initial outreach efforts until all early phase technology absorption issues have been resolved. TTC wisely recognized that their SME market is small and that any inception phase processing error on client’s fabrics could have had devastating effect on their reliability reputation and by default on the outreach. ************** - Evaluation (p.4): “EIA management of the M&E system was poor. The EIA invested little or no effort to establish baseline data or track changes in the competitiveness of SMEs within the targeted industries….” EIA Response: EIA invested considerable effort in creating baseline data through detailed and competent industry and market analysis reports that were developed in accordance with the USAID approved Life of Project Implementation Plan (LPIP) and with help from industry experts. Eight reports were developed in total, two for each of the four targeted sectors. These reports were shared with the Evaluation Team. Whether such baseline data was the most appropriate for this project is a matter of debate and we are ready to discuss this further with USAID as part of the “lessons learned” exercises. ************** - Evaluation (p. 5): “Many factors prevented the buildup of the e-BIZ centers’ top quality teams. The first relates to insufficient attention and effort to provide training for e-BIZ center staff in ICT and ICT applications in specific industries. Having in mind that e-BIZ centers are supposed to be centers of excellence for the dissemination and transfer of new high impact information and communication technologies, the EIA has done too little to build e-BIZ center staff knowledge and skills in the application of high impact ICT…” EIA Response: The reference to the first factor implies that, by the time of this evaluation, Centers have not become platforms for dissemination and transfer of new high impact ICT and that staff did not build necessary knowledge and skills which is not accurate. In fact, the three Centers (ARTECO, TTC and Transwork) have substantially achieved that goal as they currently offer the top of the line ICT based services for repeated customers, a sure sign that their teams have a top quality technical expertise. This expertise was developed through equipment manufacturer’s trainings and EIA provided consultants which in our view, is an acceptable mix. ************** 6 - Evaluation (p.6): “The TTC and Arteco e-BIZ centers were the only ones to build technology related capacities.” EIA Response: We disagree with this statement as we believe that the Transwork also managed to build strong technology related capacities. This was achieved through outsourcing the development of technological aspects of Transwork’s operations to the closely connected parent firm, which is fully in line with the approved business plan and expectations from EIA. ************** - Evaluation (p.6): “However, the number, frequency and staff coverage of the group trainings/seminars organized by the EIA could have been higher. A few organized group seminars related to marketing and financial management were organized; however, Transwork and TTC staff did not attend any of these marketing and financial management trainings and the EIA did not make any effort to repeat them after the two new e-BIZ centers had been established and included in the Activity.” EIA Response: To the best of our recollection, EIA informed Evaluators that, following the execution of the initial group trainings, the e-BIZ Centers asked for the more specific, on site, capacity building assistance. Based on the partner feedback, EIA decided to focus more on providing expert consultants and that explains the relatively small number of trainings delivered. Furthermore, the statement that “Transwork and TTC staff did not attend… trainings” is only true if we accept the premise that owners/founders were not part of Center’s staff which in our view, and in reality, was not the case. The founders have participated in EIA trainings and they were expected to apply and transfer the acquired knowledge during their Centers’ inception phase which effectively lasted until the end of EIA. ************** - Evaluation (p.7): “Operational management - The Evaluation Team could not confirm any capacity building assistance directly targeted at improving managerial skills or in adopting and integrating technology in order to gain maximum efficiency and productivity. The Evaluation Team was also unable to confirm any capacity building assistance directly targeted at improving knowledge and skills in other management areas i.e., strategic planning and human resource management.” EIA Response: For example, EIA provided SOWs for local and international consultant support given to TTC in areas of planning, management, integrating technology and production efficiency. EIA staff also participated directly and provided extensive HR support to all four Centers during their staff recruitment. Moreover, additional information on this capacity building assistance could have been provided to the Evaluators e.g. during interviews but the Evaluators never held interviews with EIA Staff who could have provided this information –namely the Business Development Managers (BDMs) or the two Chiefs of Party (COPs). ************** - Evaluation (p.7): In the case of Arteco, the EIA, on its own initiative, submitted new financial reports the following month that covered the month of July 2013. These two financial statements were fundamentally different and indicated structural changes in business operations. If, for example, we 7 look at the structure of liabilities and more specifically the liabilities towards the related legal entities1 we can see that on June 30 2013 they amounted to 118,000 BAM. Looking back through the previous annual financial statements the Evaluation Team determined that this amount of liability had existed since the beginning of the Activity. The parent company, Artisan, created this liability. Instead of contributing capital to Arteco, as it was obliged to according to the Fixed Obligations Grant, Artisan gave money to Arteco in the form of a loan…” EIA response: The above paragraphs clearly show that Evaluators are misunderstanding and, more dangerously, misinterpreting key provisions from Arteco’s Fixed Obligation Grant (FOG). Artisan was not obliged to provide capital to Arteco according to the FOG. Artisan was obliged to provide in kind and cash contribution for the operations of Arteco. Cash was indeed provided and used for various payments necessary for Arteco’s operations. Artisan decided to record that contribution as the loan to its daughter firm, which is a very common practice in BiH. EIA did not object to such recording of this cash contribution in Arteco’s financial statements as the mode of recording was not relevant to the implementation of EIA and, more importantly, for the assessment of Center’s sustainability per the approved Life of Project Implementation Plan (LPIP). EIA’s main concern was that cash was provided and spent in accordance with FOG requirements, and most importantly on development needs of Arteco. ************** - Evaluation (p. 8): “If the parent companies are taken out from equation then both Arteco and TTC are not financial sustainable. TTC generates between 80% (2012) and 90% (2013) of its total revenue through services provided to its parent company, Kismet d.o.o, and to Vocal Productions d.o.o. Arteco generates about one-third of its total revenue by providing services to its parent company, Artisan d.o.o” EIA Response: We disagree with the need to take the parent company out of the equation. The approved EIA project design accepted that such reliance on parent company will likely happen in early phases of e-BIZ Center development. Please refer as well to the assessment conducted by Cardno related to this point. And the Evaluators acknowledge this point in this very paragraph -- namely that ARTECO’s reliance is at the one third while TTC’s reliance is much higher which demonstrates that the mature Center (ARTECO) progressed to the point of lesser reliance. ************** - Evaluation (p.8): “All of the e-BIZ centers, with the exception of Transwork, had poor financial stability. Other types of financial analysis, such as Z-Score and the Kralicek Quick Test, indicated that all of the e-BIZ centers, except for Transwork, had issues with financial stability.” EIA Response: We question that the Evaluation’s assessment that Centers had poor financial stability per this statement. The Evaluators should be clearer on how these financial analysis tools are applicable to startup businesses. (see also response related to Evaluation methodology in subsequent section). ************** 1 Since Artisan and Arteco should, in line with the Project, be two independent legal entities and not a parent￾child business this method of accounting entry is also incorrect. 8 - Evaluation (p.8): “Transwork is the only e-BIZ center that achieved its targeted number of clients. TTC did not even manage to achieve 50% of its targeted number of clients. According to data received from its accountant, Exploring BiH did not manage to achieve the targeted number of 70 clients. Finally, Arteco served many clients but only 38 SMEs from Bosnia and Herzegovina.” EIA Response: It is not clear what constitutes the basis for the Evaluation’s determination about Centers not achieving the targeted number of clients. For example, we are not sure how the data received from Exploring BH’s accountant failed to provide evidence for all SME clients. During the project implementation, EIA obtained all bank statements for Exploring BH for the period 2009-2012 and they clearly show payments from Exploring BH to more than 130 local SMEs (majority hotels) that in practically all cases mean that these SME clients have benefited from Exploring BH’s processed bookings. Furthermore, more than 60 BiH SMEs have been served by Arteco at the time of the evaluation according to Arteco’s Client Relationship Management (CRM) system – in other words, achieving the project objectives in terms of meeting the numbers of SME clients served. The CRM software was developed through the support from EIA for Arteco and Transwork and this system allows for the comprehensive tracking of all services provided to each client as well as tracking specific phases for each service/project delivered including meetings, communications etc. EIA regrets that Evaluators have failed to note and examine this important client management tool. Finally, even the TTC achieved the targeted number of clients albeit in the final weeks of the EIA project. However, in our view, this is understandable considering the late start and technology absorption issues at this rural Center. ************** - Evaluation (p.9): “The majority of services provided by the e-BIZ centers were not new or non￾existent in the market. All of the services provided by TTC were either already present or can be done manually. The main contribution of TTC’s services to some of its clients came in the form of precision tailoring and faster production. Prototype development was the only unique service provided by Arteco. Transwork services can be provided by a dozen other service providers, while Exploring BiH is perceived as just another small travel agency. The results of the SME questionnaire survey also confirmed that although awareness of the services provided by the e-BIZ centers was high the majority of SMEs did not consider these services to be new or previously non-existent in the market.” EIA Response: The above statement suggests that comprehensive, “one stop shop type”, ICT-based, business support services platforms were already present in BiH wood, textiles and logistic sectors prior to the EI intervention. It is true that many of the Centers services existed in sporadic and closed business systems before EIA intervention. However, these services were never compiled into an accessible business/manufacturing support hub. ************** - Evaluation (p.9): “Financial data on changes in the volume of sales of SMEs that received services from the e-BIZ centers does not constitute the basis for the conclusion that the e-BIZ centers improved the competitiveness of their clients. In all four sectors almost half of the SMEs that received SME services experienced a decrease in their total volume of sales or did not experience any change in total sales volume.” 9 EIA Response: The financial data time range listed in the draft evaluation report should not be considered sufficient for conclusions of this type. Please see the subsequent section for additional elaboration on this point. ************** - Evaluation (p.9): “The Evaluation Team could not find evidence that the e-BIZ centers had an impact on associations. One of the reasons for this could be the attempt by the e-BIZ centers to use the trade associations as a marketing channel through which to sell their services. In addition, it might be the case that the associations and the public sector alike perceived the entire project as a private company endeavor.” EIA Response: We believe that e-BIZ Centers had an impact on associations – however, the PMP did not articulate a methodology for tracking and assessing this impact. A lesson learned is that such tracking is necessary. From an impact perspective, however, this impact was very strong in terms of helping change mindsets about application of latest technology and focus on product development as the basis for SME competitiveness. For example, the e-BIZ centers have collaborated with associations in promoting introduction and use of advanced technology (TTC) and joint collaborative efforts to help local producers access new markets (Arteco). This information was evident from project’s quarterly and annual reports that were shared with Evaluators. We are also concerned that the Evaluators have not obtained the proper source of information pertaining to the impact on associations. The report (Annex III) only provides (under Persons Interviews heading) one bullet stating “Representatives of Business Associations”. No sufficient evidence is provided for concluding that associations perceived the entire project as a private company endeavor. ************** - Evaluation (p.17): “The EIA provided lists of all organizations and individuals that had received services from the e-BIZ centers; however, these lists contained many individuals, organizations and companies that could not be considered as e-BIZ center clients, as specified by the EIA’s documents.2 We found a large number of individuals, secondary school students, furniture designers and foreign companies in these lists. The Evaluation Team used the following criteria to select SMEs from these lists for interview: balanced territorial distribution (two entities, different parts of BiH), an almost equal number of SMEs from all targeted industries and SMEs of different size (micro, small and medium).” EIA Response: The statement above is misleading as it suggests that all lists from e-BIZ Centers included the non-BIH SME clients. Only the list provided by Arteco included non-SME clients but these clients could have been easily identified and separated. As evident from the footnote the Evaluators are confusing e-BIZ Center clients from EIA project beneficiaries. The e-BIZ Centers had to keep lists of all clients served. The Evaluators requested the lists of all Center clients, not the refined lists that includes just locally registered SMEs. This was explained to the Evaluators during the meeting with them. ************** 2 All Activity documents specify that the beneficiaries of the Activity must be SMEs from Bosnia and Herzegovina. Furthermore, as defined in the Annual Report for the period January 2011 – December 2011, the unit of measure for the number of e-BIZ center customers is “the number of customers paying for e-BIZ center services”. 10 - Evaluation (p.18): “For question 2, the Evaluation Team evaluated both the financial and operational sustainability of all four e-BIZ centers. As defined by the Cooperative Agreement, financial sustainability is achieved when an e-BIZ center is able to deliver products and services to the market at a price that covers its expenses and generates a profit. In order to determine financial sustainability based on the above-mentioned definition the Evaluation Team analyzed the financial statements of the e-BIZ centers. However, since the above-mentioned definition of sustainability is extremely narrow and is used primarily as a cost-accounting tool that only covers a point in time it would not provide information on the ability of the e-BIZ centers to sustain the provision of the Activity’s benefits in the long run. Therefore, the Evaluation Team proposed a different approach and introduced a new balanced score card method along with new financial indicators to measure financial sustainability. The latter were accepted by USAID as part of the approved Evaluation Work Plan.” EIA Response: Please see the subsequent section on responses to the Evaluation methodology and approach. However, as it relates to the technical characteristics of the e-BIZ Centers in terms of performance, please note first that the definition of sustainability that was used above is not the one shared with EIA or its partners, so we were unable to provide evidence to support this definition. Moreover, please note that in August 2013, the Implementing Partners expressed concern in writing to the Evaluators that the Evaluation was using a different standard of sustainability than that which was used by the EIA since its inception (noted in footnote #5 of the Evaluation on p.7) As the letter of concern noted: “Recommendation: We respectfully suggest that the Evaluators and USAID use the standard of sustainability contained in the EI Life of Project Implementation Plan / M&E section and the Annual Work Plans or at least explain more fully why a different definition of sustainability is being used. This is the standard of sustainability that the EI project has been using for the past five years. It is also a reliable indicator in that it is the same as the one that was used for the e-BIZ Macedonia project, which established Centers that have proven to be sustainable. We would be happy to provide the PMP for the e-BIZ Macedonia project to substantiate this. The e-BIZ Centers have been successful not only in achieving this sustainability target, but they have met or exceeded all of their objectives in terms of generating in-kind investment, serving SME clients, and perhaps most importantly, catalyzing innovation in their respective sectors. To suddenly adopt a new definition of sustainability in 2013 is both to unfairly “move the goal posts” at project end and to obscure the developmental impact that the e-BIZ Centers have made. ************** - Evaluation (p. 26): „Through the interviews that were conducted, it became obvious that EIA staff as well as e-BIZ center management were not clear which clients they were supposed to serve despite the fact that this is clearly defined in the Annual Performance Report. During the interviews the definition of ‘client’ ranged from those individuals, organizations and companies that visited the e-BIZ center website to those that appeared on the e-BIZ center websites to those that visited the e-BIZ centers during the Open Day event.” EIA response: As the e-BIZ Centers are designed to serve as inclusive sector development platforms for targeted sectors they have invariably attracted individual clients (e.g. furniture designers) and foreign clients, as they needed prototypes for manufacturing orders that they were placing in BiH. EIA regrets that our sincere attempt to convey these important developmental considerations and 11 logical attempts by Centers to broaden customer base was misconstrued by Evaluators as not knowing which clients Centers were supposed to serve. ************** - Evaluation (p.27) FINDINGS – (Related to QUESTION 1: To what extent, in a measurable way, has the EI Activity succeeded in capacity building for e-biz center staff?) EIA response: Throughout this entire section ET fails to mention the involvement and on site capacity development assistance that was delivered to e-BIZ Centers by the number of EIA-procured local and international consultants. EIA provided 37 consultants on more than 50 assignments to support e-BIZ Centers with the focus ranging from planning and management to production and marketing. EIA believes that this oversight, whether by accident or on purpose, results in an incomplete picture of the support that EIA provided to Centers. ************** - Evaluation (p. 27) „Throughout the life of the activity, the EIA organized three group trainings: (i) a two-day Power of Sales seminar, (ii) a two-day Financial Management training and (iii) a one-day BiH Entrepreneurship and Business Development training. According to the attendance lists provided by EIA, the trainings delivered and staff trained were as shown below.” EIA Response: EIA disagrees with the Evaluators’ calculation of the number of staff trained as EIA believes that Center owners must be included as staff trained regardless of the fact that they are not formally registered as Center employees. Center Founders were integral to the setup of each Center as all of them served for many months of the inception phase as the regular Center staff. Furthermore, it is inaccurate to compare the number of staff trained e.g. in 2009 against the total number of all staff in the four Centers in mid-2013. The fair and proper comparison would be between the total number of trainees and the total number of staff at the two existing Centers at the time of delivery of listed trainings. ************** - Evaluation (p. 27) „The e-BIZ centers were never properly instructed or trained in how to track and record cost-share arrangements in accordance with the approved e-BIZ centers’ business plans. Because of this the e-BIZ centers financial reports do not give an accurate picture of the e-BIZ centers financial position.” EIA response: The Centers were given precise instructions from EIA related to documenting founder’s contribution to the e-BIZ Centers. Under the FOGs (see TTC FOG, Attachment C, Milestone/Invoice templates) the e-BIZ Centers were obliged to submit cost share documentation/evidence to EIA for review & approval including “receipts, original timesheets…” and estimated values of in kind software and equipment etc. Therefore, the Evaluator’s expectation that all or most of the cost share should have been entered in Centers’ financial records is without merit. Regrettably, this expectation and the Evaluators’ incorrect view of the process resulted in a series of inappropriate conclusions findings including the crucial statements that the Evaluators were unable to verify partner’s cost share. EIA delivered hundreds of documents, timesheets and receipts to Evaluators confirming that the founders’ contribution was documented and filed in compliance with FOG stipulations. 12 ************** - Evaluation (p.29): „However, information provided by one of the e-BIZ centers about the training turned out to be different from the attendance list provided by the EIA. When asked why they claimed that their staff had attended trainings organized before the e-BIZ center in question had been established, the Director of this e-BIZ center answered that they had been instructed to do so by the EIA because the EIA did not have a sufficient number of capacity building activities.3 EIA response: EIA assumes that the example listed on page 29 refers to the Transwork Center. EIA, in line with its principled position, considers the participation by Transwork’s owner at EIA training that was held during preparations for Transwork’s formal establishment as the capacity building for Transwork’s staff. This EIA position was communicated to Transwork’s current Director in much the same way as in this part of our response. Again, we are happy to reiterate that EIA believes that training to Center owners must be considered as capacity building for the Center regardless of the fact that owners have not formally registered as Center employees. Center Founders were integral to the setup of each Center and they served as regular staff for many months of Centers’ early development phases. EIA believes that our view of this matter firmly corresponds to the reality and, in essence, each owner has the formal link to their respective Center. ************** - Evaluation (p.32): “The majority of indicators signaled significant structural changes in Arteco's business operations over the period 31.12.2012 to 31.07.2013 yet they were not the result of changes in the center’s business operations but rather the result of an exercise in financial engineering. Liquidity improved significantly primarily due to cash transfers from the parent company Artisan to Arteco’s account.” EIA response: We believe that the transaction that is referred to on this page is an overdue payment from the parent firm (Artisan). The project does not understand why this considered “financial engineering” and would request that this statement be stricken from the report unless there is additional justification to suggest impropriety. ************** - Evaluation (p.33): “It was difficult to assess Arteco’s financial position due to the strong business connection with its parent company Artisan.” EIA response: Evaluators do not provide sufficient rationale as to why did the “strong business connection to its parent company” made assessment of Arteco’s financial position difficult? ************** - Evaluation (p.34): Arteco has not achieved the financial projections and capital target values as specified in the business plan. Financial projections set the value of Arteco’s capital in 2009 at 3 The interview conducted with the Director of the e-BIZ center was audio recorded. 13 711,720 BAM; however, on June 30 2013 the total value of Arteco’s capital was 2,000 BAM. Arteco has not achieved the target values for current and long-term assets set in the business plan. EIA response: The target values for long term assets included the machinery provided by EIA. However, Arteco could not have registered the value of this machinery in its official books as the title was with EIA until the completion of the EIA project. ************** - Evaluation (p.35): “As stated during the interview with the Director of Exploring BiH, in the early stages of the project it was already clear that the website/portal was a failure. The explanation given by EIA staff for the reasons behind this was that at the time the project commenced there was no booking website in BiH and nobody was able to foresee what would happen in the area of ICT within the tourism sector (booking.com etc). However, within the Market Analysis Report, the EIA recognized that, “A large number of booking agents are active in BiH but three - www.booking.com, www.venere.com and www.all-hotels.com - are dominant. The EIA also learned that several sites, including www.ljeto-zima.com and www.sobe.ba, offered detailed accommodation listings with information, photographs, prices and e-mail links for reservations through the hotel’s own websites.4 ” EIA Response:. At the time of the project inception there were no BiH sites that have offered online booking option and even the local sites listed in the Market Analysis (ljeto-zima.com and sobe.ba) have not offered that option. As the EIA Market Analysis report stated, sites such as booking.com were active in BiH at the time as they were signing up their first hotel clients. In less than two years following the EIA Market Analysis, the booking.com managed to overtake most local or international competition in BiH so EIA observation about this matter is still valid in our view. ************** - Evaluation (p.36): “From the very beginning up until the present, the Exploring BiH portal has not been sufficiently technologically advanced to allow for direct booking via the portal. Bookings were made via e-mail communication with the Director of Exploring BiH and very soon, it became clear that Exploring BiH could not compete with ‘Booking.com’.5 There is a link to the booking.com website at the Exploring BiH web site, However, that cannot be considered as a provision of possibility for direct booking.” EIA Response: Exploring BH’s portal was sufficiently technologically advanced and tested to allow the full online booking as early as fall 2009. While the actual number of online bookings was low, it is undeniable that the technological capacity existed for online booking for many years under EIA until the site was redesigned to include another booking option. And the Evaluators should have noted that the current link to the booking.com is based on the franchise type cooperation that was recommended as the course of action for Exploring BH by an international expert Jain Abishek. ************** - Evaluation (p.41): “The e-BIZ centers' financial reporting systems were inadequate; the e-BIZ centers were never instructed in how to record the USAID and parents company contributions 4 Market Analysis of the High Impact ICTs as e-BIZ Center Core Offerings in the Tourism Sector 5 http://www.booking.com 14 in financial statements. The Evaluation Team was not in a position to verify the amounts and structure of the cost-share contribution of the founders of the e-BIZ centers.” EIA Response: The above statement is not accurate as Centers received were instructed by EIA on how to document and evidence their in-kind contributions to the respective Centers per requirements from their respective FOGs. The Evaluators have received all documentation related to this, including copies of bank statements, timesheets, equipment appraisals etc. and that should have been sufficient for verifying the in￾kind contributions, in line with FOG provisions. And as it relates to the question of cost share, the requirements for cost share were included in the fixed obligation grants (FOG) issued to each of the organizations in order for them to support the E-biz centers. Cost share was documented and reported in accordance with 22 CFR 226.23.We were perplexed by the evaluator’s remarks. ************** - Evaluation (p.41): “Arteco provided services to 38 BiH SMEs over a period of 4.5 years. Out of this number, 28 companies received services only once over the 4.5 year period. Out of the 38 SMSs 13 received services of a value of less than 500 BAM (USD 356.65).” EIA Response: It is not clear how the Evaluators have determined this number. Arteco’s record from the EIA developed Client Relations Management (system) show that Arteco delivered paid services to more than 60 SMEs at the time the Evaluation was conducted. ************** - Evaluation (p.42): “TTC provided services to 19 SMEs over a period of 1.5 years. Two of these SMEs were companies owned by the owner of TTC, Kismet d.o.o., and Vocal Productions d.o.o. The services provided to these two companies generated 79.34% of TTC’s total revenue in 2012 and 90.73% in 2013. Out of 19 SMEs five (5) received a one-time service of a value of less than 500 BAM (USD 356.65) each.” EIA Response: The above number (19) is not correct as it only included TTC clients served by spring 2013. TTC was getting new clients on board until the end of EIA (August 2013) following extensive promotion efforts in late spring and early summer 2013. This was the natural pace for client acquisition considering the late start for this Center. And by the end of the project, 40 clients were being served by TTC, which meets the project objectives. ************** - Evaluation (p. 42): “The Evaluation Team was not able to determine the exact number of clients of Exploring BiH. The Evaluation Team received several diverse lists of clients served by Exploring BiH. The first list listed 48 clients, the second listed 71 clients, the third was called the “frequent clients list” also with 48 clients and, finally, the fourth list received from the Exploring BiH accountant had 45 clients in total. The Evaluation Team knows for a fact that even if a client had a contract with Exploring BiH that does not mean that a business transaction happened between them.” EIA Response: We are not sure what type of list was solicited and received from the Exploring BH’s accountant. Nevertheless, EIA collected copies of all bank statements from Exploring BH for the period August 2009 – December 2012 and during that period Exploring BH made payments to or received commission fees from more than 130 BiH SMEs All of these transactions were done for 15 Exploring BH’s executed bookings of hotel accommodation or other tourism services. If necessary, EIA is prepared to share that evidence with USAID upon request. The explanation for the list with 71 clients is simple as it included only SME clients that have signed contracts or forms that have formalized their long term relations with Exploring BH. EIA was insisting on contracts and our list was stricter in what it recognized as Exploring BH’s clients especially compared to the actual large number (130+) of sporadic paying clients that could have been extracted easily from Exploring BH bank statements. , ************** - Evaluation (p. 47): “The financial data concerning changes in the volume of sales of SMEs that received services from the e-BIZ centers does not constitute the basis for the conclusion that the e-BIZ centers improved the competitiveness of their clients. Almost half of the SMEs in all four sectors that received SME services experienced a decrease in the total volume of sales or did not experience any change in the total sales volume.” EIA response: It is unclear to why the data on year to year overall sector sales volumes was not compared against sales volumes for SMEs served by e-BIZ Centers as that would present a clearer picture whether SMEs served by e-BIZ Centers have performed better than their respective sectors at the time of the global recession. ************** - Evaluation (p.49): “There was a lack of involvement and support from the public sector. The EIA and e-BIZ centers failed to gain interest and support from public sector organizations and institutions. Therefore, the entire effort and burden of advancing all four target industries was placed with small companies struggling to achieve the brake-even point.” EIA response: EIA a a fairly small, USAID funded project, was not operating in the vacuum. Other, larger USAID projects (CCA, FIRMA) were taking the lead in mobilizing the public sector support for advancing targeted sectors such as wood and tourism. SEE NEXT PAGE FOR RESPONSES TO EVALUATION METHODOLOGY AND APPROACH 16 Responses – Evaluation Methodology and Approach Overview of Evaluation Comments There are two broad concerns that the implementing partners have with the Evaluation methodology and approach. The first is that the Evaluators used measurements for e.g. sustainability that were not related to and used through the life of the project as approved by USAID. The definition of sustainability as contained in the Start-Up Implementation and Annual Work Plan is as follows: “…At the end of the project, each e-BIZ Center has (a) an annual operational and financial plan that is based on past performance, which demonstrates that the Center will reach the break-even point within one year of the project; (b) the e-BIZ owner’s written commitment to continuing operations of the e-BIZ Center operations after the end of the project.” In August 2013, we wrote to the Evaluators respectfully suggesting that the Evaluators and USAID use the standard of sustainability contained in the or at least explain more fully why a different definition of sustainability is being used. This is the standard of sustainability that the EI project has been using for the past five years. It is also a reliable indicator in that it is the same as the one that was used for the e-BIZ Macedonia project, which established e-BIZ Centers that have proven to be sustainable. We would be happy to provide the PMP for the e-BIZ Macedonia project to substantiate this. We believe ample documentation exists to show that e-BIZ Centers have been successful not only in achieving this sustainability target, but they have met or exceeded all of their objectives in terms of generating in-kind investment, serving SME clients, and perhaps most importantly, catalyzing innovation in their respective sectors. To suddenly adopt a new definition of sustainability in 2013 is both to unfairly “move the goal posts” at project end and to obscure the developmental impact that the e-BIZ Centers have made. Furthermore, it misses entirely the fact that the sizeable in-kind investment made by the parent companies alone is a strong indicator that most of the e-BIZ Centers will continue to operate successfully going forward in their own right. The Evaluation needs to elaborate more concretely on why this definition of sustainability cannot or should not be used to measure the e-BIZ Centers’ financial health. Second, we believe that overall, the Evaluators’ approach is inconsistent with the 2011 USAID Evaluation Policy which states that evaluations should include “use of data collection and analytic methods that ensure, to the maximum extent possible, that if a different, well-qualified Evaluator were to undertake the same evaluation, he or she would arrive at the same or similar findings and conclusions.”6 In order to entertain such a scenario, measures or criteria of success would have to be clearly stated and evidence provided that align with such measures so that “readers…have sufficient information about the body of evidence and how information was gathered to make a judgment as to its reliability, validity and generalizability.”7 In particular, if the USAID-approved M&E plan did not provide the evidence necessary for the Evaluators’ questions, then it is recommended that more argumentation, context and evidence be provided as to why the Evaluators’ approach is superior rather than the flat assertions advanced (for example, the claim that the definition of sustainability used in the M&E plan was “too narrow” is baldly asserted without justification or reasoning as to what would constitute a more robust definition). Where the M&E plan’s definitions and performance indicators are rejected, concrete measures and targets are not offered in their place. 6 USAID 2011 Evaluation Policy, p. 9. 7 Ibid. 17 Finally, rather than an objective, scientific tone that would be in keeping with the 2011 USAID Evaluation policy, the report strikes an antagonistic tone that is simply not borne out by the evidence as presented in the document. Point – by – Point Response to the Evaluation - Evaluation (p.2): “…The EIA and e-BIZ centers interfered with the evaluation process at several points by adjusting documents and providing documents that were produced after the evaluation process had begun…” EIA Response: EIA submitted dozens of project reports, industry and market analysis, SOWs, consultant and training reports, business plans, FOGs and literally several hundreds of copies of Center founder’s contribution (cost share) bills, receipts, equipment assessments, in kind labor timesheets etc. EIA understands that the Evaluation Team (ET) could have had uncertainty or even suspicion about some documents as the two or three mentioned in the draft Evaluation Report. However, ET has never approached EIA for explanations or clarifications before they boldly stated that the “interference” has taken place. - Evaluation (p.43): “A review of the properties of the submitted documents revealed that, for example, the Exploring BiH survey report was created on July 25 2013, despite the fact that the survey was supposed to have taken place in 2010. However, the Evaluation Team did not investigate further in order to confirm any potential file manipulation.” EIA Response: Related to integrity of Centers’ client survey documents, EIA reviewed the above mentioned “Clients Survey” folder and noticed that several word documents (from Exploring BH and TTC) indeed show, in the properties section, that their last version was “modified” on July 24, 2013. Nevertheless, the original documents have been created by e-BIZ Centers before and Ms. Vasic (Director, Exploring BH) and Mr. Banjic (Director, TTC) e-mailed their survey files to EIA weeks before they were requested by Evaluators. EIA staff compiled and saved last versions of those word files in the ZIP folder on July 24 before sending all documents to the Evaluation Team. Upon opening files that were received via e-mail and saving them in a designated ZIP folder, some of these word documents were obviously saved as modified versions although nothing substantial was done to them. It is true, however, that in one instance (Exploring BH’s file) EIA staff made minor formatting, date and English language correction in the heading section before resaving that document. However, none of these actions mean that any substantive part of any of the survey reports have been changed or altered. We would have appreciated if the Evaluators had asked ahead of time for clarification on this, rather than stating this in the Evaluation. If requested, EIA is ready to submit all evidence related to this issue to USAID. ************** - Evaluation (p. 5): “Capacity building efforts to increase e-BIZ center staff knowledge and skills in areas of high impact information and communication technology and services were basic and insufficient.” EIA Response: As we mention several times in the section related to Question 1 (on capacity building), the Evaluators offer no concrete measure of successful capacity building to which e-Biz centers failed to conform nor evidence to demonstrate failure to conform. And when the Evaluation does provide data from the beneficiaries related to EIA’s capacity building, this data suggests that the beneficiaries had benefitted significantly from capacity building activities. Furthermore, we are concerned that it is apparent from the report that the Evaluators have failed to review or assessed the 18 instances of capacity building support delivered by the EIA procured expert consultants. We respectfully request that this conclusion be revised for the executive summary in the absence of more robust reasoning and presentation of evidence. ************** - Evaluation (p.16): “The team sent the questionnaires via email but the response rate was extremely low for both groups of SMEs (client and non-client).” EIA Response: EIA kindly requests that the Evaluators provide some context as to why email was adopted as the data collection methodology rather than face-to-face interviews. ************** - Evaluation (p.18): “Therefore, in an effort to reconstruct the baseline data, the survey questionnaire for e-BIZ center employees contained ‘before and after’ questions.” EIA Response: Here, and throughout the discussion of Question 1, it is unclear what the measure of success of capacity building activities is that the project failed to meet. Nor is there clear evidence demonstrating that the project failed to meet such measures. ************** - Evaluation (p.18): “However, since the above-mentioned definition of sustainability is extremely narrow and is used primarily as a cost-accounting tool that only covers a point in time it would not provide information on the ability of the e-BIZ centers to sustain the provision of the Activity’s benefits in the long run.” EIA Response: The PMP defined sustainability as follows “…At the end of the project, each e-BIZ Center has (a) an annual operational and financial plan that is based on past performance, which demonstrates that the Center will reach the break-even point within one year of the project; (b) the e￾BIZ owner’s written commitment to continuing operations of the e-BIZ Center operations after the end of the project.” This was approved by USAID and retained throughout the life of the project as the operative definition. EIA acknowledges that milestones and goals may change over time and would have modified this definition in partnership with USAID if requested. ************** - Evaluation (p.19): “As for questions 3 a-c, the team compared market analyses prepared for each of the four industries and the data from those surveys with the current situation within the targeted SMEs served by the four e-BIZ centers.” EIA Response: Mention is not made of subquestions 3a to 3c until page 77 and therefore the reference here is confusing. We kindly request that the Evaluators make mention of these sub-questions upfront. - Evaluation (p.19): To answer this question, and due to the survey failure, the Evaluation Team analyzed changes in the competitiveness of the targeted SME clients. The team did this by tracking changes in certain indicators, such as profitability and sales. In order to determine whether the e-BIZ centers’ SME clients had experienced an increase in the volume of sales or profitability the 19 Evaluation Team used available financial data on the performance of companies and sectors collected by the Agency for Financial Information and Consulting Services (AFIP). The Evaluation Team only managed to collect data for companies and sectors in the Federation of Bosnia and Herzegovina. EIA Response: We believe that the Evaluators could have not just used the financial performance data for serviced SMEs and compared them in terms of their year to year results, The more adequate analysis would have to include the comparison with data on the performance of the entire sector during a given period as only then we would be able to know whether the Center assisted SMEs have performed better or worse than their sector averages. This is especially important considering the global recession. We are also concerned how the Evaluation can reach credible conclusions based on data collected only in one part of Bosnia Herzegovina (Federation BiH). Furthermore, many private sector experts do not consider AFIP statements as fully reliable considering the scale of the so called grey economy in Bosnia Herzegovina. ************** - Evaluation (p.19): Although there is no evidence that the e-BIZ centers had surveyed their clients on a regular (annual) basis the EIA reported results for this indicator each year in the work plan and M&E plan. More importantly, USAID data quality assessment (DQA) of this indicator verified the existence of the survey EIA Response: EIA Response: We respectfully disagree. The M&E plan included in the Annual Work Plans for years 2009, 2010, 2011, 2012 and 2013 clearly states that this survey would be conducted in Year 3 (2010), not annually. However, it is crucial to note here that the decision on timing of this survey made complete sense from the business development perspective. The EIA’s year 3 was essentially year 2 for the e-BIZ centers which meant that the sufficient time have passed to allow for the Centers to build the client base that would offer a valid sample for surveying SMEs on whether the e-BIZ Center services have strengthened their business. In addition, the Evaluation sections of the 2011 and 2012 Annual Work Plans state (on p. 35 for 2011 and p. 16 for 2102): “Utilize a survey of e-BIZ Center customers, conducted at the end of the project by an independent evaluator, to include questions that provide data to assess the impact of the e-BIZ Center services, e.g., questions regarding whether e-BIZ Center services helped to strengthen the SME customer’s business such as by increasing sales, reducing costs, etc.” An annual survey was never expected and results for such a survey were never reported in any work plan. The M&E plan in every Annual Work Plan merely states the targets expected (for example, that “50 customers report e￾BIZ Center services strengthened their businesses, for wood, tourism, machine tools and agriculture Centers”. p. 35 of the 2010 Work Plan or p. 32-33 of the 2011 Work Plan). Where results were reported, they were preceded by the term “Actual”. No actuals for this target were ever reported by the EIA in any work plan. We would ask that specific and direct citations of the Annual Work Plans be provided to substantiate this claim. ************** - Evaluation (p.21): One of the examples of interference with the evaluation process is the provision of contradictory data on the same topic. The Evaluation Team received documents from the EIA and the e-BIZ centers yet in the majority of cases the list of SME clients received from the e￾BIZ centers and the EIA were different. 20 EIA Response: We accept that client lists that Centers provided could have been more up to date than those that EIA had on file at the time of evaluation. Even if there was some difference between the lists, that could have been easily clarified upon request. It is unfair to say that this issue is an example of interference with the evaluation process. ************** - Evaluation (p. 20): “The Evaluation Team tried to reconstruct the missing data through a questionnaire survey. The questionnaire asked selected SME clients whether the technical assistance and services provided by the e-BIZ centers had contributed to an improvement in their business operations.” EIA Response: EIA kindly requests that the Evaluators more fully explain the methodology they used to select SME clients for the survey, how many, and whom. ************** - Evaluation (p. 21): “The evaluation was limited to the approved M&E indicators for the Activity. Some of these indicators, such as the financial sustainability indicator, were inadequate to measure the accomplishments of the EIA.” EIA Response: We kindly request that the Evaluators provide more justification and evidence for the assertion that these indicators were inadequate to measure the accomplishments of the EIA. This is asserted as assumed fact, rather than being argued for and justified. In rejecting the approved M&E plan’s performance indicators, which were considered sufficient, the Evaluators should clearly and succinctly state why it is acceptable to reject these indicators and demonstrate more comprehensively why their own proposed indicators and measures of success are more suitable. ************** - Evaluation (p. 21): Other Issues that affected the Findings The Evaluation Team has strong reservations about the information and data provided by the EIA, the e-BIZ centers and the interviewed SMEs. The EIA and the e-BIZ centers interfered with the evaluation process at several points by adjusting documents and providing documents produced after the evaluation process had begun. This interference made the evaluation process difficult and makes the evaluation findings suspect. More specifically the issues listed below. • Information provided by the e-BIZ centers to the Evaluation Team on attendance at the trainings did not reflect what happened in reality. The e-BIZ centers were instructed by EIA staff to report more capacity building activities.8 EIA Response: This is simply incorrect. E-BIZ Centers were not instructed to report more capacity building activities than what happened in reality. The only guidance that Centers have received from EIA relates to providing them with information on what is considered as capacity building support (e.g. provision of EIA consultants, trainings, continuous technical support from EIA BDMs etc.). In this respect EIA was obviously interested to make sure that Centers do not mis-report all instances of 8 Transcripts of audio-recorded interviews. 21 project’s capacity building support. EIA also instructed e-BIZ centers to report their founders’ participation in EIA trainings as the Center staff training. The reason is simple, as EIA believes that founder’s participation, especially in Centers’ early pre-registration phase represented a significant capacity building assistance despite the fact that founders were not subsequently formally registered as employees of their respective Centers. Therefore, EIA and ET’s apparently have a significant difference regarding the participation of Center founders in EIA trainings. • The financial reports of the e-BIZ centers were adjusted significantly during the one-month period after the evaluation process had begun. EIA Response: EIA strongly disagrees with this statement as it implies that all Centers have adjusted financial statements. Only the parent company for one Center (ARTECO) decided to pay outstanding services payments during the mentioned time period. Nevertheless, all these actions by ARTECO made business sense, were legally permissible and were done in a transparent manner. • As previously mentioned, the lists provided by the EIA and e-BIZ centers of SMEs served by the e-BIZ centers were different; even lists of clients that came from one source were not identical. EIA Response: Comment on this was provided earlier in the text. • During the initial meeting with the EIA, the EIA staff stated that no formal survey on the competitiveness of SME clients had been performed; this was also confirmed by the EIA at the end of the process. The nearest to it were the telephone inquiries from the e-BIZ centers to their SME clients asking them about their level of satisfaction with the services provided. Nevertheless, the EIA submitted list of SMEs that had participated in the questionnaire survey. Several companies stated during field interviews that they had never participated in any kind of survey: neither through the questionnaire nor via the telephone. The majority of issues with the validity and reliability of data on the performed surveys related to the TTC and Exploring BiH e￾BIZ centers. EIA Response: ET’s statement is not quite correct. According to PMP the clients were surveyed to determine whether “e-BIZ Centers services have strengthened their business” and not on their competitiveness which is the broader area. According to our recollection, EIA staff stated that no formal external surveys by EIA or by consultants were completed other than in ARTECO’s case. EIA also stated that only surveys that were completed were done by e-BIZ Centers. Based on the information provided to EIA, Exploring BH surveyed 21 out of its 60 clients at the time of their survey (Fall 2010). When compared to the list of eight (8) Exploring BH clients that were interviewed by ET it is evident that ET met only 3 clients that have been surveyed by Exploring BH in 2010. The other 4 tourism firms that ET visited in 2013 were either not the clients of Exploring BH in 2010 or were not included in the survey sample. Hence, EIA believes that this information sufficiently explains why “several (tourism) companies stated that they had never participated in the questionnaire”. It is because they really haven’t participated as Exploring BH interviewed its other clients. EIA is unsure as to why the ET received a similar response from textile sector firms. Our field experience shows that Directors are not always aware of their firm’s participation in some activities especially in non-essential activities such as surveys that are often completed by engineers or sales personnel. 22 ************** - Evaluation (p. 22): “The financial reports of the e-BIZ centers were adjusted significantly during the one-month period after the evaluation process had begun.” EIA Response: We kindly request that the Evaluators provide more explanation and evidence to substantiate this claim. ************** - Evaluation (pp. 26-30. Findings Section for Question 1 Overall EIA Response for this section: The organization of the Findings section on Question 1 frustrates the reader’s ability to appreciate how the Evaluators drew conclusions. We request that the full data set from the Evaluators’ survey mentioned be summarized succinctly and coherently (perhaps in a summary table) to allow for an objective assessment of results, rather than presenting findings in a selective and piecemeal manner. The methodology for drawing conclusions is similarly unclear: are the conclusions of the Evaluators based on the results of the survey they administered or not? If so, the results – though presented in a confusing manner – seem to suggest that the capacity￾building activities had a very positive effect. Are the Evaluators then discounting the results of their own survey? If so, then what is the definition of success and the methodology proposed to ascertain whether it was achieved? In drawing such negative conclusions, the Evaluators owe us a more rigorous presentation of their reasoning and evidence, or should simply state that the evidence is too inconclusive to draw conclusions. ************** - Evaluation (p. 27): The Evaluation Team assessed the EIA reports on capacity building efforts and conducted an analysis of the EIA documents and reports on the training and technical assistance provided. In accordance with our Request for Information sent to the EIA, the Evaluation Team received two sets of documents: • trainings provided to e-BIZ centers; and • technical assistance provided to e-BIZ centers. EIA response: It is apparent that ET had not included numerous instances of direct, on site, consultant assistance to e-BIZ Centers when they evaluated EIA success related to this question. ************** - Evaluation (p. 27): “Financial management capacity building efforts were focused mainly on increasing the capacity of e-BIZ center staff to track and record cost-sharing arrangements based on the EIA internal cost-share tracking system. The e-BIZ centers were never properly instructed or trained in how to track and record cost-share arrangements in accordance with the approved e-BIZ centers’ business plans.” EIA Response: This is simply not true as Centers have received specific instructions/training from EIA on how to evidence and document their contributions (cost share). Because the Evaluators never state a clear measure for success of capacity building efforts or provide aligned evidence that the e-Biz centers did not meet these measures, it seems in bad faith to conclude 23 the E-biz centers did not meet that measure. It would have been more helpful to clearly demonstrate a measure of success, and then provide concrete evidence that e-Biz centers did not meet that measure. Criticisms are offered of the scope of the training, not the impact of that training, nor are details offered as to how their conception of a more robust scope for such trainings is related to their measure of success. ************** - Evaluation (p. 29): “There was no increase in technical skills at the Exploring BiH e-BIZ Center: computer skills, understanding software, hardware, troubleshooting, teamwork and the organizational structure.” EIA Response: We kindly request that the authors present the entirety of their data for all four centers in one succinct data presentation that would allow the reader to ascertain the validity of their conclusions. A more rigorous presentation of the actual data would allow the reader, at a glance, to know, for example, whether the other three centers showed an increase in skills and by how much. Such a presentation of actual data would be more in keeping with the 2011 USAID Evaluation policy objective of allowing readers to “have sufficient information about the body of evidence and how information was gathered to make a judgment as to its reliability, validity and generalizability.”9 ************** - Evaluation (p. 30): “The Evaluation Team did not have any meaningful method to evaluate the impact of day-to-day and hands on type technical assistance in a measurable way.” EIA Response: We kindly request that the Evaluators provide more justification as to why the impact of day-to-day mentoring and coaching cannot be evaluated. ************** - Evaluation (p.30): In terms of operational management, the Evaluation Team could not confirm any capacity building assistance directly targeted at improving managerial skills to adopt and integrate technology in order to gain maximum efficiency and productivity. The Evaluation Team was also unable to confirm any capacity building assistance directly targeted at improving knowledge and skills in other areas of management i.e., strategic planning and human resource management. EIA Response: EIA is not clear as to why the ET has not considered capacity building support targeted at improving above listed skills that was provided by local and international consultants (Martin McElwee, Emir Medanhodzic, Richard Bangs, etc.). Furthermore, EIA staff provided specific and detailed support in recruitment efforts for each Center thus improving Centers human resource management capacity. ************** - Evaluation (pp. 30-31): “Having in mind that e-BIZ centers are supposed to be centers of excellence for the dissemination and transfer of new high impact information and communication technologies, 9 USAID 2011 Evaluation Policy, p. 9. 24 the EIA has done too little to build e-BIZ center staff knowledge and skills in the application of high impact ICT.” EIA Response: We kindly request that the Evaluators clarify and make transparent what the concrete evidence is for this claim. The only concrete criticism levied in the discussion is of the length of the training offered, but no evidence of its suboptimal impact is provided. ************** - Evaluation (p. 31): “However, as the aforementioned definition of sustainability is extremely narrow and only covers a point in time without providing information on the ability of the e-BIZ centers to sustain the provision of the benefits of the EI Activity in the long-run, the Evaluation Team proposed a different approach. USAID accepted the latter suggestion as part of the approved Evaluation Work Plan. EIA Response: We kindly request that the Evaluators explain and justify their assertion that the definition of sustainability is narrow. A more transparent presentation of the Evaluators reasoning process would be in keeping with the 2011 evaluation policy. ************** - Evaluation (p. 32): “Other types of financial indicators analyzed by the Evaluation Team indicate that Arteco is under threat of potential bankruptcy. The value of Altman’s Z-Score test on June 30 2013 was 1.3, which falls within the range of a company at risk of bankruptcy” EIA Response: The article attached by the Evaluators to provide context on Altman’s Z-scores states that a score between 1.1 and 2.6 is clearly within the “grey zone” of values easily misclassified. Also, the Z-scores vary depending on whether it is a public manufacturing firm (Z-score), a private manufacturing firm (Z’-score), or a private, non-manufacturing firm (Z’’-score). In their footnote, the authors apply the third range (for private non-manufacturing firms), but this is inappropriate for Arteco, a private manufacturing firm, which falls under range number 2. Since Arteco is a private manufacturing firm, by the author’s standards, a score of 1.3 would fall into the “grey zone in which distress may or may not be impending”. The Evaluators should also clarify whether they used the Z-score for private companies (which weights the variables involved in the score’s calculation differently and which replaces the ratio of market capitalization with book value of equity). EIA also requests that authors include the full calculation of the Z-score as well in the document, to be transparent. The Evaluators should also mention the limitations of the Altman score, particularly the fact that it is usually not applied to firms with assets less than $1 million USD. ************** - Evaluation (p. 33): “The Kralicek Quick Test indicated that Arteco’s financial stability was poor, even after the aforementioned financial engineering exercise. On July 31 2013, the value of the Kralicek Quick Test was 0.32.” 25 EIA response: We kindly request that the Evaluators be more transparent about their use of the Kralicek Quick test and discuss, in particular, the advantages and disadvantages of using this method with SMEs such as the e-Biz centers. ************** - Evaluation (p. 33): “The Arteco e-BIZ Center has achieved the break-even point, as the measure of financial sustainability; however, other financial indicators indicate poor financial stability and the threat of potential bankruptcy.” EIA Response: EIA respectfully disagrees with this assertion as stated in the conclusion section for Question 3 and believes such a conclusion cannot in good faith be stated as by the authors’ own standards the firm’s Z-score is in the “grey zone”, so deemed because such scores are too inconclusive to be able to predict bankruptcy. EIA therefore requests that the Evaluators present this evidence in good faith as too inconclusive as to be able to predict bankruptcy. ************** - Evaluation (p. 34): “Other types of financial indicators analyzed by the Evaluation Team indicate that Exploring BiH is under extreme threat of bankruptcy. The value of Altman’s Z￾Score test has been negative throughout the entire life of the Project.” EIA Response: EIA kindly requests that the Evaluators transparently state the actual Z-score and provide the actual calculation conducted to obtain that score. ************** - Evaluation (p. 36): “Exploring BiH achieved financial sustainability by reaching the break-even point and having revenue that exceeded expenses by 1%. However, other financial analyses indicate poor financial stability and extreme bankruptcy risk. “ EIA Response: There is no cited evidence for this other than a vague reference to the Altman z-score on p. 34. ************** - Evaluation (p.39): The e-BIZ centers did not achieve the financial benchmarks set in their respective business plans. EIA Response: EIA believes that this statement is not supported by the evidence. Three out of four e￾BIZ Centers have shown better actual year 1 and 2 financial results compared to the projections in their respective business plans. One of the potential reasons for this conclusion by evaluators could be related to usage of the wrong start dates for Transwork and TTC as both Centers opened their doors in spring of 2012. Below please find the quick review income statements provided by the e-BIZ Centers. Evaluators are urged to note that Centers have shown satisfactory financial health especially in light of their business plan projections. In short, the income statements for Transwork and TTC show the following (in KM): 26 Name of Firm Total Planned for the first five quarters in terms of profit/loss after taxes Actual for the first five quarters in terms of profit/loss after taxes (2012 and 6 months of 2013) Actual 2013 in terms of profit/loss after taxes (first six months) Transwork -143,915 KM -89,878 KM 9,544 KM TTC -230,752 KM -50,227 KM 1,661 KM The income statements for Exploring BH and ARTECO show the following (in KM): Name of Firm Total Planned for the first two years in terms of profit/loss after taxes Actual for the first two years in terms of profit/loss after taxes (2010 and 2011) Actual 2012 in terms of profit/loss after taxes ARTECO -93,830 KM 12,069 KM 20,821 KM Exploring BH 20,596 KM -12,894 KM 302 KM ************** - Evaluation (p. 42): “The EIA was supposed to perform annual surveys in order to determine changes to the above-mentioned indicator.” - EIA Response: We respectfully disagree: the M&E plans included in the Annual Work Plans for years 2009, 2010, 2011, 2012 and 2013 clearly state that this survey would be conducted in Year 3 (2010), not annually. In addition, the 2011 and 2012 Annual Work Plans clearly state (on p. 35 for 2011 and p. 16 for 2102): “Utilize a survey of e-BIZ Center customers, conducted at the end of the project by an independent evaluator, to include questions that provide data to assess the impact of the e-BIZ Center services, e.g., questions regarding whether e-BIZ Center services helped to strengthen the SME customer’s business such as by increasing sales, reducing costs, etc.” ************** - Evaluation (p. 44): “Due to the survey failure, the Evaluation Team analyzed changes in the volume of sales and the profitability of SME clients.” EIA response: No mention is again made of profitability in the discussion that follows. Only results on changes in volume of sales are presented. We request that the authors present their data and evidence on profitability to allow the reader to understand better how they arrived at their conclusions. Also, the authors should describe why it is sufficient to use only a one-year difference in sales volume results to draw their conclusions. ************** 27 - Evaluation (p. 46): “The Evaluation Team interviewed six (6) of Arteco’s clients, five (5) of TTC’s clients, five (5) of Transwork’s clients and eight (8) Exploring BiH clients. Their perception of the importance of the services provided by the respective e-BIZ centers ranged from “not important at all” to “very important.” EIA Response: This is an extremely vague presentation of data results. In place of it, EIA requests that the Evaluators provide a precise summary table of data results. 28 Comments on the Recommendation Section Please note that responses to some of the recommendations have already been addressed in the sections above. Additional comment is made below for responses to recommendations that were not addressed previously or where EIA believes that reiteration of key points should be made. 1. USAID BiH should pay far more attention to the process of approving the initial work plan, M&E plan, and sub-grants. The initial EIA work plan set the indicators for the life of Activity. The Evaluation Team strongly believes that the two performance indicators proposed by the EIA and approved by USAID AOR were inadequate. Furthermore, three of the four business plans that accompanied the four FOGs contained obvious errors that USAID did not recognize when it evaluated and approved them. EIA Response: The EIA’s initial work plan and M&E approach was developed based on a proven model while the Center Business Plans have been developed in a collaborative effort from EIA and project partners and with help from external consultant experts. Additionally, each plan was reviewed, commented on and validated by industry experts. We would appreciate more clarification on why and how the work plans and particularly business plans were not adequate for this project. ************** 2. USAID BiH should focus much more on regular and rigorous monitoring of the implementation of its activities/projects. The fact that the EIA reported results based on non-existent data for several years indicates a serious problem in the way USAID BiH managed and oversaw the Activity. The lack of an adequate USAID monitoring system to track progress towards the defined objectives and goals on a regular basis resulted in some surprising findings in the evaluation. EIA Response: This is inaccurate. EIA did not report results based on non-existent data. The EIA PMP states that data following data had to be collected: e-BIZ business and operations plans, Center sales records, surveys of e-BIZ Center customers. EIA has followed the frequency of collection as stated in the PMP as we collected sales data from bank statements and surveys from Centers. ************** 8. The recipients of the USAID assistance should have recorded the cost-sharing and other financial agreements in their financial reports in accordance with the country specific accounting standards. This should have been clearly specified as an obligation of the assistance recipients and regularly monitored by the implementing organization and USAID. The e-BIZ centers were not instructed on how to record the USAID and parent company contributions within their financial statements. The Evaluation Team was not in a position to verify the amounts and structure of the cost-sharing contributions from the founders of the e-BIZ centers. EIA Response: It is evident from the draft report that the Evaluators have not accepted key provisions from the USAID approved project FOGs related to evidencing parent firm investment into the e-BIZ Center. As a result, the Evaluators have stated that they are “not in a position to verify contribution of the founders of the e-BIZ Centers” despite the detailed evidence on this contribution that EIA collected in accordance to the Fixed Obligation Grant (FOG) (or should we say 22 CFR 226.51) and presented to the Evaluators. ************** 29 10. The e-BIZ centers should have built in capacity to research and apply ICT in their production processes, not to reduce it to the marketing departments of their parent companies. Presently all ICT research and development and ICT applications in production capacities rest with the parent companies. EIA Response: This blanket statement for all EIA supported e-BIZ centers is incorrect. ARTECO and TTC clearly have the capacity to research and apply ICT in their production processes while the same capacity also exists at Transwork, albeit through the outsource option from the closely connected parent firm. We request that the Evaluators provide more specific evidence indicating lack of capacity such as in ICT. ************** 11. More effort should have been invested in attracting and retaining highly qualified staff with relevant knowledge, skills and experience. The e-BIZ centers, especially ARTECO and TTC, should have done much more in terms of creating more attractive packages to attract highly qualified employees, due to their remote location. Another option would be significant investment in education and training of the existing staff. EIA Response: EIA could agree in general about the need to attract and retain highly qualified staff. This is a fair lesson learned though more discussion is warranted about how one does this successfully in a start-up environment when time and resources are limited and given other priorities such as introducing new products to the market. ************** 12. More should have been done in raising awareness amongst SMEs about the benefits of introducing ICT into the production processes. Although the EIA and e-BIZ centers made a lot of effort to market their products they failed to raise awareness amongst SMEs concerning the importance of applying ICT in production processes. This is partially a consequence of the failure to establish meaningful cooperation with the public sector and business associations. Management of the e-BIZ centers believed that SMEs in Bosnia and Herzegovina were still not aware of the benefits they could gain through the introduction of ICT applications to the production process. Therefore, they did not manage to expand their client base and disseminate ICT applications throughout the target industries. EIA Response: The Evaluators have failed to note that there were number of activities conducted by the e-BIZ Centers, and with support from EIA, that were focused on raising awareness about applying ICT in their production processes. For example, TTC organized two large Open Day services presentations to SMEs together with the BiH Textiles Association while ARTECO and TTC also held number of presentations in cooperation with Chambers and associations that were focused on raising awareness about new technologies and use of ICT applications in production and service processes. Evaluators could have found evidence about this in EIA Quarterly and Annual Reports. ************** 14. More should have been done to establish long term cooperation between the e-BIZ centers and SMEs. The majority of SMEs were one-time clients and the e-BIZ centers did little to adequately manage their relations with them and turn this into long-term cooperation. EIA Response: We disagree with the above statement as Centers have been able to steadily attract more repeat customers. We would ask the Evaluators to provide more concrete evidence that the clients of the 30 e-BIZ Centers are predominantly one-time clients. Furthermore, ARTECO and Transwork installed the EIA-developed Client Relationship Management (CRM) systems by spring 2013 while TTC is planning to introduce this system soon. The CRM allows for integrated client management approach and we regret that Evaluators have not reviewed and assessed this important client management tool. ************** 15. Arteco and its owners should be required to meet the financial target values set by the business plan. Financial projections set the value of Arteco’s capital in 2009 at 711,720 BAM; however, on June 30 2013 the total value of Arteco’s capital was 2,000 BAM. Values for current and long-term assets set in the business plan were also not achieved, as well as ratio between long-term and current assets set in the business plan. EIA Response: The Evaluators have failed to realize that the above mentioned capital target value could not have been realized by the time of the evaluation partially due to the fact that the title for the CNC machine resided at EIA until the end of EIA project. 31 ANNEX Letters from e-BIZ Center Owners Related to the EIA Evaluation and Performance Evaluation Report ENGLISH TRANSLATIONS E-MAIL FROM THE TTC’S FOUNDER TO EIA COP From: alen karic [mailto:alenkaric@yahoo.com] Sent: Sunday, August 25, 2013 8:49 PM To: Tanovic, Jusuf Subject: obavjestenje o posjeti (information about the visit) Respected Mr. Tanovic, On Friday we had a final meeting regarding the project evaluation at the TTC in Brijesnica Mala. I can’t escape the impression and feel a moral obligation to convey may impressions and I believe that you from EDC will be surprised at the minimum. First I must say that I have received the request for the Friday meeting while I was on business trip in Germany on Wednesday. I called Mr. Pavelic and asked him to postpone this meeting for Monday (tomorrow), but that was not accepted so I had to abandon a very important activity with potential new customers and drive back to Bosnia. Mr. Pavelic told me that it would not be necessary and that they could have a meeting with Semin, the TTC’s Director. You are aware that the selection of Director was problematic at TTC and that all important activities are handled together with me and with other Kismet’s personnel (Izet Banjic and Armin Hasic). It is not the secret that the Director needs support from all of us. I was told that this would be a routine meeting but luckily I did not believe it. By the way the listed duo (Banjic and Hasic) were on vacation and our Director is not ready to answer all questions considering the dispersion of tasks to other staff and his experience of only one year. All that approach towards us really had a negative tendency and later on I was able to personally witness that. I had a feeling that we were supposed to be caught off guard so we would look clumsy to the end. Unfortunately, for all that they do believe is wrong with this project they blame you as they say for that you are paid "ten, eleven thousands of", I do not remember the currency admittedly (but if we are in BiH then its BAM, I suppose). I believe that they were looking for mistakes only in this project... I do not think that this project is bad, on the contrary ... We also invested a lot of money and that is the guarantee for my vision. Unfortunately, today I feel, and later after this meeting on Friday, as if I were a thief, and I have the impression that they want to present that you from EDC are spending money just like that and that you are giving us, the firms, the funds in teaspoons and as the lady said only 17% of the entire fund. I know, and I will repeat it if necessary before the public, that we got more than we were asking for in our application (equipment) and more than we expected as technical assistance, especially from you and your staff, and also not to mention the numerous consultants and industry experts. I want to state responsibly that they are doubting the hiring of external staff (consultants Emir, Marti etc...). When? Which date? How many days? etc. I think that written answers mean nothing to them. The tape recorder was on all the time and they want quick answers. Like in the movies when the criminals are interrogated. That is how I felt. I thought that in the legal world the written affidavit is sufficient when the data is in question, and that the interview is taken orally, but that the answers are in principal ABCD ... either yes or no, which also does not legally obligate. For example they said that we have not entered the capital in the TTC (I believe the money that Kismet was obliged to deposit to TTC’s account) and that is horrible. I do not even consider that as a mistake because we will do it by the end of the year, and as we informed you earlier, the money was already deposited of course for which we have evidence from the bank. I regret that USAID hires accountants (the ones that look at books), but not those who understand or want to understand the market, vision, mission, technological processes and also the real problems that we encounter daily. Despite all listed I am definitely convinced that we didn’t make a mistake and that USAID had not wasted its money and that I haven’t wasted my money also. I will be sorry if people from USAID get the impression that something was wrong from our side and also that your work is assessed based on the biased view which is absolutely not correct. Furthermore, the interviewing method towards customers is very wrong because our customers may think that we have embezzled something and that creates a very bad image for TTC. I believe that interviewing TTC customers in such way must stop. I suggest that gentleman come to the “Open Days” presentations and hear impressions. I remind you that those were outstanding presentations and that all participants believe that TTC services are essential and that even the USAID Director personally witness that during one of the "Open Days". One year is too short period to implement new services in the industry. Mr. Marty (US based textile expert) also said that. We are well on the way for our customers to use more and more services of TTC in the coming period. Gentleman expressed suspicion that the TTC is useful for Kismet, and I say that it is fortunate that Kismet has enough work so TTC could be sustainable and I don’t remember that they asked clients, when they were already calling, whether TTC refused someone or did not have the free capacity because it was busy working for Kismet only. TTC has available capacity and its even growing as our staff is getting better educated, and after all we are not even close to introducing the second shift, something I have no doubt will happen eventually. Finally, I want to emphasize that I am very disappointed at the way in which evaluation is carried out. I would be sorry that someone is, unfortunately, maybe trying to raise their own rating or for their firm by presenting us, the people that were involved in a hard build Center, as incompetent or dishonest. For your work and work of your associates I am ready to testify anywhere that it was prudent, committed and honest. Wish you the best... Alen Karic KISMET d.o.o 74206 Brijesnica Mala b.b. Doboj Istok Tel.: + 387 35 722 501 Fax: + 387 35 727 140 ENGLISH TRANSLATION – E-MAIL FROM THE EXPLORING BH’S DIRECTOR TO EIA COP From: sasa vasic [mailto:sasa@exploringbh.ba] Sent: Friday, August 30, 2013 9:56 AM To: Tanovic, Jusuf Subject: Exploring bh-meeting with evalautors Respected Juka, I want to pass on to you my experience that I had with the evaluators of the project which is not really so positive. The first meeting I had scheduled at the beginning of the month has been terribly stressful, people who presented themselves as project evaluators have acted as police officers and I was treated as if I stole something. Questions that were asked were not for the executive director. I, as someone who started as Director after the completion of the process of awarding grants do not need to know who sat on the committee for selection of our grant and the like. At the first meeting there were fewer issues related to my profession and plenty about ownership and selection of partners (Macedonians), the method of testing was such that I felt like I was seating on the court bench for the accused, quite uncomfortable. The second meeting took place on Monday and that one passed much better because as the questions were related exclusively to my profession. My general impression is that the evaluators have not understood the purpose of the project very well, that they have a very difficult to understand in which way we contributed to the promotion of tourism in Bosnia and Herzegovina and about how the whole sector is functioning. Considering that after the first meeting I was frankly shocked by the questioning style, I felt obliged to share that with you, as I think that as someone who has followed Exploring bh for four years ought to know this. With respect, Saša Vasić Exploringbh d.o.o. Humska 17 71 000 Sarajevo mob: +387 61 028 964 +387 63 940 111 www.exploringbh.ba From: Saša Mičić [mailto:sasa.micic@transwork.ba] Sent: Friday, February 28, 2014 11:13 AM To: Tanovic, Jusuf; 'Siniša' Cc: 'Valentina Rujević' Subject: Comments on Evaluation Document Dear Mr. Tanovic, Thank you for sending as insert of Evaluation Document and sharing it with us. It is of great significant for us to be aware of different opinion and to be able to give our comments and explanations. First of all, we would like to point out that the evaluation process started in the 2nd part of 2013 in the time when Transwork was active for only 18 months (commercial activities). We believe that such a short period of activities is not enough for our e-BIZ center to reach full volume of activities and results, although Transwork did achieve positive financial results and accomplished all tasks and goals defined in Milestone schedules. If evaluation process had happened later, the results and all findings would be much better and complete picture would be more realistic and more reliable, especially because a certain number of clients would use our service for a longer period and their own business results and competitiveness would be easier to measure. In situation that we have now, some of the clients were taken into count and analyzed after only few months of using our services, and this period is hardly enough to perceive all benefits. Transwork is the only company in the market who offers integrated GPS solutions (GPS tracking, Communications, Goods tracking, Fuel Consumption Control) with first domestic on-line freight exchange. This complete solution, which must be considered as business tool, enables our clients, transport and logistics companies, to cut costs significantly, to improve behavior of the employees, set standards, and help them to plan activities, analyze results and in final to improve their competitiveness. Other companies that offer GPS tracking services does not offer integrated services and this make us distinctive and give us competitive advantage. Additionally, if Business Plan obliged us to employ two ICT educated employees (e.g. IT engineers), we would have done so, in order to fulfill the request, but we believe that the most important is essence of the project and the final results that we have achieved. We do not agree with statement, mentioned several time in Evaluation Document, that we are marketing and sales department of parent company Satwork. Also, we believe that Evaluation Team does not have enough experience in the field of transport and logistics to realize all problems and challenges that companies in this field are dealing with and the solutions that we have offered to them. Regarding some of the quotes listed in the Evaluation documents we have comments and with do not completely agree with some of the conclusions. All comments are listed below: Comments on Evaluation Document Question 1: To what extent has the EI Activity succeeded, in a measurable way, in capacity building for e-BIZ center Staff? All seminars were organized before Transwork e-Biz center was established and its employees employed. Nevertheless, Transwork team members (two of them) were trained in previous companies and assignments for sales and marketing activities, and had a great experience in these fields. Also they had experience in financial management from previous companies. The owner of Satwork, as mentioned in Evaluation Document, did attend Power of Sale seminar, and shared his knowledge and information with Transwork employees. Regarding financial management capacity building and, EI provided training for cost – share tracking which was necessary in order to have complete financial picture and financial statement of Transwork. Financial statement of Transwork could not provide all necessary data (for example in kind contribution, costs paid by Satwork – parent company, etc.). Analyze of all documents (Financial reports, Cost – share reports, list of clients etc.) confirmed that cost sharing structure were fulfilled. QUESTION 2: Are all four USAID supported e‐BIZ centers financially and operationally sustainable? According to the Transwork Business Plan no ICT expert were not planned as member of staff. Transwork fulfilled obligation regarding employment of director, marketing and sales staff. Marketing, promotion and sale were set as one of the most important goals. Transwork staff (Director of Transwork had experience as team member in development and implementation of software in bank and consulting company, also Sales and Marketing director had previous experience as team member in implementation of bank software. Ž Also all Transwork staff contributed and were completely involved in Transwork web portal and Freight exchange development and implementation (although not ICT experts, they contributes to determine and develop different functionality, to make freight exchange simple to use, structure of the web portal, etc.). There are plenty of reasons why Transwork `failed` to become the main web-based freight-exchange web portal. First of all it is very short time of activities to compete very large international company that exists more than 20 years. Secondly, Transwork had only 4 employees dealing with not only introduction of new company, promotion of its products and services, but also fighting to achieve all Milestone that were set in Business Plan (development and implementation of new services, achieving clients, and in the same time reaching breakeven point). Thirdly, money invested in this project was not sufficient to reach such a strong impact on the market and becoming the main freight exchange. Transwork is not a marketing and sales department of its parent company Satwork. Satwork did not deal with transporters as its target group. They were mainly dealing with local B&H company in the field of micro distribution. Transwork focused mainly to the domestic and international transporters. Transwork offered new solutions to this focus group of clients offering them newly developed services as international GPS tracking, Goods tracking, Communications, Fuel consumption control. All these new services developed according to the Business Plan Milestones, Transwork contributed to the cost reduction in general and increasing of competitiveness of SME`s in transport and logistics sector. Transwork can continue its business activities as independent company, without support of parent company according to the fact that freight exchange is fully completed product, although it needs more time and effort in marketing activities in order to promote and sales to the larger and different focus groups locally and internationally. QUESTION 3: To what extent has the EI Activity succeeded, in a measurable way, in improving the co Figure 6 shows annual percentage change in the sales of Transwork’s SME clients in the period 2011- 2012. Some of the company listed in the table did not become clients of Transwork before 2013 (for example Sarajevska pivara) so data from 2011-2012 are irrelevant. Also, Raiffeisen leasing is not Transwork client at all. We jointly organized one presentation, so this company also must be excluded from the analysis. Analysis should also include other company from other part of B&H (RS). Kind regards, Doc. dr Saša Mičić Direktor Bulevar vojvode Stepe Stepanovića 132 78 000 Banja Luka Bosna i Hercegovina mob: +387 66 711 700 tel: +387 51 233 003 fax: +387 51 233 009 e-mail: sasa.micic@transwork.ba WEB: www.transwork.ba From: Elvedin Mandzukic [mailto:edo@arteco.ba] Sent: Thursday, February 27, 2014 1:47 PM To: Tanovic, Jusuf Cc: Drazen BiH; Fadil Costovic Subject: Evaluation Issues and Other Concerns for ARTECO HI Jusuf, Thank you for giving as opportunity to see the parts of the Excellence in Innovation’s project evaluation report that are focused on ARTECO. There are many comments and conclusions that we disagree about and they are: 1. Remote location insinuation is something that we completely disagree: remote from where? From Sarajevo? What wood processing industry giants they can list in Sarajevo area at the time of the project inception? Artisan was not a giant but had a vision, and you can see the results today. 2. The majority of revenue came from parent company: so what? The parent company managed to increase number of employees and to increase exports due to that fact; Artisan doubled number of employees and increased revenue by 1000%. Needless to say how ARTECO is moving boundaries through Artisan, where ARTECO supported manufacturing and development is being perceived as high end, designer furniture from Bosnia, not cheep, no-name, small margin, loan work, agents profiting, consumer gods, wannabe IKEA type that we often see at the joint exhibition at IMM. Finally, we serviced a number of small companies and individuals through our cooperation with Artisan, which was never recorded because we wanted to help those companies and individuals overcome financial gap too, and their lack of registered business; some of these companies and individuals now own their own business 3. Mostly foreign companies and individuals (mainly designers) use prototyping: through this service we show that we can do something, and often get more orders to our cooperation and partners. 4. The machine producer (Homag) did some training on the machine, but Artisan financed solid works and solid cam trainings, and purchased licensed software, as planned in our business plan agreed with EI project. 5. Our staff developed 84 new products in 2013, and communicates with so many high profile professionals within the industry and academia (very good cooperation with University of Bern in Biel in Switzerland – Artisan even provides a stipend for one young engineer at that university) that we cannot believe these paragraphs about capacity building and the employees CV’s. 6. Various analysis that were included in evaluation given with a disclaimer “due to survey failure,” as well as, data from AFIP are entirely irrelevant and confusing. Why should someone use AFIP data when we have industry that is plagued with grey economy and lack of transparency – data like this could not be admissible in any court. And so many more, but we are quite busy with our daily work, employing new people and working on new product almost every day. Our cooperation and users of services will be in a couple of years the best marketing for us. We must emphasise that we, as ARTECO represented wood technology center at IMM, and it was an excellent response from inter nation industry. However, one point about our financial practices is very troubling to us as it implies that Artisan has not contributed cost share investment for ARTECO. As you very well know, Artisan contributed all cost share in accordance to our obligations from the FOG contract with EDC. We gave you all documentation and evidence for that. You also know that we were very helpful with evaluators and we provided them all documentation they requested. We are now shocked that they could say that we didn't provide our cost share properly. First and foremost I need to confirm that our financial records are fully in accordance with BiH laws and that everything we did is legal and transparent. Therefore, we would need to know whether evaluators have a right to inspect our books in such a way and make judgements and accusations about our accounting practices and whether they have theauthority to determine whether we compleetd our cost share FOG commitments to EDC in right way. Obviously, I know that we provide cost share and I know that you have confirmed that our evidence was precise and sufficient. I need to repeat here that our impression about evaluators from last summer is still valid - they do not appear to have enough experience and understanding of real life business practices and specially about our sector. We did our best to follow perhaps too optimistic business plan, but again, when we started planing there was no sings of global economic recession. Given the fact that the majority of project duration was under the severe economic contraction, we did an excellent job. We cannot see such conclusions in this evaluation. Finally, we insist that the evaluation team should provide us with their written apology for all unfounded speculations and claims. Sincerely, Elvedin Mandžukić Arteco Wood Technology Center Medakovo bb, Tešanj Tel: 032/658-660 www.arteco.ba ENGLISH TRANSLATION – E-MAIL FROM THE TTC’S DIRECTOR TO EIA COP Sent: Friday, February 28, 2014 2:33 PM To: Tanovic, Jusuf Subject: Evaluator’s Report Respected Mr. Tanovic, Thank you for the submitted report from evaluators. We would like to give few comments on the report concerning the Textile Technology Center. We would like to take look at the report where we don’t agree with the negative approach, which was sent to you from the owner of TTC in a letter (e-mail) and unsubstantiated negative conclusions. Hereinafter, we are sending to you some points with which we don’t agree: • The report noted that the seven employees received CAD / CAM training and only three of them execute the same. As submitted in the table, nine employees have passed a variety of CAD / CAM training, of which two of them were not employed during the evaluation. Out of that number, 3 perform the function on the CAD system (Accumark software), and three of them do CAM functions ( Operate Cutter and Spreader) • In the submitted CVs it was evident that 7 seven employees had work experience in same or similar (textile manufacturing) positions. ZM (Zijad Mesic ) has 27 years of experience, but the report noted that he had none. • The report noted that according to the business plan, the TTC 's revenues for 2011 and 2012, should have been 491,130 BAM while TTC 's revenues amounted to 73,976 BAM in 2012 with the loss of 51,888. It was not taken into consideration that TTC project was delayed and that official opening of the TTC was on March 28, 2013. It is difficult to achieve revenue of 491,130 BAM within nine months. • It is true that the main revenue for TTC comes from the company Kismet and Vocal, which is why the TTC actually has a good base for operations. Connected to this, we want to state that TTC has available capacity and that no buyer was ever turned down. Considering that TTC offers new services on the market, the slow response of customers is understandable. Please note that the TTC quite correctly did its marketing to customers. Based on those facts, it is evident that the TTC offers quite equal approach towards all customers. Sincerely, Semin Banjic From: sasa vasic [mailto:sasa@exploringbh.ba] Sent: Friday, February 28, 2014 11:39 AM To: Tanovic, Jusuf Subject: Review of evaluation paper Dear Mr. Tanovic, Thank you for sending part of evaluation paper that refers to Exploringbh. As an owner of 51% of company Exploringbh, its director and the person who was involved from the beginning of the project I must send certain comments and personal view of this evaluation paper. Reading the first four paragraphs where the evaluator is explaining our financial side of business, I can freely say that the evaluator is a very good with financial analysis and good with numbers but obviously has no adequate knowledge of tourism sector in Bosnia and Herzegovina and doing business in that particular market. Opening a business in 2009 in tourism sector in BH, particularly Incoming tourism, when all you could google about BH were war and post war stories, was very risky and didn't have good prognoses for big profit at that time. The project was intended to improve this sector with a new technology and the vision was not to achieve a huge profit on a first year but to develop this sector and help it growth with a help of new technology. So the fact that Exploringbh at the end of the project became sustainable and still is, a year after being supported by USAID, speaks more than a numbers that are listed. "As stated during the interview with the Director of Exploring BiH, in the early stages of the project it was already clear that the website/portal was a failure." Now from this perspective it is obvious to see that achieving to be a "Bosnian booking.com" back in 2009 was not possible or we could say was a failure. However back in 2009, when booking.com had just few respectable hotels in their offer, when internet promotion of BH tourism was very poor and you could not search for hotel accommodation, "Bosnianan booking.com/Exploringbosnia.com" it did make a sense and it looked profitable on longer run. "The majority of interviewed clients found remembering any cooperation with Exploring BiH very difficult". Exploringbh makes a business with almost every hotel in BH, a different travel agencies and numbers of different small businesses in tourism sector. I have no idea by which criteria evaluation team had visited our clients, with some clients we made a business ones and never again due to many reasons but mostly because their service was not attractive to clients. "One rafting company owner claimed that contact with Exploring BiH brought more harm than good to his company", I cannot recall whom we brought more harm than good when we are the first and only agency that made a famous tour of "White water rafting three rivers" and brought a tour of 30 American tourist, a tour that the each and every newspaper and portal wrote about it. "The interviewed clients could not confirm any significant benefit from their cooperation with or promotion done by Exploring BiH." As I mentioned above I do not know by which criteria evaluation team had visited our clients and it does not say with how many clients they made an interview. Most probably they did not talk to the clients we make business on a daily basis. To sum up, I need to say that in my point of view, this was not an evaluation process but investigating process, the fact that the evaluation team did not write one positive thing about Exploringbh gives me a right to thing that this evaluation team did not even made an effort to understand this business and tourism sector at all. I feel very sad that one evaluation team can ruin our reputation with in my opinion not very realistic estimates, specially because I have invested a lot of money and time building this company. Best regards, Saša Vasić Director Exploringbh d.o.o. Humska 17 71 000 Sarajevo mob: +387 61 028 964 +387 63 940 111 www.exploringbh.ba ORIGINAL 02/28/2014 E-MAIL FROM THE TTC DIRECTOR TO EIA COP (IN BOSNIAN) From: direktor@ttcbh.ba [mailto:direktor@ttcbh.ba] Sent: Friday, February 28, 2014 2:33 PM To: Tanovic, Jusuf Subject: Izvjestaj evaluatora Postovani gosp.Tanovicu, Hvala Vam na dostavljenom izvjestaju od Evaluatora. Zeljeli bi dati par komentara na izvjestaj koji se tice Tekstilno tehnoloskog centra. Htjeli bi smo se osvrnuti na izvjestaj gdje se ne slazemo sa negativnim pristupom, koji Vam je poslao vlasnik TTC u pismu(mailu) i neargumentiranim negativnim zakljuccima.U daljnjem tekstu saljemo par tacaka s kojim se ne bi slozili. • U izvjestaju je navedeno da je 7 uposlenih primilo CAD/CAM treninge I da od njih samo troje izvrsava iste. Kao u dostavljenoj tabeli, 9 uposlenih je proslo razne CAD/CAM treninge od kojih njih dvoje nisu bili uposleni za vrijeme evaluacije. Od njih 3 vrsi funkciju na CAD sistemu (Accumark softwaeru), te 3 njih vrsi CAM funkcije (operiranje katerom i masinom za nalaganje). • U dostavljenim CV-ovima 7 uposlenih je imalo radnog iskustva na slicnim ili istim pozicijama. ZM (Zijad Mesic) ima 27 godina iskustva, no u izvjestaju je navedeno da nema nikako. • U izvjestaju je navedeno da po biznis planu TTC-ovi prihodi za 2011 i 2012 godinu su trebali iznositi 491130BAM, TTC-ovi prihodi su iznosili 73976BAM u 2012 sa gubitkom od 51888. U obzir nije uzeto da je projekat TTC kasnio I da je zvanicno otvaranje TTC-a bilo 28.03.2012 godine. Tesko je postici prihod od 491130 BAM u roku devet mjeseci. • Istina je da glavni promet TTC-a se svodi na firme Kismet I Vokal, zbog cega TTC ustvari ima dobru bazu za poslovanje. Navodimo s tim u vezi da TTC ima slobodnih kapaciteta I da ni jednom kupcu nikada nije odbijena narudzba. Shodno novim uslugama koje TTC na trzistu, razumivo je sporo reagovanje kupaca. Napominjemo da je TTC sasvim korektno odradio markentiski prema kupcima. Shodno ovakvim cinjenicama vidljivo je da TTC ima sasvim ravnopravan pristup prema svim kupcima. S postovanjem, Semin Banjic ORIGINAL 08/25/2013 E-MAIL FROM THE TTC’S FOUNDER TO EIA COP (IN BOSNIAN) From: alen karic [mailto:alenkaric@yahoo.com] Sent: Sunday, August 25, 2013 8:49 PM To: Tanovic, Jusuf Subject: obavjestenje o posjeti Postovani g. Tanovic, u petak je bio zavrsni sastanak vezano za evulaciju projekta koji je zavrsen u TTC-u Brijesnica Mala.Ne mogu a da se otmem dojmu I osjecam moralnu obavezu da vam prenesem dojmove,a mislim da I vi iz EDC-a ce te biti u najmanju ruku iznenadjeni... Naime moram reci da sam obavjestenje za sastanak u petak u nasim prostorijama primio tek u srijedu ,a da sam u tom momentu bio na sluzbenom putu u Njemackoj.Telefonom sam molio g. Pavelica za odgodu sastanka za ponedjeljak(sutra),ali nije bilo uvazeno stoga sam morao prekinuti veoma vazne aktivnosti sa eventualno novim kupcima da bih stigao ponovo u Bosnu.G.Pavelic mi je rekao da nije neophodno I da sastanak mogu odrzati sa direktorom TTC-a Seminom.Vama je poznato da je kod nas izbor direktora bio jako problematican I da sve vazne stvari zajedno sa direktorom licno radim ja kao I drugi ljudi iz personala Kismeta(izet Banjic I Armin Hasic).Nije tajna da director treba podrsku svih nas.Receno mi je da ce sastanak biti onako vise rutinski ,ali sva sreca da nisam povjerovao.Usput,navedeni dvojac je bio na godisnjem odmoru,a nas director I nije spreman da odgovara na sva pitanja s obzirom na disperziju obaveza na osoblje I takodje na njegovo ne iskustvo od tek jednu godinu na poslovima rukovodioca.Sav taj pristup prema nama je zaista bio tendeciozan u sta cu se I licno uvjeriti.Imam osjecaj da smo "trebali " biti zateceni I seprtljavi do kraja.Na zalost za sve sto oni smatraju da nije uredu sa ovim projektom, krive Vas jer kako kazu da ste za to placeni "deset ,jedanaest hiljada"ne sjecam se koje valute doduse(ali ako smo u BIH onda je BAM,pretpostavljam).Mislim da se trazila greska vise u projektu...Ne mislim da je projekat los cak naprotiv ...Mi smo takodje investirali veliki novac I to je garant moje vizije.Na zalost ,danas se osjecam,a poslije ovog astanka u petak,kao da sam nekakav lopov ,a imam utisak kao da se zeli pretstaviti da vi iz EDC-a trosite novac onako a nama firmama dajete novac na kasikicu kao sto gospodja rece samo 17% od ukupnog fonda.Ja znam ,a ponovicu to ako treba I pred javnosti ,da smo mi dobili I vise od onoga sto smo trazili u aplikaciji(oprema) I vise nego sto smo ocekivali kao strucnu pomoc ,prije svega Vas I Vaseg osoblja ,a takodje I da ne nabrajam silne konsultante I eksperte iz struke.Odgovorno tvrdim da se za angazman eksternog kadra(Emir ,Marti itd...) sumlja...Kad?Koji datum?Koliko? itd...Mislim da pismeni odgovori nista ne znace.Cijelo vrijeme je bio ukljucen snimac tona I brze odgovore zele.Kao u filmovima kada se kriminalac ispituje.Tako sam se I osjecao.Ja sam mislio da u pravnom svijetu je dovoljna pismena izjava kada su upitanju podaci,a da se interviju uzima usmeno ,ali odgovori su u glavnom abcd... ili da ili ne koji takodje pravno ne obavezuju.Npr. kazu da nismo upisali capital u TTC( mislim na novac koji je Kismet bio duzan uplatiti na racun TTC) I da je to strasno.Ja uopce ne smatram nikakvom greskom jer cemo to obaviti do kraja godine ,a kako smo Vas ranije I obavjestili, novac je naravno I uplacen za sta postoje dokazi banke .Zao mi je sto USAID angazuje racunovodje (one koje gledaju knjige) ,a ne I one koji razumiju ili zele da razumiju trziste,viziju,misiju,tehnoloske procese.a takodje I objektivne probleme na koje nailazimo svakodnevno.Pored svih nabrojanih definitivno sam uvjeren da nismo nista pogrijesili I da USAID nije bacio novac,a ja svoj takodje.Bit ce mi zao ,ako ljudi iz USAIDA steknu dojam da je nesto bilo pogresno sa nase strane a takodje da Vas angazman se ocijeni na osnovu tendencija ,sto apsulutno nije tacno.Takodje ,metod ispitivanja nasih kupaca je po meni jako pogresan jer nasi kupci mogu pomisliti da smo mi nesto pronevjerili I to stvara losu sliku o TTC￾u.Smatram da se hitno kupci TTC-a moraju na ovaj nacin prestati ispitivati.Neka gospoda dodje na dane otvorenih vrata pa neka cuje dojmove.Podsjecam da su bili izuzetni I da svi prisutni ocjenjuju da su usluge TTC-a neophodne u sta se uvjerio I licno director USAIDA na jednom od "dana otvorenih vrata" .Jedna godina je premali da se nove usluge u bransi implementiraju .To je rekao I g. Marti .Mi smo na dobrom putu da nasi klijenti u narednom period sve vise I vise koriste usluge TTC-a.Gospoda izrazava sumnju da TTC koristi Kismetu,a ja kazem da je sreca sto kismet ima dovoljno posla kako bi TTC bio odrziv,a ne sjecam se da su postavili pitanje klijentima ,kada su ih vec prozivali,da li je TTC nekoga odbio ili nije imao kapaciteta zato sto npr.radi z Kismet.TTC ima slobodnih kapaciteta jos I oni rastu kako se osoblje bolje educira ,na kraju krajeva nismo ni blizu da uvedemo drugu smjenu u sta uopce ne sumnjam da ce se desiti vremenom. N kraju zelim da istaknem da sam jako razocaram nacinom na koji vrsi Evulacija.Bice mi zao sto na zalost neko zeli mozda da podigne rejting sebi ili svojoj firmi tako sto ce mukotrpno gradjen centar I sve nas koji smo u njega involvirani pretstaviti kao neznalice ili nepostene .Za Vas rad I rad vasih saradnika cu svjedociti pred bilo kome da je bio domacinski,predan I posten. Svako dobro zelim... Alen Karic KISMET d.o.o 74206 Brijesnica Mala b.b. Doboj Istok Tel.: + 387 35 722 501 Fax: + 387 35 727 140 ORIGINAL 08/30/2013 E-MAIL FROM THE EXPLORING BH DIRECTOR TO EIA COP (IN BOSNIAN) From: sasa vasic [mailto:sasa@exploringbh.ba] Sent: Friday, August 30, 2013 9:56 AM To: Tanovic, Jusuf Subject: Exploringbh-sastanak sa evaluatorima Postovani Juka, Zelim da vam prenesem svoje iskustvo koje sam imala sa evaluatorima projekta a koje nije bas tako pozitivno. Prvi sastanak koji sam imala zakazan na pocetku mjeseca je uzasno bio mucan, osobe koje su se predstavile kao evaluatori projekta su se ponasali kao policajci a ja kao da sam nesto ukrala. Pitanja koja su postavljali nisu bila za izvrsnu direktoricu, ja kao neko ko je dosao na mjesto dirktorice nakon zavrsetka procesa dodjele granta ne treba da znam ko je sjedio u komisiji pri odabru granta i sl.. Na prvom sastanku manje je bilo pitanja vezano za struku a dosta za vlasinstvo I odabir partnera (makedonaca), nacin ispitivanja je bio takav da sam se osjecala kao na optuzenickoj klupi, dosta neugodan. Drugi sastanak se desio sad u ponedjeljak koji je prosao dosta bolje jer su pitanja bila vezana iskljucivo za struku. Moj generalni utisak je da evaluatori nisu bas najbolje shvatili svrhu projekta, da im je jako tesko shvatiti na koji nacin smo doprinjeli promociji turizma u BiH I kako funkcionise cijeli taj sektor. S obzirom da sam nakon prvog sastanka iskreno bila sokirana nacinom ispitivanja, osjecam duznost da to podijelim s vama, mislim da bi vi kao neko ko je Exploringbh pratio pune cetiri godine treba da znate I ovu informaciju. S postovanjem, Sasa Vasic Saša Vasić Exploringbh d.o.o. Humska 17 71 000 Sarajevo mob: +387 61 028 964 +387 63 940 111 www.exploringbh.ba Assessment of e-BIZ Centers of the USAID Bosnia & Herzegovina Excellence in Innovation Project Submitted by: Cardno Emerging Markets USA, Ltd. Submitted to: EDC Prepared by: David T. King, Ph.D. In response to: Agreement No. 14156 30 December 2013 Cardno Emerging Markets Table of Contents OVERVIEW – KEY FINDINGS AND CONCLUSIONS ......................................... 1 THE INDIVIDUAL E-BIZ CENTERS...................................................................... 6 1. Arteco Wood Technology Center ........................................................... 6 2. Exploring Bosnia...................................................................................... 6 3. Textile Technology Center ...................................................................... 7 4. Transwork ................................................................................................. 9 FINAL COMMENTS .............................................................................................. 9 Financing Growth (FG) Page ii Cardno Emerging Markets Overview – Key Findings and Conclusions This report assesses the overall development impact of the four “e-BIZ centers” created by the USAID Bosnia & Herzegovina (BiH) Excellence in Innovation project (EI, or “the Project”), which was implemented by EDC and ran from 2007 to 2013. EI Project Concept EI had an innovative – even experimental – design, which had been applied with success in Macedonia under EDC’s implementation. It emerged in the mid-2000s in USAID Washington’s EGAT (now E3) bureau / Office of Infrastructure and Engineering (I&E), when it was becoming obvious that information and communications technology was becoming a critical driver of enterprise competitiveness. ICT could help companies increase their efficiency and productivity, drive down costs, improve product quality, manage purchasing and invoicing, and shorten product development and delivery cycles. I&E featured examples from the Macedonia project in USAID was seeking ways to apply ICT to economic development, and these included such things as supporting mobile phone-based “last mile initiatives”, product bar coding, smartcards for payment processing and portable credit information, supply chain traceability PC applications, and many others. The EI concept was to solicit proposals from companies in industry who could install, manage, and employ ICT-based systems primarily for the broader benefit of the sectors in which they operated, by providing services using the systems to numerous outside SMEs. EI would purchase the indicated systems, deliver them to the selected beneficiaries and, subject to performance conditions (including co￾investment), eventually grant the systems to the beneficiaries. With values ranging from $100-250,000, these were substantial capital investments provided to the beneficiaries. In addition, EI would work closely with beneficiary companies to help them solidify their business plans and improve their operational processes and marketing and sales functions, in order to help assure a successful investment. This assessment, therefore, addresses three interrelated questions that summarize USAID’s overall purposes in designing and financing the Project: • What impact have the e-BIZ centers had on the four sectors they have served to date? • What is the likely impact of the e-BIZ centers on the sectors they have served in the longer run – e.g. five years from now? • What are the prospects for sustainability of the e-BIZ centers? The e-BIZ Centers In the order of their launching, the four e-BIZ centers established by EI are: 1. Exploring Bosnia (“Exploring”), in Sarajevo, established in August 2009 by a consortium of two existing companies, e-Tourism and Exploring Macedonia, an online portal for booking hotels and tours in BiH. EI covered the technical costs of developing this website, and subsequent promotional expenses. EI’s total investment for Exploring was approximately $125K, and Exploring invested approximately $70K. 2. Arteco Wood Technology Center (“Arteco”), in Tešanj, launched in October 2009 by Artisan, an existing furniture manufacturer, to house an advanced (computer-controlled, 5-axis) woodworking machine provided by EI. Arteco was expected to provide services to SMEs in the BiH wood processing industry based on this equipment. EI’s total investment for Arteco was approximately $305K, while the center contributed some $235K. Financing Growth (FG) Page 1 Cardno Emerging Markets 3. Textile Technology Center (TTC), in Doboj East, started in April 2012 by Kismet, an existing apparel manufacturer, to house advanced (computer-controlled, laser-based) fabric pattern plotting and cutting equipment provided by EI. TTC was expected to provide services to SMEs in the BiH apparel design and manufacturing industry, based on this equipment. EI’s total investment for TTC was about $270K, while the beneficiary invested about $285K. 4. Transwork, in Banja Luka, started in March 2012 by an existing company, Satwork, to provide services to SMEs in the freight transport industry in load information exchange, fuel usage, and fleet management. EI provided computer network servers and covered technical development costs for these services. EI’s total investment for Transwork was close to $135K, while the center put in approximately $145K. Key Findings and Conclusions i. Grantees benefitted substantially from the receipt of the systems. There was one potential risk in the basic design of the Project, which was clear from the beginning. The proposals would necessarily need to come from existing companies, since only they would have the experience in their sectors to understand and employ the proposed ICT-based systems effectively. This meant that there was always a certain probability that the systems could be used primarily for the benefit of the grantee company, rather than prioritizing services to other SMEs in the sector. To try to prevent this, the grantee companies were obligated under their contracts with EI to serve other SMEs – with a target of 40-50 for each center – and they understood this commitment. Two of the four centers – Arteco and TTC – substantially benefitted their founder-owner companies – Artisan and Kismet, respectively. The expensive equipment provided by EI was of direct use in their own production operations, and in both cases has lifted their value-added capabilities had a substantial positive impact on their businesses. While it is possible to consider these two owner companies to be clients of their respective e-BIZ subsidiaries (as, for example, a consultant hired by the project characterized Artisan’s relationship with Arteco1 ), this was not the intention of the grant. Nevertheless, this did provide positive development impact, significantly increasing sales and employment of the grantees, and sometimes overflow demand to outside producers. In the case of Kismet, thanks to the customer response and productivity improvements provided by the laser cutter, the sewing capacities of its factory are now periodically exceeded, and it is outsourcing production for its own sales to other garment producers in BiH – an impact on outside SMEs that was not anticipated by the project. Similarly, the equipment has enabled Artisan to take on larger contracts, in several cases involving multiple collaborating companies. Transwork and Exploring were not operations that could provide services to their founder-owners. For Transwork, there were cross-selling opportunities between it and Satwork, but no direct input relationship. For Exploring, a similar statement could be made for its minority owner (Exploring Macedonia), and to a lesser extent for its majority owner and the contractual grantee, e-Turizam. ii. Three of the four e-BIZ centers have performed in the manner expected, with positive development impact on their respective sectors. This means that that these centers – Arteco, TTC, and Transwork – have successfully marketed their services to SMEs in their sectors, that each has built a clientele, that this clientele is expanding, and that their revenues are significant, and for TTC and Transwork strongly growing. Vis-à-vis the contractual target of providing services to 40-50 outside clients in each sector, the actual numbers at the end of the EI project (August 2013) were 112 for Arteco, 40 for TTC, and 43 for Transwork. As seen in Table 1, all three have achieved significant revenues since startup. TTC has substantially increased the proportion of its sales to third parties, and for Arteco, while somewhat up and down, the average proportion of third-party sales over its five-year history is more than half – 54%. Interviews with clients have indicated satisfaction with the services offered2 , and this is 1 Shék Jain Trip Report, September 2010 2 Marty McElwee Trip Report covering TTC, June 2012 Financing Growth (FG) Page 2 Cardno Emerging Markets further evidenced by the extensive repeat business that all three of these centers have with many of their clients. For Arteco, 45 of its 112 clients – 40% – are repeat customers. All 29 of Transwork’s year 1 customers repeated in year 2. 60% of TTC’s first year clients repeated. For Exploring, 35 hotels have been long-term customers. This indicates that the centers have become important ongoing service providers to numerous SMEs in their sectors. Table 1: Impact Indicators e-BIZ Center EI Investment (KM) Launch Date Revenue (KM) / Clients Item 2008 2009 2010 2011 2012 2013 Exploring Bosnia 176,687 Aug-09 Net Revenue 22,176 37,296 20,530 38,992 32,000 Growth -- 68% -45% 90% -18% Clients 39 78 86 87 87 Arteco 420,000 Oct-09 Revenue 203,025 330,825 388,869 158,3793 117,809 Growth -- 63% 18% -59% -26% % 3rd Party 65% 31% 65% 71% 37% Clients 29 60 82 87 112 Transwork 199,737 Mar-12 Revenue 72,980 195,000 Growth -- 167% Clients 29 50 TTC 381,386 Apr-12 Revenue 73,977 176,000 Growth -- 138% % 3rd Party 30% 63% Clients 13 44 In the case of Arteco, the numbers of SMEs obtaining the production-related services of the system initially fell short of expectations. Arteco was not as aggressive in its early period in marketing to outside companies as expected, and its parent company, Artisan, dominated the use of the 5-axis CNC machine. The warning provided by one expert4 that Artisan might want to get the machine primarily for its own use seemed justified. With collaboration from other projects (FIRMA), Arteco was able to improve its services to third-party firms by offering CNC training, which while not exactly in line with initial expectations for production-related services, did have a positive impact on the sector. Furthermore, pressure and support from the EI project to better market services eventually had a favorable impact, as indicated by the data in Table 1. It should be noted that another recognized difficulty with the EI project concept was the “trust issue”. That is, in some cases (Arteco and TTC) the centers would be seeking to sell services – such as providing manufacturing specifications or prototypes for original designs – to firms that were competitors of their owner companies. The outside firms would have to trust that the e-BIZ centers would keep product information confidential and not share it with their parent companies. The building of trust, supported by a formal nondisclosure agreement, has progressed but has taken time – and for some third companies, these suspicions have been insurmountable. 3 Arteco’s revenues for 2009-11 included “pass-through” invoices due to its role as leader of a group of firms cooperating on large Artisan projects. This practice ceased in 2012. 4 The Wood Team Leader on USAID’s BiH FIRMA project Financing Growth (FG) Page 3 Cardno Emerging Markets Also somewhat unexpected was the positive “lighthouse” impact of EI-supported centers. When Arteco received its 5-axis CNC machine, it was the only one in the country. Engaging accomplished Bosnian and Western designers, Artisan quickly demonstrated how useful the machine was in producing market￾forward high-end furniture styles, winning design prizes at the Cologne Furniture Fair in 2010 and 2011 – one of the very few companies from non-Western European countries to do so. Other furniture producers in BiH noted this success, and now several of them have acquired these machines, some with more than one. A similar impact was felt in the apparel sector in Macedonia after the predecessor EI project there, also implemented by EDC, established a laser cutter-equipped e-BIZ center – the first such system in the country. There are now 15 such systems in Macedonia, acquired by other apparel producers who were able to see their productivity and business impact thanks to the initial e-BIZ center.5 There is no doubt that this has broadly improved the competitiveness of the entire Macedonian apparel sector, a similar impact can be expected in BiH over time. Again, this was not a planned effect. Though the acquisition of similar advanced systems by outside companies means that they would not be potential clients of the EI-supported e-BIZ center, the demonstration effects should encourage demand from smaller manufacturing operations in wood and apparel that do not have the capacity or financial resources to justify obtaining one. The overall competitiveness of both sectors is improved. iii. The impact of the e-BIZ centers on their sectors is likely to continue to grow. The sales, client numbers, and third party revenues trends for Arteco, TTC, and Transwork are generally positive. The evidence of repeat customer business indicates that services provided are of ongoing value to many outside clients. For both Arteco and TTC, the services that appear to be most useful to third party SMEs involve prototyping – using IT systems to create production designs and specifications for manufacturers that fulfill buyer needs, maximize efficiency in material use, and optimize labor and production equipment use time. These services include provision of “blueprints” in various forms for reproduction – for example, markers for fabric cutting. Prototype services may also include the production of samples. Solid growth in the demand for these services for both Arteco and TTC has been demonstrated. These services can only be produced by the kind of advanced systems they obtained from EI, which do not make economic sense for smaller manufacturers to buy. As Arteco and TTC continue to gain the trust of third party clients – that they will not divulge designs and customers to their parent companies – it is highly reasonable to expect that the positive growth trends already witnessed will continue, and that the impact of the centers on their sectors will expand over the longer term. For Arteco, technical training in the use of computer-controlled wood cutting equipment has also been a consistent source of revenue. The provision of actual production services to outside clients has fallen short of original expectations. That is, both Arteco’s 5-axis CNC wood cutting machine and TTC’s computer-operated laser fabric cutter are capable of delivering assembly-ready parts to third-party manufacturers. It appears that while most SME manufacturers find value and efficiency in the systems’ prototyping services, many have their own parts production capabilities, even if manual, and do not find it economically beneficial to buy these services from the e-BIZ centers. Certainly the parent companies’ use of the systems in their manufacturing operations will continue and grow, and third-party demand for production cutting will grow, but this service will likely remain secondary to those for prototyping. Meanwhile, Transwork’s strong expansion in a relatively short time period is quite promising. Its GPS tracking, communications and navigation, fuel-use and time monitoring, load optimization, and forthcoming broader fleet management services address a rapidly growing demand niche that has much untapped potential, based on estimates of the still-modest share of the truck fleet in Bosnia and neighboring countries that is currently equipped with these services. Indeed, as this assessment was being 5 McElwee Trip Report Financing Growth (FG) Page 4 Cardno Emerging Markets prepared, post-EI project, Transwork was awarded a major tender from the RS Postal Service to install its GPS tracking service equipment in 250 of their delivery trucks. There is little doubt that Transwork’s impact on its focus sector will expand substantially over the years to come. While not without competitors, Arteco, TTC, and Transwork have found a local clientele and based on aggressive marketing and good word of mouth are positioned to grow. iv. At least three of the four e-BIZ centers are likely to be sustainable. The question of sustainability overlaps to an extent with the immediately preceding discussion as to the likelihood of future demand growth for the centers’ services. Sustainability in addition requires that the e-BIZ centers are well￾managed and profitable. It is important to emphasize that the managers of Arteco, TTC, Transwork, and even Exploring all consider their centers to be attractive businesses in their own right and are committed to their expansion, separately from their parent companies. All four of the centers have full-time dedicated management, even though for Arteco, TTC, and Transwork key parent company owner-managers work closely with their e-BIZ center management in business planning. In all four cases, center management – entirely independent or teamed with the owner-manager – is highly entrepreneurial, capable, and committed to building the business. While Exploring is a 1-person shop, the other three have dedicated staffing: Arteco (8 persons), TTC (10), and Transwork (4). Arteco, TTC, and Transwork appear to be covering their costs, at least for personnel, or slightly profitable. With further growth they are all likely to become fully profitable. It can be expected that these three centers are sustainable for the long term. In terms of total revenues, Transwork has more than earned back EI’s investment in it, Arteco about 80% of it, Exploring about 85%, and TTC about half. Looking at all four centers as a group, the owner/founders’ view of sustainability is indicated by the fact that they have invested a total of KM 1.08 million in them, compared to KM 1.18 million provided by EI – almost half of the entire cost to date. v. The fourth center has failed mainly due to global developments, but may still have a future. Exploring is clearly the most problematic center. The basic cause is the very rapid growth of globally￾provided internet-based services, whose underestimation has been a hallmark of the information age. Exploring was seen as a critical need for the development of the tourism sector at a time when there was no booking service for hotel rooms in BiH. The grant was provided to deliver a service that did not exist. However, very soon after this center was launched, it was overtaken by very large booking services – Expedia, Travelocity, TripAdvisor, and especially Booking.com. Once these global services entered the market, Exploring’s paying clientele – local tourism services – evaporated. They no longer needed Exploring for booking services, because they were so well-served by these well-known, trusted services. However, the entrepreneur now behind Exploring is still dedicated to the business and believes it can be profitable – not as a booking engine, but as a BiH tour packager, marketing to operators in tourist￾generating countries (including such new markets as Russia) interested in sending tour groups to BiH, and intermediating between them and tourism service providers in BiH. She continues to believe in the business and is investing time and money in it, striving to make it a success. While Exploring certainly will not fulfill its principal originally-intended service – because it is a service now provided by other very large companies in the market that have enormous market footprints – it may well survive with this altered business model. Certainly that is the intention of the entrepreneur. Financing Growth (FG) Page 5 Cardno Emerging Markets The Individual e-BIZ Centers 1. Arteco Wood Technology Center Arteco was founded in October 2009 in Tešanj, FBiH, by Artisan, a successful high-end furniture designer and manufacturer with an established Western European export business. Artisan had been a beneficiary of a USAID competitiveness projects in BiH that focused on the wood sector – CCA – and indeed it was CCA that originally recommended Artisan to EI. Responding to a tender, Artisan requested and won the grant of a 5-axis CNC wood machining system. The 5-axis format allows highly accurate cutting of wood parts in a wide range of shapes without multiple setups. It greatly expands furniture design options, makes highly consistent and accurate parts, and economizes labor time and material. The e-BIZ center has a dedicated manager (Elvedin Mandžukić), and one of the owners of the parent company (Fadil Čostović) works closely with him in expanding and marketing Arteco's services. Acquisition of the system substantially benefitted the parent company, Artisan, leading to development of some award-winning designs and expanding its product line and exports. Artisan has added 20 jobs since the machine became available to it through Arteco. There was concern that the parent company’s use of the equipment was the dominant priority. However, marketing efforts intensified with the encouragement of EI, and Arteco gained SME customers, mainly at first for prototyping and CNC training. The company has 30-35 small company and individual customers that are occasional users, approximately 45 SMEs that are repeat customers using one or multiple services, and 27 foreign companies that have ordered complete designs and technical documentation, as well as prototyping for various products. Arteco’s prototyping services have also led to follow-on business for third-party producers with whom it cooperates. There are 14 of these cooperating companies that have participated in multiple Arteco￾managed projects for example in hotel furniture outfitting. The high-profile role of Artisan in the BiH furniture sector also generated trust issues – other companies were fearful that if they used Arteco’s prototyping services, their designs or customers could be stolen. Arteco has worked to establish its independence from Artisan, and Artisan now plans to move to separate premises (building permits have been issued), which will help address this issue. The center also has aggressive expansion plans: it is adding another CNC machine and a wood bending machine, in order to offer more advanced services. The original machine now often runs on three shifts. Its use can be booked online. Marketing efforts also target foreign buyers. Arteco, which has previously participated in the BiH wood industry’s joint stand (supported by FIRMA) at major international trade shows, for the first time will have its own stand at the Cologne Furniture Fair (the largest in Europe) in 2014. It is actively working to expand its client base. Arteco has 8 employees currently, which will grow to 10 when the new equipment is added, and expects to employ 15 by end-2015. Though its parts production for third parties may not have matched initial expectations, Arteco has had additional / different development impact in its active workforce training and internship programs. It is a growing and successful operation. 2. Exploring Bosnia Exploring Bosnia was established in August 2009 to address what had been identified as a significant obstacle to development of the BiH tourism sector – the absence of an online booking engine. This gap was identified during the USAID Cluster Competitiveness Activity (CCA) project, which supported the development of the tourism sector, as early as 2005, and indeed CCA had every intention of itself supporting the development of such a website. However, when EI was conceived, such an initiative was considered appropriate for its support. A similar USAID project had previously supported Exploring Macedonia, which had been successful and it eventually evolved into the country’s principal tourism promotion website. Exploring Macedonia became a 49% owner of Exploring Bosnia, and was expected Financing Growth (FG) Page 6 Cardno Emerging Markets to host the site on its servers and provide substantial software development and technical support. A BiH foundation, e-Turizam, was the 51% owner. However, given various delays both in starting the EI project and concluding the grant for Exploring, the promising original concept of a “Bosnia Expedia” hit the market only a little before major services took over the global market for these services. A Bosnia Expedia was no longer necessary – Expedia itself was here, along with Tripadvisor, Booking.com, and other major engines. When it marketed initially, Exploring was able to sign up a number of users. But only a year later, with the arrival of the major services, the value of using a local service that could never be the equal of the global ones disappeared, and many of the original customers did not renew their paid subscriptions. EI consultants provided extensive recommendations of what to Exploring needed to do to survive, but these would have required very large additional investment and even then could not have made the company competitive with the major global providers. The technical contributions of the two owners to Exploring were also disappointing. Exploring Macedonia (EM) consistently lagged in delivering technical services to Exploring Bosnia. On occasion the website was down for hours before EM addressed the problem. Eventually hosting services were removed from EM and transferred to BiH. Also, e-Turizam itself did not have adequate experience in the tourism sector, nor in world-class portal development. And as it became obvious that the original business model was doomed, the two owners became less interested. A positive factor was the entrepreneurial spirit and dedication of Ms. Saša Vasić, who became Director of Exploring, along with the active support of the EI project. With their efforts the business model was modified and Exploring evolved into a promoter and marketer of tourism to BiH and an event and tour packager, putting together the services of various local providers for foreign tourism companies interested in sending tour groups to BiH. For example, an agreement was signed with the US-based rafting tourism organization. Essentially due to the initiative of Ms. Vasić, this business developed, and as the original owners, EM and e-Turizam, lost interest, she was prepared to purchase their shares in the business and become its proprietor. This deal has not yet gone through6 , but she remains dedicated to the business and periodically active in it, using her own resources, including recent participation in the Moscow tourism fair, which generated promising contacts for bringing Russian skiers to Jahorina. (Air Moscow is initiating a weekly flight to Sarajevo at the end of this December.) Exploring could also have a role in helping to coordinate various public and private stakeholders in BiH tourism, especially as USAID and EU projects that have supported the Tourism Working Group wind down. This is something for which it could attract donor or BiH government funding. The connection of Exploring to USAID projects provides credibility in this area. It can advocate for tourism service providers in BiH. Despite a business premise that turned out to be faulty for unforseen reasons, Exploring is still alive, a year after it received its last financial support from EI. Its entrepreneurial director is still committed. If it can get out from under its current cloud, it has a chance to survive. 3. Textile Technology Center TTC was launched in April 2012 and is located in East Doboj, Federation BiH. Its owner, Kismet, won a competition held by EI for the grant of a computer-guided Gerber laser-knife fabric cutting system. (Gerber, a Connecticut-headquartered company, is the global leader in this technology.) It has a dedicated full-time manager (Semin Banjić), and the owner-manager of Kismet (Alen Karić) is closely involved in TTC’s business strategy and planning. Its manager devotes a large part of his time to marketing. 6 The transaction was on the verge of being completed in mid-2013 when it became known that the USAID￾procured evaluation of the EI project would be particularly negative with respect to Exploring Bosnia. This reputational risk threatened to reduce the value of the business, and led Ms. Vasić to drop her plan to buy out EM. Financing Growth (FG) Page 7 Cardno Emerging Markets TTC’s launch was postponed due to the EI project hiatus caused by the AED situation, so has been in operation for less than two years. TTC delayed aggressive marketing in order to ensure that it could use the laser cutter without mistakes, since those would be highly costly if an outside firm’s fabric was mis￾cut. In effect, the problems were worked out on Kismet cutting batches. TTC also has had to work to build trust with clients that using the center would not lead to losing business or designs to Kismet. This has progressed positively. TTC estimates that there are 120-130 apparel manufacturers in BiH, of which 80-90 are potential customers. It is now doing business with 44 of them (including 4 clients added since the end of the EI project), which means it is already serving approximately half of its prospective market. The most-used services are in digitizing patterns, grading them (i.e., duplicating them across a range of sizes), and plotting markers (the paper sheets that are placed on top of a stack of fabric spread on a cutting table to guide the cutter). The digital plotting and marker-making program also optimizes fabric use, reducing waste fabric to a bare minimum. This is a highly useful service to smaller apparel manufacturers who would otherwise have to work out the grading and marker making by hand, with less efficiency of material use, as well as labor time. These jobs average about € 300 each – they are small, but frequent. Kismet remains TTC’s most important single customer, because in addition to the digitizing services, it also pays TTC to cut almost all of its fabric on the laser cutter in preparation for moving it to its sewing factory. Far fewer of the outside customers using the digitizing services proceed to use TTC’s cutting services, because they have in-house manual cutting (skilled operators using specialized power cutting knives) which is relatively efficient. There are 5-6 larger companies (Granoff, Belissimo) that are using cutting services more frequently, to complement their in-house cutting units. There is no question that the availability of TTC’s services to Kismet has had a significant positive effect on Kismet’s business. Kismet has added 25 employees since TTC obtained the cutter – a 24% increase – and is now periodically itself outsourcing garment assembly work to other sewing factories, when its capacities are exceeded. TTC itself has added staff – it now has 10 employees and expects to double that over the coming years. TTC is currently co-located with Kismet, but as is recognized by management and the owner, needs more space in order to maximize its business potential. A plan is underway for Kismet to build a new factory nearby, and for TTC to take the space currently occupied in the original building by Kismet. Global market trends are a key driver of TTC’s potential business. Increasingly, major European apparel brands who outsource their cut-and-sew work want full service from their suppliers, meaning that the buyers want to provide only a sample and physical patterns for one size, plus fabric and findings (buttons, zippers, etc.), and have the apparel manufacturers take care of all of the technical details – grading, marker-making, production specifications. They expect manufacturers’ bids for work to include fabric useage estimates, and those with lower fabric use for the same number of final garments will of course be favored, since the brand is purchasing and shipping the fabric to the manufacturer. This is now possible because of digital technologies, so creates a ready market for such services. There is also a trend for European buyers to shift production back into Eastern Europe from China and Southeast Asia, due in part to potential liabilities (or at least bad publicity) highlighted by media attention to poor worker conditions and catastrophic accidents in some apparel factories in those countries. Also, with increasingly frequent style changes and the need for in-season re-orders, location matters: factories in BiH can do smaller runs (several hundred rather than several thousand pieces) and deliver them more quickly to Western Europe. Again, the availability of digital services greatly assists such responsiveness. The key challenges for TTC are (i) getting more customers, requiring among other things that they continue to overcome trust issues, (ii) finding qualified engineers, and (iii) expanding its physical space. As noted above, the latter is in process, and it will also help with the customer trust issue for TTC to be in its own, entirely separate facility from Kismet. Financing Growth (FG) Page 8 Cardno Emerging Markets 4. Transwork Transwork, located in Banja Luka, Republika Srpska, was founded in March 2012 as a subsidiary of Satwork, a company in the GPS tracking devices and software business. Transwork has a dedicated manager (Mr. Saša Mičić), and Satwork's General Director (Siniša Palić) is actively involved in strategy. The management team is highly entrepreneurial and capable. Satwork sells its units directly to consumers and to truck, auto, and auto parts retailers. Satwork saw value in establishing Transwork to expand sales of its systems in the wholesale freight and logistics market by adding extensive services to its GPS systems, and successfully bid for ICT equipment from the EI program, which consisted primarily of powerful network servers. Transwork’s software for supplying its services was developed entirely in-house, with major contributions from Satwork. Its various services provide truck fleet operators the ability to monitor all of their vehicles constantly, including their exact locations, their fuel use, their speed, and their down time. The software enables real time data exchange between an owner or fleet manager and all of his/her trucks. All of the services envisioned in its original business plan are already functional, and Transwork is now developing full route management and delivery optimization services. Transwork has a fulltime director of sales and marketing (Valentina Rujević), who finds it relatively easy to educate potential customers on the value of these services. More and more trucks do have GPS location systems installed, and Transwork’s enhanced software can be used with these. Market studies have shown than only 5-6% of all trucks in the SEE region are using such services, so substantial growth is possible. The company is systematically marketing to a customer base of some 1,000 trucking companies in BiH, Croatia, and Serbia. It is also investigating expansion to sea freight. Consequently, Transwork’s revenues and client base have grown rapidly in less than two years. The company has 50 clients (having added 4 new clients since the end of the EI project) who make monthly payments for its services, representing some 750 trucks with installed GPS equipment (including 250 RS Postal Service delivery vehicles for which Transwork just won a large competitive tender). In addition, partly to attract potential clients, Transwork offers a free online “freight exchange”, where truckers can offer availability or seek loads when they are returning from a delivery or on the way to pick one up, intermediating supply and demand and providing a means of expanded revenue for sector participants. The exchange typically displays offers from all over Europe. There are 208 active members of the freight exchange (including the 50 contract customers), so Transwork’s impact on the BiH logistics sector is considerably larger, in terms of SMEs serviced, than indicated by the number of paying customers alone. Transwork has competitors, but because it has developed all of its software in house, it can be readily updated, can more easily be customized for client needs, and is available in local languages as well as German and English. Some competitors are selling using off-the-shelf or franchised applications that are less responsive. Transwork is also now the only company in its field in BiH that provides international as well as domestic expanded GPS-based services. Transwork’s challenges include (i) building trust among buyers that a BiH firm’s software is as functional and reliable as that of German and French competitors (though it is highly price competitive with them), (ii) finding trained programming staff, because the field is relatively new. (It obtains people from Satwork, who are trained in-house.) This e-BIZ center is already profitable. This is a dynamic, competitive, growing firm, catalyzed and enabled by USAID EI. Final Comments It bears repeating that the EI model of development impact was itself innovative and quasi-experimental, having been employed only once before in the same way, in Macedonia. It extended a series of USAID efforts over time to provide substantial equity or equity-like capital to individual businesses, including in Financing Growth (FG) Page 9 Cardno Emerging Markets this region such programs as the E&E Bureau’s Enterprise Funds. We see clear progress to date in contributing to the development and competitiveness of their sectors, and it is likely that three of the four e-BIZ centers will continue to grow and prove to be sustainable, to a significant if not complete degree consistent with their initially expected service scopes. Furthermore, even the fourth center may well survive, though with a different business focus than initially planned. Given this, the project must be considered a significant success – something that USAID should be proud of. The keys to success – meaning sectoral impact and sustainability – have been: (i) informed assessment of market needs, (ii) careful selection of grantee, (iii) capable, entrepreneurial, adaptable management, and (iv) close and continuous monitoring of developments and supporting technical assistance by the sponsoring project. The scope of work for this assessment did not include recommendations for further improvement of the businesses of the centers. These have previously been provided by other consultants through EI technical assistance. Finally, while unrelated to the questions that this assessment was designed to address, it can be noted that in all four cases management of the centers and their owners provided unsolicited appreciation of critical and constructive support, both from USAID for the equipment systems, and also from the EI project itself for the technical and marketing consulting assistance received. Financing Growth (FG) Page 10 Assessment of e-BIZ Centers of the USAID Bosnia & Herzegovina Excellence in Innovation Project Submitted by: Cardno Emerging Markets USA, Ltd. Submitted to: EDC Prepared by: David T. King, Ph.D. In response to: Agreement No. 14156 30 December 2013 Cardno Emerging Markets Table of Contents OVERVIEW – KEY FINDINGS AND CONCLUSIONS ......................................... 1 THE INDIVIDUAL E-BIZ CENTERS...................................................................... 7 1. Arteco Wood Technology Center ........................................................... 7 2. Exploring Bosnia...................................................................................... 8 3. Textile Technology Center ...................................................................... 9 4. Transwork ............................................................................................... 10 FINAL COMMENTS ............................................................................................ 11 Financing Growth (FG) Page ii Cardno Emerging Markets Overview – Key Findings and Conclusions This report assesses the overall development impact of the four “e-BIZ centers” created by the USAID Bosnia & Herzegovina (BiH) Excellence in Innovation project (EI, or “the Project”), which was implemented by EDC and ran from 2007 to 2013. EI Project Concept EI had an innovative – even experimental – design, which had been applied with success in Macedonia under EDC’s implementation. It emerged in the mid-2000s in USAID Washington’s EGAT (now E3) bureau / Office of Infrastructure and Engineering (I&E), when it was becoming obvious that information and communications technology was becoming a critical driver of enterprise competitiveness. ICT could help companies increase their efficiency and productivity, drive down costs, improve product quality, manage purchasing and invoicing, and shorten product development and delivery cycles. I&E featured examples from the Macedonia project in USAID was seeking ways to apply ICT to economic development, and these included such things as supporting mobile phone-based “last mile initiatives”, product bar coding, smartcards for payment processing and portable credit information, supply chain traceability PC applications, and many others. The EI concept was to solicit proposals from companies in industry who could install, manage, and employ ICT-based systems primarily for the broader benefit of the sectors in which they operated, by providing services using the systems to numerous outside SMEs. EI would purchase the indicated systems, deliver them to the selected beneficiaries and, subject to performance conditions (including co￾investment), eventually grant the systems to the beneficiaries. With values ranging from $100-250,000, these were substantial capital investments provided to the beneficiaries. In addition, EI would work closely with beneficiary companies to help them solidify their business plans and improve their operational processes and marketing and sales functions, in order to help assure a successful investment. This assessment, therefore, addresses three interrelated questions that summarize USAID’s overall purposes in designing and financing the Project: • What impact have the e-BIZ centers had on the four sectors they have served to date? • What is the likely impact of the e-BIZ centers on the sectors they have served in the longer run – e.g. five years from now? • What are the prospects for sustainability of the e-BIZ centers? The e-BIZ Centers In the order of their launching, the four e-BIZ centers established by EI are: 1. Exploring Bosnia (“Exploring”), in Sarajevo, established in August 2009 by a consortium of two existing companies, e-Tourism and Exploring Macedonia, an online portal for booking hotels and tours in BiH. EI covered the technical costs of developing this website, and subsequent promotional expenses. EI’s total investment for Exploring was approximately $125K, and Exploring invested approximately $70K. 2. Arteco Wood Technology Center (“Arteco”), in Tešanj, launched in October 2009 by Artisan, an existing furniture manufacturer, to house an advanced (computer-controlled, 5-axis) woodworking machine provided by EI. Arteco was expected to provide services to SMEs in the BiH wood processing industry based on this equipment. EI’s total investment for Arteco was approximately $305K, while the center contributed some $235K. Comment [ET1]: This assessment might have been useful if sound methodological approach was used, if there was a critical assessment of information and data received from the EI Activity and if the Author tried to answer real evaluation questions in a measurable way. The real evaluation questions were: 1. To what extent has the EII Activity succeeded in a measurable way in capacity building for eBIZ Center Staff? 2. Are all four USAID supported eBIZ Centers financially and operationally sustainable? 3. To what extent has the EI Activity succeeded in a measurable way in improving the competitiveness of the SMEs served by the four eBIZ Centers? a. What was the situation of the targeted SME population before the EI Activity began? b. What is the situation of the targeted SME population now? c. Did EI respond to their identified needs? 4. What impacts have USAID supported e BIZ Centers had on associations, sectors, and industries targeted by the interventions? 5. What have been the main obstacles in the implementation of the EI Activity? Financing Growth (FG) Page 1 Cardno Emerging Markets 3. Textile Technology Center (TTC), in Doboj East, started in April 2012 by Kismet, an existing apparel manufacturer, to house advanced (computer-controlled, laser-based) fabric pattern plotting and cutting equipment provided by EI. TTC was expected to provide services to SMEs in the BiH apparel design and manufacturing industry, based on this equipment. EI’s total investment for TTC was about $270K, while the beneficiary invested about $285K. 4. Transwork, in Banja Luka, started in March 2012 by an existing company, Satwork, to provide services to SMEs in the freight transport industry in load information exchange, fuel usage, and fleet management. EI provided computer network servers and covered technical development costs for these services. EI’s total investment for Transwork was close to $135K, while the center put in approximately $145K. Key Findings and Conclusions i. Grantees benefitted substantially from the receipt of the systems. There was one potential risk in the basic design of the Project, which was clear from the beginning. The proposals would necessarily need to come from existing companies, since only they would have the experience in their sectors to understand and employ the proposed ICT-based systems effectively. This meant that there was always a certain probability that the systems could be used primarily for the benefit of the grantee company, rather than prioritizing services to other SMEs in the sector. To try to prevent this, the grantee companies were obligated under their contracts with EI to serve other SMEs – with a target of 40-50 for each center – and they understood this commitment. Two of the four centers – Arteco and TTC – substantially benefitted their founder-owner companies – Artisan and Kismet, respectively. The expensive equipment provided by EI was of direct use in their own production operations, and in both cases has lifted their value-added capabilities had a substantial positive impact on their businesses. While it is possible to consider these two owner companies to be clients of their respective e-BIZ subsidiaries (as, for example, a consultant hired by the project characterized Artisan’s relationship with Arteco1 ), this was not the intention of the grant. Nevertheless, this did provide positive development impact, significantly increasing sales and employment of the grantees, and sometimes overflow demand to outside producers. In the case of Kismet, thanks to the customer response and productivity improvements provided by the laser cutter, the sewing capacities of its factory are now periodically exceeded, and it is outsourcing production for its own sales to other garment producers in BiH – an impact on outside SMEs that was not anticipated by the project. Similarly, the equipment has enabled Artisan to take on larger contracts, in several cases involving multiple collaborating companies. Transwork and Exploring were not operations that could provide services to their founder-owners. For Transwork, there were cross-selling opportunities between it and Satwork, but no direct input relationship. For Exploring, a similar statement could be made for its minority owner (Exploring Macedonia), and to a lesser extent for its majority owner and the contractual grantee, e-Turizam. ii. Three of the four e-BIZ centers have performed in the manner expected, with positive development impact on their respective sectors. This means that that these centers – Arteco, TTC, and Transwork – have successfully marketed their services to SMEs in their sectors, that each has built a clientele, that this clientele is expanding, and that their revenues are significant, and for TTC and Transwork strongly growing. Vis-à-vis the contractual target of providing services to 40-50 outside clients in each sector, the actual numbers at the end of the EI project (August 2013) were 112 for Arteco, 40 for TTC, and 43 for Transwork. As seen in Table 1, all three have achieved significant revenues since startup. TTC has substantially increased the proportion of its sales to third parties, and for Arteco, while somewhat up and down, the average proportion of third-party sales over its five-year history is more than half – 54%. Interviews with clients have indicated satisfaction with the services offered2 , and this is 1 Shék Jain Trip Report, September 2010 2 Marty McElwee Trip Report covering TTC, June 2012 Comment [ET2]: That is exactly the Evaluation Team finding. The equipment procured for Arteco enabled its parent company Artisan to take larger contracts. However, their production capacity was not sufficient and they developed a small wood processing cluster of 25-30 small wood shops around Artisan, hiring them to produce specific parts of furniture for the Artisan. Artisan indeed improved businesses of these small shops. However, these small shops and their business operations remain entirely dependent on the jobs received from Artisan. Please see excerpt from Annex 17 – Interview with Arteco’s management – “They have a lot of companies that are attached to them as their suppliers (about 17 companies). These companies and shops exist thanks to them. For example, there is a small firm that produces metal frames for them and they could close their business if there are no orders from Artisan.” Other Arteco clients were one-time clients, receiving small value service, than foreign ... Comment [ET3]: Only Transwork performed as expected when it comes to the number of clients served. Comment [ET4]: The EIA provided to the Evaluation Team a list of 30 companies served by TTC (pls see Annex 2). However, TTC accounting department submitted to the Evaluation Team a list of 28 companies served by TTC (Please see Annex 2). Out of 28 enlisted, 9 SMEs allegedly received service but did not pay for it. Additional two SMEs are Kismet doo and Vocal Production doo, both owned by the TTC owner. Therefore the Evaluation ... Comment [ET5]: Out of total 73,976.74 KM TTC’s revenues in 2012, 51,988.51 KM was received for services provided to the parent company, Kismet doo, and additional 6,703.6 KM for services provided to the Vocal Production doo, which is owned by the owner of TTC - Kismet doo. Only 20.66% of total TTC revenues in 2012 came from, as Mr. King defined them, “outside clients”. In 2013 the situation even worsen since only 9.28% of total ... Comment [ET6]: Author used different statistics for different eBIZ Centres, trying to select favorable ones only. For TTC, the Author claims that the proportion of its sales to third parties increased over the time (which did not happen) and, for Arteco, the Author is trying to explain Arteco’s success by using five-year average third party sales and he also includes in this statistics the year 2009 which is actually outlier. However, Author does not mention that third-party sales were continuously ... Comment [ET7]: In majority of cases, interviews with clients showed complete irrelevance of one￾time services provided by the e-BIZ Centers for business operations of the interviewed clients. In majority of cases interviewed SMEs could not or they had hard time recalling they received service from an eBIZ Centre. Please see the excerpt from audio recordings of the interviews with SME clients Financing Growth (FG) Page 2 Cardno Emerging Markets further evidenced by the extensive repeat business that all three of these centers have with many of their clients. For Arteco, 45 of its 112 clients – 40% – are repeat customers. All 29 of Transwork’s year 1 customers repeated in year 2. 60% of TTC’s first year clients repeated. For Exploring, 35 hotels have been long-term customers. This indicates that the centers have become important ongoing service providers to numerous SMEs in their sectors. Table 1: Impact Indicators e-BIZ Center EI Investment (KM) Launch Date Revenue (KM) / Clients Item 2008 2009 2010 2011 2012 2013 Exploring Bosnia 176,687 Aug-09 Net Revenue 22,176 37,296 20,530 38,992 32,000 Growth -- 68% -45% 90% -18% Clients 39 78 86 87 87 Arteco 420,000 Oct-09 Revenue 203,025 330,825 388,869 158,3793 117,809 Growth -- 63% 18% -59% -26% % 3rd Party 65% 31% 65% 71% 37% Clients 29 60 82 87 112 Transwork 199,737 Mar-12 Revenue 72,980 195,000 Growth -- 167% Clients 29 50 TTC 381,386 Apr-12 Revenue 73,977 176,000 Growth -- 138% % 3rd Party 30% 63% Clients 13 44 The Evaluation Team Comment No.1: Arteco's Revenues 2009 2010 2011 2012 7 months 2013 Artisan 9,273 108,193 116,064 37,496 35,128 Third Party SMEs Sales 220,853 165,971 54,968 23,888 9,900 Total (KM) 230,126 274,164 171,032 61,384 45,028 Third Party SMEs Sales (%) 95.97 60.54 32.14 38.92 21.99 In the case of Arteco, the numbers of SMEs obtaining the production-related services of the system initially fell short of expectations. Arteco was not as aggressive in its early period in marketing to outside 3 Arteco’s revenues for 2009-11 included “pass-through” invoices due to its role as leader of a group of firms cooperating on large Artisan projects. This practice ceased in 2012. Comment [ET8]: Please see Annex 3 and (document provided by the Arteco) where it is visible that all SME clients received one-time service Financing Growth (FG) Page 3 Cardno Emerging Markets companies as expected, and its parent company, Artisan, dominated the use of the 5-axis CNC machine. The warning provided by one expert4 that Artisan might want to get the machine primarily for its own use seemed justified. With collaboration from other projects (FIRMA), Arteco was able to improve its services to third-party firms by offering CNC training, which while not exactly in line with initial expectations for production-related services, did have a positive impact on the sector. Furthermore, pressure and support from the EI project to better market services eventually had a favorable impact, as indicated by the data in Table 1. It should be noted that another recognized difficulty with the EI project concept was the “trust issue”. That is, in some cases (Arteco and TTC) the centers would be seeking to sell services – such as providing manufacturing specifications or prototypes for original designs – to firms that were competitors of their owner companies. The outside firms would have to trust that the e-BIZ centers would keep product information confidential and not share it with their parent companies. The building of trust, supported by a formal nondisclosure agreement, has progressed but has taken time – and for some third companies, these suspicions have been insurmountable. Also somewhat unexpected was the positive “lighthouse” impact of EI-supported centers. When Arteco received its 5-axis CNC machine, it was the only one in the country. Engaging accomplished Bosnian and Western designers, Artisan quickly demonstrated how useful the machine was in producing market￾forward high-end furniture styles, winning design prizes at the Cologne Furniture Fair in 2010 and 2011 – one of the very few companies from non-Western European countries to do so. Other furniture producers in BiH noted this success, and now several of them have acquired these machines, some with more than one. A similar impact was felt in the apparel sector in Macedonia after the predecessor EI project there, also implemented by EDC, established a laser cutter-equipped e-BIZ center – the first such system in the country. There are now 15 such systems in Macedonia, acquired by other apparel producers who were able to see their productivity and business impact thanks to the initial e-BIZ center.5 There is no doubt that this has broadly improved the competitiveness of the entire Macedonian apparel sector, a similar impact can be expected in BiH over time. Again, this was not a planned effect. Though the acquisition of similar advanced systems by outside companies means that they would not be potential clients of the EI-supported e-BIZ center, the demonstration effects should encourage demand from smaller manufacturing operations in wood and apparel that do not have the capacity or financial resources to justify obtaining one. The overall competitiveness of both sectors is improved. iii. The impact of the e-BIZ centers on their sectors is likely to continue to grow. The sales, client numbers, and third party revenues trends for Arteco, TTC, and Transwork are generally positive. The evidence of repeat customer business indicates that services provided are of ongoing value to many outside clients. iii.iv. The Evaluation Team Comment No.2 v. Comment for Arteco revenue trends from SME clients Arteco's Revenues 2009 2010 2011 2012 7 months 2013 Artisan 9,273 108,193 116,064 37,496 35,128 Third Party SMEs Sales 220,853 165,971 54,968 23,888 9,900 Total (KM) 230,126 274,164 171,032 61,384 45,028 4 The Wood Team Leader on USAID’s BiH FIRMA project 5 McElwee Trip Report Comment [ET9]: Out of 28 clients that received training service by Arteco over the five-year period, five (5) clients were SMEs and 23 clients were individuals. Please see attachment Annex 4 Comment [ET10]: Please see comments no.5 no. 6 and no. 7 Comment [ET11]: Over the five-year period only two (2) SMEs from BiH were provided with prototype development service (Standard from Sarajevo and Sportex from Kljuc). Please see Annex 4 Comment [ET12]: Out of the 2012 TTC’s total revenue of 73,976.74 KM, 51,988.51 KM was received for the services provided to the parent company Kismet doo and additional 6,703.6 KM for the services provided to Vocal Production doo owned by the same owner of TTC and Kismet doo. Only 20.66% of total TTC revenues in 2012 came from, as the Assessment Author defined them, “outside clients”. In 2013 situation even deteriorated given that only 9.28% of total TTC’s revenues came from “outside clients”. Please see data provided by TTC in Annex 2. Financing Growth (FG) Page 4 Cardno Emerging Markets Third Party SMEs Sales (%) 95.97 60.54 32.14 38.92 21.99 For both Arteco and TTC, the services that appear to be most useful to third party SMEs involve prototyping – using IT systems to create production designs and specifications for manufacturers that fulfill buyer needs, maximize efficiency in material use, and optimize labor and production equipment use time. These services include provision of “blueprints” in various forms for reproduction – for example, markers for fabric cutting. Prototype services may also include the production of samples. Solid growth in the demand for these services for both Arteco and TTC has been demonstrated. These services can only be produced by the kind of advanced systems they obtained from EI, which do not make economic sense for smaller manufacturers to buy. As Arteco and TTC continue to gain the trust of third party clients – that they will not divulge designs and customers to their parent companies – it is highly reasonable to expect that the positive growth trends already witnessed will continue, and that the impact of the centers on their sectors will expand over the longer term. For Arteco, technical training in the use of computer-controlled wood cutting equipment has also been a consistent source of revenue. The provision of actual production services to outside clients has fallen short of original expectations. That is, both Arteco’s 5-axis CNC wood cutting machine and TTC’s computer-operated laser fabric cutter are capable of delivering assembly-ready parts to third-party manufacturers. It appears that while most SME manufacturers find value and efficiency in the systems’ prototyping services, many have their own parts production capabilities, even if manual, and do not find it economically beneficial to buy these services from the e-BIZ centers. Certainly the parent companies’ use of the systems in their manufacturing operations will continue and grow, and third-party demand for production cutting will grow, but this service will likely remain secondary to those for prototyping. Meanwhile, Transwork’s strong expansion in a relatively short time period is quite promising. Its GPS tracking, communications and navigation, fuel-use and time monitoring, load optimization, and forthcoming broader fleet management services address a rapidly growing demand niche that has much untapped potential, based on estimates of the still-modest share of the truck fleet in Bosnia and neighboring countries that is currently equipped with these services. The Evaluation Team Comment No.3 Comment on Transwork’s package of services and breakdown of total revenues per service delivered: Transwork provides three types of services: GPS tracking - 92.4% of total revenues, fuel consumption control – 0.09% of total revenues, installation of sensor on the fuel tank cover – 7.47% and other services – 0.04% RB Vrsta proizvoda/usluge jedinična cijena u KM % učešća u ukupnom prihodu 2012 % učešća u ukupnom prihodu 2013 1 GPS uređaj 205 57.2 36.6 2 Usluga pretplata 20 37.8 52.3 3 Usluga ugradnja 20 3.3 3.5 4 Usluga kontrole potrošnje goriva 3 0 0.05 5 Usluga demontaža uređaja 20 0.04 0.02 6 Usluga premještanje uređaja 20 0.06 0.02 Comment [ET13]: Over the five-year period Arteco provided prototype development service to two (2) SMEs from BiH only (Standard from Sarajevo and Sportex from Kljuc). Please see Annex 4 Comment [ET14]: Out of 28 clients that received training service (all types of training) from Arteco over the five-year period, five (5) clients were SMEs (one of these five is REZ – Regional Development Agency) and 23 clients were individuals. Please see attachment Annex 4. Comment [ET15]: Over the five-year period Arteco provided prototype development service to two (2) SMEs from BiH only (Standard from Sarajevo and Sportex from Kljuc). Please see Annex 4 Financing Growth (FG) Page 5 Cardno Emerging Markets 7 Usluga ugradnja senzora na poklopac goriva 70 1.4 7.47 8 Ostale usluge 0.2 0.04 100 100 The same services were / are provided by: Excerpt from Trasnwrok’s Business Plan: “It has been estimated that in BiH at this time there are around 6000 users of GPS services: - Tim System d.o.o Sarajevo - has around 2000 users - Satwork d.o.o. Banja Luka - has 1500 users - Limes d.o.o. Sarajevo – has around 1000 users - Others - have around 1500 users combined “ However there were also other providers of GPS tracking and fuel consumption tracking services: http://www.starnet.ba http://www.gpstim.ba/onama/ (they have their own in-house developed software) http://www.satwork.net http://www.alfapro.ba/cijene.html http://sourcecode.ba http://www.gps-ba.net http://www.trackstar.ba (entire system designed and produced in BiH) http://www.systech.ba/index.ph Indeed, as this assessment was being prepared, post-EI project, Transwork was awarded a major tender from the RS Postal Service to install its GPS tracking service equipment in 250 of their delivery trucks. There is little doubt that Transwork’s impact on its focus sector will expand substantially over the years to come. While not without competitors, Arteco, TTC, and Transwork have found a local clientele and based on aggressive marketing and good word of mouth are positioned to grow. iv.vi. At least three of the four e-BIZ centers are likely to be sustainable. The question of sustainability overlaps to an extent with the immediately preceding discussion as to the likelihood of future demand growth for the centers’ services. Sustainability in addition requires that the e-BIZ centers are well￾managed and profitable. It is important to emphasize that the managers of Arteco, TTC, Transwork, and even Exploring all consider their centers to be attractive businesses in their own right and are committed to their expansion, separately from their parent companies. All four of the centers have full-time dedicated management, even though for Arteco, TTC, and Transwork key parent company owner-managers work closely with their e-BIZ center management in business planning. In all four cases, center management – entirely independent or teamed with the owner-manager – is highly entrepreneurial, capable, and committed to building the business. While Exploring is a 1-person shop, the other three have dedicated staffing: Arteco (8 persons), TTC (10), and Transwork (4). Financing Growth (FG) Page 6 Cardno Emerging Markets Arteco, TTC, and Transwork appear to be covering their costs, at least for personnel, or slightly profitable. With further growth they are all likely to become fully profitable. It can be expected that these three centers are sustainable for the long term. In terms of total revenues, Transwork has more than earned back EI’s investment in it, Arteco about 80% of it, Exploring about 85%, and TTC about half. Looking at all four centers as a group, the owner/founders’ view of sustainability is indicated by the fact that they have invested a total of KM 1.08 million in them, compared to KM 1.18 million provided by EI – almost half of the entire cost to date. v.vii. The fourth center has failed mainly due to global developments, but may still have a future. Exploring is clearly the most problematic center. The basic cause is the very rapid growth of globally￾provided internet-based services, whose underestimation has been a hallmark of the information age. Exploring was seen as a critical need for the development of the tourism sector at a time when there was no booking service for hotel rooms in BiH. The grant was provided to deliver a service that did not exist. However, very soon after this center was launched, it was overtaken by very large booking services – Expedia, Travelocity, TripAdvisor, and especially Booking.com. Once these global services entered the market, Exploring’s paying clientele – local tourism services – evaporated. They no longer needed Exploring for booking services, because they were so well-served by these well-known, trusted services. However, the entrepreneur now behind Exploring is still dedicated to the business and believes it can be profitable – not as a booking engine, but as a BiH tour packager, marketing to operators in tourist￾generating countries (including such new markets as Russia) interested in sending tour groups to BiH, and intermediating between them and tourism service providers in BiH. She continues to believe in the business and is investing time and money in it, striving to make it a success. While Exploring certainly will not fulfill its principal originally-intended service – because it is a service now provided by other very large companies in the market that have enormous market footprints – it may well survive with this altered business model. Certainly that is the intention of the entrepreneur. The Individual e-BIZ Centers 1. Arteco Wood Technology Center Arteco was founded in October 2009 in Tešanj, FBiH, by Artisan, a successful high-end furniture designer and manufacturer with an established Western European export business. Artisan had been a beneficiary of a USAID competitiveness projects in BiH that focused on the wood sector – CCA – and indeed it was CCA that originally recommended Artisan to EI. Responding to a tender, Artisan requested and won the grant of a 5-axis CNC wood machining system. The 5-axis format allows highly accurate cutting of wood parts in a wide range of shapes without multiple setups. It greatly expands furniture design options, makes highly consistent and accurate parts, and economizes labor time and material. The e-BIZ center has a dedicated manager (Elvedin Mandžukić), and one of the owners of the parent company (Fadil Čostović) works closely with him in expanding and marketing Arteco's services. Acquisition of the system substantially benefitted the parent company, Artisan, leading to development of some award-winning designs and expanding its product line and exports. Artisan has added 20 jobs since the machine became available to it through Arteco. There was concern that the parent company’s use of the equipment was the dominant priority. However, marketing efforts intensified with the encouragement of EI, and Arteco gained SME customers, mainly at first for prototyping and CNC training. The company has 30-35 small company and individual customers that are occasional users, approximately 45 SMEs that are repeat customers using one or multiple services, and 27 foreign companies that have ordered complete designs and technical documentation, as well as prototyping for various products. Arteco’s prototyping services have also led to follow-on business for third-party producers with whom it Comment [ET16]: Please see excerpt from Annex 8 –interview with Mr. Mandzukic - “What would be different if there was no USAID assistance? With USAID, Artisan technologically improved for three levels and their 20 regular clients advanced technologically at least for one level. Each new person trained how to read technical drawings is a progress.” Comment [ET17]: Please see previous comments Financing Growth (FG) Page 7 Cardno Emerging Markets cooperates. There are 14 of these cooperating companies that have participated in multiple Arteco￾managed projects for example in hotel furniture outfitting. The high-profile role of Artisan in the BiH furniture sector also generated trust issues – other companies were fearful that if they used Arteco’s prototyping services, their designs or customers could be stolen. Arteco has worked to establish its independence from Artisan, and Artisan now plans to move to separate premises (building permits have been issued), which will help address this issue. The center also has aggressive expansion plans: it is adding another CNC machine and a wood bending machine, in order to offer more advanced services. The original machine now often runs on three shifts. Its use can be booked online. Marketing efforts also target foreign buyers. Arteco, which has previously participated in the BiH wood industry’s joint stand (supported by FIRMA) at major international trade shows, for the first time will have its own stand at the Cologne Furniture Fair (the largest in Europe) in 2014. It is actively working to expand its client base. Arteco has 8 employees currently, which will grow to 10 when the new equipment is added, and expects to employ 15 by end-2015. Though its parts production for third parties may not have matched initial expectations, Arteco has had additional / different development impact in its active workforce training and internship programs. It is a growing and successful operation. 2. Exploring Bosnia Exploring Bosnia was established in August 2009 to address what had been identified as a significant obstacle to development of the BiH tourism sector – the absence of an online booking engine. This gap was identified during the USAID Cluster Competitiveness Activity (CCA) project, which supported the development of the tourism sector, as early as 2005, and indeed CCA had every intention of itself supporting the development of such a website. However, when EI was conceived, such an initiative was considered appropriate for its support. A similar USAID project had previously supported Exploring Macedonia, which had been successful and it eventually evolved into the country’s principal tourism promotion website. Exploring Macedonia became a 49% owner of Exploring Bosnia, and was expected to host the site on its servers and provide substantial software development and technical support. A BiH foundation, e-Turizam, was the 51% owner. However, given various delays both in starting the EI project and concluding the grant for Exploring, the promising original concept of a “Bosnia Expedia” hit the market only a little before major services took over the global market for these services. A Bosnia Expedia was no longer necessary – Expedia itself was here, along with Tripadvisor, Booking.com, and other major engines. When it marketed initially, Exploring was able to sign up a number of users. But only a year later, with the arrival of the major services, the value of using a local service that could never be the equal of the global ones disappeared, and many of the original customers did not renew their paid subscriptions. EI consultants provided extensive recommendations of what to Exploring needed to do to survive, but these would have required very large additional investment and even then could not have made the company competitive with the major global providers. The technical contributions of the two owners to Exploring were also disappointing. Exploring Macedonia (EM) consistently lagged in delivering technical services to Exploring Bosnia. On occasion the website was down for hours before EM addressed the problem. Eventually hosting services were removed from EM and transferred to BiH. Also, e-Turizam itself did not have adequate experience in the tourism sector, nor in world-class portal development. And as it became obvious that the original business model was doomed, the two owners became less interested. A positive factor was the entrepreneurial spirit and dedication of Ms. Saša Vasić, who became Director of Exploring, along with the active support of the EI project. With their efforts the business model was modified and Exploring evolved into a promoter and marketer of tourism to BiH and an event and tour Financing Growth (FG) Page 8 Cardno Emerging Markets packager, putting together the services of various local providers for foreign tourism companies interested in sending tour groups to BiH. For example, an agreement was signed with the US-based rafting tourism organization. Essentially due to the initiative of Ms. Vasić, this business developed, and as the original owners, EM and e-Turizam, lost interest, she was prepared to purchase their shares in the business and become its proprietor. This deal has not yet gone through6 , but she remains dedicated to the business and periodically active in it, using her own resources, including recent participation in the Moscow tourism fair, which generated promising contacts for bringing Russian skiers to Jahorina. (Air Moscow is initiating a weekly flight to Sarajevo at the end of this December.) Exploring could also have a role in helping to coordinate various public and private stakeholders in BiH tourism, especially as USAID and EU projects that have supported the Tourism Working Group wind down. This is something for which it could attract donor or BiH government funding. The connection of Exploring to USAID projects provides credibility in this area. It can advocate for tourism service providers in BiH. Despite a business premise that turned out to be faulty for unforseen reasons, Exploring is still alive, a year after it received its last financial support from EI. Its entrepreneurial director is still committed. If it can get out from under its current cloud, it has a chance to survive. 3. Textile Technology Center TTC was launched in April 2012 and is located in East Doboj, Federation BiH. Its owner, Kismet, won a competition held by EI for the grant of a computer-guided Gerber laser-knife fabric cutting system. (Gerber, a Connecticut-headquartered company, is the global leader in this technology.) It has a dedicated full-time manager (Semin Banjić), and the owner-manager of Kismet (Alen Karić) is closely involved in TTC’s business strategy and planning. Its manager devotes a large part of his time to marketing. TTC’s launch was postponed due to the EI project hiatus caused by the AED situation, so has been in operation for less than two years. TTC delayed aggressive marketing in order to ensure that it could use the laser cutter without mistakes, since those would be highly costly if an outside firm’s fabric was mis￾cut. In effect, the problems were worked out on Kismet cutting batches. TTC also has had to work to build trust with clients that using the center would not lead to losing business or designs to Kismet. This has progressed positively. TTC estimates that there are 120-130 apparel manufacturers in BiH, of which 80-90 are potential customers. It is now doing business with 44 of them (including 4 clients added since the end of the EI project), which means it is already serving approximately half of its prospective market. The most-used services are in digitizing patterns, grading them (i.e., duplicating them across a range of sizes), and plotting markers (the paper sheets that are placed on top of a stack of fabric spread on a cutting table to guide the cutter). The digital plotting and marker-making program also optimizes fabric use, reducing waste fabric to a bare minimum. This is a highly useful service to smaller apparel manufacturers who would otherwise have to work out the grading and marker making by hand, with less efficiency of material use, as well as labor time. These jobs average about € 300 each – they are small, but frequent. Kismet remains TTC’s most important single customer, because in addition to the digitizing services, it also pays TTC to cut almost all of its fabric on the laser cutter in preparation for moving it to its sewing factory. Far fewer of the outside customers using the digitizing services proceed to use TTC’s cutting services, because they have in-house manual cutting (skilled operators using specialized power cutting 6 The transaction was on the verge of being completed in mid-2013 when it became known that the USAID￾procured evaluation of the EI project would be particularly negative with respect to Exploring Bosnia. This reputational risk threatened to reduce the value of the business, and led Ms. Vasić to drop her plan to buy out EM. Comment [ET18]: Please see previous comments Comment [ET19]: Frequent?? – Over the two years of TTC operations, 18 SMEs have paid for TTC’s service. Out of these 18, two (2) companies are owned by the TTC’s owner. Please see Annex 2 Comment [ET20]: 90 % of total TTC’s revenues Financing Growth (FG) Page 9 Cardno Emerging Markets knives) which is relatively efficient. There are 5-6 larger companies (Granoff, Belissimo) that are using cutting services more frequently, to complement their in-house cutting units. There is no question that the availability of TTC’s services to Kismet has had a significant positive effect on Kismet’s business. Kismet has added 25 employees since TTC obtained the cutter – a 24% increase – and is now periodically itself outsourcing garment assembly work to other sewing factories, when its capacities are exceeded. TTC itself has added staff – it now has 10 employees and expects to double that over the coming years. TTC is currently co-located with Kismet, but as is recognized by management and the owner, needs more space in order to maximize its business potential. A plan is underway for Kismet to build a new factory nearby, and for TTC to take the space currently occupied in the original building by Kismet. Global market trends are a key driver of TTC’s potential business. Increasingly, major European apparel brands who outsource their cut-and-sew work want full service from their suppliers, meaning that the buyers want to provide only a sample and physical patterns for one size, plus fabric and findings (buttons, zippers, etc.), and have the apparel manufacturers take care of all of the technical details – grading, marker-making, production specifications. They expect manufacturers’ bids for work to include fabric useage estimates, and those with lower fabric use for the same number of final garments will of course be favored, since the brand is purchasing and shipping the fabric to the manufacturer. This is now possible because of digital technologies, so creates a ready market for such services. There is also a trend for European buyers to shift production back into Eastern Europe from China and Southeast Asia, due in part to potential liabilities (or at least bad publicity) highlighted by media attention to poor worker conditions and catastrophic accidents in some apparel factories in those countries. Also, with increasingly frequent style changes and the need for in-season re-orders, location matters: factories in BiH can do smaller runs (several hundred rather than several thousand pieces) and deliver them more quickly to Western Europe. Again, the availability of digital services greatly assists such responsiveness. The key challenges for TTC are (i) getting more customers, requiring among other things that they continue to overcome trust issues, (ii) finding qualified engineers, and (iii) expanding its physical space. As noted above, the latter is in process, and it will also help with the customer trust issue for TTC to be in its own, entirely separate facility from Kismet. 4. Transwork Transwork, located in Banja Luka, Republika Srpska, was founded in March 2012 as a subsidiary of Satwork, a company in the GPS tracking devices and software business. Transwork has a dedicated manager (Mr. Saša Mičić), and Satwork's General Director (Siniša Palić) is actively involved in strategy. The management team is highly entrepreneurial and capable. Satwork sells its units directly to consumers and to truck, auto, and auto parts retailers. Satwork saw value in establishing Transwork to expand sales of its systems in the wholesale freight and logistics market by adding extensive services to its GPS systems, and successfully bid for ICT equipment from the EI program, which consisted primarily of powerful network servers. Transwork’s software for supplying its services was developed entirely in-house, with major contributions from Satwork. Its various services provide truck fleet operators the ability to monitor all of their vehicles constantly, including their exact locations, their fuel use, their speed, and their down time. The software enables real time data exchange between an owner or fleet manager and all of his/her trucks. All of the services envisioned in its original business plan are already functional, and Transwork is now developing full route management and delivery optimization services. Transwork has a fulltime director of sales and marketing (Valentina Rujević), who finds it relatively easy to educate potential customers on the value of these services. More and more trucks do have GPS location systems installed, and Transwork’s enhanced software can be used with these. Market studies have shown than only 5-6% of all trucks in the SEE region are using such services, so substantial growth Comment [ET21]: Granof using cutting services more frequently ???– total value of the received services in two (2) years was 447 US$ whilst Belissima is not even on the list of the companies that paid TTC’s services. Please see Annex 2 Comment [ET22]: By whom from the Transwork? Transwork has zero employees with any kind of technical expertise. All software is Satwork’s software. Transwork is just Satwork’s sales department with no ICT design and development capacity at all. Please see Annex 7 Comment [ET23]: According to the Transwork Director, their fleet management software does not provide accurate information on fuel consumption and cannot be installed on some types of trucks (Volvo, etc). Therefore, they offer to their clients a measurement of the fuel consumption by the introducing physical sonde into the fuel tank (not very sophisticated technology). Please see Annex 6 Financing Growth (FG) Page 10 Cardno Emerging Markets is possible. The company is systematically marketing to a customer base of some 1,000 trucking companies in BiH, Croatia, and Serbia. It is also investigating expansion to sea freight. Consequently, Transwork’s revenues and client base have grown rapidly in less than two years. The company has 50 clients (having added 4 new clients since the end of the EI project) who make monthly payments for its services, representing some 750 trucks with installed GPS equipment (including 250 RS Postal Service delivery vehicles for which Transwork just won a large competitive tender). In addition, partly to attract potential clients, Transwork offers a free online “freight exchange”, where truckers can offer availability or seek loads when they are returning from a delivery or on the way to pick one up, intermediating supply and demand and providing a means of expanded revenue for sector participants. The exchange typically displays offers from all over Europe. There are 208 active members of the freight exchange (including the 50 contract customers), so Transwork’s impact on the BiH logistics sector is considerably larger, in terms of SMEs serviced, than indicated by the number of paying customers alone. Transwork has competitors, but because it has developed all of its software in house, it can be readily updated, can more easily be customized for client needs, and is available in local languages as well as German and English. Some competitors are selling using off-the-shelf or franchised applications that are less responsive. Transwork is also now the only company in its field in BiH that provides international as well as domestic expanded GPS-based services. Transwork’s challenges include (i) building trust among buyers that a BiH firm’s software is as functional and reliable as that of German and French competitors (though it is highly price competitive with them), (ii) finding trained programming staff, because the field is relatively new. (It obtains people from Satwork, who are trained in-house.) This e-BIZ center is already profitable. This is a dynamic, competitive, growing firm, catalyzed and enabled by USAID EI. Final Comments It bears repeating that the EI model of development impact was itself innovative and quasi-experimental, having been employed only once before in the same way, in Macedonia. It extended a series of USAID efforts over time to provide substantial equity or equity-like capital to individual businesses, including in this region such programs as the E&E Bureau’s Enterprise Funds. We see clear progress to date in contributing to the development and competitiveness of their sectors, and it is likely that three of the four e-BIZ centers will continue to grow and prove to be sustainable, to a significant if not complete degree consistent with their initially expected service scopes. Furthermore, even the fourth center may well survive, though with a different business focus than initially planned. Given this, the project must be considered a significant success – something that USAID should be proud of. The keys to success – meaning sectoral impact and sustainability – have been: (i) informed assessment of market needs, (ii) careful selection of grantee, (iii) capable, entrepreneurial, adaptable management, and (iv) close and continuous monitoring of developments and supporting technical assistance by the sponsoring project. The scope of work for this assessment did not include recommendations for further improvement of the businesses of the centers. These have previously been provided by other consultants through EI technical assistance. Finally, while unrelated to the questions that this assessment was designed to address, it can be noted that in all four cases management of the centers and their owners provided unsolicited appreciation of critical and constructive support, both from USAID for the equipment systems, and also from the EI project itself for the technical and marketing consulting assistance received. Comment [ET24]: The EI Evaluation Team and Mr. King did not interview the same people. Transwork staff interviewed by the Evaluation Team stated that the online fright exchange was a failure and that they could not compete with Timcom. According to Transwork staff, making their online fright exchange useful to transport companies would require a large financial investment that is beyond Satwork’s and Transwork’s financial capacity and expansion to all Balkan countries through establishment of cooperation with similar online fright exchange providers in the region. Please see Annex 6 and audio recording from the interview with Transwork. Comment [ET25]: There are other BiH companies providing same services and in-house developed software (even before Satwork) Some of them are: http://www.gpstim.ba/onama/ (they have their own in-house developed software – 250 clinets in BiH) http://sourcecode.ba http://www.trackstar.ba (the entire system designed and produced in BiH) Comment [ET26]: To provide GPS tracking service that was also provided by 11 other companies at the time the EIA Activity stepped in with its assistance and money. When asked about the value added of the EIA Activity intervention, Transwork’s Director response was: faster development of the Satwork’s software. He also stated that this software would have been developed any way but, with the EIA Activity money, it was developed sooner. Please see Annex 6 Comment [ET27]: By providing only partial information and anecdotal evidences, the Author is trying to convince audience of all four eBIZ Centres’ bright future. Although eBIZ Centers do not do exactly what they were designed to do, that does not bother the Author since he believes it is likely they will continue growing. Financing Growth (FG) Page 11 Cardno Emerging Markets Financing Growth (FG) Page 12 DRAFT – 2/27/14 (v2) EXCELLENCE IN INNOVATION ACTIVITY PERFORMANCE EVALUATION – RESPONSES February 28, 2014 1. To what extent has the EI Activity succeeded in a measurable way in capacity building for eBIZ Center Staff? 2. Are all four USAID supported eBIZ Centers financially and operationally sustainable? 3. To what extent has the EI Activity succeeded in a measurable way in improving the competitiveness of the SMEs served by the four eBIZ Centers? a. What was the situation of the targeted SME population before the EI Activity began? b. What is the situation of the targeted SME population now? c. Did EI respond to their identified needs? 4. What impacts have USAID supported e BIZ Centers had on associations, sectors, and industries targeted by the interventions? 5. What have been the main obstacles in the implementation of the EI Activity? Overview FHI360 and EIA (“Implementing Partners”) welcome this opportunity to provide responses (“Response”) to the USAID-funded Excellence in Innovation Activity (EIA) Performance Evaluation (“Evaluation”). The Response below is divided into two categories: 1) responses to the Evaluation from a technical standpoint, including the EIA’s impact on the economic sectors the project served, sustainability of the e￾BIZ Centers, and capacity building of the e-BIZ Centers and client SMEs; and 2) responses to the Evaluation in terms of its methodology and approach. As part of the Response, the Implementing Partners would like to make the following statements related to the Evaluation and to the EIA: 1) We have been and continue to be focused first and foremost on helping support USAID’s goal of fostering growth in the economy of Bosnia and Herzegovina (BiH) and stimulating innovation in the economic sectors that EIA served; 2) We appreciate the thoroughness of the Evaluation, its lessons learned raised by the Evaluation, and look forward to applying these lessons learned to other similar efforts; 1 3) We note with concern that the Evaluation makes a significant number of assertions both about the EIA’s technical performance and conduct during the Evaluation process that we believe are inaccurate. We lay out these concerns in detail below. 4) The Evaluators have not provided an analysis of the impact of capacity building provided by the numerous local and international consultants that EIA procured during the course of the project. 5) The Evaluators followed the terms of the EIA Cooperative Agreement (CA) and Life of Project Implementation Plan (LPIP) when they were determining the number of Center clients (insisting on BiH SMEs only); however, the Evaluators seemed to disregard the CA and LPIP elements when defining criteria for assessing Center sustainability. In this respect, the Evaluators used measurements for Center sustainability that were not related to and used through the life of the project as approved by USAID. And, problematically, the Evaluators have never shared the changed definition of sustainability with EIA even after such definition was accepted by USAID and used during evaluation of EIA. Echoing the point above, the Evaluation used measurements of sustainability that were never applied to the EIA from the outset of its design, including the Performance Monitoring Plan / Life of Project Implementation Plan through the life of its implementation. As such, the “goalposts were moved” in terms of measuring a crucial element of the EIA’s success and impact. 6) As it relates to the question of cost share, the requirements for cost share were included in the fixed obligation grants (FOG) issued to each of the organizations in order for them to support the E-biz centers. Cost share was documented and reported in accordance with 22 CFR 226.23.We were perplexed by the evaluator’s remarks. 7) Similar to most if not all development activities carried out over a period of several years with multiple stakeholders across several economic sectors, EIA should have done some things differently. We acknowledge this below and appreciate where the Evaluation raises this. However, the overall tenor of the Evaluation unfairly discounts the role of EIA in generating strong impact in the economic sectors, achieving project goals in terms of driving innovations in the sectors, nor strengthening SME capacity – all core factors of the approved cooperative agreement with USAID. We believe there is ample evidence to demonstrate that EIA achieved virtually all of the project goals and objectives as set forth in the Cooperative Agreement and Implementation Plan, and USAID/BiH’s objectives as pertains to the EIA. In this respect we would like to confirm whether the Evaluators have reviewedall relevant documentation that was provided to them or interviewed all key stakeholders in this process. 8) We are concerned and disappointed that the Evaluation states that EIA interfered with the Evaluation process. We do not believe that EIA interfered in this manner and document below the numerous instances where EIA provided copious support to the Evaluation and also show our difference of opinion about specific claims e.g. that documents were amended or that e-BIZ Center staff was instructed to provide inflated training attendance data. 9) While we believe that the Evaluation approached this evaluation with a set methodology, there are many instances in the Evaluation where the underlying assumptions and approaches used led to inaccurate or misleading conclusions or statements. We document these below. 10) During the course of the evaluation, EIA diligently provided supporting materials, data, and actively engaged with USAID throughout this process. However, we feel that there are several areas where the Evaluation missed opportunities to fully engage the knowledge of the project and the Comment [ET1]: The Evaluation Team has been receiving significantly different financial reports from eBIZ Centers in just one month period (30 June -30 July). When asked why they adjusted their financial reports, Arteco’s management replied that they were instructed by the EI Activity to do so in order to improve financial indicators. Please see Annex 11 ARTECO INTERVIEW TRANSCRIPT a (audio recording of the interview with Arteco is also available) Transwork management reported that their staff participated in trainings organized by the EI Activity and that was not true. When asked why they did so, they explained they were asked by the EI Activity to do so. Please see Annex 12 TRANSWORK TRANSCRIPT CAPACITY BUILDING (audio recording of interview with Transwork is also available); When it comes to surveys, TTC submitted the list of surveyed clients. Olimp doo, a company from Gracanica, was the first on the list. When the Evaluation Team interviewed Olimp management,... Comment [ET2]: Not much to say. Please see the excerpt from Annex 6 & Annex 7 - Interview Minutes Transwork -“Capacity building" "There was nobody they could turn to and learn what to do when establishing Transwork. There are no experts in that particular area. They trained ... Comment [ET3]: Not correct. We did use the CA definition of sustainability. However, we used other methods as well, which USAID approved. It is our understanding that Ms. Whitmore was consulted on 3/26/13 about both definitions and agreed that the measures used by the EIA were not adequate. Her email to Ms. Vukmanovic is attached. Please see ... Comment [ET4]: This is something that should be discussed with USAID. It is our understanding, however, that the USAID Evaluation Policy does not recommend/ require anything like that. In accordance with the Policy, only SoWs and draft evaluation reports should be shared with various ... Comment [ET5]: In accordance with 22 CFR 226.23, “(a) All contributions, including cash and third party inkind, shall be accepted as part of the recipient’s cost sharing or matching when such contributions meet all of the following criteria. (1) Are verifiable from the recipient’s records.“ ... Comment [ET6]: Exploring BiH – The Evaluation Team's understanding was that all parties agreed that Exploring BiH failed to deliver and to achieve project goals. TTC was, among other things, expected to “Execute wide ranging marketing activities, aimed at reaching the targeted number of client SMEs (40) set in the e- ... Comment [ET7]: Please see Comment No.1 Comment [ET8]: Audio recordings of all interviews were handed to USAID/BiH. USAID hired a certified, external translator to translate a certain number of interview excerpts. These excerpts will be used throughout the documents. 2 beneficiaries. Additionally, EIA feels that a more fully articulated evaluation should have been provided per the USAID Evaluation Policy. 11) The Evaluators have never shared their overall methodology with EIA regardless of the fact that disclosing evaluation methodology to the subject of evaluation is one of the pillars of the evaluation process. 12) The Evaluators did not interview key project personnel (EIA Business Development Manager and the two Chiefs of Party. We believe that failing to do this led to an incomplete understanding of EIA. 13) According to the verbal feedback and written statements from e-BIZ Centers, the project beneficiaries felt that the Evaluation process was more like an audit than an Evaluation interview, and that statements by the Evaluators appeared to indicate a bias to portray the EIA and its staff in a negative light. The beneficiaries reported being extremely uncomfortable with the aggressive manner in which questions were apparently posed to them, and with apparently sensitive information about EIA staff that was also discussed with them. This approach appears to contradict the principles of USAID’s Evaluation Policy that local beneficiaries should be treated with respect. Please refer to the Appendix for statements testifying to this from the e-BIZ Centers. Comment [ET9]: Addressed above. Comment [ET10]: Not correct. We had a group interview with the COP, main Business Development Manager. COP was interviewed at least two times. Please see Annex 10 - The EIA Interview Minutes 1 . Audio recording of these interviews are available; Comment [ET11]: As we learned from USAID/BiH senior management, project implementers and project beneficiaries should be giving accurate information to USAID. When we first realized that we were not getting accurate information, which was at the very beginning of the evaluation process, we informed USAID and kept them informed on a regular basis until the end of the field work. 3 Responses – Technical and Performance Components Overview of Evaluation Comments – Technical and Performance The Evaluation took strong issue with the inherent design of the e-BIZ Project, particularly related to the relationship between parent company and the e-BIZ Center, and the level of outreach of the e-BIZ Centers to the SMEs. It is implied as well that through this design and relatively low levels of capacity building by EIA, the SMEs did not benefit significantly from this project, nor the sectors themselves, including in terms of innovation introduced into the sectors, The Evaluation implies that the design and way in which the EIA was carried out also did not enable the e-BIZ Centers to be sustainable in the long run. We strongly disagree with this assessment. Nevertheless, before providing a point-by-point response to the Evaluation along these lines, the implementing partners would first like to rebut these claims by drawing on an assessment of the e-BIZ Centers that was carried out by David T. King of Cardno Emerging Markets of EIA in December 2013, and which was funded by EIA. The implementing partners believe that the Evaluators should examine closely the findings from this assessment and refer to them when making revisions to the draft Evaluation. The full assessment is included as an annex. An excerpt of these assessment findings are as follows: Key Findings and Conclusions i. Grantees benefitted substantially from the receipt of the systems. There was one potential risk in the basic design of the Project, which was clear from the beginning. The proposals would necessarily need to come from existing companies, since only they would have the experience in their sectors to understand and employ the proposed ICT-based systems effectively. This meant that there was always a certain probability that the systems could be used primarily for the benefit of the grantee company, rather than prioritizing services to other SMEs in the sector. To try to prevent this, the grantee companies were obligated under their contracts with EI to serve other SMEs – with a target of 40-50 for each center – and they understood this commitment. ii. Three of the four e-BIZ centers have performed in the manner expected, with positive development impact on their respective sectors. This means that that these centers – Arteco, TTC, and Transwork – have successfully marketed their services to SMEs in their sectors, that each has built a clientele, that this clientele is expanding, and that their revenues are significant, and for TTC and Transwork strongly growing. iii. The impact of the e-BIZ centers on their sectors is likely to continue to grow. The sales, client numbers, and third party revenues trends for Arteco, TTC, and Transwork are generally positive. The evidence of repeat customer business indicates that services provided are of ongoing value to many outside clients. * The Evaluation Team Comment 1: Comment on positive third party SMEs revenues trend for Arteco and TTC: Comment [ET12]: The Evaluation Team only showed evidence that TTC works almost exclusively for its parent company and that, instead of expected 40 SMEs, TTC served 19 SMEs (including 2 parent companies). The Evaluation Team believes that is an important issue. According to records received from the eBiZ Centers, other eBIZ Centers, except for Transwork, failed to serve targeted number of SMEs. Comment [ET13]: The EI Activity does not have any information that could, in a measurable way, support a conclusion that four targeted sectors experienced increase of competitiveness due to services provided by eBIZ Centers. The Evaluation Team also could not find evidence that would support conclusion of increased competitiveness of four sectors. Comment [ET14]: It is fair to call that document “an assessment” and not evaluation. No comment on the “assessment” performed by the former COP of the EIA twin project FIRMA, also financed by USAD BiH. Comment [ET15]: The Evaluation Team does not understand how EIA funded the assessment in December 2013 if their activity ended in August 2013. This is more a question/comment for USAID. Comment [ET16]: The Evaluation Team examined these findings. Document with comments attached. In short - no methodology and no info on sources used. The assessment reads like the Activity presentation prepared by the EIA staff (with some anecdotal evidence). The author, the former USAID/FIRMA COP, failed to mention the experience USAID/FIRMA activity had with EIA, which is something the Evaluation Team learned when we interviewed USAID/FIRMA employees. Comment [ET17]: Excellent summary. TTC was expected to serve 40 SMEs and managed to provide services only to 16 SMEs over the 16 months period. Potential risk that the eBIZ Center wiil be used primarily for the benefit of the granty company materialized completely in the case of TTC. 90% of total eBIZ Center revenues are coming from services provided to the parent company. ARTECO was expected to serve at least 50 SMEs and managed to provide services to 39 SMEs over the ... Comment [ET18]: Except for Transwork, eBIZ Centers failed to reach expected number of SME clients; share of revenues from services provided to third party SMEs in total revenues of TTC and Arteco is small. ... Comment [ET19]: Out of total 73,976.74 KM of the TTC’s revenues in 2012, 51,988.51 KM was received for services provided to the parent company, Kismet doo, and additional 6,703.6 KM for services provided to the Vocal Production doo, which is owned by the owner of TTC - Kismet doo. Only 20.66% of total TTC revenues in 2012 came ... 4 Arteco's Revenues 2009 2010 2011 2012 7 months 2013 Artisan (KM) 9,273 108,193 116,064 37,496 351,28 Third Party SMEs Sales (KM) 220,853 165,971 54,968 23,888 9,900 Total (KM) 230,126 274,164 171,032 61,384 45,028 Third Party SMEs Sales (%) 95.97 60.54 32.14 38.92 21.99 TTC’s Revenues 2012 7 months 2013 Kismet & Vocal Production (KM) 58,692.11 86,473.45 Third Party SMEs Sales (KM) 15,284.63 8,842.53 Total (KM) 73,976.74 95,315.98 Third Party SMEs Sales (%) 20.66 9.28 For both Arteco and TTC, the services that appear to be most useful to third party SMEs involve prototyping – using IT systems to create production designs and specifications for manufacturers that fulfill buyer needs, maximize efficiency in material use, and optimize labor and production equipment use time… iv. At least three of the four e-BIZ centers are likely to be sustainable. The question of sustainability overlaps to an extent with the immediately preceding discussion as to the likelihood of future demand growth for the centers’ services. Sustainability in addition requires that the e-BIZ centers are well￾managed and profitable. It is important to emphasize that the managers of Arteco, TTC, Transwork, and even Exploring all consider their centers to be attractive businesses in their own right and are committed to their expansion, separately from their parent companies v. The fourth center has failed mainly due to global developments, but may still have a future. Exploring is clearly the most problematic center. The basic cause is the very rapid growth of globally￾provided internet-based services, whose underestimation has been a hallmark of the information age. Exploring was seen as a critical need for the development of the tourism sector at a time when there was no booking service for hotel rooms in BiH. The grant was provided to deliver a service that did not exist. However, very soon after this center was launched, it was overtaken by very large booking services – Expedia, Travelocity, TripAdvisor, and especially Booking.com. ************** Comment [ET20]: Over the five-year period only two (2) SMEs from BiH were provided with prototype development service (Standard from Sarajevo and Sportex from Kljuc). Please see Annex 4 - Arteco - List of Clinets and Type of Services 5 Before proceeding to the point-by-point response, the EIA must stress that it believes that differing views on several core project postulates are the cause for disagreements on number of evaluation’s findings and conclusions. They are as follows: • EIA believes that at least two, if not all four Centers should have been treated as bona fide startups during this evaluation. It is apparent that Evaluators have failed to treat them as startups during their analysis of Centers’ financial and operational sustainability. • EIA believes that owners have been and still are the pivotal staff members of respective Centers regardless of the fact that they have not formally registered as Center employees. This is an important point since this is the evidence provided in the report of training attendance misreporting. • As it relates to the question of cost share, the requirements for cost share were included in the fixed obligation grants (FOG) issued to each of the organizations in order for them to support the E-biz centers. Cost share was documented and reported in accordance with 22 CFR 226.23.We were perplexed by the evaluator’s remarks. • Transwork’s close integration with Satwork and reliance on its technological support is logical and desirable from a startup business development standpoint and certainly expected from the planning perspective. The Evaluators assert that they do not believe that such integration is desirable but does not provide a substantiated reason for this assertion. ** The Evaluation Team comment 2: Comment on Transwork’s package of services and breakdown of total revenues per service delivered (Annex 5 – TRANSWORK TYPE of SERVICES & THEIR SHARE IN TOTAL REVENUES): Transwork provides three types of services: GPS tracking - 92.4% of total revenues, fuel consumption control – 0.09% of total revenues, installation of sensor on the fuel tank cover – 7.47% and other services – 0. 04% No Product /Service Unit Price (BAM) % in total revenues 2012 % in total revenues 2013 1 GPS device 205 57.2 36.6 2 Subscription for GPS tracking service 20 37.8 52.3 3 Device installation 20 3.3 3.5 Comment [ET21]: USAID posed no such requirement to the Evaluation Team. The Evaluation Team does not believe that different treatment of certain eBIZ Center would change evaluation results and whether they achieved expected goals and results. Comment [ET22]: No comment. Comment [ET23]: In accordance with 22 CFR 226.23, “(a) All contributions, including cash and third party in-kind, shall be accepted as part of the recipient’s cost sharing or matching when such contributions meet all of the following criteria. (1) Are verifiable from the recipient’s records.“ The Evaluation Team tried to verify cost share arrangements through recipients' financial records and we found out there was no record about it in the recipients' financial records. We used the same financial reports to determine financial sustainability of the eBIZ centers. Lack of records of cost share arrangements in their financial reports caused a distorted picture on their financial position. Comment [ET24]: The Evaluation Team never claimed that something was desirable or not. The Evaluation Team claims that Transwork does not have any technical capacity in area of ICT and that it serves only as Satwork’s sales and marketing department. Reasons behind this claim: (i) Transwork has four employees with no ICT background or experience. (ii) All of them are economists. (iii) Their job is to market and sell Satwork’s ICT products (mainly GPS tracking). (iv) That is the same service you can buy directly from Satwork http://www.satwork.net , but now there is another marketing and sales channel (Transwork) supported by USAID. 6 4 Fuel consumption control service 3 0 0.05 5 Device de-installation service 20 0.04 0.02 6 Device movement service 20 0.06 0.02 7 Sensor installation on fuel tank lid 70 1.4 7.47 8 Other services 0.2 0.04 100 100 The same services were / are provided by: Excerpt from Trasnwork’s Business Plan: “It has been estimated that in BiH at this time there are around 6000 users of GPS services: - Tim System d.o.o Sarajevo - has around 2000 users - Satwork d.o.o. Banja Luka - has 1500 users - Limes d.o.o. Sarajevo – has around 1000 users - Others - have around 1500 users combined “ However there were also other providers of GPS tracking and fuel consumption tracking services:: http://www.starnet.ba http://www.gpstim.ba/onama/ (they have their own in-house developed software) http://www.satwork.net http://www.alfapro.ba/cijene.html http://sourcecode.ba http://www.gps-ba.net http://www.trackstar.ba (entire system designed and produced in BiH) http://www.systech.ba/index.ph • ARTECO’s prototyping services for foreign clients represents a pivotal indirect support to the BiH wood processing sector as these services complete the crucial precondition before those foreign clients could proceed to give manufacturing orders to local SMEs. The Evaluators view 7 the recent dominance of such foreign clients in ARTECO’s revenue structure as a negative development. • The late launch of Transwork and TTC that was caused by external factors, including AED’s suspension that resulted in a delay in issuing the FOGs for the textile and logistics e-BIZ centers.This should have been noted by Evaluators. The fact that these two Centers were operational for less than 16 months at the time of evaluation does not appear to be considered by Evaluators when they were assessing the development progress and achievements of these two Centers. Point – by – Point Response to the Evaluation Comments - Evaluation (p.4): “Each of the e-BIZ centers had a different impact on the competitiveness of theSMEs within their respective industries. Yet when considering the increased competitiveness of the SMEs the only certain thing is that the EIA significantly raised the competitiveness of e-BIZ centers’ parent companies. The EIA distorted the market in some of the industries; for example, in logistics and transportation by supporting one of the numerous companies that provide a GPS monitoring and tracking service.” EIA Response: The suggestion that EIA distorted the market in the BiH logistics and transportation sector needs to be reconsidered, and it is inaccurate to imply that that the primary focus in this sector was to support only the provision GPS monitoring and tracking service. In fact, the primary sector development assistance that EIA helped introduce to the BiH marketwas creation of a comprehensive transportation & logistics support platform with an array of ICT based services including the first domestic online freight exchange; in the case of the e-BIZ Center (Transwork), GPS services was one of many services. *** The Evalation Team Comment 3: Comment on Transwork’s package of services and breakdown of total revenues per service delivered: Transwork provides three types of services: GPS tracking - 92.4% of total revenues, fuel consumption control – 0.09% of total revenues, installation of sensor on the fuel tank cover – 7.47% and other services – 0. 04% No Product /Service Unit Price (BAM) % in total revenues 2012 % in total revenues 2013 1 GPS device 205 57.2 36.6 2 Subscription for GPS tracking service 20 37.8 52.3 Comment [ET25]: No. The Evaluation Team only tried to establish whether there was a proof that Arteco served at least 50 BiH SMEs as it was expected. What we found out was that Arteco served 39 BiH SMEs. Also, what we learned from interviews with Arteco and USAID was that this kind of cooperation, where Arteco attempted to connect foreign buyers and BiH SMEs, was stopped by USAID. According to Ms. Boba Vukmanovic "that was not what they were expected to do". “1) Finding international buyers for the BiH wood processing SMEs and contributing to an increase in the export of processed wood products from BiH They applied two models in delivering this service: (1) They establish contact with foreign company / customer who has its own product. Arteco finds local producer for them and take 10% commission. Example is company Scano from Norway. (2) Arteco develops product and technical documentation, hires local wood processors and than exports products. Arteco was providing these services from 2009 to 2011. They stoped with it because Ms. Dobrila Vukmanovic asked them to do so. She claimed that this was not the purpose of the project. “ Please see Annex 8 - ARTECO Interview Minutes 2 (audio recording of the interview with Arteco is also available). Comment [ET26]: It is our understanding that this was the very reason why the EI Activity was granted extension. The Second Extension document clearly recognized that TTC “The Center is in its nascent phase and had some early problems with malfunctioning of the new high tech equipment (solved by distributor fairly quickly) and errors by young operators (solved by additional training paid by the owner).” However, the Second Extension document states that it is expected TTC to serve at least 40 SMEs. 8 3 Device installation 20 3.3 3.5 4 Fuel consumption control service 3 0 0.05 5 Device de-installation service 20 0.04 0.02 6 Device movement service 20 0.06 0.02 7 Sensor installation on fuel tank lid 70 1.4 7.47 8 Other services 0.2 0.04 100 100 The same services were / are provided by: Excerpt from Trasnwrok’s Business Plan: “It has been estimated that in BiH at this time there are around 6000 users of GPS services: - Tim System d.o.o Sarajevo - has around 2000 users - Satwork d.o.o. Banja Luka - has 1500 users - Limes d.o.o. Sarajevo – has around 1000 users - Others - have around 1500 users combined “ However there were also other providers of GPS tracking and fuel consumption tracking services: http://www.starnet.ba http://www.gpstim.ba/onama/ (they have their own in-house developed software) http://www.satwork.net http://www.alfapro.ba/cijene.html http://sourcecode.ba http://www.gps-ba.net http://www.trackstar.ba (entire system designed and produced in BiH) http://www.systech.ba/index.ph More broadly, the Evaluators are urged to be more specific about the instances of “market distortion” in the case of EIA and clear about how “market distortion” ran counter to the project design (approved by USAID) to introduce new innovations into economic sectors by working with specific businesses as catalysts for driving these new innovations (the e-BIZ Centers). ************** - Evaluation (p.4):“The E-BIZ center Arteco made quite an important contribution to the wood processing sector by creating an outsourcing opportunity for SMEs in the development of product technical documentation and prototypes. However, since the design of new products is not so frequent in BiH present demand amongst SMEs within the wood processing sector for this service remains low. Greater demand for prototype development came from foreign customers.” 9 EIA Response: EIA regrets that Evaluators have failed to recognize Arteco’s strong efforts to educate and train local SMEs on importance of the product design for their overall competitiveness. Although, the demand from local SMEs is still low, it is steadily improving in recent years. More importantly, the Evaluationdoes not consider that “foreign customers” that have developed prototypes at ARTECO are, in vast majority of cases, customers that were about to place significant manufacturing orders from a variety of BiH producers, including SMEs other than Artisan/ARTECO. The high technology product prototyping at ARTECO was necessary before these orders could proceed. Hence, ARTECO’s servicing of the “foreign customer” for prototyping actually represents indirect servicing of local SMEs i.e. by providing a critical, previously non-existent R&D function. ************** - Evaluation (p.4):“Almost 90% of BiH textile companies work for foreign customers on basic cutting and sewing operations based on received patterns; however, ICT capacities in the majority of these companies are non-existent. Therefore, if adequately mentored and managed, TTC could become the ICT resource center for many BiH textile companies presently without any ICT capacity. Unfortunately, after being operational for almost two years TTC’s outreach remains low and it continues to use its resources mainly in support of production in its parent company.” EIA Response: The Evaluation’s statement about TTC’s length of operation is incorrect; therefore the ET’s conclusion about low outreach should be reconsidered. At the time of evaluation, TTC was in operation for only 15 months as their official opening day was April 20, 2012. Furthermore, the Evaluators were informed that TTC slowed their initial outreach efforts until all early phase technology absorption issues have been resolved. TTC wisely recognized that their SME market is small and that any inception phase processing error on client’s fabrics could have had devastating effect on their reliability reputation and by default on the outreach. ************** - Evaluation (p.4):“EIA management of the M&E system was poor. The EIA invested little or no effort to establish baseline data or track changes in the competitiveness of SMEs within the targeted industries….” EIA Response: EIA invested considerable effort in creating baseline data through detailed and competent industry and market analysis reports that were developed in accordance with the USAID approved Life of Project Implementation Plan (LPIP) and with help from industry experts. Eight reports were developed in total, two for each of the four targeted sectors. These reports were shared with the Evaluation Team. Whether such baseline data was the most appropriate for this project is a matter of debate and we are ready to discuss this further with USAID as part of the “lessons learned” exercises. ************** - Evaluation (p. 5): “Many factors prevented the buildup of the e-BIZ centers’ top quality teams. The first relates to insufficient attention and effort to provide training for e-BIZ center staff in ICT and ICT applications in specific industries. Having in mind that e-BIZ centers are supposed to be centers of excellence for the dissemination and transfer of new high impact information and communication technologies, the EIA has done too little to build e-BIZ center staff knowledge and skills in the application of high impact ICT…” Comment [ET27]: The Evaluation Team cannot argue about whether someone “is about to place significant manufacturing orders”. We never claimed that prototype development service is not important. We never claimed that provision of services to foreign customers is bad. The Evaluation Team only claims that Arteco provided services to 39 BiH SMEs. Comment [ET28]: The Evaluation Team cannot reconsider the following facts from the Second Extension related to TTC: “Activities during the Extension Period: Execute wide ranging marketing activities, aimed at reaching the targeted number of client SMEs (40) set in the e-BIZ Centers business plans; Development of Client Relationship Management, the ICT based system that will help the e-BIZ Center improve its marketing and client relationship capability; Finalization of all internal processes (organizational, staffing, cost saving etc.) aimed at achieving full operational and financial sustainability; of international outreach to EU based clients aimed at securing new value-added manufacturing orders for BiH SMEs that would utilize TTC’s ICT based manufacturing support services; and Resolving anticipated technology based production bottlenecks identified in the international consultant’s report.” The Evaluation Team cannot reconsider the fact that TTC provided service to 19 SMEs (including 2 parent companies). Please see Annex 2 - TTC CLIENTS SHARE IN TOTAL REVENUES Comment [ET29]: That is the reason why the EI Activity was granted an extension. The Second Extension document clearly recognized that TTC “The Center is in its nascent phase and had some early problems with malfunctioning of the new high tech equipment (solved by distributor fairly quickly) and errors by young operators (solved by additional training paid by the owner).” However, according to the Second Extension document, TTC is still expected to deliver activities enlisted in the comment 28. Comment [ET30]: All these reports do not provide baseline data about competitiveness of targeted SMEs and about competitiveness of targeted sector. Not a single report provides baseline data on capacities of eBIZ Centers staff. These were baseline data the Evaluation Team needed as a starting point to answer evaluation questions and they were not collected by the EI Actiivty. 10 EIA Response: The reference to the first factor implies that, by the time of this evaluation, Centers have not become platforms for dissemination and transfer of new high impact ICT and that staff did not build necessary knowledge and skills which is not accurate. In fact, the three Centers (ARTECO, TTC and Transwork) have substantially achieved that goal as they currently offer the top of the line ICT based services for repeated customers, a sure sign that their teams have a top quality technical expertise. This expertise was developed through equipment manufacturer’s trainings and EIA provided consultants which in our view, is an acceptable mix. ************** Comment [ET31]: ARTECO - Out of 39 SMEs served in total, 30 were one-time clients. Client retention in their five-year history is almost non￾existent. Out of these 30 SME clients, 13 received one-time service worth less than 510 $ US each. When the Evaluation Team interviewed SME clients, they had hard time to remember they received some service and even harder for them to understand what we were asking them when discussing whether these services increased their competitiveness. TTC – Out of 17 third party SMEs served by TTC, 12 were one-time clients. 11 - Evaluation (p.6):“The TTC and Arteco e-BIZ centers were the only ones to build technology related capacities.” EIA Response: We disagree with this statement as we believe that the Transwork also managed to build strong technology related capacities. This was achieved through outsourcing the development of technological aspects of Transwork’s operations to the closely connected parent firm, which is fully in line with the approved business plan and expectations from EIA. ************** - Evaluation (p.6): “However, the number, frequency and staff coverage of the group trainings/seminars organized by the EIA could have been higher. A few organized group seminars related to marketing and financial management were organized; however, Transwork and TTC staff did not attend any of these marketing and financial management trainings and the EIA did not make any effort to repeat them after the two new e-BIZ centers had been established and included in the Activity.” EIA Response: To the best of our recollection, EIA informed Evaluators that, following the execution of the initial group trainings, the e-BIZ Centers asked for the more specific, on site, capacity building assistance. Based on the partner feedback, EIA decided to focus more on providing expert consultants and that explains the relatively small number of trainings delivered. Furthermore, the statement that “Transwork and TTC staff did not attend… trainings” is only true if we accept the premise that owners/founders were not part of Center’s staff which in our view, and in reality, was not the case. The founders have participated in EIA trainings and they were expected to apply and transfer the acquired knowledge during their Centers’ inception phase which effectively lasted until the end of EIA. ************** - Evaluation (p.7):“Operational management - The Evaluation Team could not confirm any capacity building assistance directly targeted at improving managerial skills or in adopting and integrating technology in order to gain maximum efficiency and productivity. The Evaluation Team was also unable to confirm any capacity building assistance directly targeted at improving knowledge and skills in other management areas i.e., strategic planning and human resource management.” EIA Response: For example, EIA provided SOWs for local and international consultant support given to TTC in areas of planning, management, integrating technology and production efficiency. EIA staff also participated directly and provided extensive HR support to all four Centers during their staff recruitment. Moreover, additional information on this capacity building assistance could have been provided to the Evaluators e.g. during interviews but the Evaluators never held interviews with EIA Staff who could have provided this information –namely the Business Development Managers (BDMs) or the two Chiefs of Party (COPs). ************** - Evaluation (p.7): In the case of Arteco, the EIA, on its own initiative, submitted new financial reports the following month that covered the month of July 2013. These two financial statements were fundamentally different and indicated structural changes in business operations. If, for example, we Comment [ET32]: Transwork is an independent company which does not have any in-house technology (ICT) related capacity. The Evaluation Team will not argue whether having a money to pay someone to develop technology for you can be considered as your own technology development capacity. Comment [ET33]: No comment. Please see evaluation question 1. The Activity even reported participation of Transwork’s staff to this trainings in its 2011 Annual Report (page 3): “Together with other EI supported e-BIZ Centers, the Satwork owner and Transwork staff attended a two-day Marketing and Sales Skills seminar that the EI project held during the last week of March 2011.” Comment [ET34]: Not true again. It does not matter how many times it is repeated. The Evaluation Team had interviews with the EIA staff (including COP and Business Development Managers) two times. Beside that, the Team Leader met COP two times. During the second interview, where also Mr. Tetelman was present, at some point after 45 minutes from beginning of interview, Mr. Tetelman, COP and Business Development Manager, stopped answering the Evaluation Team questions. The only answer the Evaluation Team was able to get was “ We will get back to you on that”. The Evaluation Team interviewed eBIZ Centers' managers and staff and asked about direct assistance provided by local and international consultants. At minimum, half of these consultancies for which the Evaluation Team received SOW, they could not remember or they were not very positive about assistance provided. (Audio recordings from these interviews are available); We analyzed SOWs and there were no specific tasks related to strategic planning and human resources management. As the Evaluation Team stated in its reports, all consultancy SOWs, except in case of Transwork, were related to marketing, promotion and sales. 12 look at the structure of liabilities and more specifically the liabilities towards the related legal entities1 we can see that on June 30 2013 they amounted to 118,000 BAM. Looking back through the previous annual financial statements the Evaluation Team determined that this amount of liability had existed since the beginning of the Activity. The parent company, Artisan, created this liability. Instead of contributing capital to Arteco, as it was obliged to according to the Fixed Obligations Grant, Artisan gave money to Arteco in the form of a loan…” EIA response: The above paragraphs clearly show that Evaluators are misunderstanding and, more dangerously, misinterpreting key provisions from Arteco’s Fixed Obligation Grant (FOG). Artisan was not obliged to provide capital to Arteco according to the FOG. Artisan was obliged to provide in kind and cash contribution for the operations of Arteco.Cash was indeed provided and used for various payments necessary for Arteco’s operations. Artisan decided to record that contribution as the loan to its daughter firm, which is a very common practice in BiH. EIA did not object to such recording of this cash contribution in Arteco’s financial statements as the mode of recording was not relevant to the implementation of EIA and, more importantly, for the assessment of Center’s sustainability per the approved Life of Project Implementation Plan (LPIP). EIA’s main concern was that cash was provided and spent in accordance with FOG requirements, and most importantly on development needs of Arteco. ************** - Evaluation (p. 8):“If the parent companies are taken out from equation then both Arteco and TTC are not financial sustainable. TTC generates between 80% (2012) and 90% (2013) of its total revenue through services provided to its parent company, Kismet d.o.o, and to Vocal Productions d.o.o. Arteco generates about one-third of its total revenue by providing services to its parent company, Artisan d.o.o” EIA Response: We disagree with the need to take the parent company out of the equation. The approved EIA project design accepted that such reliance on parent company will likely happen in early phases of e-BIZ Center development. Please refer as well to the assessment conducted by Cardno related to this point. And the Evaluators acknowledge this point in this very paragraph -- namely that ARTECO’s reliance is at the one third while TTC’s reliance is much higher which demonstrates that the mature Center (ARTECO) progressed to the point of lesser reliance. ************** - Evaluation (p.8):“All of the e-BIZ centers, with the exception of Transwork, had poor financial stability. Other types of financial analysis, such as Z-Score and the Kralicek Quick Test, indicated that all of the e-BIZ centers, except for Transwork, had issues with financial stability.” EIA Response: We question that the Evaluation’s assessment that Centers had poor financial stability per this statement. The Evaluators should be clearer on how these financial analysis tools are applicable to startup businesses. (see also response related to Evaluation methodology in subsequent section). ************** 1 Since Artisan and Arteco should, in line with the Project, be two independent legal entities and not a parent￾child business this method of accounting entry is also incorrect. Comment [ET36]: The above paragraph tells, that the EI Activity instructed Arteco to erase liability from its financial records in order to improve financial indicators. Pleas see Annex 11 - ARTECO INTERVIEW TRANSCRIPT - FINANCE. Also audio recording of the interview is available. Instead of investing in eBIZ Center, Artisan indebted Arteco and than for this loan received free services from Arteco. Artisan did not pay for Arteco services since they offset Arteco’s accounts receivables with that “loan”. But, as the EIA representative is often stating, it might be that it was a common practice in transitional countries such as Bosnia and Herzegovina and thus it was ok. Comment [ET35]: Please se Arteco’s Business Plan (attachment to FOG) and financial statements and projections and amounts of planned capital and amounts of planned liabilities (debt). It matters whether someone provides money as capital or in a a form of loan. Comment [ET37]: No comment on “common practice”. Comment [ET38]: The Evaluation Team again states that, if the parent companies are taken out from equation, then both Arteco and TTC are not financial sustainable. These eBIZ Centers are not designed to serve parent companies but to serve BiH SMEs in targeted industries. It is easy to achieve financial sustainability if each time you fail to acquire enough clients and sell enough services, you parent company jumps in and gives you job and / or transfers money to cover for your operating costs. One factor of success is how many BiH SMEs these centers served and whether eBIZ Centers can operate as sound businesses from the revenues coming from provision of these services. Not acquiring enough third party SME clients leads to two conclusions: (i) either services offered are not needed in the market, or (ii) parent company did not bother too much to acquire enough external SME clients since they needed eBIZ Center production capacity for its own production. If an eBIZ Center did not achieve financial sustainability from revenues generated exclusively from sales of services to third party SMEs, we than must consider that the reason behind it is one of the two previously mentioned. If that is not a point, why bother - they can work only for parent companies and have no worries about financial sustainability. That is a fact whether ... Comment [ET39]: The Evaluation Team has no comment on the EIA questioning the methodology. The EI is not familiar with some special methods for start-up company analysis of financial reports. On the other hand, the applied methodology is accepted and approved by the USAID. 13 - Evaluation (p.8):“Transwork is the only e-BIZ center that achieved its targeted number of clients. TTC did not even manage to achieve 50% of its targeted number of clients. According to data received from its accountant, Exploring BiH did not manage to achieve the targeted number of 70 clients. Finally, Arteco served many clients but only 38 SMEs from Bosnia and Herzegovina.” EIA Response: It is not clear what constitutes the basis for the Evaluation’s determination about Centers not achieving the targeted number of clients. For example, we are not sure how the data received from Exploring BH’s accountant failed to provide evidence for all SME clients. During the project implementation, EIA obtained all bank statements for Exploring BH for the period 2009-2012 and they clearly show payments from Exploring BH to more than 130 local SMEs (majority hotels) that in practically all cases mean that these SME clients have benefited from Exploring BH’s processed bookings. Furthermore, more than 60 BiH SMEs have been served by Arteco at the time of the evaluation according to Arteco’s Client Relationship Management (CRM) system – in other words, achieving the project objectives in terms of meeting the numbers of SME clients served. The CRM software was developed through the support from EIA for Arteco and Transwork and this system allows for the comprehensive tracking of all services provided to each client as well as tracking specific phases for each service/project delivered including meetings, communications etc. EIA regrets that Evaluators have failed to note and examine this important client management tool. Finally, even the TTC achieved the targeted number of clients albeit in the final weeks of the EIA project. However, in our view, this is understandable considering the late start and technology absorption issues at this rural Center. ************** - Evaluation (p.9):“The majority of services provided by the e-BIZ centers were not new or non￾existent in the market. All of the services provided by TTC were either already present or can be done manually. The main contribution of TTC’s services to some of its clients came in the form of precision tailoring and faster production. Prototype development was the only unique service provided by Arteco. Transwork services can be provided by a dozen other service providers, while Exploring BiH is perceived as just another small travel agency. The results of the SME questionnaire survey also confirmed that although awareness of the services provided by the e-BIZ centers was high the majority of SMEs did not consider these services to be new or previously non-existent in the market.” EIA Response: The above statement suggests that comprehensive, “one stop shop type”, ICT-based, business support services platforms were already present in BiH wood, textiles and logistic sectors prior to the EI intervention. It is true that many of the Centers services existed in sporadic and closed business systems before EIA intervention. However, these services were never compiled into an accessible business/manufacturing support hub. ************** - Evaluation (p.9):“Financial data on changes in the volume of sales of SMEs that received services from the e-BIZ centers does not constitute the basis for the conclusion that the e-BIZ centers improved the competitiveness of their clients. In all four sectors almost half of the SMEs that received SME services experienced a decrease in their total volume of sales or did not experience any change in total sales volume.” Comment [ET40]: Please find attached Annex1 - ARTECO TYPE OF CLIENTS AND SERVICES and Annex 3 ARTECO CLIENTS SHARE IN TOTAL REVENUES (documents submitted by Arteco) where all Arteco’s clients are divided in several categories. In the category of BiH SMEs they enlisted 39 companies. Comment [ET41]: The Evaluation Team will not comment what allegedly happened after completion of the evaluation and the EI Activity. The Evaluation Team received a document (please see Annex2 TTC CLIENTS SHARE IN TOTAL REVENUES) from TTC where they enlisted 28 SMEs served by TTC. Out of these 28 SMEs, 9 did not pay for services and the Evaluation Team could not verify them as clients as per the EI Activity definition of a client. The Evaluation Team was in position to confirm only 19 SMEs that paid for services received from TTC. Out of these 19 SMEs, two SMEs are parent companies (Kismet doo and Vocal Production doo). Therefore the Evaluation team was able to verify only 17 “third party SMEs” as clients. Not to mention that the Evaluation Team verified as a client even SMEs that received one-time service worth 15 $US (Contessa doo) and 63 $US (Sana Linea ad). Nevertheless, this is less than 50% of targeted 40 SMEs. Comment [ET42]: That is why the EI Activity was granted an extension 14 EIA Response: The financial data time range listed in the draft evaluation report should not be considered sufficient for conclusions of this type. Please see the subsequent section for additional elaboration on this point. ************** - Evaluation (p.9):“The Evaluation Team could not find evidence that the e-BIZ centers had an impact on associations. One of the reasons for this could be the attempt by the e-BIZ centers to use the trade associations as a marketing channel through which to sell their services. In addition, it might be the case that the associations and the public sector alike perceived the entire project as a private company endeavor.” EIA Response: We believe that e-BIZ Centers had an impact on associations – however, the PMP did not articulate a methodology for tracking and assessing this impact. A lesson learned is that such tracking is necessary. From an impact perspective, however, this impact was very strong in terms of helping change mindsets about application of latest technology and focus on product development as the basis for SME competitiveness. For example, the e-BIZ centers have collaborated with associations in promoting introduction and use of advanced technology (TTC) and joint collaborative efforts to help local producers access new markets (Arteco). This information was evident from project’s quarterly and annual reports that were shared with Evaluators. We arealso concerned that the Evaluators have not obtained the proper source of information pertaining to the impact on associations. The report (Annex III) only provides (under Persons Interviews heading) one bullet stating “Representatives of Business Associations”. No sufficient evidence is provided for concluding that associations perceived the entire project as a private company endeavor. ************** - Evaluation (p.17):“The EIA provided lists of all organizations and individuals that had received services from the e-BIZ centers; however, these lists contained many individuals, organizations and companies that could not be considered as e-BIZ center clients, as specified by the EIA’s documents.2 We found a large number of individuals, secondary school students, furniture designers and foreign companies in these lists. The Evaluation Team used the following criteria to select SMEs from these lists for interview: balanced territorial distribution (two entities, different parts of BiH), an almost equal number of SMEs from all targeted industries and SMEs of different size (micro, small and medium).” EIA Response: The statement above is misleading as it suggests that all lists from e-BIZ Centers included the non-BIH SME clients. Only the list provided by Arteco included non-SME clients but these clients could have been easily identified and separated. As evident from the footnote the Evaluators are confusing e-BIZ Center clients from EIA project beneficiaries. The e-BIZ Centers had to keep lists of all clients served. The Evaluators requested the lists of all Center clients, not the refined lists that includes just locally registered SMEs. This was explained to the Evaluators during the meeting with them. ************** 2 All Activity documents specify that the beneficiaries of the Activity must be SMEs from Bosnia and Herzegovina. Furthermore, as defined in the Annual Report for the period January 2011 – December 2011, the unit of measure for the number of e-BIZ center customers is “the number of customers paying for e-BIZ center services”. Comment [ET43]: The Evaluation Team interviewed management of all eBIZ Centers and asked them whether they managed to establish any meaningful cooperation with business associations. The answers received from all eBIZ Centers was negative. There was no reason to investigate further on this issue. Audio recordings of these interviews are available. Management of Transwork informed us that, when they attempted to establish cooperation with the association and present Transwork to membership of the association, they were asked to pay if they want to present their business to association’s membership. They refused to do so. The association asked to pay for presentation of Transwork’s products and services to its membership because there was no prior cooperation with the association and because presenting commercial products and services to association membership is a private company endeavor. Comment [ET44]: The Evaluation Team statement is not misleading and does not suggest anything. We received two different lists of Arteco’s clients. One with 88 and another one with 122. These lists contained many individuals, NGOs, foreign companies. Sending such lists could be considered misleading by the Evaluation Team. Also, there was an issue whether Mr. Tetelman, COP and Business Development Manager knew who could be considered and verified as an eBIZ Center's client. The Evaluation Team could not get straight answer during the interview with COP, Mr. Tetelman and Business Development Manager. They claimed that clients are all enlisted in the documents submitted. They also claimed that clients were even the companies that visited center during the open day event. The claimed that clients are companies that clicked once on the Exploring web site. This confusion ended by the EI Activity team telling the Evaluation Team that they would get back to us on this. Audio recording of this interview is available. Comment [ET45]: The Evaluation Team asked for the list of clients as per project definition of the client. The Evaluation Team did not have any reason to interview individuals, students, NGOs in order to determine whether services they received from EBIZ Centers increased their competitiveness 15 - Evaluation (p.18):“For question 2, the Evaluation Team evaluated both the financial and operational sustainability of all four e-BIZ centers. As defined by the Cooperative Agreement, financial sustainability is achieved when an e-BIZ center is able to deliver products and services to the market at a price that covers its expenses and generates a profit. In order to determine financial sustainability based on the above-mentioned definition the Evaluation Team analyzed the financial statements of the e-BIZ centers. However, since the above-mentioned definition of sustainability is extremely narrow and is used primarily as a cost-accounting tool that only covers a point in time it would not provide information on the ability of the e-BIZ centers to sustain the provision of the Activity’s benefits in the long run. Therefore, the Evaluation Team proposed a different approach and introduced a new balanced score card method along with new financial indicators to measure financial sustainability. The latter were accepted by USAID as part of the approved Evaluation Work Plan.” EIA Response: Please see the subsequent section on responses to the Evaluation methodology and approach. However, as it relates to the technical characteristics of the e-BIZ Centers in terms of performance, please note first that the definition of sustainability that was used aboveis not the one shared with EIA or its partners, so we were unable to provide evidence to support this definition. Moreover, please note that in August 2013, the Implementing Partners expressed concern in writing to the Evaluators that the Evaluation was using a different standard of sustainability than that which was used by the EIA since its inception (noted in footnote #5 of the Evaluation on p.7) As the letter of 2 concern noted: “Recommendation: We respectfully suggest that the Evaluators and USAID use the standard of sustainability contained in the EI Life of Project Implementation Plan / M&E section and the Annual Work Plans or at least explain more fully why a different definition of sustainability is being used. This is the standard of sustainability that the EI project has been using for the past five years. It is also a reliable indicator in that it is the same as the one that was used for the e-BIZ Macedonia project, which established Centers that have proven to be sustainable. We would be happy to provide the PMP for the e-BIZ Macedonia project to substantiate this. The e-BIZ Centers have been successful not only in achieving this sustainability target, but they have met or exceeded all of their objectives in terms of generating in-kind investment, serving SME clients, and perhaps most importantly, catalyzing innovation in their respective sectors. To suddenly adopt a new definition of sustainability in 2013 is both to unfairly “move the goal posts” at project end and to obscure the developmental impact that the e-BIZ Centers have made. ************** - Evaluation (p. 26): „Through the interviews that were conducted, it became obvious that EIA staff as well as e-BIZ center management were not clear which clients they were supposed to serve despite the fact that this is clearly defined in the Annual Performance Report. During the interviews the definition of ‘client’ ranged from those individuals, organizations and companies that visited the e-BIZ center website to those that appeared on the e-BIZ center websites to those that visited the e-BIZ centers during the Open Day event.” EIA response: As the e-BIZ Centers are designed to serve as inclusive sector development platforms for targeted sectors they have invariably attracted individual clients (e.g. furniture designers) and foreign clients, as they needed prototypes for manufacturing orders that they were placing in BiH. EIA regrets that our sincere attempt to convey these important developmental considerations and Comment [ET46]: We did use the CA definition of sustainability. The Evaluation Team analyzed sustainability as a brake-even point. However, we used other methods as well, which USAID approved. It is our understanding that Ms. Whitmore was consulted on 3/26/13 about both definitions and agreed that the measures used by the EIA were not adequate. Her email to Ms. Vukmanovic is attached. Please see Annex 13 - EMAIL EXCHANGE SUSTAINABILITY DEFINITION This is something that should be discussed with USAID. It is our understanding, however, that the USAID Evaluation Policy does not recommend/ require anything like that. In accordance with the Policy, only SoWs and draft evaluation reports should be shared with various stakeholders. The Evaluation Work Plan is our contract deliverable. Comment [ET47]: Not correct. The Evaluation Team used the CA definition of sustainability. However, we used other methods as well, which USAID approved. It is our understanding that Ms. Whitmore was consulted on 3/26/13 about both definitions and agreed that the measures used by the EIA were not adequate. Her email to Ms. Vukmanovic is attached. Please see Annex 13 EMAIL EXCHANGE SUSTAINABILITY DEFINITION 16 logical attempts by Centers to broaden customer base was misconstrued by Evaluators as not knowing which clients Centers were supposed to serve. ************** - Evaluation (p.27)FINDINGS – (Related to QUESTION 1: To what extent, in a measurable way, has the EI Activity succeeded in capacity building for e-biz center staff?) EIA response: Throughout this entire section ET fails to mention the involvement and on site capacity development assistance that was delivered to e-BIZ Centers by the number of EIA-procured local and international consultants. EIA provided 37 consultants on more than 50 assignments to support e-BIZ Centers with the focus ranging from planning and management to production and marketing. EIA believes that this oversight, whether by accident or on purpose, results in an incomplete picture of the support that EIA provided to Centers. ************** - Evaluation (p. 27) „Throughout the life of the activity, the EIA organized three group trainings: (i) a two-day Power of Sales seminar, (ii) a two-day Financial Management training and (iii) a one-day BiH Entrepreneurship and Business Development training.According to the attendance lists provided by EIA, the trainings delivered and staff trained were as shown below.” EIA Response: EIA disagrees with the Evaluators’ calculation of the number of staff trained as EIA believes that Center owners must be included as staff trained regardless of the fact that they are not formally registered as Center employees. Center Founders were integral to the setup of each Center as all of them served for many months of the inception phase as the regular Center staff.Furthermore, it is inaccurate to compare the number of staff trained e.g. in 2009 against the total number of all staff in the four Centers in mid-2013. The fair and proper comparison would be between the total number of trainees and the total number of staff at the two existing Centers at the time of delivery of listed trainings. ************** - Evaluation (p. 27) „The e-BIZ centers were never properly instructed or trained in how to track and record cost-share arrangements in accordance with the approved e-BIZ centers’ business plans. Because of this the e-BIZ centers financial reports do not give an accurate picture of the e-BIZ centers financial position.” EIA response: TheCenters were given precise instructions from EIA related to documenting founder’s contribution to the e-BIZ Centers. Under the FOGs (see TTC FOG, Attachment C, Milestone/Invoice templates) the e-BIZ Centers were obliged to submit cost share documentation/evidence to EIA for review & approval including “receipts, original timesheets…” and estimated values of in kind software and equipment etc. Therefore, the Evaluator’s expectation that all or most of the cost share should have been entered in Centers’ financial records is without merit. Regrettably, this expectation and the Evaluators’ incorrect view of the process resulted in a series of inappropriate conclusions findings including the crucial statements that the Evaluators were unable to verify partner’s cost share. EIA delivered hundreds of documents, timesheets and receipts to Evaluators confirming that the founders’ contribution was documented and filed in compliance with FOG stipulations. Comment [ET48]: Audio recording of the interview with Mr. Tetelman, COP, Business Development Manager is available. They could not agree among themselves who the clients were and after internal consultation they decided they would “get back to us on that”. The Evaluation Team considers this as lack of clear understanding “which clients they were supposed to serve” Comment [ET49]: The Evaluation Team recognized in its report that the EI Activity invested a lot of efforts in provision of technical assistance in the areas of marketing, promotion and sales. That is also the case with these 37 consultants (except in case of Transwork) who were mainly engaged to perform tasks related to marketing, promotion and sales of services Comment [ET50]: No comment on the remark that owners are considered staff / employees. Comment [ET51]: The staff of eBIZ Centers received or did not receive training, simple as that. Also, it is important to know how much of capacity building assistance was retained with eBIZ Centers. Comment [ET52]: Please re-read our comment carefully. We refer to the business plans on which all four FOGs were based. We strongly claim that the eBiZ centers did not meet their cost share obligations as specified in the business plans, especially in terms of equity requirements. We believe that USAID would at least expect EDC and FHI/360 to follow cost share requirements for all four FOGs as required from both implementing partners for their own Cooperative Agreement overall cost share as follows: Cooperative Agreement – Accounting for Cost-Share page 44 and page 51 (303): "Cost share refers to the resources a recipient contributes to the total cost of an agreement. Cost share becomes a condition of an award when it is part of the approved budget. The cost share must be verifiable from the recipient's records and can be audited. Cost sharing applies throughout the life of an agreement and the AOR should monitor the recipient’s financial reports to ensure that the recipient is making progress toward the required cost share." 17 ************** - Evaluation (p.29): „However, information provided by one of the e-BIZ centers about the training turned out to be different from the attendance list provided by the EIA. When asked why they claimed that their staff had attended trainings organized before the e-BIZ center in question had been established, the Director of this e-BIZ center answered that they had been instructed to do so by the EIA because the EIA did not have a sufficient number of capacity building activities.3 EIA response: EIA assumes that the example listed on page 29 refers to the Transwork Center. EIA, in line with its principled position, considers the participation by Transwork’s owner at EIA training that was held during preparations for Transwork’s formal establishment as the capacity building for Transwork’s staff. This EIA position was communicated to Transwork’s current Director in much the same way as in this part of our response. Again, we are happy to reiterate that EIA believes that training to Center owners must be considered as capacity building for the Center regardless of the fact that owners have not formally registered as Center employees. Center Founders were integral to the setup of each Center and they served as regular staff for many months of Centers’ early development phases. EIA believes that our view of this matter firmly corresponds to the reality and, in essence, each owner has the formal link to their respective Center. ************** - Evaluation (p.32):“The majority of indicators signaled significant structural changes in Arteco's business operations over the period 31.12.2012 to 31.07.2013 yet they were not the result of changes in the center’s business operations but rather the result of an exercise in financial engineering. Liquidity improved significantly primarily due to cash transfers from the parent company Artisan to Arteco’s account.” EIA response: We believe that the transaction that is referred to on this page is an overdue payment from the parent firm (Artisan). The project does not understand why this considered “financial engineering” and would request that this statement be stricken from the report unless there is additional justification to suggest impropriety. ************** - Evaluation (p.33): “It was difficult to assess Arteco’s financial position due to the strong business connection with its parent company Artisan.” EIA response: Evaluators do not provide sufficient rationale as to why did the “strong business connection to its parent company” made assessment of Arteco’s financial position difficult? ************** - Evaluation (p.34): Arteco has not achieved the financial projections and capital target values as specified in the business plan. Financial projections set the value of Arteco’s capital in 2009 at 711,720 BAM; however, on June 30 2013 the total value of Arteco’s capital was 2,000 BAM. Arteco has not achieved the target values for current and long-term assets set in the business plan. 3 The interview conducted with the Director of the e-BIZ center was audio recorded. Comment [ET53]: Transwork reported participation of staff, not owners, to these trainings. Their staff did not participate at these trainings. Even in its 2011 Annual Report, the Activity reported participation of Satwork’s owner and Transwork’s staff (please see page 3 of the Report). Please see Annex 12 - TRANSWORK TRANSCRIPT CAPACITY BUILDING Comment [ET54]: Please see Annex 11 - ARTECO INTEVIES TRANSCRIPT 2 FINANCE where Arteco’s management states that they were instructed to adjust financial reports in order to “ improve the picture, the condition of Arteco, the coefficients, gearing ratio, profitability ratio, you understand? They were asking for this as of 31 July… So, we were simply acting on instructions and we did that…” Audio recording of this interview is available. 18 EIA response: The target values for long term assets included the machinery provided by EIA. However, Arteco could not have registered the value of this machinery in its official books as the title was with EIA until the completion of the EIA project. ************** - Evaluation (p.35): “As stated during the interview with the Director of Exploring BiH, in the early stages of the project it was already clear that the website/portal was a failure.The explanation given by EIA staff for the reasons behind this was that at the time the project commenced there was no booking website in BiH and nobody was able to foresee what would happen in the area of ICT within the tourism sector (booking.com etc). However, within the Market Analysis Report, the EIA recognized that, “A large number of booking agents are active in BiH but three - www.booking.com, www.venere.com and www.all-hotels.com - are dominant. The EIA also learned that several sites, including www.ljeto-zima.com and www.sobe.ba, offered detailed accommodation listings with information, photographs, prices and e-mail links for reservations through the hotel’s own websites.4 ” EIA Response:. At the time of the project inception there were no BiH sites that have offered online booking option and even the local sites listed in the Market Analysis (ljeto-zima.com and sobe.ba) have not offered that option. As the EIA Market Analysis report stated, sites such as booking.com were active in BiH at the time as they were signing up their first hotel clients. In less than two years following the EIA Market Analysis, the booking.com managed to overtake most local or international competition in BiH so EIA observation about this matter is still valid in our view. ************** - Evaluation (p.36): “From the very beginning up until the present, the Exploring BiH portal has not been sufficiently technologically advanced to allow for direct booking via the portal. Bookings were made via e-mail communication with the Director of Exploring BiH and very soon, it became clear that Exploring BiH could not compete with ‘Booking.com’.5 There is a link to the booking.com website at the Exploring BiH web site, However, that cannot be considered as a provision of possibility for direct booking.” EIA Response: Exploring BH’s portal was sufficiently technologically advanced and tested to allow the full online booking as early as fall 2009. While the actual number of online bookings was low, it is undeniable that the technological capacity existed for online booking for many years under EIA until the site was redesigned to include another booking option. And the Evaluators should have noted that the current link to the booking.com is based on the franchise type cooperation that was recommended as the course of action for Exploring BH by an international expert Jain Abishek. ************** - Evaluation (p.41): “The e-BIZ centers' financial reporting systems were inadequate; the e-BIZ centers were never instructed in how to record the USAID and parents company contributions in financial statements. The Evaluation Team was not in a position to verify the amounts and structure of the cost-share contribution of the founders of the e-BIZ centers.” 4 Market Analysis of the High Impact ICTs as e-BIZ Center Core Offerings in the Tourism Sector 5 http://www.booking.com Comment [ET55]: The business plan foresees a total value of CNC machine at 420.000 BAM (of which 400,000 is given by USAID). The equity value in the projected balance sheet amounts to 711.720 BAM (Annex D1 p.117-118). It implies that the difference in amount of 311.720 BAM (711.720- 400.000) is planned as equity increase by Artisan through various forms (Annex D 1 p.79). But on June 30 2013, after four years of operation, the total value of Arteco's capital was still 2.000 BAM (Annex D 2) We know that Arteco could not record an increase in assets (because it does not legally own the machine), but under current accounting rules both in the United States (US GAAP) and internationally (IFRS), it must be recorded in the off-balance-sheet account. But this was not done. That raises another question, who and how calculated the depreciation on that equipment. The accountant of Arteco was fully aware of this mistake what is evident from the interview. To sum up, the Centers conducted (intentionally or not) two mistakes: (1) recored their obliged cost share via liabilities vs equity, (2) did not record machines in off balance sheet which made their financial statements inaccurate. According to the Financial Accounting Standards Board (FASB), "Financial reporting should provide information that is useful to present and potential investors and creditors and other users in making rational investment, credit, and similar decisions". Comment [ET56]: Please see Annex 16 – Exploring BH Interview Minutes “Ms. Vasic is of opinion that internet booking is a complete failure. Hotels are not paying any more. The only thing left from the original project are customers’ inquiries but even that does not go through the web site. Customers inquiries are received via e-mail exchange.” Audio recording of two interviews with Ms. Vasic are also available. Comment [ET57]: Please see Annex 16 – Exploring BH Interview Minutes “Ms. Vasic is of opinion that internet booking is a complete failure. Hotels are not paying any more. The only thing left from the original project are customers’ inquiries but even that does not go through the web site. Customers inquiries are received via e-mail exchange.” Audio recording of two interviews with Ms. Vasic are also available. 19 EIA Response: The above statement is not accurate as Centers received were instructed by EIA on how to document and evidence their in-kind contributions to the respective Centers per requirements from their respective FOGs. The Evaluators have received all documentation related to this,including copies of bank statements, timesheets, equipment appraisals etc. and that should have been sufficient for verifying the in￾kind contributions, in line with FOG provisions. And as it relates to the question of cost share, the requirements for cost share were included in the fixed obligation grants (FOG) issued to each of the organizations in order for them to support the E-biz centers. Cost share was documented and reported in accordance with 22 CFR 226.23.We were perplexed by the evaluator’s remarks. ************** - Evaluation (p.41): “Arteco provided services to 38 BiH SMEs over a period of 4.5 years. Out of this number, 28 companies received services only once over the 4.5 year period. Out of the 38 SMSs 13 received services of a value of less than 500 BAM (USD 356.65).” EIA Response: It is not clear how the Evaluators have determined this number. Arteco’s record from the EIA developed Client Relations Management (system) show that Arteco delivered paid services to more than 60 SMEs at the time the Evaluation was conducted. ************** - Evaluation (p.42): “TTC provided services to 19 SMEs over a period of 1.5 years. Two of these SMEs were companies owned by the owner of TTC, Kismet d.o.o., and Vocal Productions d.o.o. The services provided to these two companies generated 79.34% of TTC’s total revenue in 2012 and 90.73% in 2013. Out of 19 SMEs five (5) received a one-time service of a value of less than 500 BAM (USD 356.65) each.” EIA Response: The above number (19) is not correct as it only included TTC clients served by spring 2013. TTC was getting new clients on board until the end of EIA (August 2013) following extensive promotion efforts in late spring and early summer 2013. This was the natural pace for client acquisition considering the late start for this Center. And by the end of the project, 40 clients were being served by TTC, which meets the project objectives. ************** - Evaluation (p. 42): “The Evaluation Team was not able to determine the exact number of clients of Exploring BiH. The Evaluation Team received several diverse lists of clients served by Exploring BiH. The first list listed 48 clients, the second listed 71 clients, the third was called the “frequent clients list” also with 48 clients and, finally, the fourth list received from the Exploring BiH accountant had 45 clients in total. The Evaluation Team knows for a fact that even if a client had a contract with Exploring BiH that does not mean that a business transaction happened between them.” EIA Response: We are not sure what type of list was solicited and received from the Exploring BH’s accountant. Nevertheless, EIA collected copies of all bank statements from Exploring BH for the period August 2009 – December 2012 and during that period Exploring BH made payments to or received commission fees from more than 130 BiH SMEs. All of these transactions were done for Exploring BH’s executed bookings of hotel accommodation or other tourism services. If necessary, EIA is prepared to share that evidence with USAID upon request. The explanation for the list with 71 clients is simple as it included only SME clients that have signed contracts or forms that have Comment [ET58]: The Evaluation Team does not argue whether there is a track record and evidence of in-kind contributions and cost-share. The Evaluation Team stated that there was no evidence about it in the eBIZ Centers (recipients) financial statements. Comment [ET59]: Please see documents received from Arteco (Annex1 ARTECO TYPE OF CLIENTS AND SERVICES and Annex4 ARTECO CLIENTS SHARE IN TOTAL REVENUES). Comment [ET60]: Please see Annex 2 – TTC CLINETS SAHRE TOTAL REVENUES - the document received from TTC where TTC’s accountant provides list of SMEs that received services and amounts received for these services. 28 companies were enlisted and 19 paid for service. Out of these 19, two are companies owned by the owner of TTC – Kismet doo and Vocal Production doo. Comment [ET61]: The Evaluation Team asked all eBIZ Centers to submit the same documents:  List of clients  Breakdown of clients as per their share in total revenues (per annum)  Breakdown of client per type of business relation: a) long term contract, b) one-time services, c) in process of negotiation 20 formalized their long term relations with Exploring BH. EIA was insisting on contracts and our list was stricter in what it recognized as Exploring BH’s clients especially compared to the actual large number (130+) of sporadic paying clients that could have been extracted easily from Exploring BH bank statements. , ************** - Evaluation (p. 47): “The financial data concerning changes in the volume of sales of SMEs that received services from the e-BIZ centers does not constitute the basis for the conclusion that the e-BIZ centers improved the competitiveness of their clients. Almost half of the SMEs in all four sectors that received SME services experienced a decrease in the total volume of sales or did not experience any change in the total sales volume.” EIA response: It is unclear to why the data on year to year overall sector sales volumes was not compared against sales volumes for SMEs served by e-BIZ Centers as that would present a clearer picture whether SMEs served by e-BIZ Centers have performed better than their respective sectors at the time of the global recession. ************** - Evaluation (p.49): “There was a lack of involvement and support from the public sector. The EIA and e-BIZ centers failed to gain interest and support from public sector organizations and institutions. Therefore, the entire effort and burden of advancing all four target industries was placed with small companies struggling to achieve the brake-even point.” EIA response: EIA a a fairly small, USAID funded project, was not operating in the vacuum. Other,larger USAID projects (CCA, FIRMA) were taking the lead in mobilizing the public sector support for advancing targeted sectors such as wood and tourism. SEE NEXT PAGE FOR RESPONSES TO EVALUATION METHODOLOGY AND APPROACH Comment [ET62]: No comment. 21 Responses – Evaluation Methodology and Approach Overview of Evaluation Comments There are two broad concerns that the implementing partners have with the Evaluation methodology and approach. The first is that the Evaluators used measurements for e.g. sustainability that were not related to and used through the life of the project as approved by USAID. The definition of sustainability as contained in the Start-Up Implementation and Annual Work Plan is as follows: “…At the end of the project, each e-BIZ Center has (a) an annual operational and financial plan that is based on past performance, which demonstrates that the Center will reach the break-even point within one year of the project; (b) the e-BIZ owner’s written commitment to continuing operations of the e-BIZ Center operations after the end of the project.” In August 2013, we wrote to the Evaluators respectfully suggesting that the Evaluators and USAID use the standard of sustainability contained in the or at least explain more fully why a different definition of sustainability is being used. This is the standard of sustainability that the EI project has been using for the past five years. It is also a reliable indicator in that it is the same as the one that was used for the e-BIZ Macedonia project, which established e-BIZ Centers that have proven to be sustainable. We would be happy to provide the PMP for the e-BIZ Macedonia project to substantiate this. We believe ample documentation exists to show that e-BIZ Centers have been successful not only in achieving this sustainability target, but they have met or exceeded all of their objectives in terms of generating in-kind investment, serving SME clients, and perhaps most importantly, catalyzing innovation in their respective sectors. To suddenly adopt a new definition of sustainability in 2013 is both to unfairly “move the goal posts” at project end and to obscure the developmental impact that the e-BIZ Centers have made. Furthermore, it misses entirely the fact that the sizeable in-kind investment made by the parent companies alone is a strong indicator that most of the e-BIZ Centers will continue to operate successfully going forward in their own right. The Evaluation needs to elaborate more concretely on why this definition of sustainability cannot or should not be used to measure the e-BIZ Centers’ financial health. Second, we believe that overall, the Evaluators’ approach is inconsistent with the 2011 USAID Evaluation Policy which states that evaluations should include “use of data collection and analytic methods that ensure, to the maximum extent possible, that if a different, well-qualified Evaluator were to undertake the same evaluation, he or she would arrive at the same or similar findings and conclusions.”6 In order to entertain such a scenario, measures or criteria of successwould have to be clearly stated and evidence provided that align with such measures so that “readers…have sufficient information about the body of evidence and how information was gathered to make a judgment as to its reliability, validity and generalizability.”7 In particular, if the USAID-approved M&E plan did not provide the evidence necessary for the Evaluators’ questions, then it is recommended that more argumentation, context and evidence be provided as to why the Evaluators’ approach is superior rather than the flat assertions advanced (for example, the claim that the definition of sustainability used in the M&E plan was “too narrow” is baldly asserted without justification or reasoning as to what would constitute a more robust definition). Where the M&E plan’s definitions and performance indicators are rejected, concrete measures and targets are not offered in their place. 6 USAID 2011 Evaluation Policy, p. 9. 7 Ibid. Comment [ET63]: The Evaluation Team did use the CA definition of sustainability. However, we used other methods as well, which USAID approved. It is our understanding that Ms. Whitmore was consulted on 3/26/13 about both definitions and agreed that the measures used by the EIA were not adequate. Her email to Ms. Vukmanovic is attached. Please see Annex 13 EMAIL EXCHANGE SUSTAINABILITY DEFINITION Comment [ET64]: It seems that, for some reason, the Evaluation Team was provided with the documentation that tells different story from this one 22 Finally, rather than an objective, scientific tone that would be in keeping with the 2011 USAID Evaluation policy, the report strikes an antagonistic tone that is simply not borne out by the evidence as presented in the document. Point – by – Point Response to the Evaluation - Evaluation (p.2): “…The EIA and e-BIZ centers interfered with the evaluation process at several points by adjusting documents and providing documents that were produced after the evaluation process had begun…” EIA Response: EIA submitted dozens of project reports, industry and market analysis, SOWs, consultant and training reports, business plans, FOGs and literally several hundreds of copies of Center founder’s contribution (cost share) bills, receipts, equipment assessments, in kind labor timesheets etc. EIA understands that the Evaluation Team (ET)could have had uncertainty or even suspicion about some documents as the two or three mentioned in the draft Evaluation Report. However, ET has never approached EIA for explanations or clarifications before they boldly stated that the “interference” has taken place. - Evaluation (p.43): “A review of the properties of the submitted documents revealed that, for example, the Exploring BiH survey report was created on July 25 2013, despite the fact that the survey was supposed to have taken place in 2010. However, the Evaluation Team did not investigate further in order to confirm any potential file manipulation.” EIA Response: Related to integrity of Centers’ client survey documents, EIA reviewed the above mentioned “Clients Survey” folder and noticed that several word documents (from Exploring BH and TTC) indeed show,in the properties section, that their last version was “modified” on July 24, 2013. Nevertheless, the original documents have been created by e-BIZ Centers before and Ms. Vasic (Director, Exploring BH) and Mr. Banjic (Director, TTC) e-mailed their survey files to EIA weeks before they were requested by Evaluators. EIA staff compiled and saved last versions of those word files in the ZIP folder on July 24 before sending all documents to the Evaluation Team. Upon opening files that were received via e-mail and saving them in a designated ZIP folder, some of these word documents were obviously saved as modified versions although nothing substantial was done to them. It is true, however, that in one instance (Exploring BH’s file) EIA staff made minor formatting, date and English language correction in the heading section before resaving that document. However, none of these actions mean that any substantive part of any of the survey reports have been changed or altered. We would have appreciated if the Evaluators had asked ahead of time for clarification on this, rather than stating this in the Evaluation. If requested, EIA is ready to submit all evidence related to this issue to USAID. ************** - Evaluation (p. 5): “Capacity building efforts to increase e-BIZ center staff knowledge and skills in areas of high impact information and communication technology and services were basic and insufficient.” EIA Response: As we mention several times in the section related to Question 1 (on capacity building), the Evaluators offer no concrete measure of successful capacity building to which e-Biz centers failed to conform nor evidence to demonstrate failure to conform. And when the Evaluation does provide data from the beneficiaries related to EIA’s capacity building, this data suggests that the beneficiaries had benefitted significantly from capacity building activities. Furthermore, we are concerned that it is apparent from the report that the Evaluators have failed to review or assessed the Comment [ET65]: Measure of successful capacity building to which e-Biz centers failed to conform: ARTECO VISION – “Within five years, ARTECO will become the leading company of applied technologies for the wood processing industry in BiH, a central location of knowledge for the wood industry and an organization that will link international buyers with domestic producers.” ARTECO MISSION is a) to provide technical services to SMEs in the wood processing industry in Bosnia￾Herzegovina (BiH) to increase their competitiveness so they can compete in downstream international markets, and b) to link international buyers with local customers of the ARTECO. To its clients, ARTECO will provide: Large scope of universally accessible services, supported by a unique price/quality ratio to be provided by superior experts and sub-contractors. Access to modern ICT applications, Permanent improvement of business processes to improve the competitiveness of the BiH wood processing industry. TTC VISION - Vision is to have a textile-technology center with innovative and creative skilled staff, efficient organization, provision of state-of-the-art and advanced services to satisfied clients. TTC mission is to become the leader in providing technology-based services and know-how for the BiH textile industry. TRANSWORK VISION - In the next five years, the Transwork e-biz Center (hereinafter: the Transwork EBC) shall become the leading company in BiH and the region providing ICT services to companies whosemain activity is related to transport and logistics. The Transwork EBC shall become the leading force that will make a major contribution to enabling logistic and transport companies from BiH to be well equipped to compete in local, regional and international markets. The mission of the Transwork e-BIZ Center is to increase the competitiveness of BiH transport and logistics companies by introducing modern ICT solutions in their daily operations enabling these companies to increase their customer base, profits, reduce costs and penetrate the European market. 23 instances of capacity building support delivered by the EIA procured expert consultants. We respectfully request that this conclusion be revised for the executive summary in the absence of more robust reasoning and presentation of evidence. ************** - Evaluation (p.16): “The team sent the questionnaires via email but the response rate was extremely low for both groups of SMEs (client and non-client).” EIA Response: EIA kindly requests that the Evaluators provide some context as to why email was adopted as the data collection methodology rather than face-to-face interviews. ************** - Evaluation (p.18): “Therefore, in an effort to reconstruct the baseline data, the survey questionnaire for e-BIZ center employees contained ‘before and after’ questions.” EIA Response: Here, and throughout the discussion of Question 1, it is unclear what the measure of success of capacity building activities is that the project failed to meet. Nor is there clear evidence demonstrating that the project failed to meet such measures. ************** - Evaluation (p.18): “However, since the above-mentioned definition of sustainability is extremely narrow and is used primarily as a cost-accounting tool that only covers a point in time it would not provide information on the ability of the e-BIZ centers to sustain the provision of the Activity’s benefits in the long run.” EIA Response: The PMP defined sustainability as follows “…At the end of the project, each e-BIZ Center has (a) an annual operational and financial plan that is based on past performance, which demonstrates that the Center will reach the break-even point within one year of the project; (b) the e￾BIZ owner’s written commitment to continuing operations of the e-BIZ Center operations after the end of the project.” This was approved by USAID and retained throughout the life of the project as the operative definition. EIA acknowledges that milestones and goals may change over time and would have modified this definition in partnership with USAID if requested. ************** - Evaluation (p.19): “As for questions 3 a-c, the team compared market analyses prepared for each of the four industries and the data from those surveys with the current situation within the targeted SMEs served by the four e-BIZ centers.” EIA Response: Mention is not made of subquestions 3a to 3c until page 77 and therefore the reference here is confusing. We kindly request that the Evaluators make mention of these sub-questions upfront. - Evaluation (p.19):To answer this question, and due to the survey failure, the Evaluation Team analyzed changes in the competitiveness of the targeted SME clients. The team did this by tracking changes in certain indicators, such as profitability and sales. In order to determine whether the e-BIZ centers’ SME clients had experienced an increase in the volume of sales or profitability the Comment [ET66]: Do not understand the question 24 Evaluation Team used available financial data on the performance of companies and sectors collected by the Agency for Financial Information and Consulting Services (AFIP). The Evaluation Team only managed to collect data for companies and sectors in the Federation of Bosnia and Herzegovina. EIA Response: We believe that the Evaluators could have not just used the financial performance data for serviced SMEs and compared them in terms of their year to year results, The more adequate analysis would have to include the comparison with data on the performance of the entire sector during a given period as only then we would be able to know whether the Center assisted SMEs have performed better or worse than their sector averages. This is especially important considering the global recession. We are also concerned how the Evaluation can reach credible conclusions based on data collected only in one part of Bosnia Herzegovina (Federation BiH). Furthermore, many private sector experts do not consider AFIP statements as fully reliable considering the scale of the so called grey economy in Bosnia Herzegovina. ************** - Evaluation (p.19): Although there is no evidence that the e-BIZ centers had surveyed their clients on a regular (annual) basis the EIA reported results for this indicator each year in the work plan and M&E plan. More importantly, USAID data quality assessment (DQA) of this indicator verified the existence of the survey EIA Response: EIA Response: We respectfully disagree. The M&E plan included in the Annual Work Plans for years 2009, 2010, 2011, 2012 and 2013 clearly states that this survey would be conducted in Year 3 (2010), not annually. However, it is crucial to note here that the decision on timing of this survey made complete sense from the business development perspective. The EIA’s year 3 was essentially year 2 for the e-BIZ centers which meant that the sufficient time have passed to allow for the Centers to build the client base that would offer a valid sample for surveying SMEs on whether the e-BIZ Center services have strengthened their business. In addition, the Evaluation sections of the 2011 and 2012 Annual Work Plans state (on p. 35 for 2011 and p. 16 for 2102): “Utilize a survey of e-BIZ Center customers, conducted at the end of the project by an independent evaluator, to include questions that provide data to assess the impact of the e-BIZ Center services, e.g., questions regarding whether e-BIZ Center services helped to strengthen the SME customer’s business such as by increasing sales, reducing costs, etc.” An annual survey was never expected and results for such a survey were never reported in any work plan. The M&E plan in every Annual Work Plan merely states the targets expected (for example, that “50 customers report e￾BIZ Center services strengthened their businesses, for wood, tourism, machine tools and agriculture Centers”. p. 35 of the 2010 Work Plan or p. 32-33 of the 2011 Work Plan). Where results were reported, they were preceded by the term “Actual”. No actuals for this target were ever reported by the EIA in any work plan. We would ask that specific and direct citations of the Annual Work Plans be provided to substantiate this claim. ************** - Evaluation (p.21): One of the examples of interference with the evaluation process is the provision of contradictory data on the same topic. The Evaluation Team received documents from the EIA and the e-BIZ centers yet in the majority of cases the list of SME clients received from the e￾BIZ centers and the EIA were different. 25 EIA Response: We accept that client lists that Centers provided could have been more up to date than those that EIA had on file at the time of evaluation. Even if there was some difference between the lists, that could have been easily clarified upon request. It is unfair to say that this issue is an example of interference with the evaluation process. ************** - Evaluation (p. 20): “The Evaluation Team tried to reconstruct the missing data through a questionnaire survey. The questionnaire asked selected SME clients whether the technical assistance and services provided by the e-BIZ centers had contributed to an improvement in their business operations.” EIA Response: EIA kindly requests that the Evaluators more fully explain the methodology they used to select SME clients for the survey, how many, and whom. ************** - Evaluation (p. 21): “The evaluation was limited to the approved M&E indicators for the Activity. Some of these indicators, such as the financial sustainability indicator, were inadequate to measure the accomplishments of the EIA.” EIA Response: We kindly request that the Evaluators provide more justification and evidence for the assertion that these indicators were inadequate to measure the accomplishments of the EIA. This is asserted as assumed fact, rather than being argued for and justified. In rejecting the approved M&E plan’s performance indicators, which were considered sufficient, the Evaluators should clearly and succinctly state why it is acceptable to reject these indicators and demonstrate more comprehensively why their own proposed indicators and measures of success are more suitable. ************** - Evaluation (p. 21): Other Issues that affected the Findings The Evaluation Team has strong reservations about the information and data provided by the EIA, the e-BIZ centers and the interviewed SMEs. The EIA and the e-BIZ centers interfered with the evaluation process at several points by adjusting documents and providing documents produced after the evaluation process had begun. This interference made the evaluation process difficult and makes the evaluation findings suspect. More specifically the issues listed below. • Information provided by the e-BIZ centers to the Evaluation Team on attendance at the trainings did not reflect what happened in reality. The e-BIZ centers were instructed by EIA staff to report more capacity building activities.8 EIA Response: This is simply incorrect. E-BIZ Centers were not instructed to report more capacity building activities than what happened in reality. The only guidance that Centers have received from EIA relates to providing them with information on what is considered as capacity building support (e.g. provision of EIA consultants, trainings, continuous technical support from EIA BDMs etc.). In this respect EIA was obviously interested to make sure that Centers do not mis-report all instances of 8 Transcripts of audio-recorded interviews. 26 project’s capacity building support. EIA also instructed e-BIZ centers to report their founders’ participation in EIA trainings as the Center staff training. The reason is simple, as EIA believes that founder’s participation, especially in Centers’ early pre-registration phase represented a significant capacity building assistance despite the fact that founders were not subsequently formally registered as employees of their respective Centers. Therefore, EIA and ET’s apparently have a significant difference regarding the participation of Center founders in EIA trainings. • The financial reports of the e-BIZ centers were adjusted significantly during the one-month period after the evaluation process had begun. EIA Response: EIA strongly disagrees with this statement as it implies that all Centers have adjusted financial statements. Only the parent company for one Center (ARTECO) decided to pay outstanding services payments during the mentioned time period. Nevertheless, all these actions by ARTECO made business sense, were legally permissible and were done in a transparent manner. • As previously mentioned, the lists provided by the EIA and e-BIZ centers of SMEs served by the e-BIZ centers were different; even lists of clients that came from one source were not identical. EIA Response: Comment on this was provided earlier in the text. • During the initial meeting with the EIA, the EIA staff stated that no formal survey on the competitiveness of SME clients had been performed; this was also confirmed by the EIA at the end of the process. The nearest to it were the telephone inquiries from the e-BIZ centers to their SME clients asking them about their level of satisfaction with the services provided. Nevertheless, the EIA submitted list of SMEs that had participated in the questionnaire survey. Several companies stated during field interviews that they had never participated in any kind of survey: neither through the questionnaire nor via the telephone. The majority of issues with the validity and reliability of data on the performed surveys related to the TTC and Exploring BiH e￾BIZ centers. EIA Response: ET’s statement is not quite correct. According to PMP the clients were surveyed to determine whether “e-BIZ Centers services have strengthened their business” and not on their competitiveness which is the broader area. According to our recollection, EIA staff stated that no formal external surveys by EIA or by consultants were completed other than in ARTECO’s case. EIA also stated that only surveys that were completed were done by e-BIZ Centers. Based on the information provided to EIA, Exploring BH surveyed 21 out of its 60 clients at the time of their survey (Fall 2010). When compared to the list of eight (8) Exploring BH clients that were interviewed by ET it is evident that ET met only 3 clients that have been surveyed by Exploring BH in 2010. The other 4 tourism firms that ET visited in 2013 were either not the clients of Exploring BH in 2010 or were not included in the survey sample. Hence, EIA believes that this information sufficiently explains why “several (tourism) companies stated that they had never participated in the questionnaire”. It is because they really haven’t participated as Exploring BH interviewed its other clients. EIA is unsure as to why the ET received a similar response from textile sector firms. Our field experience shows that Directors are not always aware of their firm’s participation in some activities especially in non-essential activities such as surveys that are often completed by engineers or sales personnel. 27 ************** - Evaluation (p. 22): “The financial reports of the e-BIZ centers were adjusted significantly during the one-month period after the evaluation process had begun.” EIA Response: We kindly request that the Evaluators provide more explanation and evidence to substantiate this claim. ************** - Evaluation (pp. 26-30. Findings Section for Question 1 Overall EIA Response for this section: The organization of the Findings section on Question 1 frustrates the reader’s ability to appreciate how the Evaluators drew conclusions. We request that the full data set from the Evaluators’ survey mentioned be summarized succinctly and coherently (perhaps in a summary table) to allow for an objective assessment of results, rather than presenting findings in a selective and piecemeal manner. The methodology for drawing conclusions is similarly unclear: are the conclusions of the Evaluators based on the results of the survey they administered or not? If so, the results – though presented in a confusing manner – seem to suggest that the capacity￾building activities had a very positive effect. Are the Evaluators then discounting the results of their own survey? If so, then what is the definition of success and the methodology proposed to ascertain whether it was achieved? In drawing such negative conclusions, the Evaluators owe us a more rigorous presentation of their reasoning and evidence, or should simply state that the evidence is too inconclusive to draw conclusions. ************** - Evaluation (p. 27): The Evaluation Team assessed the EIA reports on capacity building efforts and conducted an analysis of the EIA documents and reports on the training and technical assistance provided. In accordance with our Request for Information sent to the EIA, the Evaluation Team received two sets of documents: • trainings provided to e-BIZ centers; and • technical assistance provided to e-BIZ centers. EIA response: It is apparent that ET had not included numerous instances of direct, on site, consultant assistance to e-BIZ Centers when they evaluated EIA success related to this question. ************** - Evaluation (p. 27): “Financial management capacity building efforts were focused mainly on increasing the capacity of e-BIZ center staff to track and record cost-sharing arrangements based on the EIA internal cost-share tracking system. The e-BIZ centers were never properly instructed or trained in how to track and record cost-share arrangements in accordance with the approved e-BIZ centers’ business plans.” EIA Response: This is simply not true as Centers have received specific instructions/training from EIA on how to evidence and document their contributions (cost share). Because the Evaluators never state a clear measure for success of capacity building efforts or provide aligned evidence that the e-Biz centers did not meet these measures, it seems in bad faith to conclude 28 the E-biz centers did not meet that measure. It would have been more helpful to clearly demonstrate a measure of success, and then provide concrete evidence that e-Biz centers did not meet that measure. Criticisms are offered of the scope of the training, not the impact of that training, nor are details offered as to how their conception of a more robust scope for such trainings is related to their measure of success. ************** - Evaluation (p. 29): “There was no increase in technical skills at the Exploring BiH e-BIZ Center: computer skills, understanding software, hardware, troubleshooting, teamwork and the organizational structure.” EIA Response: We kindly request that the authors present the entirety of their data for all four centers in one succinct data presentation that would allow the reader to ascertain the validity of their conclusions. A more rigorous presentation of the actual data would allow the reader, at a glance, to know, for example, whether the other three centers showed an increase in skills and by how much. Such a presentation of actual data would be more in keeping with the 2011 USAID Evaluation policy objective of allowing readers to “have sufficient information about the body of evidence and how information was gathered to make a judgment as to its reliability, validity and generalizability.”9 ************** - Evaluation (p. 30): “The Evaluation Team did not have any meaningful method to evaluate the impact of day-to-day and hands on type technical assistance in a measurable way.” EIA Response: We kindly request that the Evaluators provide more justification as to why the impact of day-to-day mentoring and coaching cannot be evaluated. ************** - Evaluation (p.30):In terms of operational management, the Evaluation Team could not confirm any capacity building assistance directly targeted at improving managerial skills to adopt and integrate technology in order to gain maximum efficiency and productivity. The Evaluation Team was also unable to confirm any capacity building assistance directly targeted at improving knowledge and skills in other areas of management i.e., strategic planning and human resource management. EIA Response:EIA is not clear as to why the ET has not considered capacity building support targeted at improving above listed skills that was provided by local and international consultants (Martin McElwee, Emir Medanhodzic, Richard Bangs, etc.). Furthermore, EIA staff provided specific and detailed support in recruitment efforts for each Center thus improving Centers human resource management capacity. ************** - Evaluation (pp. 30-31): “Having in mind that e-BIZ centers are supposed to be centers of excellence for the dissemination and transfer of new high impact information and communication technologies, 9 USAID 2011 Evaluation Policy, p. 9. 29 the EIA has done too little to build e-BIZ center staff knowledge and skills in the application of high impact ICT.” EIA Response: We kindly request that the Evaluators clarify and make transparent what the concrete evidence is for this claim. The only concrete criticism levied in the discussion is of the length of the training offered, but no evidence of its suboptimal impact is provided. ************** - Evaluation (p. 31): “However, as the aforementioned definition of sustainability is extremely narrow and only covers a point in time without providing information on the ability of the e-BIZ centers to sustain the provision of the benefits of the EI Activity in the long-run, the Evaluation Team proposed a different approach. USAID accepted the latter suggestion as part of the approved Evaluation Work Plan. EIA Response: We kindly request that the Evaluators explain and justify their assertion that the definition of sustainability is narrow. A more transparent presentation of the Evaluators reasoning process would be in keeping with the 2011 evaluation policy. ************** - Evaluation (p. 32): “Other types of financial indicators analyzed by the Evaluation Team indicate that Arteco is under threat of potential bankruptcy. The value of Altman’s Z-Score test on June 30 2013 was 1.3, which falls within the range of a company at risk of bankruptcy” EIA Response: The article attached by the Evaluators to provide context on Altman’s Z-scores states that a score between 1.1 and 2.6 is clearly within the “grey zone” of values easily misclassified. Also, the Z-scores vary depending on whether it is a public manufacturing firm (Z-score), a private manufacturing firm (Z’-score), or a private, non-manufacturing firm (Z’’-score). In their footnote, the authors apply the third range (for private non-manufacturing firms), but this is inappropriate for Arteco, a private manufacturing firm, which falls under range number 2. Since Arteco is a private manufacturing firm, by the author’s standards, a score of 1.3 would fall into the “grey zone in which distress may or may not be impending”. The Evaluators should also clarify whether they used the Z-score for private companies (which weights the variables involved in the score’s calculation differently and which replaces the ratio of market capitalization with book value of equity). EIA also requests that authors include the full calculation of the Z-score as well in the document, to be transparent. The Evaluators should also mention the limitations of the Altman score, particularly the fact that it is usually not applied to firms with assets less than $1 million USD. ************** - Evaluation (p. 33): “TheKralicek Quick Test indicated that Arteco’s financial stability was poor, even after the aforementioned financial engineering exercise. On July 31 2013, the value of the Kralicek Quick Test was 0.32.” Comment [ET67]: Please see Annex A 30 EIA response: We kindly request that the Evaluators be more transparent about their use of the Kralicek Quick test and discuss, in particular, the advantages and disadvantages of using this method with SMEs such as the e-Biz centers. ************** - Evaluation (p. 33): “The Arteco e-BIZ Center has achieved the break-even point, as the measure of financial sustainability;however, other financial indicators indicate poor financial stability and the threat of potential bankruptcy.” EIA Response: EIA respectfully disagrees with this assertion as stated in the conclusion section for Question 3 and believes such a conclusion cannot in good faith be stated as by the authors’ own standards the firm’s Z-score is in the “grey zone”, so deemed because such scores are too inconclusive to be able to predict bankruptcy. EIA therefore requests that the Evaluators present this evidence in good faith as too inconclusive as to be able to predict bankruptcy. ************** - Evaluation (p. 34): “Other types of financial indicators analyzed by the Evaluation Team indicate that Exploring BiH is under extreme threat of bankruptcy. The value of Altman’s Z￾Score test has been negative throughout the entire life of the Project.” EIA Response: EIA kindly requests that the Evaluators transparently state the actual Z-score and provide the actual calculation conducted to obtain that score. ************** - Evaluation (p. 36): “Exploring BiH achieved financial sustainability by reaching the break-even point and having revenue that exceeded expenses by 1%. However, other financial analyses indicate poor financial stability and extreme bankruptcy risk. “ EIA Response: There is no cited evidence for this other than a vague reference to the Altman z-score on p. 34. ************** - Evaluation (p.39):The e-BIZ centers did not achieve the financial benchmarks set in their respective business plans. EIA Response: EIA believes that this statement is not supported by the evidence. Three out of four e￾BIZ Centers have shown better actual year 1 and 2 financial results compared to the projections in their respective business plans. One of the potential reasons for this conclusion by evaluators could be related to usage of the wrong start dates for Transwork and TTC as both Centers opened their doors in spring of 2012. Below please find the quick review income statements provided by the e-BIZ Centers. Evaluators are urged to note that Centers have shown satisfactory financial health especially in light of their business plan projections. In short, the income statements for Transwork and TTC show the following (in KM): Comment [ET68]: The statement is well supported by the evidence. Please see Annex B 31 Name of Firm Total Planned for the first five quarters in terms of profit/loss after taxes Actual for the first five quarters in terms of profit/loss after taxes (2012 and 6 months of 2013) Actual 2013 in terms of profit/loss after taxes (first six months) Transwork -143,915 KM -89,878 KM 9,544 KM TTC -230,752 KM -50,227 KM 1,661 KM The income statements for Exploring BH and ARTECO show the following (in KM): Name of Firm Total Planned for the first two years in terms of profit/loss after taxes Actual for the first two years in terms of profit/loss after taxes (2010 and 2011) Actual 2012 in terms of profit/loss after taxes ARTECO -93,830 KM 12,069 KM 20,821 KM Exploring BH 20,596 KM -12,894 KM 302 KM ************** - Evaluation (p. 42): “The EIA was supposed to perform annual surveys in order to determine changes to the above-mentioned indicator.” - EIA Response: We respectfully disagree: the M&E plans included in the Annual Work Plans for years 2009, 2010, 2011, 2012 and 2013 clearly state that this survey would be conducted in Year 3 (2010), not annually. In addition, the 2011 and 2012 Annual Work Plans clearly state (on p. 35 for 2011 and p. 16 for 2102): “Utilize a survey of e-BIZ Center customers, conducted at the end of the project by an independent evaluator, to include questions that provide data to assess the impact of the e-BIZ Center services, e.g., questions regarding whether e-BIZ Center services helped to strengthen the SME customer’s business such as by increasing sales, reducing costs, etc.” ************** - Evaluation (p. 44): “Due to the survey failure, the Evaluation Team analyzed changes in the volume of sales and the profitability of SME clients.” EIA response: No mention is again made of profitability in the discussion that follows. Only results on changes in volume of sales are presented. We request that the authors present their data and evidence on profitability to allow the reader to understand better how they arrived at their conclusions. Also, the authors should describe why it is sufficient to use only a one-year difference in sales volume results to draw their conclusions. ************** 32 - Evaluation (p. 46): “The Evaluation Team interviewed six (6) of Arteco’s clients, five (5) of TTC’s clients, five (5) of Transwork’s clients and eight (8) Exploring BiH clients. Their perception of the importance of the services provided by the respective e-BIZ centers ranged from “not important at all” to “very important.” EIA Response: This is an extremely vague presentation of data results. In place of it, EIA requests that the Evaluators provide a precise summary table of data results. Comment [ET69]: Agree – removed from the Report 33 Comments on the Recommendation Section Please note that responses to some of the recommendations have already been addressed in the sections above. Additional comment is made below for responses to recommendations that were not addressed previously or where EIA believes that reiteration of key points should be made. 1. USAID BiH should pay far more attention to the process of approving the initial work plan, M&E plan, and sub-grants. The initial EIA work plan set the indicators for the life of Activity. The Evaluation Team strongly believes that the two performance indicators proposed by the EIA and approved by USAID AOR were inadequate. Furthermore, three of the four business plans that accompanied the four FOGs contained obvious errors that USAID did not recognize when it evaluated and approved them. EIA Response: The EIA’s initial work plan and M&E approach was developed based on a proven model while the Center Business Plans have been developed in a collaborative effort from EIA and project partners and with help from external consultant experts. Additionally, each plan was reviewed, commented on and validated by industry experts. We would appreciate more clarification on why and how the work plans and particularly business plans were not adequate for this project. ************** 2. USAID BiH should focus much more on regular and rigorous monitoring of the implementation of its activities/projects. The fact that the EIA reported results based on non-existent data for several years indicates a serious problem in the way USAID BiH managed and oversaw the Activity. The lack of an adequate USAID monitoring system to track progress towards the defined objectives and goals on a regular basis resulted in some surprising findings in the evaluation. EIA Response: This is inaccurate. EIA did not report results based on non-existent data. The EIA PMP states that data following data had to be collected: e-BIZ business and operations plans, Center sales records, surveys of e-BIZ Center customers. EIA has followed the frequency of collection as stated in the PMP as we collected sales data from bank statements and surveys from Centers. ************** 8. The recipients of the USAID assistance should have recorded the cost-sharing and other financial +agreements in their financial reports in accordance with the country specific accounting standards. This should have been clearly specified as an obligation of the assistance recipients and regularly monitored by the implementing organization and USAID.The e-BIZ centers were not instructed on how to record the USAID and parent company contributions within their financial statements. The Evaluation Team was not in a position to verify the amounts and structure of the cost-sharing contributions from the founders of the e-BIZ centers. EIA Response: It is evident from the draft report that the Evaluators have not accepted key provisions from the USAID approved project FOGs related to evidencing parent firm investment into the e-BIZ Center. As a result, the Evaluators have stated that they are “not in a position to verify contribution of the founders of the e-BIZ Centers” despite the detailed evidence on this contribution that EIA collected in accordance to the Fixed Obligation Grant (FOG) (or should we say 22 CFR 226.51) and presented to the Evaluators. ************** Comment [ET70]: Business plans are full of errors and possibly fraud. We submitted a detailed analysis of each eBiZ center's business plan and pointed to the errors. For example, Transwork's business plan (part of the FOG) included a balance sheet that didn't balance and negative equity. Similar examples can be provided for all four centers. Thus, we question expertise of the experts who reviewed the business plans. We believe that any comment after this is not necessary. See Annex C for a quick review of business plans/FOG errors. 34 10. The e-BIZ centers should have built in capacity to research and apply ICT in their production processes, not to reduce it to the marketing departments of their parent companies. Presently all ICT research and development and ICT applications in production capacities rest with the parent companies. EIA Response: This blanket statement for all EIA supported e-BIZ centers is incorrect. ARTECO and TTC clearly have the capacity to research and apply ICT in their production processes while the same capacity also exists at Transwork, albeit through the outsource option from the closely connected parent firm. We request that the Evaluators provide more specific evidence indicating lack of capacity such as in ICT. ************** 11. More effort should have been invested in attracting and retaining highly qualified staff with relevant knowledge, skills and experience. Thee-BIZ centers, especially ARTECO and TTC, should have done much more in terms of creating more attractive packages to attract highly qualified employees, due to their remote location. Another option would be significant investment in education and training of the existing staff. EIA Response: EIA could agree in general about the need to attract and retain highly qualified staff. This is a fair lesson learned though more discussion is warranted about how one does this successfully in a start-up environment when time and resources are limited and given other priorities such as introducing new products to the market. ************** 12. More should have been done in raising awareness amongst SMEs about the benefits of introducing ICT into the production processes. Although the EIA and e-BIZ centers made a lot of effort to market their products they failed to raise awareness amongst SMEs concerning the importance of applying ICT in production processes. This is partially a consequence of the failure to establish meaningful cooperation with the public sector and business associations. Management of the e-BIZ centers believed that SMEs in Bosnia and Herzegovina were still not aware of the benefits they could gain through the introduction of ICT applications to the production process. Therefore, they did not manage to expand their client base and disseminate ICT applications throughout the target industries. EIA Response: The Evaluators have failed to note that there were number of activities conducted by the e-BIZ Centers, and with support from EIA, that were focused on raising awareness about applying ICT in their production processes. For example, TTC organized two large Open Day services presentations to SMEs together with the BiH Textiles Association while ARTECO and TTC also held number of presentations in cooperation with Chambers and associations that were focused on raising awareness about new technologies and use of ICT applications in production and service processes. Evaluators could have found evidence about this in EIA Quarterly and Annual Reports. ************** 14. More should have been done to establish long term cooperation between the e-BIZ centers and SMEs. The majority of SMEs were one-time clients and the e-BIZ centers did little to adequately manage their relations with them and turn this into long-term cooperation. EIA Response: We disagree with the above statement as Centers have been able to steadily attract more repeat customers. We would ask the Evaluators to provide more concrete evidence that the clients of the 35 e-BIZ Centers are predominantly one-time clients. Furthermore, ARTECO and Transwork installed the EIA-developed Client Relationship Management (CRM) systems by spring 2013 while TTC is planning to introduce this system soon. The CRM allows for integrated client management approach and we regret that Evaluators have not reviewed and assessed this important client management tool. ************** 15. Arteco and its owners should be required to meet the financial target values set by the business plan. Financial projections set the value of Arteco’s capital in 2009 at 711,720 BAM; however, on June 30 2013 the total value of Arteco’s capital was 2,000 BAM. Values for current and long-term assets set in the business plan were also not achieved, as well as ratio between long-term and current assets set in the business plan. EIA Response: The Evaluators have failed to realize that the above mentioned capital target value could not have been realized by the time of the evaluation partially due to the fact that the title for the CNC machine resided at EIA until the end of EIA project. Comment [ET71]: ARTECO - Out of 39 SMEs served in total, 29 were one-time clients. Client retention in their five-year history is almost non￾existent. Out of these 29 SME clients, 13 received one-time service worth less than 510 $ US. When the Evaluation Team interviewed SME clients, they had hard time to remember they received some service and even harder for them to understand what we were asking them when discussing whether these services increased their competitiveness. Please se Annex 3 - ARTECO CLIENTS SHARE IN TOTAL REVENUES. TTC - Over the 16 months of TTC operations, 18 SMEs paid for TTC’s service. Out of these 18, 12 were one-time clients, two (2) companies are owned by the TTC’s owner. Please see Annex 2 - TTC CLINETS SHARE IN TOTAL REVENUES 36 1 EXCELLENCE IN INNOVATION ACTIVITY STATEMENT OF DIFFERENCES May 5, 2014 Overview FHI 360 and EIA (“Implementing Partners”) welcome this opportunity to provide this Statement of Differences (“Statement”) to the USAID-funded Excellence in Innovation Activity (EIA) Performance Evaluation (“Evaluation”). The Statement below is divided into two categories: 1) responses to the Evaluation from a technical standpoint, including the EIA’s impact on the economic sectors the project served, sustainability of the e￾BIZ Centers, and capacity building of the e-BIZ Centers and client SMEs; and 2) responses to the Evaluation in terms of its methodology and approach. As part of the Statement, the Implementing Partners would like to make the following comments related to the Evaluation and to the EIA: 1) We have been and continue to be focused on supporting USAID’s goal of fostering growth in the economy of Bosnia and Herzegovina (BiH) and stimulating innovation in the economic sectors that EIA served; 2) We appreciate the thoroughness of the Evaluation, its lessons learned raised by the Evaluation, and will apply these lessons learned, as relevant, to other similar efforts; 3) We note with concern that the Evaluation makes a significant number of assertions both about the EIA’s technical performance and conduct during the Evaluation process that we believe are inaccurate. We lay out these concerns in detail below. 4) The Evaluators missed an opportunity to provide an analysis of the impact of capacity building provided by the numerous local and international consultants that EIA procured during the course of the project. 5) The Evaluators followed the terms of the EIA Cooperative Agreement (CA) and Life of Project Implementation Plan (LPIP) to determine the number of Center clients (insisting on BiH SMEs only); however, the Evaluators seemed to disregard the CA and LPIP elements when defining criteria for assessing Center sustainability. This is a lack of internal consistency in the evaluation – using the CA and LPIP for some parts of the evaluation but not others. In this respect, the Evaluators used measurements for Center sustainability that were not related to and used through the life of the project as approved by USAID. As well, the Evaluators never shared the changed definition of sustainability with EIA even after such definition was accepted by USAID and used during evaluation of EIA. As such, the evaluation measured a crucial element of the EIA’s success and impact against a sustainability definition that was never introduced into the project during its implementation. A lesson learned is that a critical review of impact assumptions and indicator definitions and measurement should be completed by the project and USAID at least midway through implementation. 2 6) As per the question of cost share, the requirements for cost share were included in the fixed obligation grants (FOG) issued to each of the organizations in order for them to support the E-biz centers. Cost share was documented and reported in accordance with 22 CFR 226.23. The EIA met its cost share and documented its cost share in compliance with the CFR. 7) Similar to most if not all development activities carried out over a period of several years with multiple stakeholders across several economic sectors, EIA should have done some things differently. We acknowledge this below and appreciate where the Evaluation raises this point. However, the overall tenor of the Evaluation unfairly discounts the role of EIA in generating strong impact in the economic sectors, achieving project goals in terms of driving innovations in the sectors, nor strengthening SME capacity – all core factors of the approved cooperative agreement. There is ample evidence to demonstrate that EIA achieved virtually all of the project goals and objectives as set forth in the Cooperative Agreement and Implementation Plan, and USAID/BiH’s objectives as pertains to the EIA. 8) We are concerned by the Evaluation’s assertion that EIA interfered with the Evaluation process. We do not believe that EIA interfered in this manner and document below the numerous instances where EIA provided copious support to the Evaluation and also show our difference of opinion about specific claims e.g. that documents were amended or that e-BIZ Center staff was instructed to provide inflated training attendance data. As part of our internal code of conduct and commitment to uphold the integrity of external evaluations, we would not interfere in this manner. 9) During the course of the evaluation, EIA provided supporting materials, data, and actively engaged with USAID. However, we feel that there are several areas where the Evaluation missed opportunities to fully engage the knowledge of the project and the beneficiaries. As well, EIA feels that a more fully articulated evaluation should have been provided per the USAID Evaluation Policy. 10) The Evaluators have never shared their overall methodology with EIA; disclosing evaluation methodology to the subject of evaluation is standard protocol of the evaluation process. 11) There are many instances in the Evaluation where the underlying assumptions and approaches used led to inaccurate or misleading conclusions or statements. We document these below. 12) The Evaluators did not conduct one-on-one interviews with key project personnel (EIA Business Development Manager and the two Chiefs of Party). We believe that failing to do this led to an incomplete understanding of EIA. 13) According to the verbal feedback and written statements from e-BIZ Centers, the project clients felt that statements by the Evaluators appeared to reflect a bias to portray the EIA and its staff in a negative light. The project clients reported to EIA being extremely uncomfortable with the aggressive manner in which questions were apparently posed to them, and with apparently sensitive information about EIA staff that was also discussed with them. This approach appears to contradict the principles of USAID’s Evaluation Policy that local beneficiaries and clients should be treated with respect. Please refer to the Appendix for statements testifying to this from the e-BIZ Centers. 3 Responses – Technical and Performance Components Overview of Evaluation Comments – Technical and Performance The Evaluation took strong issue with the inherent design of the e-BIZ Project, particularly related to the relationship between parent company and the e-BIZ Center, and the level of outreach of the e-BIZ Centers to the SMEs. It is implied as well that through this design and relatively low levels of capacity building by EIA, the SMEs did not benefit significantly from this project, nor the sectors themselves, including in terms of innovation introduced into the sectors, The Evaluation implies that the design and way in which the EIA was carried out also did not enable the e-BIZ Centers to be sustainable in the long run. We strongly disagree with this assessment. The implementing partners would first like to rebut these claims by drawing on an assessment of the EIA e￾BIZ Centers carried out by David T. King of Cardno Emerging Markets in December 2013, and which was funded by EDC. The implementing partners believe that the Evaluators should examine closely the findings from this assessment and refer to them when making revisions to the draft Evaluation. The full assessment is included as an annex. An excerpt of these assessment findings are as follows: Key Findings and Conclusions i. Grantees benefitted substantially from the receipt of the systems. There was one potential risk in the basic design of the Project, which was clear from the beginning. The proposals would necessarily need to come from existing companies, since only they would have the experience in their sectors to understand and employ the proposed ICT-based systems effectively. This meant that there was always a certain probability that the systems could be used primarily for the benefit of the grantee company, rather than prioritizing services to other SMEs in the sector. To try to prevent this, the grantee companies were obligated under their contracts with EI to serve other SMEs – with a target of 40-50 for each center – and they understood this commitment. ii. Three of the four e-BIZ centers have performed in the manner expected, with positive development impact on their respective sectors. This means that that these centers – Arteco, TTC, and Transwork – have successfully marketed their services to SMEs in their sectors, that each has built a clientele, that this clientele is expanding, and that their revenues are significant, and for TTC and Transwork strongly growing. iii. The impact of the e-BIZ centers on their sectors is likely to continue to grow. The sales, client numbers, and third party revenues trends for Arteco, TTC, and Transwork are generally positive. The evidence of repeat customer business indicates that services provided are of ongoing value to many outside clients. For both Arteco and TTC, the services that appear to be most useful to third party SMEs involve prototyping – using IT systems to create production designs and specifications for manufacturers that fulfill buyer needs, maximize efficiency in material use, and optimize labor and production equipment use time… iv. At least three of the four e-BIZ centers are likely to be sustainable. The question of sustainability overlaps to an extent with the immediately preceding discussion as to the likelihood of future demand growth for the centers’ services. Sustainability in addition requires that the e-BIZ centers are well￾managed and profitable. 4 It is important to emphasize that the managers of Arteco, TTC, Transwork, and even Exploring all consider their centers to be attractive businesses in their own right and are committed to their expansion, separately from their parent companies v. The fourth center has failed mainly due to global developments, but may still have a future. Exploring is clearly the most problematic center. The basic cause is the very rapid growth of globally￾provided internet-based services, whose underestimation has been a hallmark of the information age. Exploring was seen as a critical need for the development of the tourism sector at a time when there was no booking service for hotel rooms in BiH. The grant was provided to deliver a service that did not exist. However, very soon after this center was launched, it was overtaken by very large booking services – Expedia, Travelocity, TripAdvisor, and especially Booking.com. ************** Before proceeding to the point-by-point response, the EIA must stress that it believes that differing views on several core project postulates are the cause for disagreements on number of evaluation’s findings and conclusions. They are as follows:  EIA believes that at least two, if not all four Centers should have been treated as bona fide startups during this evaluation. It is apparent that Evaluators have failed to treat them as startups during their analysis of Centers’ financial and operational sustainability.  EIA believes that owners have been and still are the pivotal staff members of respective Centers regardless of the fact that they have not formally registered as Center employees. This is an important point since this is the evidence provided in the report of training attendance misreporting.  As it relates to the question of cost share, the requirements for cost share were included in the fixed obligation grants (FOG) issued to each of the organizations in order for them to support the E-biz centers. Cost share was documented and reported in accordance with 22 CFR 226.23.We were perplexed by the evaluator’s remarks since EIA complied with the CFR.  Transwork’s close integration with Satwork and reliance on its technological support is logical and desirable from a startup business development standpoint and certainly expected from the planning perspective. The Evaluators assert that they do not believe that such integration is desirable but does not provide a substantiated reason for this assertion.  ARTECO’s prototyping services for foreign clients represents important indirect support to the BiH wood processing sector; these types of services complete the crucial precondition before those foreign clients could proceed to give manufacturing orders to local SMEs. It would have been helpful for the Evaluators to recognize the recent dominance of such foreign clients in ARTECO’s revenue structure as a positive development.  The late launch of Transwork and TTC that was caused by external factors, including AED’s suspension that resulted in a delay in issuing the FOGs for the textile and logistics e-BIZ centers. This should have been noted by Evaluators. The fact that these two Centers were operational for less than 16 months at the time of evaluation does not appear to be considered by Evaluators when they were assessing the development progress and achievements of these two Centers. 5 Point – by – Point Response to the Evaluation Comments - Evaluation (p.4): “Each of the e-BIZ centers had a different impact on the competitiveness of the SMEs within their respective industries. Yet when considering the increased competitiveness of the SMEs the only certain thing is that the EIA significantly raised the competitiveness of e-BIZ centers’ parent companies. The EIA distorted the market in some of the industries; for example, in logistics and transportation by supporting one of the numerous companies that provide a GPS monitoring and tracking service.” EIA Response: The suggestion that EIA distorted the market in the BiH logistics and transportation sector needs to be reconsidered, and it is inaccurate to imply that that the primary focus in this sector was to support only the provision GPS monitoring and tracking service. In fact, the primary sector development assistance that EIA helped introduce to the BiH marketwas creation of a comprehensive transportation & logistics support platform with an array of ICT based services including the first domestic online freight exchange; in the case of the e-BIZ Center (Transwork), GPS services was one of many services. More broadly, the Evaluators are urged to be more specific about the instances of “market distortion” in the case of EIA and clear about how “market distortion” ran counter to the project design (approved by USAID) to introduce new innovations into economic sectors by working with specific businesses as catalysts for driving these new innovations (the e-BIZ Centers). ************** - Evaluation (p.4):“The E-BIZ center Arteco made quite an important contribution to the wood processing sector by creating an outsourcing opportunity for SMEs in the development of product technical documentation and prototypes. However, since the design of new products is not so frequent in BiH present demand amongst SMEs within the wood processing sector for this service remains low. Greater demand for prototype development came from foreign customers.” EIA Response: EIA regrets that Evaluators have failed to recognize Arteco’s strong efforts to educate and train local SMEs on the importance of product design for their overall competitiveness. Although, the demand from local SMEs is still low, it is steadily improving in recent years. More importantly, the Evaluation does not consider that “foreign customers” that have developed prototypes at ARTECO are, in vast majority of cases, customers that were about to place significant manufacturing orders from a variety of BiH producers, including SMEs other than Artisan/ARTECO. The high technology product prototyping at ARTECO was necessary before these orders could proceed. Hence, ARTECO’s servicing of the “foreign customer” for prototyping actually represents indirect servicing of local SMEs i.e. by providing a critical, previously non-existent R&D function. ************** - Evaluation (p.4): “Almost 90% of BiH textile companies work for foreign customers on basic cutting and sewing operations based on received patterns; however, ICT capacities in the majority of these companies are non-existent. Therefore, if adequately mentored and managed, TTC could become the ICT resource center for many BiH textile companies presently without any ICT capacity. Unfortunately, after being operational for almost two years TTC’s outreach remains low and it continues to use its resources mainly in support of production in its parent company.” EIA Response: The Evaluation’s statement about TTC’s length of operation is incorrect; therefore the ET’s conclusion about low outreach should be reconsidered. At the time of evaluation, TTC was in 6 operation for only 15 months as their official opening day was April 20, 2012. Furthermore, the Evaluators were informed that TTC slowed their initial outreach efforts until all early phase technology absorption issues have been resolved. TTC wisely recognized that their SME market is small and that any inception phase processing error on client’s fabrics could have had devastating effect on their reliability reputation and by default on the outreach. ************** - Evaluation (p.4): “EIA management of the M&E system was poor. The EIA invested little or no effort to establish baseline data or track changes in the competitiveness of SMEs within the targeted industries….” EIA Response: EIA invested considerable effort in creating baseline data through detailed and competent industry and market analysis reports that were developed in accordance with the USAID approved Life of Project Implementation Plan (LPIP) and with help from industry experts. Eight reports were developed in total, two for each of the four targeted sectors. These reports were shared with the Evaluation Team. Whether such baseline data were the most appropriate for this project is a matter of debate and we are ready to discuss this further with USAID as part of the “lessons learned” exercises. ************** - Evaluation (p. 5): “Many factors prevented the buildup of the e-BIZ centers’ top quality teams. The first relates to insufficient attention and effort to provide training for e-BIZ center staff in ICT and ICT applications in specific industries. Having in mind that e-BIZ centers are supposed to be centers of excellence for the dissemination and transfer of new high impact information and communication technologies, the EIA has done too little to build e-BIZ center staff knowledge and skills in the application of high impact ICT…” EIA Response: The reference to the first factor implies that, by the time of this evaluation, Centers have not become platforms for dissemination and transfer of new high impact ICT and that staff did not build necessary knowledge and skills which is not accurate. In fact, the three Centers (ARTECO, TTC and Transwork) have substantially achieved that goal as they currently offer the top of the line ICT based services for repeated customers, a sure sign that their teams have a top quality technical expertise. This expertise was developed through equipment manufacturer’s trainings and EIA provided consultants which in our view, is an acceptable mix. ************** Evaluation (p.6): “The TTC and Arteco e-BIZ centers were the only ones to build technology related capacities.” EIA Response: We disagree with this statement; Transwork also managed to build strong technology￾related capacities. This was achieved through outsourcing the development of technological aspects of Transwork’s operations to the closely connected parent firm, which is fully in line with the approved business plan and expectations from EIA. ************** 7 - Evaluation (p.6): “However, the number, frequency and staff coverage of the group trainings/seminars organized by the EIA could have been higher. A few organized group seminars related to marketing and financial management were organized; however, Transwork and TTC staff did not attend any of these marketing and financial management trainings and the EIA did not make any effort to repeat them after the two new e-BIZ centers had been established and included in the Activity.” EIA Response: EIA informed Evaluators that, following the execution of the initial group trainings, the e-BIZ Centers asked for the more specific, on-site, capacity building assistance. Based on partner feedback, EIA focused more on providing expert consultants and that explains the relatively small number of trainings delivered. Furthermore, the statement that “Transwork and TTC staff did not attend… trainings” is only true if we accept the premise that owners/founders were not part of Center’s staff which in our view, and in reality, was not the case. It is important to note that in the case of March 2011 Sales and Marketing Seminar, Transwork and TTC owners attended together with, then designated, Directors of their respective Centers. As founders have participated in EIA trainings, they were expected to apply and transfer the acquired knowledge during their Centers’ inception phase which effectively lasted until the end of EIA. ************** - Evaluation (p.7): “Operational management - The Evaluation Team could not confirm any capacity building assistance directly targeted at improving managerial skills or in adopting and integrating technology in order to gain maximum efficiency and productivity. The Evaluation Team was also unable to confirm any capacity building assistance directly targeted at improving knowledge and skills in other management areas i.e., strategic planning and human resource management.” EIA Response: For example, EIA provided SOWs for local and international consultant support given to TTC in areas of planning, management, integrating technology and production efficiency. EIA staff also participated directly and provided extensive HR support to all four Centers during their staff recruitment. Moreover, additional information on this capacity building assistance could have been provided to the Evaluators e.g. during interviews. However, the Evaluators did not conduct in-depth, issue-specific, one-on-one interviews with EIA Staff who could have provided this information – namely the Business Development Managers (BDMs) or the two Chiefs of Party (COPs). ************** - Evaluation (p.7): In the case of Arteco, the EIA, on its own initiative, submitted new financial reports the following month that covered the month of July 2013. These two financial statements were fundamentally different and indicated structural changes in business operations. If, for example, we look at the structure of liabilities and more specifically the liabilities towards the related legal entities1 we can see that on June 30 2013 they amounted to 118,000 BAM. Looking back through the previous annual financial statements the Evaluation Team determined that this amount of liability had existed since the beginning of the Activity. The parent company, Artisan, created this liability. Instead of contributing capital to Arteco, as it was obliged to according to the Fixed Obligations Grant, Artisan gave money to Arteco in the form of a loan…” 1 Since Artisan and Arteco should, in line with the Project, be two independent legal entities and not a parent￾child business this method of accounting entry is also incorrect. 8 EIA response: The above paragraphs reflect the Evaluators misunderstanding of and, more dangerously, misinterpretation of key provisions from Arteco’s Fixed Obligation Grant (FOG). Artisan was not obliged to provide capital to Arteco according to the FOG. Artisan was obliged to provide in kind and cash contribution for the operations of Arteco. Cash was indeed provided and used for various payments necessary for Arteco’s operations. Artisan decided to record that contribution as the loan to its daughter firm, which is a very common practice in BiH. EIA did not object to such recording of this cash contribution in Arteco’s financial statements as the mode of recording was not relevant to the implementation of EIA and, more importantly, for the assessment of Center’s sustainability per the approved Life of Project Implementation Plan (LPIP). EIA’s main concern was that cash was provided and spent in accordance with FOG requirements, and most importantly on development needs of Arteco. ************** - Evaluation (p. 8): “If the parent companies are taken out from equation then both Arteco and TTC are not financial sustainable. TTC generates between 80% (2012) and 90% (2013) of its total revenue through services provided to its parent company, Kismet d.o.o, and to Vocal Productions d.o.o. Arteco generates about one-third of its total revenue by providing services to its parent company, Artisan d.o.o” EIA Response: We disagree with the need to take the parent company out of the equation. The approved EIA project design accepted that such reliance on parent company will likely happen in early phases of e-BIZ Center development. Please refer as well to the assessment conducted by Cardno related to this point. And the Evaluators acknowledge this point in this very paragraph -- namely that ARTECO’s reliance is at the one third while TTC’s reliance is much higher which demonstrates that the mature Center (ARTECO) progressed to the point of lesser reliance. ************** - Evaluation (p.8): “All of the e-BIZ centers, with the exception of Transwork, had poor financial stability. Other types of financial analysis, such as Z-Score and the Kralicek Quick Test, indicated that all of the e-BIZ centers, except for Transwork, had issues with financial stability.” EIA Response: We question that the Evaluation’s assessment that Centers had poor financial stability. The Evaluators should be clearer on how these financial analysis tools are applicable to startup businesses. (see also response related to Evaluation methodology in subsequent section). ************** - Evaluation (p.8): “Transwork is the only e-BIZ center that achieved its targeted number of clients. TTC did not even manage to achieve 50% of its targeted number of clients. According to data received from its accountant, Exploring BiH did not manage to achieve the targeted number of 70 clients. Finally, Arteco served many clients but only 38 SMEs from Bosnia and Herzegovina.” EIA Response: It is not clear what constitutes the basis for the Evaluation’s determination about Centers not achieving the targeted number of clients. For example, during project implementation, EIA obtained all bank statements for Exploring BH for the period 2009-2012 and they clearly show payments from Exploring BH to more than 130 local SMEs (majority hotels) that in practically all cases mean that these SME clients have benefited from Exploring BH’s processed bookings. 9 In total, more than 60 BiH SMEs had been served by Arteco at the time of the evaluation according to Arteco’s Client Relationship Management (CRM) system – in other words, achieving the project objectives in terms of meeting the numbers of SME clients served. The CRM software was developed through the support from EIA for Arteco and Transwork and this system allows for the comprehensive tracking of all services provided to each client as well as tracking specific phases for each service/project delivered including meetings, communications etc. Finally, even the TTC achieved the targeted number of clients albeit in the final weeks of the EIA project. ************** - Evaluation (p.9): “The majority of services provided by the e-BIZ centers were not new or non￾existent in the market. All of the services provided by TTC were either already present or can be done manually. The main contribution of TTC’s services to some of its clients came in the form of precision tailoring and faster production. Prototype development was the only unique service provided by Arteco. Transwork services can be provided by a dozen other service providers, while Exploring BiH is perceived as just another small travel agency. The results of the SME questionnaire survey also confirmed that although awareness of the services provided by the e-BIZ centers was high the majority of SMEs did not consider these services to be new or previously non-existent in the market.” EIA Response: The above statement suggests that comprehensive, “one stop shop type”, ICT-based, business support services platforms were already present in BiH wood, textiles and logistic sectors prior to the EI intervention. It is true that many of the Centers services existed in sporadic and closed business systems before EIA intervention. However, these services were never compiled into an accessible business/manufacturing support hub. ************** - Evaluation (p.9): “Financial data on changes in the volume of sales of SMEs that received services from the e-BIZ centers does not constitute the basis for the conclusion that the e-BIZ centers improved the competitiveness of their clients. In all four sectors almost half of the SMEs that received SME services experienced a decrease in their total volume of sales or did not experience any change in total sales volume.” EIA Response: The financial data time range listed in the draft evaluation report should not be considered sufficient for conclusions of this type. Please see the subsequent section for additional elaboration on this point. ************** - Evaluation (p.9): “The Evaluation Team could not find evidence that the e-BIZ centers had an impact on associations. One of the reasons for this could be the attempt by the e-BIZ centers to use the trade associations as a marketing channel through which to sell their services. In addition, it might be the case that the associations and the public sector alike perceived the entire project as a private company endeavor.” EIA Response: We believe that e-BIZ Centers had an impact on associations – however, the PMP did 10 not articulate a methodology for tracking and assessing this impact. From an impact perspective, EIA helped change mindsets about application of latest technology and focus on product development as the basis for SME competitiveness. For example, the e-BIZ centers have collaborated with associations in promoting introduction and use of advanced technology (TTC) and joint collaborative efforts to help local producers access new markets (Arteco). This information was evident from project’s quarterly and annual reports that were shared with Evaluators. No sufficient evidence is provided for concluding that associations perceived the entire project as a private company endeavor. ************** - Evaluation (p.17): “The EIA provided lists of all organizations and individuals that had received services from the e-BIZ centers; however, these lists contained many individuals, organizations and companies that could not be considered as e-BIZ center clients, as specified by the EIA’s documents.2 We found a large number of individuals, secondary school students, furniture designers and foreign companies in these lists. The Evaluation Team used the following criteria to select SMEs from these lists for interview: balanced territorial distribution (two entities, different parts of BiH), an almost equal number of SMEs from all targeted industries and SMEs of different size (micro, small and medium).” EIA Response: The statement above is misleading as it suggests that all lists from e-BIZ Centers included the non-BIH SME clients. Only the list provided by Arteco included non-SME clients but these clients could have been easily identified and separated. As evident from the footnote the Evaluators are confusing e-BIZ Center clients from EIA project beneficiaries. The e-BIZ Centers had to keep lists of all clients served. The Evaluators requested the lists of all Center clients, not the refined lists that includes just locally registered SMEs. This was explained to the Evaluators during the meeting with them. ************** - Evaluation (p.18): “For question 2, the Evaluation Team evaluated both the financial and operational sustainability of all four e-BIZ centers. As defined by the Cooperative Agreement, financial sustainability is achieved when an e-BIZ center is able to deliver products and services to the market at a price that covers its expenses and generates a profit. In order to determine financial sustainability based on the above-mentioned definition the Evaluation Team analyzed the financial statements of the e-BIZ centers. However, since the above-mentioned definition of sustainability is extremely narrow and is used primarily as a cost-accounting tool that only covers a point in time it would not provide information on the ability of the e-BIZ centers to sustain the provision of the Activity’s benefits in the long run. Therefore, the Evaluation Team proposed a different approach and introduced a new balanced score card method along with new financial indicators to measure financial sustainability. The latter were accepted by USAID as part of the approved Evaluation Work Plan.” EIA Response: Please see the subsequent section on responses to the Evaluation methodology and approach. However, as it relates to the technical characteristics of the e-BIZ Centers in terms of performance, please note first that the definition of sustainability that was used above is not the one shared with EIA or its partners, so we were unable to provide evidence to support this definition. 2 All Activity documents specify that the beneficiaries of the Activity must be SMEs from Bosnia and Herzegovina. Furthermore, as defined in the Annual Report for the period January 2011 – December 2011, the unit of measure for the number of e-BIZ center customers is “the number of customers paying for e-BIZ center services”. 11 Moreover, please note that in August 2013, the Implementing Partners expressed concern in writing to the Evaluators that the Evaluation was using a different standard of sustainability than that which was used by the EIA since its inception (noted in footnote #5 of the Evaluation on p.7) As the letter of 2 concern noted: “Recommendation: We respectfully suggest that the Evaluators and USAID use the standard of sustainability contained in the EI Life of Project Implementation Plan / M&E section and the Annual Work Plans or at least explain more fully why a different definition of sustainability is being used. This is the standard of sustainability that the EI project has been using for the past five years. It is also a reliable indicator in that it is the same as the one that was used for the e-BIZ Macedonia project, which established Centers that have proven to be sustainable. We would be happy to provide the PMP for the e-BIZ Macedonia project to substantiate this. The e-BIZ Centers have been successful not only in achieving this sustainability target, but they have met or exceeded all of their objectives in terms of generating in-kind investment, serving SME clients, and perhaps most importantly, catalyzing innovation in their respective sectors. The only way to evaluate the achievement of sustainability by the project is to use the definition submitted by the project and approved by USAID; this is what the project worked towards during its implementation. Whether a different definition of sustainability would have been more appropriate is a different issue, and one which EIA and USAID should have discussed during the project period of performance. However, this evaluation must only use the working definitions for sustainability. ************** - Evaluation (p. 26): Through the interviews that were conducted, it became obvious that EIA staff as well as e-BIZ center management were not clear which clients they were supposed to serve despite the fact that this is clearly defined in the Annual Performance Report. During the interviews the definition of ‘client’ ranged from those individuals, organizations and companies that visited the e-BIZ center website to those that appeared on the e-BIZ center websites to those that visited the e-BIZ centers during the Open Day event.” EIA response: As the e-BIZ Centers are designed to serve as inclusive sector development platforms for targeted sectors they have invariably attracted individual clients (e.g. furniture designers) and foreign clients, as they needed prototypes for manufacturing orders that they were placing in BiH. EIA regrets that our sincere attempt to convey these important developmental considerations and logical attempts by Centers to broaden customer base was misconstrued by Evaluators as not knowing which clients Centers were supposed to serve. ************** - Evaluation (p.27) FINDINGS – (Related to QUESTION 1: To what extent, in a measurable way, has the EI Activity succeeded in capacity building for e-biz center staff?) EIA response: Throughout this entire section ET fails to mention the involvement and on site capacity development assistance that was delivered to e-BIZ Centers by the number of EIA-procured local and international consultants. EIA provided 37 consultants on more than 50 assignments to support e-BIZ Centers with the focus ranging from planning and management to production and marketing. This oversight results in an incomplete picture of the support that EIA provided to Centers. 12 ************** - Evaluation (p. 27): Throughout the life of the activity, the EIA organized three group trainings: (i) a two-day Power of Sales seminar, (ii) a two-day Financial Management training and (iii) a one-day BiH Entrepreneurship and Business Development training .According to the attendance lists provided by EIA, the trainings delivered and staff trained were as shown below.” EIA Response: EIA disagrees with the Evaluators’ calculation of the number of staff trained because Center owners must be included as staff trained regardless of the fact that they are not formally registered as Center employees. Center Founders were integral to the setup of each Center as all of them served for many months of the inception phase as the regular Center staff. ************** - Evaluation (p. 27): The e-BIZ centers were never properly instructed or trained in how to track and record cost-share arrangements in accordance with the approved e-BIZ centers’ business plans. Because of this the e-BIZ centers financial reports do not give an accurate picture of the e-BIZ centers financial position.” EIA response: The Centers were given precise instructions from EIA related to documenting founder’s contribution to the e-BIZ Centers. Under the FOGs (see TTC FOG, Attachment C, Milestone/Invoice templates), the e-BIZ Centers were obliged to submit cost share documentation/evidence to EIA for review & approval including “receipts, original timesheets…” and estimated values of in kind software and equipment etc. Therefore, the Evaluator’s expectation that all or most of the cost share should have been entered in Centers’ financial records is without merit. ************** - Evaluation (p.29): However, information provided by one of the e-BIZ centers about the training turned out to be different from the attendance list provided by the EIA. When asked why they claimed that their staff had attended trainings organized before the e-BIZ center in question had been established, the Director of this e-BIZ center answered that they had been instructed to do so by the EIA because the EIA did not have a sufficient number of capacity building activities.3 EIA response: EIA assumes that the example listed on page 29 refers to the Transwork Center. EIA, in line with its principled position, considers the participation by Transwork’s owner at EIA training that was held during preparations for Transwork’s formal establishment as the capacity building for Transwork’s staff. This EIA position was communicated to Transwork’s current Director in much the same way as in this part of our response. Again, EIA believes that training to Center owners must be considered as capacity building for the Center regardless of the fact that owners have not formally registered as Center employees. ************** 3The interview conducted with the Director of the e-BIZ center was audio recorded. 13 - Evaluation (p.32): “The majority of indicators signaled significant structural changes in Arteco's business operations over the period 31.12.2012 to 31.07.2013 yet they were not the result of changes in the center’s business operations but rather the result of an exercise in financial engineering. Liquidity improved significantly primarily due to cash transfers from the parent company Artisan to Arteco’s account.” EIA response: We believe that the transaction that is referred to on this page is an overdue payment from the parent firm (Artisan), and is not “financial engineering”. ************** - Evaluation (p.34): Arteco has not achieved the financial projections and capital target values as specified in the business plan. Financial projections set the value of Arteco’s capital in 2009 at 711,720 BAM; however, on June 30 2013 the total value of Arteco’s capital was 2,000 BAM. Arteco has not achieved the target values for current and long-term assets set in the business plan. EIA response: The target values for long term assets included the machinery provided by EIA. However, Arteco could not have registered the value of this machinery in its official books as the title was with EIA until the completion of the EIA project. Nevertheless, EIA accepts that beneficiaries should have recorded the donated machinery in their off-balance-sheet accounts to conform to international accounting standards. ************** - Evaluation (p.35): “As stated during the interview with the Director of Exploring BiH, in the early stages of the project it was already clear that the website/portal was a failure. The explanation given by EIA staff for the reasons behind this was that at the time the project commenced there was no booking website in BiH and nobody was able to foresee what would happen in the area of ICT within the tourism sector (booking.com etc). However, within the Market Analysis Report, the EIA recognized that, “A large number of booking agents are active in BiH but three - www.booking.com, www.venere.com and www.all-hotels.com - are dominant. The EIA also learned that several sites, including www.ljeto-zima.com and www.sobe.ba, offered detailed accommodation listings with information, photographs, prices and e-mail links for reservations through the hotel’s own websites.4 ” EIA Response: At project inception, there were no BiH sites that have offered online booking option and even the local sites listed in the Market Analysis (ljeto-zima.com and sobe.ba) have not offered that option. As the EIA Market Analysis report stated, sites such as booking.com were active in BiH at the time as they were signing up their first hotel clients. In less than two years following the EIA Market Analysis, the booking.com managed to overtake most local or international competition in BiH so EIA observation about this matter is still valid in our view. ************** - Evaluation (p.36): “From the very beginning up until the present, the Exploring BiH portal has not been sufficiently technologically advanced to allow for direct booking via the portal. Bookings were made via e-mail communication with the Director of Exploring BiH and very soon, it became clear that Exploring BiH could not compete with ‘Booking.com’.5 There is a link to the 4Market Analysis of the High Impact ICTs as e-BIZ Center Core Offerings in the Tourism Sector 5http://www.booking.com 14 booking.com website at the Exploring BiH web site, However, that cannot be considered as a provision of possibility for direct booking.” EIA Response: Exploring BH’s portal was sufficiently technologically advanced and tested to allow the full online booking as early as fall 2009. While the actual number of online bookings was low, it is undeniable that the technological capacity existed for online booking for many years under EIA until the site was redesigned to include another booking option. The current link to booking.com is based on the franchise type cooperation that was recommended as the course of action for Exploring BH by an international expert Jain Abishek. ************** - Evaluation (p.41): “The e-BIZ centers' financial reporting systems were inadequate; the e-BIZ centers were never instructed in how to record the USAID and parents company contributions in financial statements. The Evaluation Team was not in a position to verify the amounts and structure of the cost-share contribution of the founders of the e-BIZ centers.” EIA Response: The above statement is inaccurate as Centers were instructed by EIA on how to document and evidence their in-kind contributions to the respective Centers per requirements from their respective FOGs. The Evaluators have received all documentation related to this, including copies of bank statements, timesheets, equipment appraisals etc. and that should have been sufficient for verifying the in-kind contributions, in line with FOG provisions. And as it relates to the question of cost share, the requirements for cost share were included in the fixed obligation grants (FOG) issued to each of the organizations in order for them to support the E-biz centers. Cost share was documented and reported in accordance with 22 CFR 226.23. ************** - Evaluation (p.41): “Arteco provided services to 38 BiH SMEs over a period of 4.5 years. Out of this number, 28 companies received services only once over the 4.5 year period. Out of the 38 SMSs 13 received services of a value of less than 500 BAM (USD 356.65).” EIA Response: It is unclear how the Evaluators determined this number. Arteco’s record from the EIA developed Client Relations Management (system) show that Arteco delivered paid services to more than 60 SMEs at the time the Evaluation was conducted. ************** - Evaluation (p.42): “TTC provided services to 19 SMEs over a period of 1.5 years. Two of these SMEs were companies owned by the owner of TTC, Kismet d.o.o., and Vocal Productions d.o.o. The services provided to these two companies generated 79.34% of TTC’s total revenue in 2012 and 90.73% in 2013. Out of 19 SMEs five (5) received a one-time service of a value of less than 500 BAM (USD 356.65) each.” EIA Response: The above number (19) is incorrect as it only included TTC clients served by spring 2013. TTC was getting new clients on board until the end of EIA (August 2013) following extensive promotion efforts in late spring and early summer 2013. This was the natural pace for client acquisition considering the late start for this Center. And by the end of the project, 40 clients were being served by TTC, which meets the project objectives. 15 ************** - Evaluation (p. 42): “The Evaluation Team was not able to determine the exact number of clients of Exploring BiH. The Evaluation Team received several diverse lists of clients served by Exploring BiH. The first list listed 48 clients, the second listed 71 clients, the third was called the “frequent clients list” also with 48 clients and, finally, the fourth list received from the Exploring BiH accountant had 45 clients in total. The Evaluation Team knows for a fact that even if a client had a contract with Exploring BiH that does not mean that a business transaction happened between them.” EIA Response: It is unclear sure what type of list was solicited and received from the Exploring BH’s accountant. However, EIA collected copies of all bank statements from Exploring BH for the period August 2009 – December 2012 and during that period Exploring BH made payments to or received commission fees from more than 130 BiH SMEs. All of these transactions were done for Exploring BH’s executed bookings of hotel accommodation or other tourism services. , ************** - Evaluation (p. 47): “The financial data concerning changes in the volume of sales of SMEs that received services from the e-BIZ centers does not constitute the basis for the conclusion that the e-BIZ centers improved the competitiveness of their clients. Almost half of the SMEs in all four sectors that received SME services experienced a decrease in the total volume of sales or did not experience any change in the total sales volume.” EIA response: The data on year to year overall sector sales volumes should be compared against sales volumes for SMEs served by e-BIZ Centers to determine whether SMEs served by e-BIZ Centers have performed better than their respective sectors at the time of the global recession. ************** - Evaluation (p.49): “There was a lack of involvement and support from the public sector. The EIA and e-BIZ centers failed to gain interest and support from public sector organizations and institutions. Therefore, the entire effort and burden of advancing all four target industries was placed with small companies struggling to achieve the brake-even point.” EIA response: EIA was a fairly small, USAID funded project and worked with larger USAID projects (CCA, FIRMA) who were the lead in mobilizing the public sector support for advancing targeted sectors such as wood and tourism. SEE NEXT PAGE FOR RESPONSES TO EVALUATION METHODOLOGY AND APPROACH 16 Responses – Evaluation Methodology and Approach Overview of Evaluation Comments There are two broad concerns that the implementing partners have with the Evaluation methodology and approach. The first is that the Evaluators used measurements for e.g. sustainability that were not related to and used through the life of the project as approved by USAID. The definition of sustainability as contained in the Start-Up Implementation and Annual Work Plan is as follows: “…At the end of the project, each e-BIZ Center has (a) an annual operational and financial plan that is based on past performance, which demonstrates that the Center will reach the break-even point within one year of the project; (b) the e-BIZ owner’s written commitment to continuing operations of the e-BIZ Center operations after the end of the project.” In August 2013, we wrote to the Evaluators suggesting that the Evaluators and USAID use the standard of sustainability contained in the approved project documents that the EI project had used throughout the period of performance. It is a reliable indicator in that it is the same indicator was used for the e-BIZ Macedonia project, which established e-BIZ Centers that have proven to be sustainable. Ample documentation exists to show that e-BIZ Centers have been successful not only in achieving this sustainability target, but they have met or exceeded all of their objectives in terms of generating in-kind investment, serving SME clients, and perhaps most importantly, catalyzing innovation in their respective sectors. The evaluation must measure sustainability success against the sustainability indicator and definition used by the project as its goal. Second, we believe that overall, the Evaluators’ approach is inconsistent with the 2011 USAID Evaluation Policy which states that evaluations should include “use of data collection and analytic methods that ensure, to the maximum extent possible, that if a different, well-qualified Evaluator were to undertake the same evaluation, he or she would arrive at the same or similar findings and conclusions.”6 In order to entertain such a scenario, measures or criteria of success would have to be clearly stated and evidence provided that align with such measures so that “readers…have sufficient information about the body of evidence and how information was gathered to make a judgment as to its reliability, validity and generalizability.”7 Point – by – Point Response to the Evaluation - Evaluation (p.2): “…The EIA and e-BIZ centers interfered with the evaluation process at several points by adjusting documents and providing documents that were produced after the evaluation process had begun…” EIA Response: EIA submitted dozens of project reports, industry and market analysis, SOWs, consultant and training reports, business plans, FOGs and literally several hundreds of copies of Center founder’s contribution (cost share) bills, receipts, equipment assessments, in kind labor timesheets etc. EIA understands that the Evaluation Team (ET)could have had uncertainty or even suspicion about some documents as the two or three mentioned in the draft Evaluation Report. However, ET has never approached EIA for explanations or clarifications before they boldly stated that the “interference” has taken place. 6 USAID 2011 Evaluation Policy, p. 9. 7 Ibid. 17 - Evaluation (p.43): “A review of the properties of the submitted documents revealed that, for example, the Exploring BiH survey report was created on July 25 2013, despite the fact that the survey was supposed to have taken place in 2010. However, the Evaluation Team did not investigate further in order to confirm any potential file manipulation.” EIA Response: EIA reviewed the above mentioned “Clients Survey” folder and noticed that several word documents (from Exploring BH and TTC) indeed showed in the properties section, that their last version was “modified” on July 24, 2013. Nevertheless, the original documents have been created by e-BIZ Centers before and Ms. Vasic (Director, Exploring BH) and Mr. Banjic (Director, TTC) e￾mailed their survey files to EIA weeks before they were requested by Evaluators. EIA staff compiled and saved last versions of those word files in the ZIP folder on July 24 before sending all documents to the Evaluation Team. Upon opening files that were received via e-mail and saving them in a designated ZIP folder, some of these word documents were obviously saved as modified versions although nothing substantial was done to them. It is true, however, that in one instance (Exploring BH’s file) EIA staff made minor formatting, date and English language correction in the heading section before resaving that document. However, none of these actions mean that any substantive part of any of the survey reports have been changed or altered. ************** - Evaluation (p. 5): “Capacity building efforts to increase e-BIZ center staff knowledge and skills in areas of high impact information and communication technology and services were basic and insufficient.” EIA Response: As we mention in the section related to Question 1 (capacity building), the Evaluators offer no concrete measure of successful capacity building to which e-Biz centers failed to conform nor evidence to demonstrate failure to conform. We respectfully request that this conclusion be revised for the executive summary in the absence of more robust reasoning and presentation of evidence. ************** - Evaluation (p.16): “The team sent the questionnaires via email but the response rate was extremely low for both groups of SMEs (client and non-client).” EIA Response: EIA kindly requests that the Evaluators provide some context as to why email was adopted as the data collection methodology rather than face-to-face interviews. ************** - Evaluation (p.18): “Therefore, in an effort to reconstruct the baseline data, the survey questionnaire for e-BIZ center employees contained ‘before and after’ questions.” EIA Response: Here, and throughout the discussion of Question 1, it is unclear what the measure of success of capacity building activities is that the project failed to meet. ************** - Evaluation (p.18): “However, since the above-mentioned definition of sustainability is extremely narrow and is used primarily as a cost-accounting tool that only covers a point in time it would not 18 provide information on the ability of the e-BIZ centers to sustain the provision of the Activity’s benefits in the long run.” EIA Response: The PMP defined sustainability as follows “…At the end of the project, each e-BIZ Center has (a) an annual operational and financial plan that is based on past performance, which demonstrates that the Center will reach the break-even point within one year of the project; (b) the e￾BIZ owner’s written commitment to continuing operations of the e-BIZ Center operations after the end of the project.” This was approved by USAID and retained throughout the life of the project as the operative definition. EIA acknowledges that milestones and goals may change over time and would have modified this definition in partnership with USAID if requested. As well, the project may have benefitted from a review of the indicator and its definition by EIA and USAID midway through project implementation. ************** - Evaluation (p.19): “As for questions 3 a-c, the team compared market analyses prepared for each of the four industries and the data from those surveys with the current situation within the targeted SMEs served by the four e-BIZ centers.” EIA Response: Mention is not made of sub-questions 3a to 3c until page 77 and therefore the reference here is confusing. We kindly request that the Evaluators make mention of these sub￾questions upfront. - Evaluation (p.19): To answer this question, and due to the survey failure, the Evaluation Team analyzed changes in the competitiveness of the targeted SME clients. The team did this by tracking changes in certain indicators, such as profitability and sales. In order to determine whether the e-BIZ centers’ SME clients had experienced an increase in the volume of sales or profitability the Evaluation Team used available financial data on the performance of companies and sectors collected by the Agency for Financial Information and Consulting Services (AFIP). The Evaluation Team only managed to collect data for companies and sectors in the Federation of Bosnia and Herzegovina. EIA Response: We believe that the Evaluators could have not just used the financial performance data for serviced SMEs and compared them in terms of their year to year results. The more adequate analysis would include the comparison with data on the performance of the entire sector during a given period as only then we would be able to know whether the Center assisted SMEs have performed better or worse than their sector averages. This is especially important considering the global recession. We are also concerned that data were collected in only one part of Bosnia Herzegovina (Federation BiH). Finally, many private sector experts do not consider AFIP statements fully reliable considering the scale of the so called grey economy in Bosnia Herzegovina. ************** - Evaluation (p.19): Although there is no evidence that the e-BIZ centers had surveyed their clients on a regular (annual) basis the EIA reported results for this indicator each year in the work plan and M&E plan. More importantly, USAID data quality assessment (DQA) of this indicator verified the existence of the survey EIA Response: We respectfully disagree. The M&E plan included in the Annual Work Plans for years 2009, 2010, 2011, 2012 and 2013 clearly states that this survey would be conducted in Year 3 (2010), not annually. 19 In addition, the Evaluation sections of the 2011 and 2012 Annual Work Plans state (on p. 35 for 2011 and p. 16 for 2102): “Utilize a survey of e-BIZ Center customers, conducted at the end of the project by an independent evaluator, to include questions that provide data to assess the impact of the e-BIZ Center services, e.g., questions regarding whether e-BIZ Center services helped to strengthen the SME customer’s business such as by increasing sales, reducing costs, etc.” An annual survey was never expected and results for such a survey were never reported in any work plan. ************** - Evaluation (p.21): One of the examples of interference with the evaluation process is the provision of contradictory data on the same topic. The Evaluation Team received documents from the EIA and the e-BIZ centers yet in the majority of cases the list of SME clients received from the e￾BIZ centers and the EIA were different. EIA Response: We accept that client lists that Centers provided could have been more up to date than those that EIA had on file at the time of evaluation. ************** - Evaluation (p. 20): “The Evaluation Team tried to reconstruct the missing data through a questionnaire survey. The questionnaire asked selected SME clients whether the technical assistance and services provided by the e-BIZ centers had contributed to an improvement in their business operations.” EIA Response: EIA kindly requests that the Evaluators more fully explain the methodology they used to select SME clients for the survey, how many, and whom. ************** - Evaluation (p. 21): Other Issues that affected the Findings The Evaluation Team has strong reservations about the information and data provided by the EIA, the e-BIZ centers and the interviewed SMEs. The EIA and the e-BIZ centers interfered with the evaluation process at several points by adjusting documents and providing documents produced after the evaluation process had begun. This interference made the evaluation process difficult and makes the evaluation findings suspect. More specifically the issues listed below.  Information provided by the e-BIZ centers to the Evaluation Team on attendance at the trainings did not reflect what happened in reality. The e-BIZ centers were instructed by EIA staff to report more capacity building activities.8 EIA Response: This is incorrect. E-BIZ Centers were not instructed to report more capacity building activities than what happened in reality. The only guidance that Centers have received from EIA relates to providing them with information on what is considered as capacity building support (e.g. provision of EIA consultants, trainings, continuous technical support from EIA BDMs etc.). In this respect EIA was obviously interested to make sure that Centers do not mis-report all instances of 8Transcripts of audio-recorded interviews. 20 project’s capacity building support. EIA also instructed e-BIZ centers to report their founders’ participation in EIA trainings as the Center staff training. The reason is simple, as EIA believes that founder’s participation, especially in Centers’ early pre-registration phase represented a significant capacity building assistance despite the fact that founders were not subsequently formally registered as employees of their respective Centers.  The financial reports of the e-BIZ centers were adjusted significantly during the one-month period after the evaluation process had begun. EIA Response: This statement implies that all Centers have adjusted financial statements. Only the parent company for one Center (ARTECO) decided to pay outstanding services payments during the mentioned time period. Nevertheless, all these actions by ARTECO made business sense, were legally permissible and were done in a transparent manner.  As previously mentioned, the lists provided by the EIA and e-BIZ centers of SMEs served by the e-BIZ centers were different; even lists of clients that came from one source were not identical. EIA Response: Comment on this was provided earlier in the text.  During the initial meeting with the EIA, the EIA staff stated that no formal survey on the competitiveness of SME clients had been performed; this was also confirmed by the EIA at the end of the process. The nearest to it were the telephone inquiries from the e-BIZ centers to their SME clients asking them about their level of satisfaction with the services provided. Nevertheless, the EIA submitted list of SMEs that had participated in the questionnaire survey. Several companies stated during field interviews that they had never participated in any kind of survey: neither through the questionnaire nor via the telephone. The majority of issues with the validity and reliability of data on the performed surveys related to the TTC and Exploring BiH e￾BIZ centers. EIA Response: ET’s statement is not quite correct. According to PMP the clients were surveyed to determine whether “e-BIZ Centers services have strengthened their business” and not on their competitiveness which is the broader area. According to our recollection, EIA staff stated that no formal external surveys by EIA or by consultants were completed other than in ARTECO’s case. EIA also stated that only surveys that were completed were done by e-BIZ Centers. Based on the information provided to EIA, Exploring BH surveyed 21 out of its 60 clients at the time of their survey (Fall 2010). When compared to the list of eight (8) Exploring BH clients that were interviewed by ET it is evident that ET met only 3 clients that have been surveyed by Exploring BH in 2010. The other 4 tourism firms that ET visited in 2013 were either not the clients of Exploring BH in 2010 or were not included in the survey sample. Hence, EIA believes that this information sufficiently explains why “several (tourism) companies stated that they had never participated in the questionnaire”. It is because they really haven’t participated as Exploring BH interviewed its other clients. EIA is unsure as to why the ET received a similar response from textile sector firms. Our field experience shows that Directors are not always aware of their firm’s participation in some activities especially in non-essential activities such as surveys that are often completed by engineers or sales personnel. ************** 21 - Evaluation (p. 22): “The financial reports of the e-BIZ centers were adjusted significantly during the one-month period after the evaluation process had begun.” EIA Response: We kindly request that the Evaluators provide evidence to substantiate this claim. ************** - Evaluation (pp. 26-30. Findings Section for Question 1 Overall EIA Response for this section: We request that the full data set from the Evaluators’ survey mentioned be summarized succinctly and coherently (perhaps in a summary table) to allow for an objective assessment of results, rather than presenting findings in a selective and piecemeal manner. The methodology for drawing conclusions is similarly unclear: are the conclusions of the Evaluators based on the results of the survey they administered? If so, the results suggest that the capacity￾building activities had a very positive effect. Are the Evaluators then discounting the results of their own survey? If so, then what is the definition of success and the methodology proposed to ascertain whether it was achieved? ************** - Evaluation (p. 27): The Evaluation Team assessed the EIA reports on capacity building efforts and conducted an analysis of the EIA documents and reports on the training and technical assistance provided. In accordance with our Request for Information sent to the EIA, the Evaluation Team received two sets of documents:  Trainings provided to e-BIZ centers; and  Technical assistance provided to e-BIZ centers. EIA response: It is clear that ET had not included numerous instances of direct, on site, consultant assistance to e-BIZ Centers when they evaluated EIA success related to this question. ************** - Evaluation (p. 27): “Financial management capacity building efforts were focused mainly on increasing the capacity of e-BIZ center staff to track and record cost-sharing arrangements based on the EIA internal cost-share tracking system. The e-BIZ centers were never properly instructed or trained in how to track and record cost-share arrangements in accordance with the approved e-BIZ centers’ business plans.” EIA Response: Centers have received specific instructions/training from EIA on how to document their contributions (cost share). ************** - Evaluation (p. 29): “There was no increase in technical skills at the Exploring BiH e-BIZ Center: computer skills, understanding software, hardware, troubleshooting, teamwork and the organizational structure.” EIA Response: We kindly request that the authors present the entirety of their data for all four centers in one succinct data presentation that would allow the reader to ascertain the validity of their conclusions. A more rigorous presentation of the actual data would allow the reader, at a glance, to know, for example, whether the other three centers showed an increase in skills and by how much. 22 ************** - Evaluation (p. 30): “The Evaluation Team did not have any meaningful method to evaluate the impact of day-to-day and hands on type technical assistance in a measurable way.” EIA Response: We kindly request that the Evaluators provide more justification as to why the impact of day-to-day mentoring and coaching cannot be evaluated. ************** - Evaluation (p.30): In terms of operational management, the Evaluation Team could not confirm any capacity building assistance directly targeted at improving managerial skills to adopt and integrate technology in order to gain maximum efficiency and productivity. The Evaluation Team was also unable to confirm any capacity building assistance directly targeted at improving knowledge and skills in other areas of management i.e., strategic planning and human resource management. EIA Response: It is unclear why the ET has not considered capacity building support targeted at improving above listed skills that was provided by local and international consultants (Martin McElwee, Emir Medanhodzic, Richard Bangs, etc.). EIA staff provided specific and detailed support in recruitment efforts for each Center thus improving Centers human resource management capacity. ************** - Evaluation (p. 32): “Other types of financial indicators analyzed by the Evaluation Team indicate that Arteco is under threat of potential bankruptcy. The value of Altman’s Z-Score test on June 30 2013 was 1.3, which falls within the range of a company at risk of bankruptcy” EIA Response: The article attached by the Evaluators to provide context on Altman’s Z-scores states that a score between 1.1 and 2.6 is clearly within the “grey zone” of values easily misclassified. Also, the Z-scores vary depending on whether it is a public manufacturing firm (Z-score), a private manufacturing firm (Z’-score), or a private, non-manufacturing firm (Z’’-score). In their footnote, the authors apply the third range (for private non-manufacturing firms), but this is inappropriate for Arteco, a private manufacturing firm, which falls under range number 2. Since Arteco is a private manufacturing firm, by the author’s standards, a score of 1.3 would fall into the “grey zone in which distress may or may not be impending”. ************** - Evaluation (p. 33): “The Kralicek Quick Test indicated that Arteco’s financial stability was poor, even after the aforementioned financial engineering exercise. On July 31 2013, the value of the Kralicek Quick Test was 0.32.” EIA response: We kindly request that the Evaluators be more transparent about their use of the Kralicek Quick test and discuss the advantages and disadvantages of using this method with SMEs such as the e-Biz centers. ************** 23 - Evaluation (p. 33): “The Arteco e-BIZ Center has achieved the break-even point, as the measure of financial sustainability; however, other financial indicators indicate poor financial stability and the threat of potential bankruptcy.” EIA Response: Ssuch a conclusion cannot be supported as by the authors’ own standards the firm’s Z-score is in the “grey zone”, because such scores are too inconclusive to be able to predict bankruptcy. EIA therefore requests that the Evaluators present this evidence in good faith as too inconclusive as to be able to predict bankruptcy. ************** - Evaluation (p. 34): “Other types of financial indicators analyzed by the Evaluation Team indicate that Exploring BiH is under extreme threat of bankruptcy. The value of Altman’s Z￾Score test has been negative throughout the entire life of the Project.” EIA Response: EIA kindly requests that the Evaluators state the actual Z-score and provide the calculation conducted to obtain that score. ************** - Evaluation (p. 36): “Exploring BiH achieved financial sustainability by reaching the break-even point and having revenue that exceeded expenses by 1%. However, other financial analyses indicate poor financial stability and extreme bankruptcy risk. “ EIA Response: There is no evidence for this other than a vague reference to the Altman z-score, p. 34. ************** - Evaluation (p.39): The e-BIZ centers did not achieve the financial benchmarks set in their respective business plans. EIA Response: EIA believes that this statement is not supported by the evidence. Three out of four e￾BIZ Centers have shown better actual year 1 and 2 financial results compared to the projections in their respective business plans. One of the potential reasons for this conclusion by evaluators could be related to usage of the wrong start dates for Transwork and TTC as both Centers opened their doors in spring of 2012. Below please find the quick review income statements provided by the e-BIZ Centers. Evaluators are urged to note that Centers have shown satisfactory financial health especially in light of their business plan projections. In short, the income statements for Transwork and TTC show the following (in KM): Name of Firm Total Planned for the first five quarters in terms of profit/loss after taxes Actual for the first five quarters in terms of profit/loss after taxes (2012 and 6 months of 2013) Actual 2013 in terms of profit/loss after taxes (first six months) 24 Transwork -143,915 KM -89,878 KM 9,544 KM TTC -230,752 KM -50,227 KM 1,661 KM The income statements for Exploring BH and ARTECO show the following (in KM): Name of Firm Total Planned for the first two years in terms of profit/loss after taxes Actual for the first two years in terms of profit/loss after taxes (2010 and 2011) Actual 2012 in terms of profit/loss after taxes ARTECO -93,830 KM 12,069 KM 20,821 KM Exploring BH 20,596 KM -12,894 KM 302 KM ************** - Evaluation (p. 42): “The EIA was supposed to perform annual surveys in order to determine changes to the above-mentioned indicator.” - EIA Response: The M&E plans included in the Annual Work Plans for years 2009, 2010, 2011, 2012 and 2013 clearly state that this survey would be conducted in Year 3 (2010), not annually. In addition, the 2011 and 2012 Annual Work Plans clearly state (on p. 35 for 2011 and p. 16 for 2102): “Utilize a survey of e-BIZ Center customers, conducted at the end of the project by an independent evaluator, to include questions that provide data to assess the impact of the e-BIZ Center services, e.g., questions regarding whether e-BIZ Center services helped to strengthen the SME customer’s business such as by increasing sales, reducing costs, etc.” ************** - Evaluation (p. 44): “Due to the survey failure, the Evaluation Team analyzed changes in the volume of sales and the profitability of SME clients.” EIA response: No mention is again made of profitability in the discussion that follows. Only results on changes in volume of sales are presented. We request that the authors present their data and evidence on profitability to allow the reader to understand better how they arrived at their conclusions. Also, the authors should describe why it is sufficient to use only a one-year difference in sales volume results to draw their conclusions. ************** - Evaluation (p. 46): “The Evaluation Team interviewed six (6) of Arteco’s clients, five (5) of TTC’s clients, five (5) of Transwork’s clients and eight (8) Exploring BiH clients. Their perception of the importance of the services provided by the respective e-BIZ centers ranged from “not important at all” to “very important.” EIA Response: This is an extremely vague presentation of data results. In place of it, EIA requests that the Evaluators provide a precise summary table of data results. 25 Comments on the Recommendation Section Please note that responses to some of the recommendations have already been addressed in the sections above. Additional comment is made below for responses to recommendations that were not addressed previously or where EIA believes that reiteration of key points should be made. 1. USAID BiH should pay far more attention to the process of approving the initial work plan, M&E plan, and sub-grants. The initial EIA work plan set the indicators for the life of Activity. The Evaluation Team strongly believes that the two performance indicators proposed by the EIA and approved by USAID AOR were inadequate. Furthermore, three of the four business plans that accompanied the four FOGs contained obvious errors that USAID did not recognize when it evaluated and approved them. EIA Response: The EIA’s initial work plan and M&E approach were developed based on a proven model while the Center Business Plans have been developed in a collaborative effort from EIA and project partners and with help from external consultant experts. Additionally, each plan was reviewed, commented on and validated by industry experts. ************** 2. USAID BiH should focus much more on regular and rigorous monitoring of the implementation of its activities/projects. The fact that the EIA reported results based on non-existent data for several years indicates a serious problem in the way USAID BiH managed and oversaw the Activity. The lack of an adequate USAID monitoring system to track progress towards the defined objectives and goals on a regular basis resulted in some surprising findings in the evaluation. EIA Response: EIA did not report results based on non-existent data. The EIA PMP states that data following data had to be collected: e-BIZ business and operations plans, Center sales records, surveys of e-BIZ Center customers. EIA has followed the frequency of collection as stated in the PMP as we collected sales data from bank statements and surveys from Centers. ************** 8. The recipients of the USAID assistance should have recorded the cost-sharing and other financial +agreements in their financial reports in accordance with the country specific accounting standards. This should have been clearly specified as an obligation of the assistance recipients and regularly monitored by the implementing organization and USAID.The e-BIZ centers were not instructed on how to record the USAID and parent company contributions within their financial statements. The Evaluation Team was not in a position to verify the amounts and structure of the cost-sharing contributions from the founders of the e-BIZ centers. EIA Response: It is evident from the draft report that the Evaluators have not accepted key provisions from the USAID approved project FOGs related to evidencing parent firm investment into the e-BIZ Center. As a result, the Evaluators have stated that they are “not in a position to verify contribution of the founders of the e-BIZ Centers” despite the detailed evidence on this contribution that EIA collected in accordance to the Fixed Obligation Grant (FOG) (22 CFR 226.51) and presented to the Evaluators. ************** 10. The e-BIZ centers should have built in capacity to research and apply ICT in their production 26 processes, not to reduce it to the marketing departments of their parent companies. Presently all ICT research and development and ICT applications in production capacities rest with the parent companies. EIA Response: This blanket statement for all EIA supported e-BIZ centers is incorrect. ARTECO and TTC clearly have the capacity to research and apply ICT in their production processes while the same capacity also exists at Transwork, albeit through the outsource option from the closely connected parent firm. We request that the Evaluators provide more specific evidence indicating lack of capacity such as in ICT. ************** 11. More effort should have been invested in attracting and retaining highly qualified staff with relevant knowledge, skills and experience. Thee-BIZ centers, especially ARTECO and TTC, should have done much more in terms of creating more attractive packages to attract highly qualified employees, due to their remote location. Another option would be significant investment in education and training of the existing staff. EIA Response: EIA could agree in general about the need to attract and retain highly qualified staff. ************** 12. More should have been done in raising awareness amongst SMEs about the benefits of introducing ICT into the production processes. Although the EIA and e-BIZ centers made a lot of effort to market their products they failed to raise awareness amongst SMEs concerning the importance of applying ICT in production processes. This is partially a consequence of the failure to establish meaningful cooperation with the public sector and business associations. Management of the e-BIZ centers believed that SMEs in Bosnia and Herzegovina were still not aware of the benefits they could gain through the introduction of ICT applications to the production process. Therefore, they did not manage to expand their client base and disseminate ICT applications throughout the target industries. EIA Response: The Evaluators have failed to note that there were number of activities conducted by the e-BIZ Centers, and with support from EIA, that were focused on raising awareness about applying ICT in their production processes. For example, TTC organized two large Open Day services presentations to SMEs together with the BiH Textiles Association while ARTECO and TTC also held number of presentations in cooperation with Chambers and associations that were focused on raising awareness about new technologies and use of ICT applications in production and service processes. ************** 14. More should have been done to establish long term cooperation between the e-BIZ centers and SMEs. The majority of SMEs were one-time clients and the e-BIZ centers did little to adequately manage their relations with them and turn this into long-term cooperation. EIA Response: ARTECO and Transwork installed the EIA-developed Client Relationship Management (CRM) systems by spring 2013 while TTC is planning to introduce this system soon. The CRM allows for integrated client management approach and we regret that Evaluators have not reviewed and assessed this important client management tool. ************** 27 15. Arteco and its owners should be required to meet the financial target values set by the business plan. Financial projections set the value of Arteco’s capital in 2009 at 711,720 BAM; however, on June 30 2013 the total value of Arteco’s capital was 2,000 BAM. Values for current and long-term assets set in the business plan were also not achieved, as well as ratio between long-term and current assets set in the business plan. EIA Response: The Evaluators have failed to realize that the above mentioned capital target value could not have been realized by the time of the evaluation partially due to the fact that the title for the CNC machine resided at EIA until the end of EIA project.