January 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by International Business & Technical Consultants, Inc. (IBTCI). Final Evaluation: Bolivian Productivity and Competitiveness (BPC) Project FINAL EVALUATION: BOLIVIAN - PRODUCTIVITY AND COMPETITIVENESS PROJECT (BPC) Authors: Milton Núñez-Garcés Luis Alejandro Bernal Sergio G. Villarroel Böhrt January 2013 Project Number AID-511-O-12-00042 REQ-511-12-000057 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i ABBREVIATIONS & ACRONYMS APP/ PPA Public‐Private Alliances ABN Association of the Andes (humus production on the shores of Titicaca) ATPDEA Andean Trade Promotion and Drug Eradication Act BCA Bolivia Communications Activity BCCN Bolivia Trade and Business Competitiveness ‐ (USAID ‐ First phase: USAID‐2003‐2005 and Second Phase: 2005‐2009) BPC / PC‐B Bolivia Productivity and Competitiveness Project , Proyecto de Productividad y Competitividad‐Bolivia CADEFOR Amazon Center for Forest Development / Centro Amazónico de Desarrollo Forestal CADEX Chamber of Exporters of Santa Cruz / Cámara de Exportadores de Santa Cruz   CADEXCO Chamber Exporters of Cochabamba / Cámara de Exportadores de Cochabamba CAF Andean Development Corporation / Corporación Andina de Fomento CAMEX /Chamber of Exporters of La Paz/ Cámara de Exportadores de La Paz CDCS Country Development Cooperation Strategy CEDES Council for Sustainable Development / Consejo Empresarial para el Desarrollo Sostenible CEDETEX Textile Development Center / Centro Desarrollo Productos Textiles CEPB Confederation of Bolivian Entrepreneurs / Confederación de Empresarios Privados de Bolivia   CNI   National Chamber of Industries /Cámara Nacional de Industrias COTEXBO Cooperative Textile Producers of Bolivia COTR Contracting Officer’s Technical Representative CP Commercial Promotion CPTS Center for Promotion of Sustainable Technologies / Centro Promoción de Tecnologías Sostenibles DANIDA Danish International Development Agency   DESIERTOS BLANCOS Salt Transformation Cooperative EPC/UCB School for Productivity and Competitiveness – Bolivian Catholic University / Escuela de la Producción y la Competitividad/Universidad Católica Boliviana FAN Friends of Nature Foundation / Fundación Amigos de la Naturaleza FAUTAPO Foundation for the Support of Universities at Tarija and Potosi / Fundación de Apoyo a Universidades de Tarija y Potosí FST Service Delivery Fund (Fondo de Servicios Técnicos Proyecto PC‐Bolivia)   ii FUNDAPRO Production Foundation / Fundación para la Producción FUNDEMPRESA Foundation for Entrepreneurial Development / Fundación para el Desarrollo Empresarial GDA Global Development Alliance (PPA) GNP / PIB Gross National Product / Producto Interno Bruto IBCE Bolivian Institute of Foreign Trade / Instituto Boliviano de Comercio Exterior IBNORCA Bolivian Institute for Quality and Standarization / Instituto Boliviano de Normalización y Calidad   IBTCI International Business & Technical Consultants, Inc. ICAP Institute for Training / Instituto de Capacitación   IDB/BID‐FOMIN Inter‐American Development Bank (and Multilateral Investment Funds)   IDEPRO Institute for Development of MSMEs / Instituto para el Desarrollo de la Pequeña Unidad Productiva IESM Institutions Strenghtened Directly IFSP Integrated Food Security Project (Proyecto de USAID) IMB Bolivian Institute for Furniture Producers INE National Institute of Statistics / Instituto Nacional de Estadística ISD Institutions Strengthened Directly   ISI Institutions Strengthened with Indirect Methods JICA Japan International Cooperation Agency M&E Monitoring and Evaluation / Monitoreo y Evaluación MAPA USAID Project of Market Access and Poverty Alleviation / Acceso al Mercado y Alivio a la Pobreza (Proyecto de USAID) MOU Memorandum of Understanding (Convenio de Entendimiento) MSMEs / MIPYMEs Micro‐Small‐Medium‐size Enterprises / Micro‐pequeña‐mediana Empresas NGO Non‐Government Organizations PAI Amazon Integrated Program / Programa Amazónico Integrado (Proyecto de USAID ver también IBA) PELETBOL Pellet Industry of Bolivia PML‐SISO Clean Production – Industrial Accident Prevention / Producción Más Limpia‐Seguridad Industrial y Salud Ocupacional PMP Performance Monitoring Plan PND National Development Plan / Plan Nacional de Desarrollo PPA Public‐Private Alliance PROINPA Foundation for the promotion of agricultural products Seeds of the iii Andes   PROSOL Fruit Dehydration MSME QAS Quality Assurance System RFA Request for Application (donations and PPA) RFP Request for Proposal SBK Standard Bank SDF Service Delivery Fund SEGE Sustainable Economic Growth and Environment Office/USAID (Oficina de Desarrollo Económico Sostenible y Medio Ambiente de USAID/Bolivia) SENASAG National Services for Animal Health and Food Security / Servicio Nacional de Sanidad Agropecuaria e Inocuidad Alimentaria SUMITA LTDA Camelids Proliferation and Production Ltd.    TA Technical Assistance TALLER TELLEZ Workshop for the production of color stamping equipment UNEC NGO for the promotion of Nontraditional Products Union (Tarija) USAID US Agency for International Development  / Agencia de los Estados Unidos para el Desarrollo Internacional WI Workforce Inclusion iv EXECUTIVE SUMMARY This report presents the final performance evaluation of the Bolivian Productivity and Competitiveness (BPC) Project of USAID/Bolivia conducted by International Business and Technical Consultants, Inc. (IBTCI). The purpose of the Final Evaluation (FE) was to: (a) evaluate the performance of project results and deliverables from September 30, 2009 through June 30, 2012; (b) highlight lessons learned and best practices developed for MSMEs and public-Private-Alliances (PPAs) that could be replicated; (c) provide recommendations on strategies to promote PPAs in new programs under the Country Development Cooperative Strategy (CDCS) for the period 2012 through 2017 and in the design of new projects; and (d) provide important information of activities that should be taken into account in future projects, methodologies and strategies to be replicated for technical assistance, training and developing PPAs. The BPC project was contracted with Chemonics International, Inc. for a total estimated budget (TEC) of US $ 9,771,913; for a period of four years. However, due new geopolitical conditions the project has been reduced in time (only 3-1/2 years) and budget reductions of approximately 18% (for a total of US$ 8,013.000). Supposedly most USAID programs dealing with MSME promotion will have an early close-out; in the case of the BPC it will end it in January or February 2013. The evaluation, which took place from October to December 2012, included a review of each of the project objectives: (1) Improve the competitiveness and productivity of selected value chains (2) Strengthen local institutions that support the development of MSMEs; (3) Promote dialogue between key stakeholders to improve the business environment and competitiveness of value chains and MSMEs; and, (4) Leverage funding from the private sector, other donors and public institutions using public-private alliance (PPA)1 funds and other sources of funds. The framework criteria applied in the evaluation, in accordance with USAID Monitoring and Evaluation (M&E) -evaluation directives, looked into the following project parameters: (1) Relevance, (2) Effectiveness, and (3) Sustainability. Activities for improved productivity and competitiveness of MSMEs and value chains The BPC regularly evaluated its work and performance in terms of improvement of productivity at the MSMEs’ level. The indicators were the ratio between sales and number of workers before and after its intervention in the enterprise2 . Results reveal that measuring productivity in the manner helped to detect real increase in productivity at an estimated 29% overall. Measurement of sales at the level of MSMEs showed an average increase of 27% over the evaluation coverage period. In parallel, 26% MSMEs claimed increased productivity due to the project intervention. Profitability registered an average increase of 32%. However, only 26% of MSMEs were able to provide formal accounting data. 1 Conceptually, PPAs are the same as Global Development Alliances, or GDAs. These collaborations leverage the resources of the local private sector and other stakeholders to partner with the US Government on development initiatives. (See www.idea.usaid.gov) 2 This measurement was proposed by Chemonics and accepted by USAID as valid measure of productivity thus, for evaluation effects, it has to be considered and accepted. It is however a very debatable, to say the least, view of productivity and measurement thereof, as it may be studied in many authoritative publications.   v Data on job creation showed 15% of MSMEs stated having created new jobs because of BPC interventions. The responding MSMEs reported to have an increased labor demand in an average of five direct new stable workers hired. In addition, a couple of respondents argued they created a higher number of indirect jobs, up to 100 in the case of a bio-products firm. Most of the interviewed MSMEs had positive views and experiences with help received from the BPC Project. This experience translated in terms of level of satisfaction or high satisfaction was reported by 55% of MSME respondents while 45% had no formed-opinion about the project. Most MSMEs expressed the opinion that this sector of the Bolivian economy needs strong support in technical assistance (TA) and training and that education and workforce development is a structural problem of the country. Additionally, MSME expressed any effort made in providing training opportunities to their productive sector will impact it positively. The experience of BPC with value chains is, perhaps, the most interesting aspect of the evaluation, both from the perspective of project design and project implementation. Except for the associations working in collective projects, such as PPAs, this evaluation did not find evidence that the MSMEs were geared to think toward a larger productive sector, group, cluster, or value-chain, or similar economic-productive circle. A. Activities to Strengthen local institutions that supported the development of MSMEs The Evaluation team detected that BPC implemented institutional strengthening activities by classifying and concentrating efforts in two different groups: institutions strengthened directly (ISD) by the BPC project and Institutions that were strengthen indirectly. The first group, comprised of four well-organized ISDs, strongly agreed that involvement in the BPC project was very positive and it helped them increase their capabilities to provide technical services to MSMEs. Other forms of assistance, such as the equipment provided, technical training, and the improvement in their procedures were also appreciated. The project allowed them to increase their universe of clients by reaching a broader number of MSMEs. In the case of Institutions Strengthened Indirectly (ISIs), 75% of respondents stated the most common outcome of the BPC projects was the spill-over effect from the opportunity given by working or collaborating with the BPC project because they were able to offer their technical or training services to a larger client-base audience. The work with a larger demand market allowed them to invest in improved technical knowledge and trainers, plus the increased demand-driven improvement in services. The project also allowed them to accumulate more experience, which in turn helped them to improve their technical capabilities and adjusted their own techniques for reaching a broader universe of MSMEs. The ISDs had good capacities to compete for resources before working with the BPC, with the exception of COTEXBO. But the contact with the BPC helped reinforce and maintain these capacities, especially with respect to equipment acquired. In the case of ISIs, a large number of institutions expressed that their capacities to compete for resources increased after their contact with BPC Project. On the other hand, having worked with BPC Project taught all ISIs the need to analyze and plan every intervention in terms of sales and jobs, a mandatory requirement for USAID assistance. When asked about the usefulness or relevance of workshops to facilitate contact and collaboration with the BPC project, institutions responded that the courses were useful and vi very useful. But 90% mentioned that personal contacts were more effective (telephone calls, e￾mails, meetings and visits) between the ISD and administrative personnel from the BPC. For future collaboration alliances, some of the consulting institutions or non-governmental organizations (NGOs) interviewed expressed concern with a potential challenge, which will cause a reduction in market opportunities. For most of them, they felt that the Government of Bolivia (GOB) seems to “Black-ball” institutions that had participated in US government programs or projects by reducing or barring them from access to opportunities with GOB programs. B. Activities to Promote dialogue between key stakeholders to improve the business environment and competitiveness of value chains and MSMEs Due to a number of factors not favorable in promoting dialogue among key stakeholders for MSMEs development or business-environment improvement, the BPC invested or concentrated very limited resources and personnel to this endeavor. Because it meant supporting activities and actions to promote legal and procedural business-friendly reforms, the BPC encountered a lack of openness and/or willingness to collaborate by GOB institutions, and that also discouraged larger business and sectorial institutions, to embark in what they sensed as arid or pointless activities. C. Activities to leverage funding from the private sector, other donors and public institutions using public-private alliances (PPAs) and other sources of funds The figures reported up to June 2012 show that a total of 27 public-private alliance (PPA) operations were accomplished; with an investment of US$ 633,311 from USAID. This created a leverage of counter-private investment of US$ 1.09 million dollars. It should be stated that the evaluation was able to account that the capital invested by the private parties was larger (in-situ revisions) than what they had reported because it was not necessary as long they reached the amounts agreed upon. The evaluation of PPA performance and results showed that although the number of cases was lower than planned, the BPC Project achieved or exceeded its expected 1:1 goal, and translated it into a more positive result by a factor of 1:1.76 (USAID‘s US$ 633,312 and the PPA partners US$ 1,096.590). Therefore in this activity’s objective, the overall result was achieved. It should be noted that a special model of methodology for promoting the incubation and growth of MSMEs through a form of Public-Private-Alliances was designed and agreed between the stakeholders. This model was chosen as a solution for Bolivia since the public GOB sector did not participate therefore USAID’s funds were the only public sector investment available. The private sector portion of the investment came from the interested MSMEs (which normally is partly provided by angel-funds or investment funds). This solution seems to have become the optimal and most effective tool to accomplish the objectives of helping start-ups with new and innovative ideas, especially those that use local bio-raw materials in their finished products and/or installed eco-friendly processes. Survey results showed that all beneficiary respondents were satisfied with PPAs supported by the BPC project. Additionally all the MSMEs with PPAs responded that they are willing to sign another PPA agreement (83.3% would sign even if the counterpart contribution was greater than the 1:1 match). vii The evaluation team gave additional effort to find out if the beneficiaries recognized who was the funding donor, or angel. Results showed all 18 MSMEs recognized their benefactor was USAID and were very grateful for the support received. These results present an encouraging picture for the ongoing work of USAID in Bolivia. D. Best Practices in Promoting MSMEs’ Development, Institutional strengthening, and Public￾Private-Alliances The Evaluation Team identified some best practices from the BPC project, because of its potential scalability and transferability to other contexts or programs. The “Little schools” or in-work training had the aim of directly inserting workers to fulfill specific work opportunities as well as specific needs in collaborating MSMEs and mid-size industries. This on-the-job training model aimed to create qualified employees for jobs that are available due to MSMEs expansion. The effect of increased productivity has forced additional demand for trained workers. It is a theoretical-practical training model on the workplace for a period of three months. The hirer and trainee work together to improve workforce production and real job opportunities according to market demands; leading to a sense of commitment among all parties involved, Integral Technical Assistance: Although usually MSMEs focus their needs only on production and technical bottlenecks, they learn that 80% of their needs will occur in developing their capabilities in market management (marketing) as well as their abilities to mobilize new excess production, a product of gained productivity. The reality is usually more complex and therefore demands assistance in areas of management, organization and, particularly, marketing and sales management. These additional needs have been encountered by all participating MSMEs, and the Evaluation team determined that this could be achieved by contracting/working with service providers able to integrate a technical assistance and training that includes all enterprise functional areas in a multi-disciplinary approach. The PPA-Bolivia model could work for any number of projects and could even be replicated in other countries with similar characteristics as Bolivia. E. Lessons Learned MSMEs should be aware that solving technical and production bottlenecks are only a small proportion of a business endeavor. The introduction and practice of marketing concepts (market management) becomes crucial to any MSME before embarking in any new business endeavor or expansion. Otherwise it becomes a great financial risk and even personal investment. The comprehensive approach to strengthen PPAs was implemented only in a few cases, placing greater emphasis on resolving production and technical bottlenecks to increase productivity. But all MSMEs later learned that the biggest bottleneck was managing new markets and sales for the added production. In most cases, it also translated into an improvement in competitiveness because larger production also meant a decrease in per unit costs. In general, MSMEs’ needs and weaknesses were discovered to be in administration processes and procedures, opening new markets and expanding existing ones, developing niche markets, creating a new mix of products, and introducing practices to improve gender and environmental consideration as part of an improved social conscience. viii Public communication and media announcements (national press) were very effective in reaching a larger audience and increased transparency in bidding processes. This solution proved to be an effective and practical method to procure funds by specialized MSME-start-ups as well as to ensure the allocation of funds in areas of innovation, new technologies or industrial eco-friendly technologies. The PPA model of solution may be useful in the design and development of future projects dealing with promotion and development of MSMEs businesses, although special attention must be placed to the different degrees of development needs and strengths of specific MSMEs sectors, in assigning adequate types of solutions or help needed by each beneficiary before signing any PPA agreements. Depending on the demand for resources and the importance of the sector, it is clear that the PPA model is a viable and valuable tool to promote start-up of businesses in areas of specialized technologies, eco-friendly industries, sustainable use of local natural raw materials, as well as new business ideas and/or dealing with innovation. Recommendations for future actions Any existing and/or new USAID project in Bolivia should be mindful in installing lean processes to reduce delays and requirements for implementing PPA, TA support, and support for developing MSMEs. The evaluation showed that prolonged processing time between application and final investment caused the receiving partner to lose interest and effect of the business initiative was reduced. Although the BPC project was able to react and adjust the conditions of PPA operations quickly, the adjustments were not fully reflected in the manuals and tools developed for this purpose. In terms of sustainability, the Evaluation surveys showed two different sides to the issue: (a) most of the individual MSMEs showed measurable market-opportunity strengths, which helped them to a greater degree to qualify as potentially “sustainable businesses”; however (b) the MSME-program, within the BPC itself, could not be sustainable due to the lack of support or partnership (coordination) from any GOB institution because it requires contextual institutional support, facilitation and, perhaps, some supplementary public investment, from the business and GOB environment. Since the BPC is coming to an early closeout, many activities have not achieved completion level, yet. It may be a good idea that few successful activities to be assimilated by other USAID projects. Perhaps BPC-institutional partners could track them and support their completion to improve their corporate image. ix Table of Contents ABBREVIATIONS & ACRONYMS .................................................................................................................................... i EXECUTIVE SUMMARY .................................................................................................................................................... iv INDEX OF TABLES, PHOTOS AND GRAPHICS ..................................................................................................... xi 1. INTRODUCTION .......................................................................................................................................................... 1 1.1Purposes for Evaluating the BPC Project: (as per the Statement of Work (SOW): ............................... 1 2. BPC - Project Background ............................................................................................................................................ 3 2.1 BPC Project Objectives and Development Hypothesis ................................................................................ 5 3.METHODOLOGY ........................................................................................................................................................... 6 4.EVALUATION ANALYSIS: .......................................................................................................................................... 11 4.1 Analysis of Activity: Improved Productivity and Competitiveness of MSMEs and Value Chains ........ 11 4.2 Analysis of Activity: Institutional Strengthening to Develop MSMEs ......................................................... 15 4.3 Analysis of Activity: BPC Project – Leverage of USAID Resources and PPAs ....................................... 18 5. FINDINGS ...................................................................................................................................................................... 25 5.1 FINDINGS - Activity: Improved Productivity and Competitiveness of MSMEs and Value Chains…. 27 5.2 FINDINGS - Activity: Institutional Strengthening to Develop MSMEs ....................................................... 30 5.3 FINDINGS: Leveraging Funds through Public-Private Alliances (PPAs) ..................................................... 32 5.4 BPC and workforce development ...................................................................................................................... 34 5.5 BPC work in the improvement of the Business Environment ...................................................................... 34 6. CONCLUSIONS ........................................................................................................................................................... 34 7. RECOMMENDATIONS .............................................................................................................................................. 36 8. LESSONS LEARNED .................................................................................................................................................... 37 9. BEST PRACTICES ......................................................................................................................................................... 39 10. FUTURE DIRECTIONS ............................................................................................................................................. 40 Annex 1: Evaluation Statement of Work ........................................................................................................ 41 Annex 2: General Conceptualization of Evaluation Approach vs. Project Objectives ......................... 52 Annex 3: BPC Organizational Structure ......................................................................................................... 55 Annex 4: MSMEs Interviewed List ................................................................................................................... 56 Annex 5: List of People Individually Interviewed .......................................................................................... 58 Annex 6: Summary of Evaluation Findings in Directly Strengthened Institutions (ISDMs) .................. 60 Annex 7: Summary of Evaluation Findings in Indirectly Strengthened Institutions (IDIs) .................... 61 Annex 8: Institutional Strengthening Methodology Plan ............................................................................. 61 Annex 8: Institutional Strengthening Methodology Plan ............................................................................. 62 Annex 9: Institutional Strengthening Coverage............................................................................................. 63 Annex 10: BPC: MSMEs Experience with BPC Objectives ......................................................................... 64 Annex 11: Guides for Conducting Surveys, Focus-Groups, Key Personal Interviews ......................... 67 x Annex 12: PROTOCOLS (INSTRUMENTS) ................................................................................................ 70 Annex 13: Bibliography and documentary References ................................................................................ 80 Annex 14: Description of Technical abilities Personnel of IBTCI – Evaluation- Team........................ 83 Annex 15: Conflict of Interest Forms ............................................................................................................. 85 xi INDEX OF TABLES, PHOTOS AND GRAPHICS TABLE No. DESCRIPTION Page No. 1 BPC Project General Details 4 2 Field Work Plan and Sampling Instrument 8 3 Sample Calculation for La Paz 10 4 Sample Calculation for Cochabamba 10 5 Sample Calculation for Santa Cruz de la Sierra 10 6 Sample Calculation for Tarija 10 7 BPC – Value-Chain Total Value Increment 13 8 BPC – Institutional Perception of BPC 16 9 BPC – Public Private Alliance 18 10 BPC – Leverage of Resources (PPAs) 19 11 BPC – Comparative BPC -List of Partner Contribution 21 12 BPC – Survey and Personal Interviews Information and Analysis 23 13 BPC –Accomplishments as of June 2012 26 14 BPC –Accumulated Results June 2012 27 15 BPC –MSME Assistance by Size 27 16 BPC –MSME Assistance by Sector 28 17 SME – Technical Assistance to MSMEs by Region, June 2012 28 18 BPC – Trade Promotion Activities 29 19 BPC – Productivity in MSMEs Assisted Increased (June 2012) 29 20 BPC – Increased Sales in MSMEs assisted (June, 2012) 29 21 BPC – Employment Generation by MSMEs Assisted (June,2012) 30 22 BPC – Institutional Strengthening Through Direct Mechanisms 31 23 BPC – Workforce Training for MSMEs 34 24 BPC – MSME Supported Trade Promotion Activities 34 25 BPC – Overall Evaluation Grading by Evaluation Technical Team 36 Figure No. 1 BPC: Main and Specific Objective 5 2 Map of Bolivia BPC areas of intervention-type of value-chain benefited 11 3 PPAs Portfolio according to Productive Chain 38 4 PPA Grant Implementation Process 39 PHOTO No. 1 A Apparel Detailing PPA-AAA Tajara - District of Tarija (Nov/2012) 19 2 PPA -Bolivian Association of Producers of Natural Products (ABN), La Paz, Nov. 2012 19 3 UNC Oregano Multiplication Greenhouse Field Dryers at District of Tarija 22 4 PPA Commercial Promotion and TA to bring MSME Furniture-Fair , Tarija, Bolivia 24 5 PPA Partner PIAPEC Café, El Alto, Bolivia 26 6 Community benefited with PPA – Tarija District (Nov/2012) 31 7 BPC – PPA Apparel MSME (TA from YESMAR) Santa Cruz, Nov/2012 32 8 MSME Pharmaceutical Laboratory – Montero –Santa Cruz – District - Nov. / 2012) 40 1 1. INTRODUCTION USAID contracted International Business & Technical Consultants, Inc. (IBTCI) to conduct a performance evaluation of the Bolivian Productivity and Competitiveness (BPC) Project by USAID. The implementation of the evaluation contract took place during Oct 2012 to Jan – 2013, in Bolivia and with desk work in the United States of America. 1.1 Purposes for Evaluating the BPC Project: (as per the Statement of Work (SOW): As detailed in IBTCI’s evaluation Statement of Work (SOW), the purposes and uses of the evaluation were to, 1. Evaluate the performance of project results and deliverables from September 30, 2009 through June 30, 2012 (The documentation and information received only covers this period); 2. Highlight lessons learned and best practices developed for micro, small and medium- sized enterprises (MSMEs) and Public Private Alliances (PPAs) that could be replicated; 3. Provide recommendations on strategies to promote PPAs in new programs under the Country Development Cooperative Strategy (CDCS) for the period 2012 through 2017 and in the design of new projects; and provide important information of activities that should be taken into account in future projects, methodologies and strategies to be replicated for technical assistance, training and developing PPAs. 4. Learn to what extent the project’s objectives and goals have been achieved 5. Know what extent has the development hypothesis been achieved, and if it fed the SDE results framework 6. Explain lessons learned and best practices that can be shared throughout the Agency to improve future programs in Bolivia or other Latin American countries. The main audience of the evaluation report will be USAID/Bolivia, especially the SEGE team, LAC Bureau and the implementing partner. This evaluation will provide an additional report sanitized in order to share it with the Bolivian Government officials, private sector representatives and other international donors as needed Bolivian Productivity and Competitiveness Project (BPC) Country Context To understand the general country environment where the BPC developed its activities, it is necessary to have an understanding of the country’s social, political, and (macro-) economic conditions. Economic Context of BPC3 : Bolivia is a perfect example of “Unbalanced Growth Theory”4 . Balanced economic development theory precludes that development must take place simultaneously in many (economic) activities to provide the element of mutual support and this is not present in the Bolivian Economy. Instead, growth is limited to only few successful economic activities, such as production in minerals, gas, and oil. Additionally, growth in these sectors seem divorced from the competitive development of other sectors. According to the Millennium Foundation in its Report No. 325 , for 2011, the Bolivian economy was still enjoying a worldwide boom condition with continued high prices for domestic exports and therefore, this was reflected in growing imports, significant reserve accumulation and relative price stability. Although the world economy slowed, Bolivia’s economy, according to the Millennium Foundation6 , is going through a period of prosperity. 3 Business environment analysis was developed in collaboration of USAID-M&E contractor team and the BPC management staff. 4 “The Strategy of Economic Development”, Albert O. Hirschman, Westview Encore Edition, Bolder and London, 1952 (Chap. 4, pg. 62) 5 Millennium Foundation "Millennium Report on the Economy, Management 2011", June 2012, No.32, La Paz, Bolivia. 6 Millennium Foundation "Millennium Report on the Economy, First Semester, October 2012, No. 33, La Paz, Bolivia. 2 Exports are high due to rising raw material prices. This in turn pushes higher overall income and expands aggregate demand. The socio-economic platform of the national economic environment between 2011 and 2012:7 is shown, in a nutshell, in the following of macroeconomic outcomes: • In March 2012, GDP grew by 5.16% compared to 5.63% in 2011. • At current prices, GDP is U.S. $ 24.4 billion and $ 2,282.60 - per capita. • The production and service sectors that grew most through March 2012 include financial activities (8.3%), construction and public works (11.5%), and oil and natural gas (7.7%). Manufacturing grew 5.0% and 2011 growth in this sector was mainly due to the increase in the manufacture of cement, it is the main input for the construction industry. • FDI increased 38.6%, from USD 201.1 million to USD 278.6 million. • Gross international reserves at the Central Bank of Bolivia increased by 15.7%, from USD 10.751.2 billion in June 2011 to USD 12.438.4 billion in June 2012. • Between June 2011 and June 2012, tax revenue increased by 29.32%. At current prices these revenues account for about USD 3.400 billion. Of all of these taxes, the Direct Tax on Hydrocarbons accounts for 23.44% of the total collected. • In the banking sector, public deposits increased by 21.6% from USD 7.8215 billion in January 2012 to 9.5071 billion in June 2012). Of the June 2012 amount, 33.5% were time deposits and 30% were savings accounts deposits. • The loan portfolio from January to June 2012 increased by 19.45% from USD 5.9893 billion to $ 7.1541 billion. Of the total bank loans as of June 2012, 27.4% was directed to wholesale and retail, 24.5% in real estate, 17.7% in manufacturing, and 11.9% in construction. • Lending interest rates to April 2012 showed a downward trend, reaching about 11% on average and deposit rates around 1.25%. • Accumulated inflation to June 2012 is 1.96% compared to a rate of 4.27% in June of 2011, confirming the relative stability of the national economy. The remarkable growth of domestic exports between 2011 and 2012 is mainly explained by increases in prices of raw materials exported by the country. Exports grew 26.56% from January to June 2012. Total exports are led by are exports of hydrocarbons (51.4%), minerals (31.1%), and non-traditional exports of all products, including wood and textiles (17.4%). The Central Bank of Bolivia reported that the most important destinations for Bolivian raw material and mineral exports in 2011 were: Brazil (33%), Argentina (11.6%), U.S. (9.6%), Japan (5.9%), and South Korea (5%). Social Context8 of BPC: Social and political conflicts have remained in Bolivia, impacting citizens’ activities in general. In 2011 there were 884 conflicts9 , setting a historical record compared to the early 1980's. Of all recorded conflicts, 23% occurred between October and December. In the first half of 2012, 412 recorded events were recorded. 7 According to information provided by the Millennium Foundation. 8 Economic data compiled by the Millennium Foundation for Conflict Observatory data of CERES. 9 Published in the national press. 3 Generally, conflicts were related to job demands, sector claims related to: (a) indigenous marches east in defense of TIPNIS, (b) demands of the Ayllus of Mallku Khota to force the government to the reversal of mining concessions in the hands of foreign companies and (c) Bolivian police revolt over pay and institutional needs. 2. BPC - PROJECT BACKGROUND The Bolivian Productivity and Competitiveness Project (BPC) is an initiative of The United States Agency for International Development (USAID/Bolivia) steered towards increasing productivity and sales of micro, small and medium enterprises (MSMEs). The BPC started in September 30, 2009 and estimated to end in February 2013. The new closeout date has been moved to February 2013, earlier than originally planned due to USAID’s decision to reduce the budget from some programs, which includes the BPC Project. The project was implemented to help the development of the business (MSMEs) sectors dealing with textile and apparel, manufactures of wood products, processed foods, bio-products and handcrafts sectors. The geographic areas of the intervention were: the departments of La Paz, Cochabamba, Santa Cruz, Tarija and Chuquisaca. (See Table 1 for BPC Project Details) 4 Table 1: BPC Project General Details Purpose: Increase productivity and sales of MSMEs in order to generate/sustain employment and increase incomes in urban and peri-urban areas. Implementer: Chemonics International Inc. (Washington, DC) Duration (planned) September 30, 2009 –August 30, 2013 Actual duration (early closeout) From information gathered during the evaluation the project is estimated to end in Feb-2013 to assure close out processes and procedures are met TEC (Total Estimated cost) $US 9.771.913 Actual estimated cost Estimated final funds used will be 18% less than initially budgeted. Therefore ending budget should reach approximately US$ 8,012,968 Regional coverage (Departments) La Paz, Cochabamba, Santa Cruz de la Sierra, Tarija, and Chuquisaca. (NOTE: Only 4 out of 5 cities was assessed) Sectorial coverage Textile and apparel, Wood manufactures, Processed foods, Bio-products, and Handicrafts MSME productivity and management assistance: Good manufacturing systems for processed food; Support MSMEs to attend trade fairs and, business roundtables; and, Training to strengthen the labor force and improve management skills. Value chain Improvement Events (e.g. workshops, fairs, business round tables and forums) sponsored to promote strategic commercial alliances for BPC Project Institutional Strengthening: Technical Assistance, Training Technical Assistance, Training Supports SEGE Strategic Objective “Improved livelihoods through promotion of sustainable agriculture and natural resources-based business and productive opportunities”. The project supports directly the Intermediate Result 3 (IR 3); “Improve competitiveness and productivity of micro, small and medium-sized businesses and their services providers to generate sustainable employment opportunities and increased sales”. 5 2.1 BPC Project Objectives and Development Hypothesis BPC’s main purpose was to increase productivity and sales of MSMEs in order to generate and sustain employment and increase incomes in urban and peri-urban areas. By strengthening value chains and developing new and existing domestic and international market linkages, it was expected to generate gains in market transactions and cash sales for MSME’s. In the process, high-value labor intensive products and new market niches will be identified and interventions will improve profitability and increase value at all stages of the chain. Moreover, BPC seeks to strengthen the capacity of local organizations by providing services to their productive sector. The services aim to improve the business enabling environment and workforce development to support competitiveness. Specifically, the BPC project’s objectives are to: 1. Improve the competitiveness and productivity of selected value chains: Textile and apparel, wood manufacturing products (e.g. furniture and similar); bio-products, processed foods and handicrafts; 2. Strengthen local institutions that support the development of MSMEs; 3. Promote dialogue between key stakeholders (e.g. private sector organizations, public sector, social organizations) to improve the business environment and competitiveness of value chains and MSMEs; and, 4. Leverage funding from the private sector, other donors and public institutions using public-private alliance (PPA) funds and other sources of funds. The BPC project development hypothesis is; Increasing the Productivity and Sales of Urban and Peri-Urban MSMEs, Employment will be Generated and Incomes will Increase. Figure 1 illustrates BPC’s four specific objectives to achieve the overarching objective to develop MSMS sales and productivity. Figure 1: BPC: Main and Specific Objectives The BPC Project is guided by the development hypothesis that increasing the productivity and sales of urban and peri-urban MSMEs will lead to rising employment and sustained higher incomes. Bolivia’s economy largely consists of small-scale entrepreneurs, with an estimated 500,000 MSMEs employing 80 percent of the labor force. However, the MSME environment is challenged by low levels of productivity, limited access to markets, high transaction costs, limited access to credit, low levels of engagement with other private sector agents, and 6 operations in the informal sector. By strengthening value chains and developing new and existing market linkages, there will be gains in market transactions and cash sales for MSME’s, which have faced numerous domestic and regional barriers to growth. The project also has a significant component to support public-private partnerships (PPPs), also called public￾private alliances (PPAs) with the local private sector and donor groups. This component is in line with USAID’s Global Development Alliance to forge market-based partnerships. 3. METHODOLOGY Evaluation Methodology of BPC Performance: This section discusses the detailed methods and approaches the team applied to carry out the evaluation. Evaluation Criteria Framework: The general framework criteria for our evaluation, which was applied in accordance with USAID M&E-evaluation directives; was agreed upon between IBTCI and USAID-Bolivia to be concentrated in the following parameters: (1) Relevance, (2) Effectiveness, and (3) Sustainability. To make more sense of our work we developed a comparative matrix showing the relationship between Project Objectives vs. Project Indicators vs. Evaluation Questions and Evaluation methodology for performance. – These are shown in Annex 2. The evaluation team maintained equilibrium among: a) performance measures based on parameters as defined by USAID recommendations, b) BPC methodology and indicators, and 3) the evaluation questions to be answered. As IBTCI was requested to conduct a performance evaluation, we used a mix –method design in order to base evaluation findings on both quantitative and qualitative data. Quantitative methods involved the review of primary and secondary statistical data and conducting descriptive analytical techniques. Qualitative evaluation methods involved site visits, key informant interviews, and one focus group discussion Purpose for Evaluating BPC: The purposes of evaluating the BPC Project as mentioned in the task order Statement of Work (SOW) are to: A. Evaluate the performance of project results and deliverables from September 30, 2009 through June 30, 2012 (The documentation and information received only covers this period); B. Highlight lessons learned and best practices developed for micro, small and medium- sized enterprises (MSMEs) and Public Private Alliances (PPAs) that could be replicated; C. Provide recommendations on strategies to promote PPAs in new programs under the Country Development Cooperative Strategy (CDCS) for the period 2012 through 2017 and in the design of new projects; and D. Provide important information of activities that should be taken into account in future projects, methodologies and strategies to be replicated for technical assistance, training and developing PPAs. Key evaluation questions as detailed in the Statement of work (SOW): The purpose of the evaluation is guided by five key questions: 1. To what extent have project activities been effective in achieving their intended objectives? To what extent did the BPC Project contribute to the current SEGE program objectives? 2. What are the most relevant elements that contributed to/impeded increase productivity and sales of MSMEs? To what extent the PPA mechanisms were effective in leveraging funds from the private sector 7 and other donors? Were they adequate considering the current economic and political context in Bolivia? How can they be incorporated into future mission strategies? 3. Over the course of implementing the project, what new best practices and lessons learned were identified or revalidated that should be continued in the Mission’s future programs? 4. How have investments made in strengthening the current value chains contributed to improved competitiveness and productivity of supported sectors? Were they strengthened in an integral approach? Are there other priority sectors that should be supported in the future? 5. To what extent have BPC activities strengthened local institutions capacities in being able to provide adequate services to MSMEs, compete for resources, address MSMEs’ issues, and manage USAID funds or concepts? Sampling Methodology and Data Collection To obtain project-level information to effectively answer the evaluation questions, the team leveraged three main sources of data: First, the evaluation team leveraged monitoring and evaluation data collected at the activity level for the BPC Project, – which had been provided to USAID by the implementing partner; Second, the team referenced all available project documents, including quarterly reports, annual reports, BPC manuals, and other relevant documents; Third, we also obtained lists and information of all major activities, events and how these support services were delivered to project beneficiaries, i.e. individuals, MSMEs, potential workers, innovators, sectorial institutions, associations and, in some case, indigenous communities. This information provided an overall understanding of the universe and characteristics of beneficiary population. For the field-based data collection activities, the evaluation designed an approach to access directly a sample of BPC beneficiaries. To ensure unbiased results, the team applied sampling to represent the a) different cities, b) different institutional subsectors, and c) business sizes, which we considered important dimensions of beneficiaries. This ensured that interviewees were a balanced sample and representative of the beneficiary population and would provide in￾depth knowledge and experiences regarding the BPC project. After careful consideration and dialogue with the main stakeholders, it was determined that since the total numbers of beneficiaries in most of the activities were comparatively small, strict standard sampling methodology could not be applied. Therefore it was more practical to implement (a) two opened surveys (one for the institutional and one for the PPA components, (b) an extensive plan of personal interviews based on city location and activity type; and (c) two focus-groups: one with the institutional beneficiaries and a second with a small sample of MSME-beneficiaries. The most effective and practical method to evaluate the performance of the BPC was the personal visits and interviews strategy, which allowed the team to define the degree of efficiency, relevance and potential sustainability as an effect of the project’s work with the beneficiaries. To implement the process and application of personal interview methodology in different cities, different sectors, and innovative industries, the evaluation team defined a strategy shown in Table 2 (below). Evaluation Implementation per City of intervention Table 3 is the overall field work planning schedule by location and target beneficiaries. This information guided the team’s visits and personal interviews. Special attention was given to aspects of size of MSME and industrial sector. 8 Table 2: Field Work Planning Schedule and Sampling Instrument Applied Type of Beneficiaries Sample vs. universe) La Paz/ El Alto Cochabamba Tarija Santa Cruz Training Beneficiaries Perception & Quality Total Beneficiaries of intervention Technical Assistance (with subcontractors) 15/131 5/31 1/3 10/90 Methodology: Information was captured from M&E data documented by BPC (Evaluation). Sample approx. 2% of the universe. 232 Public-Private Alliances 5/13 1/1 2/3 2/2 27 Commercial Promotion 3/53 3/31 3/4 3/64 152 MSMES (Employers) 2/42 2/13 0 2/12 67 Institutional Development (Direct and Indirect Assistance) 28/28 10/10 1/1 4/20 61 Training Events 12/125 4/33 0 7/57 230 Total Intervention 65 25 7 28 Man-Level of Effort 5 days 4 days 2 days 7 days Responsible for M&E execution (IBTCI Consultant) The Team as a whole Sergio Villarroel Milton Núñez Luis Bernal/ Milton Núñez (2 days) All members of Evaluation team Dates: Nov. 6 - 9, 2012 Nov. 12-15, 2012 Nov. 12 - 13, 2012 Nov. 12 - 17, 2012 NOTE: Definition of factors: x/y = sample size / number of beneficiaries in the city of intervention A. Samples by City: Since the main effort from the BPC project had been in La Paz and El Alto, the work load for the Evaluation team was mainly concentrated in these two cities. The larger shares of targeted respondents were accordingly planned in these two cities. The sample size obtained in Cochabamba was 21 respondents, which represents 24% of the stakeholder population. The largest share of the respondents came from implementing institutions. Sampling in Santa Cruz resulted in 11% coverage of the stakeholder population. This was because it is a large city and distances between assignments were long, with extremely tangled and slow traffic. Therefore, the team added an extra interviewer in order to accomplish what had been planned within a reasonable time frame, which in this case had been designed to be a full six-day week. In Tarija, seven out of the stakeholder population of 11 was sampled. Most of them came from the PPA and commercial production sectors. Tables 3 to 6 detail the distribution of respondents per city. 9 Methodological Limitations of the Current Evaluation Due to time and logistical constraints, the evaluation team did not have the opportunity to pre-test evaluation instruments. The small sample size obtained after the sampling also did not necessitate pre-testing. Thus, the evaluation team developed instruments using questions that have been proven to be valid and useful in the evaluation of small business and entrepreneurship projects. These were sourced from the empirical literature, previous small business evaluation data, and each team member’s prior experience in SME evaluations. The small sample size obtained also limited the analysis of data to simple descriptive statistics. However, the findings were sufficient to answer the key evaluation questions. The purposive sampling procedure also limits the findings to the interviewed beneficiaries. Inferences cannot be made to the general beneficiary population. 10 Operative Matrix - BPC Project Evaluation Visits made in SANTA CRUZ Sample Universe 1 Technical Assistance 10 90 2 Public-Priv ate Alliance 2 2 3 Commercial Promotion 3 64 4 MSMES 2 12 5 Implementing Institutions 4 20 TOTAL Sta Cruz 21 188 Service Operation Matrix - BPC Project Evaluation Visits made in LA PAZ Sample Universe 1 Technical Assistance 15 131 2 Public-Priv ate Alliance 7 13 3 Commercial Promotion 3 53 4 MSMES 2 42 5 Implementing Partner 29 28 TOTAL LA PAZ 56 267 Service Visits made in TARIJA Sample Universe 1 Technical Assistance 1 3 2 Public-Private Alliance 2 3 3 Commercial Promotion 3 4 4 MSMES 0 0 5 Implementing Institutions 1 1 TOTAL TARIJA 7 11 Service Table 4: Sample for Cochabamba Table 5: Sample for Santa Cruz Table 6: Sample for in Tarija Table 3: Sample La Paz Operative Matrix - BPC Project M&E Visits made in COCHABAMBA Sample Universe 1 Technical Assistance 5 31 2 Public-Priv ate Alliance 1 1 3 Commercial Promotion 3 31 4 MSMES 2 13 5 Implementing Institutions 10 10 TOTAL 21 86 Service 11 4. EVALUATION ANALYSIS:10 Figure 2. In general, most members of the Evaluation team grade the performance of the BPC as “Good” or “Acceptable” in most activities. It is also important to note that evaluation findings show the BPC has performed remarkably well in the development of PPAs. 4.1 Analysis of Activity: Improved Productivity and Competitiveness of MSMEs and Value Chains In terms of sales, 32% of the 46 interviewed firms reported increases due to the project. Three firms responded that their sales were not affected by the BPC while the rest either did not respond or provide information. The average new sales among those who stated having increases was 49.8%. In parallel, 12 firms or 26% of respondents claimed increased productivity due to the project. On average, productivity improved by 56% although it could not be determined if the increases were due to technical assistance, commercial promotion services, workforce inclusion, or training.11 Improvements in productivity were also associated with cost reductions, and 26% of respondents claimed this result due to involvement with the project. However, not all of them were able or willing to provide figures. Among those who did, cost reduction averaged 17%. Regarding profitability, 15 firms or 32% of respondents reported higher profitability because of BPC. However, only 26% were able to provide figures. Among those who did, the increment in profits averaged 79%. 10 The evaluation analysis was based on the data and information FINDINGS shown and discussed numeral 8 11 BPCs definition and measurement of productivity as shown by the level of sales. It is more common to measure productivity for MSMEs as determined by the manufacturing production or outputs. BPC – Areas of Intervention and type of Value-chain benefited Results September - 2011 Geographical Areas: Urban and Rural Areas in the Districs of La Paz, Cochabamba y Santa Cruz and rural areas of Chuquisaca y Tarija Value-Chains Selected: Textiles and confections Wood products and furniture Processed Food Bio-comercio and crafts 12 On job creation, seven firms or 15% of respondents stated having created new jobs because of BPC, with an average of five direct new ones. In addition, a couple of respondents stated they created a higher number of indirect jobs, up to 100 in one case. The evaluation findings show that, except for a couple of cases in which owners regarded BPC’s intervention as poorly executed or irrelevant for their firms, a good number of the interviewed MSMEs had positive views of the experience. The smaller firms tend to have higher BPC effects in terms on the firm’s overall management improvement, formalization, and in getting a wider vision of the business as an enterprise and not merely a livelihood. This perception is oftentimes reinforced by the fact that the firm is experiencing a generational change from the time when a father managed the business in very personal and artisan way to the present time when the children want to see the old workshop transformed into a sustainable business managed with modern practices. A parallel, or complementary, perception of a firm’s overall improvement is due to production methods, organization, work environment, better product quality and manufacturing efficiency. Those owners are satisfied with a better way to do things, thanks to BPCs intervention. A total of 35% of respondents declared positive effects in terms of their firm’s general improvement, whether it was managerial, organizational, or technical. However, the idea that BPCs trade and trade-fairs promotion interventions have no sustainable effects is the predominant view among those firms that received that kind of support. In their view, the types of commercial events BPC promoted were interesting and were good opportunities to learn about their work and their competitors’ operations. But it did not necessarily lead to more commercial opportunities or, more importantly, to the development of greater market linkages or a deeper commercial vision. In retrospect, they wished commercial assistance provided by projects such as the BPC were more hands-on, lasting, and practically oriented to obtain a more sustainable commercial route for their companies. When several MSMEs’ were asked about their satisfaction with USAID, 45% had no opinion or knowledge about USAID’s role or involvement in the project. The other 55% were satisfied or highly satisfied, which is credited to the MSMEs’ direct contact and relationship with those professionals who provided technical assistance, training or other kind of support, directly to their firms. Regardless of their level or satisfaction, the most frequent “suggestion” was that any assistance received was limited in time, scope and duration. Evaluation findings showed evidence that work was achieving project objectives. However, some respondents did not feel to be part of a larger whole such as “Value Chain”, “Cluster” or similar entity. This is surprising as BPC proudly claims that it has been a project policy to work not with MSMEs individually but with collectives that make up horizontal or vertical chains acting as mutual suppliers and getting potential scale economies or joining commercial forces. As BPC worked through implementing organizations that rendered services on its behalf, one possible explanation is that the MSMEs groups were not aware of the link between the implementing organization and the larger BPC initiative. Again, probably the view of MSMEs working under a PPA differs in this regard. BPC improvement of Value Chains The experience of BPC with value chains is perhaps the most interesting aspect for evaluation purposes, both from the perspective of project design and project implementation. Initially BPC had a wide vision of chains12 and 12 BPC contractual document defines: “The value chain system includes: end markets, be they international or national, the business enabling environment within which the chain operates, the vertical (among actors from input supply to final market) and horizontal linkages amongst cooperating actors (mainly at the production level but could exist elsewhere), and the supporting services markets (both cross-cutting and sector-specific) which provide finance and business services to value chain actors”. 13 sectors with specific challenges the project aimed to overcome. They were stated in BPC’s first quarterly report as follows:  Textiles and apparel: “Therefore, project’s challenge is to support this chain to increase the national production, competitively substitute imports (legal and smuggled) and widening its export options to regional and worldwide markets”.  Camelid-hair textiles and apparel. “Project challenge is to support integral initiatives –in coordination with other international cooperation programs, public and private sectors) to incentivize camelid’s stockbreeding with emphasis in Llamas, so the industry is able to process their hair into fibers, textiles and apparel. The project would support entrepreneurial initiatives –through design, product development and manufacturing technical assistance- so that there is an increase of national production of apparel made out of Llama and Alpaca fibers to be exported worldwide”.  Wood manufacturing: “Project’s challenge is to support MSMEs so that they strengthen their market position nationwide, based on products’ quality rather than low prices that do not generate profits to save, invest and grow”.  Processed Foods. “The project faces the challenge of supporting with specific interventions the building of sub-sectorial food manufacturing complexes to increase the national offer of processed food products oh high quality that will be also competitive for exports”  Bio-products. “Project’s challenge is to become, working with PAI, a promoter of technical initiatives that allow these products to better position themselves in national markets, the same way that there are hundreds of organic, natural and environmentally-friendly foreign products, while contributing so that those initiatives achieve production, quality and competitive standards that allow their products to be exported”. To assess the impact on value chains, BPC adopted a proxy multiplier proposed by another USAID initiative (for agro-farming production and foods industry), which is based on sales increments. According to that multiplier, there has been an increment in the total value of value chains, as showed in Table 7. Table 7: BPC: Value Chain Total Value Increment (USD) Value Chain ∆ Value Processed Foods 4.385.650 Textiles and apparel 11.823.435 Wood manufacturing 6.392.443 Other Chains 1.466.085 BPC also applied a PPA model with 27 MSMEs. BPC reports impacts on value chains in terms of creating new market linkages. These included:  Business roundtables and associations of forestry producers and furniture manufacturers that have found potential buyers;  One event for apparel manufacturers. Participants hoped this event could promote their products with national commercial and institutional buyers to substitute the imports;  Trade missions to Chile and Brazil that allowed for new contacts to furniture and apparel manufacturers 14  Incoming trade missions from USA, Mexico, Russia, Spain and Australia that have visited apparel manufacturers; and  Incoming trade missions from USA, Russia and Europe interested in wood intermediate products for the building industry have opened new contacts. BPC and the supply of Business services for MSMEs As BPC sought to strengthen the capacity of local organizations by working them to improve the quality and supply of business services and support, it was necessary to survey MSMEs for their perceptions regarding BPC effects in strengthening their capacities as suppliers of business development services. To that end the following organizations were contacted: Cámara de Exportaciones CAMEX, Fundación IDEA, IBNORCA, CPTS, IDEPRO, Cámara Nacional de Industrias CNI. These institutions were asked: What type of new (or better pre-existing) services to MSMEs did they receive thanks to BPC? Additionally, they were asked whether they think those services are appropriate in nature, quality, quantity and price to the actual needs of MSMEs? Each organization’s response is provided below. CAMEX: “Thanks to BPC support, we have improved our training methodologies for MSMEs. Particularly by using the identification and categorization methodology BPC transferred to us. With this methodology we have increased our capacities to better outreach to those firms. There are new services being implemented such as “Reverse Fairs” consisting in bringing buyers to Bolivia. Likewise we have strengthened our business roundtables that were too basic and now are more systematic. BPC supported us in creating catalogs and advertising materials that benefit MSMEs”. IDEA: “Our activity with BPC moved us to create a new institutional office: “The MSME Productivity and Competitiveness Unit”, as well as new services in education and training for various value chains, such as workshops on cash flow management, production costs reduction or imports from China. Although BPC helped by subsidizing training costs to MSMEs, it limited its support to those value chains the project included. Now IDEA has enhanced its outreach to other sectors” IBNORCA: “With BPC we worked in developing a new product, the “Handmade Certification”, together we worked with OIPAC (a cultural-oriented artisans’ association) to outreach to artisans and interest them on the certification”. CPTS: “CPTS was provided by BPC with computers, educational software and support to developing promotional materials, all of which have allowed CPTS to improve the quality of our training courses”. IDEPRO: “Based on BPC support we have new training services in administration aimed to business owners, as well as workers training for workforce insertion. We have kept offering technical assistance to MSMEs”. CNI: “BPC helped us to generate new services as we not only created the MSMEs’ Unit within the Chamber, but also started our services in El Alto where we provide training on product packaging and labeling, as well as in Fairs and Business Roundtables, we have also provided support to El Alto Municipality in Fairs for Metalworking and Processed Food; besides we have kept providing technical assistance”. 15 4.2 Analysis of Activity: Institutional Strengthening to Develop MSMEs In terms of effectiveness, BPC’s work with four major institutions to strengthen them through the direct mechanism was considered satisfactory due to the expectations and results attained by these institutional activities. While the original BPC plan was to strengthen 14 institutions, only four fulfilled the conditions and characteristics to qualify for BPC support. However this smaller group helped the achievement of objectives more manageable. Much time and effort was invested in trying to strengthen six other institutions (CADEFOR, EPC, IMB, BIONATIVA, Red OEPAIC and IBNORCA) but these efforts were not successful. Moreover, even if these six institutions were strengthened, a gap of four institutions would remain. The lack of resources availability to satisfy all the institutions’ requirements was a valid argument. However, the question surfaces as a potential weakness in the form of inadequate planning and preparation in the supply and demand side of the equation, and this could also have been a factor of challenge, which must be carefully considered. During interviews with professional consulting institutions and NGOs, a very challenging issue was mentioned by several managers about the possibility of an unspoken cloud of uncertainty, or being “black balled,” by GOB institutions due to work done with US government projects or receiving benefits from US government funds. There was the thought of being barred by the GOB from access to programs and other opportunities. Many consulting institutions’ and NGOs’ main sources of income have always been governmental programs and funds. This has been noted as one potential cause for the limited success for the BPC project in the area of attracting institutions to become stronger and capable of accessing USAID funds and benefits. Having this in mind, the BPC could be graded with moderate to low degrees of effectiveness, although as mentioned before some mitigation can be assigned to the socio-political environment. Analyzing the general situation, could these institutions be sustainable on their own? We believe that with the exception of COTEXBO, the remaining three ISDMs are large institutions with consolidated activities and low rates of personnel rotation, which allows them to enjoy an acceptable degree of sustainability. The training centers that were equipped for CNI and IDEPRO can actually charge the students for the services and therefore guarantee their sustainability. COTEXBO is also taking steps towards financial sustainability by diversifying the sources of funding. There is also the fact that similar wood-drying technologies have higher market prices, two times higher than the one created by CPTS. This makes it affordable to more MSMEs and, therefore, facilitates the prevalence and sustainability of the new oven. Considering these issues, the degree of sustainability can be easily identified as high. Indirectly Strengthened Institutions (ISI) In the case of ISIs, information given by the institutions related to the improvement of their technical capacities after working with the BPC was affirmative in 83% of the cases. The most common reason for improvement was the spillover effect from having the opportunity to offer services to a larger client-base. This opportunity led to more work with a larger variety of clients and gave institutions more experience and improved technical capabilities. Institutions adjusted their own techniques for reaching a broader universe of MSMEs. These ISI institutions already enjoyed built-in systems capabilities because the targeted institutions had received technical and financial support, in the last two years from other cooperating agencies, mostly in technical areas. Therefore, in order to investigate the positive effects of the BPC project, a cross reference table was used to isolate the effects of other donor’s interventions. 16 The effectiveness of the BPC in improving the quality of services delivery by the ISI group will be examined through a test to compare the results obtained with the contact-intensity reached in each case.13 To define specific ranges of contact-intensity, we assumed that only one contract signed would be equivalent to low contact-intensity. Between two and three contracts were considered medium and four contracts or more were high contact-intensity. Table 8 presents the information disaggregated according to these categories. The statistical procedure in this case will be the computation of conditional probabilities of the institution’s perceptions that their technical capacities to provide services to the MSMEs have improved, given certain level of contact-intensity. The evaluation of the cause-effect relationship depended on the capacity of each institution to perceive their own technical capabilities to provide adequate services to MSMEs and if those had improved. Using the information provided in the cross reference table, the probability will be high. In other words, if we randomly choose one institution from the ISI group that didn’t receive any institutional strengthening except the one provided by the BPC, the probability that this institution answers that their capacities to provide technical services to MSMEs have improved is 82.4%. This approximation is of course valid only for the subgroup that didn’t receive strengthening support from other agencies. However, for the remaining group, the question in the survey reflects also the correct institution’s perception because they were asked to separate only the positive effect attributable to the contact with the BPC. The conclusion is that this indirect channel of strengthening becomes a very high grade for the BPC project. Even with low contact-intensity, the probability is high, which is consistent with the findings shown in the previous table. In summary, the results were better than planned for a specific group of institutions that worked more closely with the BPC, signing many SBKs, grants, or MoUs. Thus, a high degree of effectiveness can be assigned. The evaluation proceeded to look into a mix of different institutions and determined that the larger ones can achieve better sustainability due to their capacities. They were also at a more developed stage and their activities were consolidated. Interestingly, field testing and interviews revealed that larger institutions are the ones that obtain the higher benefit from the indirect channel of strengthening even though smaller institutions have greater need for support and improvement. Nevertheless, this contradiction was compensated to a large extent with medium to high levels of contact-intensity. In other words, small institutions that signed many contracts with the BPC (SBKs, grants or MoUs) had better chances to reach sustainability due to the experience gained and the curriculum enrichment that allowed them to compete for more resources. The final effect varies according to the size of the implementing institution so only a medium degree of sustainability is appropriate in this case. 13 Contact-intensity means the number of contracts signed with the BPC (either SBKs, grants or MoUs). The more contracts signed with the BPC, the more intense the contact would be. Table 8: Institutional Perception Items Institution's perception that technical capacities have improved Institution's perception that technical capacities have not improved Total Low contact-intensity # contracts (SBK, Grants and MoUs) = 1 17 5 22 Medium contact-intensity # contracts (SBK, Grants and MoUs) = 2 or 3 10 2 12 High contact-intensity # contracts (SBK, Grants and MoUs) = 4 or more 80 8 Total 35 7 42 17 Strengthening of Institutions’ Capacities to Address MSMEs’ Issues It has been assumed that the capacity to address MSMEs issues can be achieved mainly through: i) the institution’s capacity to understand the functioning of the value chain; and ii) the institution’s capacity to identify MSMEs needs. Both aspects were included in the questionnaires prepared for ISDMs and ISIs in order to test the perception of the institutions with regard to these issues. Institutions Strengthened through Direct Mechanism (ISDM): In a survey applied to all four ISDMs and in a focus group with three of them, the responding institutions stated that the BPC had an affirmative effect in their capacities to identify MSMEs’ needs plus a better understanding of how the value chains and clusters chosen interrelate and work to help each other’s business. Furthermore, ISDMs stated the importance of the indirect channel of strengthening because, in the institutions’ perceptions, both capacities were not enhanced due to the equipment provided or the improvement in some processes. The enhancement was more a result of the contact with the BPC as subcontractors. Therefore, from the point of view of effectiveness, it is interesting to note that the ISDMs gained these capacities even though they were not planned as part of the initial strengthening package. Nonetheless, the results must be qualified as positive in terms of planned versus accomplished outputs. The conclusion is that the degree of effectiveness by the BPC was high in the area of institutional strengthening through direct mechanisms. In the same manner, did direct mechanisms reduce the risk of failure? Thanks to the low rotation of personnel, it can be construed as a higher degree of sustainability. However value-chains and MSMEs characteristics and needs are dynamic. So, in the absence continuous contact with the BPC, interaction with MSMEs is attained through other means. So, sustainability is uncertain or at most medium grade can be assigned. Indirectly Strengthened Institutions (ISI): Looking at the ISI-group of institutions and their capacities to address MSMEs issues, ISIs stated that the BPC project had a direct and important impact in their daily activities. Findings show 81% of institutions stated that their abilities to identify MSMEs’ needs have improved and 69% have a better understanding of how value chains work and are structured. Findings seem more evident only in the case of small institutions and due to the capacities gained from curriculum-enrichment. It is possible that their sustainability could be reduced unless the curricula and services continue to improve or the institutions take steps to increase investing in equipment and hiring more permanent technical personnel. Strengthened of Institution’s Capacities to Manage USAID Funds When asked about the usefulness or relevance of the workshops to facilitate the administrative contact with the BPC project, two ISDMs (IDEPRO and COTEXBO) responded that the courses were useful and one (CNI) reported the courses very useful. But more important than the workshops were the personal contacts - telephone calls, e-mails, meetings and visits - between the ISDMs and administrative personnel from the BPC. Three institutions (including the CPTS) responded that this type of contact was very useful and one (COTEXBO) said it was useful. A high degree of relevance is therefore justified given the recognition of clear benefits. The effectiveness of BPC intervention is measured by the number of persons trained in the institutions, which showed a 92% achievement rate. Specifically, 138 persons from the originally 150 targeted were actually trained in administrative procedures. Even though the respondent sample included all implementing institutions, not only ISDMs, a good number of participants (16 of the 138) belonged to the ISDMs. That is why a high degree of effectiveness can be assigned. In managing USAID funds for the ISDM group, three of them stated that administrative requirements of the BPC were normal and only COTEXBO (being a relatively new organization) answered that the administrative paperwork was very bureaucratic. This show how ISDMs are used to handle resources provided by foreign donors, which is a capacity gained through time. 18 Indirectly Strengthened Institutions (ISI): In the ISIs case, 55% of the institutions participated in the workshops given by the BPC to strengthen the administrative capacities of the employees. When asked about the relevance or usefulness of the workshops, the institutions that participated declared that the courses were useful or very useful and no institution found it of little use. In the evaluation exercise it was determined that 100% of all participating ISIs found that the workshop curricula was useful (or “very useful”), therefore it follows that the curriculum design was adequate. Regarding the importance these ISIs placed on direct contacts with BPC’s personnel (through telephone calls, e￾mails, meetings and visits) to facilitate administrative procedures, most responded that this type of communication was very useful and the rest thought that it was useful. The findings reveal that the direct contacts were even more important than the workshops. One important finding was the opinion given by ISIs regarding the complexity of BPC’s administrative procedures. The vast majority of the institutions responded that the administrative requirements were normal. The perception on administrative requirements is understandable in the case of big institutions but not necessarily in the case of small ones. The explanation was found in the interviews when small institutions clarified that the administrative requirements of state-funded initiatives were far more complicated and bureaucratic. Some experienced delays in the disbursements that lasted for several months. Given the strong cause-effect connection, a high degree of relevance can be assigned to this component. The number of persons trained within the IDIs supported was 138 persons out of 150 targeted (92% achievement), which is a strong indication of effectiveness. Managing USAID funds by ISIs can only be sustainable if administrative personnel rotation is low. In the interviews, all institutions confirmed that this condition is normal. They all know that a well-trained administrative department is highly treasured asset. Taking this into account, the level of possible sustainability of these ISIs can be considered as high. 4.3 Analysis of Activity: BPC Project – Leverage of USAID Resources and PPAs A parallel support mechanism for MSMEs that BPC has been implementing is the Public-Private Alliances (PPAs). These alliances are meant to assist in solving problems or bottlenecks to groups of firms through a grant mechanism to acquire mostly equipment or other resources that will benefit firms. Alliances forged under from 2010 to 2012 are shown in Table 9. Table 9: BPC: Public-Private Alliances Value Chain FY 2010 FY 2011 FY 2012 Jun 2012 Accumulated June 2012 % Education 1 1 4% Processed Foods 10 6 16 59% CP/EE 4 4 15% Bio-commerce 3 2 5 19% Textiles 1 1 4% TOTAL 1 13 13 27 100% 19 Examining the problem of institutional weaknesses and symptoms for small interest in the formation of MSMEs, the business community expressed that, among other reasons, the business sector did not enjoy support from the academic sector although they were the major clients of the graduates produced by schools and universities. The incubation and development of MSMEs is a complex issue and needs special support because MSMEs could help alleviate or mitigate the lack of job opportunities. For this purpose a PPA agreement and donations were channeled through the School of Public Management at the Bolivian Catholic University to educate and form a group of 18 teachers and professors at the Public University of El Alto which, in turn, will be capable to help start MSMEs in their development process. Based on the review of information generated by the BPC project, it was determined that from project start￾up until June 2012, a total of 27 Public-Private Alliances (PPAs) were achieved. These were geographically distributed in the cities of La Paz / El Alto, Cochabamba, Santa Cruz, Oruro, Tarija, Chuquisaca and Potosi Eight out of the 18 enterprises surveyed were in an initial stage in the business life cycle, seven were in the growth stage, and three were in maturity stage. In terms of firm size, the respondents included two micro enterprises, 11 small firms, and five medium-sized firms. In terms of gender, the respondents consisted of 14 men and four women. Eleven of the 14 men were MSMEs managers while three were employees. In the case of women, three were managers and one was an associate. In terms of geographical coverage, 15 firms have a market reach of only local sales, 12 have nationwide sales, one firm sells its products at a regional level, and four MSMEs export their products. Analysis of the Effectiveness of BPC in leveraging funds14: According to the BPC plans, 13 PPA operations should have been completed by the end of 2012 and all of the US$ 1 Million in resources allocated and used in those operations. Funds should have been used on a pro-rated distribution basis in each of the value 14 For BPC Project implementation purposes, leveraging funds refers to a programs costs or not borne by the Federal Government. It also could involve one or more partners proposing contributions that will be spent in parallel to the USAID-funded activity. But not expend by grant recipients or subcontract awardees. Leveraged funds could also come in the form of the entity’s ability to get other supporters to provide their own form of assistance directly to the same end-users PC-B Project Leverage of resources through PPA mechanism up to June, 2012 (Expressed in US$) PPA Partners Contributions (in USD) Percentaje USAID (BPC) 633.312 36,61% Private partners (MSMEs and Production Associations) 1.096.590 63,39% TOTAL (for a total of 27 PPAs) 1.728.902 100% Table 10: BPC Project – Leverage of Resources (Photo # 1 at apparel detailing - PPA - AAA-Tajzara / District of Tarija – Municipality of Yunchara/ Nov-2012) (Photo #2 of PPA Bolivia Natural Food Products ABN, La Paz – Nov.2012) 20 chains. In the same manner in evaluating the effectiveness in allocating PPA funds, the BPC figures reported up to June 2012 showed that a total of 27 PPA operations were done and a total of US$ 1.09 Million were contributed by PPA partners, as shown in Table 11. These results were beyond expectations of BPC management and USAID. The following table is a comparison of project funds granted to MSME-start-up. 21 Table 11: Bolivian Productivity and Competitiveness Project –PPA-Comparative List of Partner Contribution: Public-Private Alliances Matrix: Comparative list Partner Contributions for PPAs Source: BPC files as reported to USAID-Bolivia Nº Name of Beneficiary Contributions by partner TOTAL BPC Project % Beneficiary % 1 Association AFIPAC 24,207 38% 39,963 62% 64,170 2 Association APAJIMPA Padilla 6,814 38% 11,127 62% 17,941 3 Association APAFAM El Villar 9,462 40% 14,335 60% 23,798 4 Association APROMAJI Muyupampa 8,976 38% 14,595 62% 23,572 5 Association ASOPROMANI Yacuiba 8,976 35% 16,840 65% 25,816 6 Cooperative CIAPEC 15,489 38% 25,126 62% 40,615 7 Association APROMAJI - PEDERNAL 7,406 38% 12,139 62% 19,545 8 OLEUNS BEAUTY 8,435 36% 15,269 64% 23,703 9 COSNATVAL 15,443 17% 73,396 83% 88,839 10 Association NAKHAKI 22,867 15% 131,979 85% 154,846 11 Catholic University of Bolivia (Master’s Program) 143,162 50% 143,219 50% 286,381 12 UNEC 116,057 31% 259,626 69% 375,683 13 SUMITA LTDA 14,877 37% 25,157 63% 40,033 14 PELETBOL 17,608 36% 31,231 64% 48,839 15 TELLEZ Workshop 15,408 40% 22,813 60% 38,222 16 PROSOL 27,551 45% 33,674 55% 61,224 17 Bolivia Natural Products - ABN 25,264 45% 30,603 55% 55,866 18 White Deserts 20,131 26% 58,209 74% 78,341 19 S.A.C- BIOLAC 27,405 45% 33,495 55% 60,900 20 AGROCAINE 15,936 45% 19,741 55% 35,678 21 HandycraftLABORory DERMA￾FARMACÉUTICAL ADA 12,259 49% 12,730 51% 24,989 22 INMUNO – VIDA 12,630 50% 12,849 50% 25,479 23 NOEMY - Thinking of you 12,138 50% 12,213 50% 24,351 24 Association APLENAT 14,985 49% 15,320 51% 30,305 25 Association San Carlos 9,941 49% 10,314 51% 20,255 26 Andes Association 9,941 49% 10,314 51% 20,255 27 Association New Hope 9,941 49% 10,314 51% 20,255 28 Tannery San Juan 29 Tannery San Lorenzo 30 PROINPA TOTAL FINANCING 633,311.28 37% 1,096,589.86 63% 1,729,901.14 Contributions by Partner BPC Project 633,311.28 37% Beneficiaries 1,096,589.86 63% 22 Table 11 shows that the project actually exceeded its targets and that PPA funding matched USAID in a 1.71:1 ratio. It was initially planned that USAID funds were to be equally matched with MSME funding. Moreover, the initial amount of US$ 1 Million was to be disbursed for four years. Since the project was cut short and budgets were reduced by 18%, the original PPA funds for three years could be adjusted to US$ 750,000 for three years for a completion rate of 84.4%. Alternatively, the 18% cut meant a budget of US$ 820,000, which means a completion rate of 77.2% of its disbursement goal for PPA activities. Measuring effectiveness in terms of PPA activities meant assessing planned versus achieved funding leveraged from the private sector, other donors, and public institutions using a public-private alliance (PPA) or other funding source. Examining the investment placed by BPC in several PPA-MSME organizations, the planned funding match was a 1:1 ratio. But the outcome translated into a more positive ratio of 1:76:1 (USAID‘s US$ 633,312 and the APP partners US$ 1,096.590). Therefore in this objective’s overall result is very positive. An analysis of the information received from PPA partners showed that 11 out of 18 interviewed/surveyed MSMEs mentioned that their projects were fully implemented. Seven stated that they were partially implemented, with one saying the project was 50 to 53% complete and the other stating a completion rate of 70 to 85%. During the evaluation’s data collection period, PPA-donations of equipment for improving productivity had been done in all 12 MSMEs covered. The discrepancy between partial completion rates reported and 100% rates during the evaluation was largely due to differences in reporting. Not all 27 MSMEs had been able to fully install and operate the equipment due to: (a) limited technical or financial capacity to achieve full installation and production, and (b) lack of market for the extra production. Thus, MSMEs could not report full completion rates. This meant that in all cases they had placed full attention to the technical and production aspects, but not in marketing the excess production. Our opinion is that the PPAs should have been helped and developed in an integral manner. Level of Satisfaction with the PPA model and application: In relation to the level of satisfaction with the support received from the BPC project, 14 out of the 18 enterprises (78%) reported being very satisfied and the remaining four were satisfied (22%). Of the four MSMEs that were satisfied, two based their answers on the degree of project implementation while one MSME stated having other unmet needs. But these respondents also stated their own lack of preparedness. (Photo no. 3 UNEC Oregano multiplication greenhouse and field dryers at District of Tarija – Nov.2012) 23 Table 12: Survey and Personal Interviews Information and Analysis of Results (Nov./2012) Efectiveness Pertinence Gender Environment (2) No. Name of MSMES BPC Delivery of PPA Agreement How did know about BPC PPA mechanism? Who was identified as the donor? Beneficiary Satisfaction Beneficiaries Opinion According to Evaluator Are MSMES Gender issues? Environmental Kardex? 1 SAN CARLOS Association 50% Consulting Firm Minkanaku USAID Very Satisfied Yes Yes No Not mentioned 2 SUMITA 100% Intercom Consulting Firm USAID Satisfied Yes Yes Yes Yes 3 CIAPEC LTDA. 100% Receiv ed an e-mail USAID Satisfied Yes Yes Yes Yes 4 PROSOL 80% Univ ersity colleague USAID Very Satisfied Yes Yes Yes Yes 5 Alimentos Boliv ia Natural (ABN) 100% Receiv ed Technical Assistance from USAID USAID Very Satisfied Yes Yes Not y et Yes 6 Noemy Thinking of y ou 75% Ideas Emprendedoras Contest USAID Very Satisfied Yes Yes Not y et No 7 TELLEZ w orkshop 75% Fly er of Ideas Emprendedoras Contest USAID Satisfied Yes Yes Not y et No 8 COSNATVAL Workshop 100% Ideas Emprendedoras Contest USAID Very Satisfied Yes Yes Yes They already hav e a 9 UNEC 100% FTDA-VALLES USAID Very Satisfied Yes N/A Yes Yes 10 INMUNO VIDA 100% Fundacion Nuev o Norte in La Paz. USAID Very Satisfied Yes Yes Not y et No 11 OLEUNS BEAUTY 100% Business Association USAID Very Satisfied Yes Yes Yes Not necessary 12 DERMA-Pharmaceutical Small Lab. ADA 100% FAN, Concurso proy ectos innov adores 2011 USAID Very Satisfied Yes Yes Yes Not necessary 13 PELLETBOL 100% Ideas Emprendedoras Contest USAID Satisfied Yes Yes N/A No 14 Association APROMAJI￾MUYUPAMPA 100% Other USAID Very Satisfied Does not mention Yes Yes Not necessary 15 Agri-Industry CHONCHOCORO Ltd. 100% Prev ious w ork w ith BPC Project USAID Very Satisfied Yes Yes Does not mention Yes 16 Natural Producs White Deserts Ltd. 90% e-mail USAID Very Satisfied Yes Yes Yes Yes 17 Catholic Univ ersity of Boliv ia 53% Prev ious w ork w ith BPC Project USAID Very Satisfied Yes Yes Does not mention Not applicable 18 Producers Assoc. for hot￾peppers & peanuts Municipalities of Padilla and Adilla APAJIMPA-I 100% e-mail USAID Very Satisfied Yes Yes Yes Yes USAID-Bolivia - BPC Evaluation (M&E) Donor Identification and Branding Sustainability (1) Will continue to work (1) The sustainability w as measured by Ev aluator according to a) ex istence and potential of market supply and increasing demand, b) Accounting Sy stems (use some ty pe of accounting sy stem); c) Management Sy stem (HR, pay ments, accounts pay able , receiv ables, social security , sales and basic marketing function and a production structure. (2) Some of the beneficiaries mentioned thay they not required to fill an env ironmental kardex due that they consider that are w orking in fav or of the nature conserv ancy , w ith organic products. SOURCE: BPC files and IBTCI M&E technical Team Public-Private Alliances (PPAs): Implementation Results of Interviews Information Summary Analysis of Personal Interviews and Survey The evaluation team placed additional efforts to find out if the beneficiaries’ were able to separate the complex web of institutions that offered services, the delivery of technical assistance, and the donations to determine what institution was behind all the assistance. Results showed that all 18 enterprises recognized USAID as the donor funding and all expressed to be very grateful with the support received. (Table12). In addition, all interviewed/surveyed MSMEs mentioned that they would sign another PPA agreement while 15 MSMEs (83%) considered they would sign another agreement even if the counterpart contribution were greater 24 than 50%. Two firms (11%) stated possibly signing a new PPA agreement and one (5.5%) did not answer the question. The main reasons for agreeing to a new PPA with USAID were: (a) a limited or complete lack of available resources for very small or small entrepreneurs; (b) any help is appreciated because it has helped them to implement their ideas; and (c) USAID is a more flexible donor although still full of complicated rules, requirements, and paperwork. All interviewed/surveyed MSMEs reported positive aspects of PPAs. The most relevant are: improved productivity, reduced costs, increased sales, higher product quality, good communication with project staff, and a good follow-up of activities from project staff within the alliance. The information clearly shows that entrepreneurs are willing to work with USAID in the future, which is a very good sign of the potential use of PPAs as a mechanism for USAID continued support to the Bolivian business environment. The Effects of PPA Investments (Donations) on current value chains and MSMEs: In using the PPA strategy as a mechanism to strengthen selected value￾chains with the purpose of increasing productivity, the evaluation team observed that 13 MSMEs (72%) mentioned they were able to increase their productivity. Four MSMEs did not have an opinion because it was too early in their operations or in their enterprise life cycle. In one case, signs of effects have a longer timeline since the donation was for 30 university teachers to guide entrepreneurs crystallizing their ideas. This can only count as an indirect service provided to the MSMEs and the value-chains. Nine out of the 18 MSMEs (50%) mentioned that they are more competitive now thanks to the intervention of the BPC. This accomplishment was measured in increased sales, greater production capacity, and increased product diversification, better knowledge of the products, production process automation, and improved ability to meet the customer requirements. All of these made the MSME more equipped in a fully competitive market. Eleven MSMEs (61%) believed that the increase of productivity meant greater competitiveness due to increased profits. Twelve MSMEs (67%) mentioned that the increase in productivity and competitiveness is directly related to an increase in their own demand for goods and services from third party enterprises. But the evaluation team found that many PPAs who had already received machinery, lab and testing equipment and other in-kind donations were not operating fully. The answers from 14 out of 18 interviewed/surveyed MSMEs mentioned that the plans developed for the execution of each PPA had been compiled by the BPC project. But they continue with additional actions for full installation and operation of the machinery donated. Nine enterprises (50%) felt that the PPA had provided them with enough technical and follow-up support to continue their full development and six (33%) mentioned that the support received was acceptable. The increase in sales from five enterprises was between US$ 1,000 and 5,000. In three MSMEs, it was between US$ 6,000 and 10,000. One case each was found in increased between US$ 11,000 and 15,000; US$ 16,000 to 25,000; and exceeding US$ 26,000. In relation to new employment opportunities created, the evaluation of PPA activity revealed the following increased labor demand from MSMEs: (a) Nine MSMEs needed one to five additional employees, (b) One MSME needed six to new 10 employees; (c) One MSME needed 11 to 25 new employees; and (d) One MSME needed 26 to 50 employees. (Photo No. 4 PPA-for Commercial promotion and TA to bring the MSME – furniture – fairs / Tarija- Nov-2012) 25 5. FINDINGS 5.a BPC Relevant information in Institutional Compliance, as of June 30th / 2012: Organizational Structure of Bolivian Productivity and Competitiveness Project: The administrative structure of the BPC project is shown in Annex 3. All personnel were from Bolivia and were charged to accomplish the project objectives and activities as proposed by the USAIDs and mandated by the project ToR For the supply of business services, business training, and technical assistance, the BPC sub-contracted with technical local institutions, NGOs and business-services firms with proven expertise. This ensured some degree of technical know-how and assistance, training, and hand-holding for MSMEs. But at the same time the BPC did contract technical experts to improve the quality of knowledge of those subcontractors in order to improve technical services and to be best-practices at the national levels. Other areas of direct intervention was the “Out-reach” process for procuring and attracting MSMEs interested in research and production of innovative products, which used or incorporated natural raw materials or practiced innovative industrial processes, e.g. PML and EE practices introduced into industrial production. Evaluation findings show that the services BPC provided made it a foundation to be a generator of new ideas, new products. The BPC activities created an environment to bring innovative ideas to fruition, train people (through sub-contractors) to be able and ready when MSMEs were also in need of capable workers. To the extent that these were the overarching goals and objectives of the BPC, the project can be considered effective and efficient to the degree allowed by local circumstances. 26 Table 13: BPC Accomplishments as of June 2012 and 2013 Plans15 BPC Accomplishments as of June/2012 compared to 2013 Plans Indicators FY 2011 Result FY 2011 Goal % Delivered FY 2010-11 Accumulate d Result Goal 2010- 2013 % Delivered 1 Number of people trained 5.190 3.200 162% 6.662 10.300 65% 2 Number of students trained who obtained jobs 469 380 123% 506 1.200 42% 3 Firms assisted in management, administration, and processes 162 110 147% 179 380 47% 4 Number of MIPYMES created 11 11 100% 12 38 32% 5 Number of direct and indirect jobs 1.837 2.574 71% 2.660 7500 35% 6 Value of sales (USD millions) $ 2,74 $ 5,50 50% $ 3,78 $ 17 22% 7 Increased level of productivity 49% 5% n.a. 49% 5% n.a. 8 Economic impact in selected value chains (USD millions) $ 6,84 13,75 50 $ 9,45 $ 43 22% 9 Number of strengthened institutions 0 5 0% 1 15 7% 10 Number of people trained in strengthened institutions 69 46 150% 69 150 46% 11 Number of events for promotion dialogue and commercial alliances 33 22 150% 51 76 67% 12 Value of sales intentions $ 6,33 $ 7,20 88% 13 $ 29 45% 13 Institutions that received support for events and courses 21 6 350% 38 24 158% 14 Number of PPA operations and PPA types 13 4 325% 14 13 108% 15 Counterpart resources in PPA and PPA type (USD millions) 0,58 0,28 205% 0,72 $ 1 72% SOURCE: Provided by BPC project M&E- staff - in La Paz Bolivia 5.1 FINDINGS - Activity: Improved Productivity and Competitiveness of MSMEs and Value Chains An inventory of project information and documentation showed the following to have been completed and submitted to USAID: (a) strategic plans for a four-year period, (b) Diagnostics and Plans for technical interventions in pre-determined Value-Chains-, (c) M&E plans and M&E-manuals of all BPC activities, (d) Services Delivery Fund/Grants Manual, (e) Institutional Strengthening Plans, (f) BPC institutional Communications strategies and plans. The BPC project developed its own M&E tracking system that 15 Performance Monitoring data and figures as Reported to USAID – Bolivia Photo 5: PPA partner CIAPEC Café – El Alto. 27 measured the outputs of every single activity and action in compliance with its contractual requirements. The results and outputs, according to what has been informed to USAID-Bolivia are developed in different matrices as shown below in tables: Table 14: BPC Accumulated Results as of June 2012 INDICATOR Accumulated Results Oct 2009-June 2012 Project Target 2009-2013 % Progress 1. Beneficiaries 68,457 89.760 76% 2. Assisted MSMEs 298 380 78% 3. Sales (MM USD) $ 9,71 $ 15,20 64% 4. Trained Individuals 9,009 10.450 86% 5. Trainees who obtain a job 1.040 1.200 87% 6. Employment 4.548 6.500 70% 7. Sales Intentions (MM USD) $ 19,12 $ 28,80 66% 8. Leveraged Resources (MM USD) $ 1,09 $ 1,00 109% Table 14 shows in the first column the parameters or indicators that the BPC had to monitor during the project’s life, while the second shows accumulated results of three years’ work. Column three shows the expected targets results between 2009 and 2013. The resulting performance, in most cases is above 80%. An analysis of BPC’s work with different types of MSMEs and firm sizes showed that more effort and investment was provided to assist microenterprises and small enterprises, as shown in Table 15. Table 15: BPC: MSMEs assisted by size (June 30, 2012) Size of Assisted Firms FY 2010 FY 2011 FY 2012 (Jun 2012) Accumulated June 2012 % Micro enterprises 10 77 43 130 44% Small enterprises 6 58 46 110 37% Medium enterprises 1 5 10 16 5% Associations 0 22 20 42 14% TOTAL 17 162 119 298 100% Table 16 shows that most of the BPC assistance was provided to firms in Textile and Apparel and Processed Food operations while minimal support was given to businesses in Handicrafts. 28 Table 16: BPC: Technical Assistances by Sector (June 30, 2012) Firms with Technical Assistances in Progress FY 2010 FY 2011 FY 2012 (Jun 2012) Accumulated June 2012 % Textiles & Apparel 17 63 58 138 46% Processed Food 0 26 33 59 20% Bio-commerce 0 18 17 35 12% Wood Manufactures 0 44 11 55 18% Handcrafts 0 11 0 11 4% TOTAL 17 162 119 298 100% BPC’s geographical distribution of its assistance to business, shown in Table 17, closely resembles the overall geographical distribution of Bolivia’s economy, where La Paz/El Alto and Santa Cruz de la Sierra, have the majority of urban and peri-urban economy. Table 17 BPC: Technical Assistance to MSMEs by Region FY 2010 FY 2011 FY 2012 (Jun 2012) Accumulated June 2012 % La Paz 9 47 13 69 23% El Alto 8 37 26 71 24% Santa Cruz 0 48 42 90 30% Cochabamba 0 17 30 47 16% Beni 0 8 7 15 5.0% Potosí 0 1 0 1 0.3% Oruro 0 1 1 2 0.7% Tarija 0 3 0 3 1.0% Total 17 162 119 298 100% An additional type of BPC assistance to MSMEs has been in the area of trade promotional events such as trade fairs, business roundtables, and trade missions. These agreements were delivered to the beneficiaries as planned, as presented in Table 20. 29 Table 18: BPC: Trade Promotion Activities Type of Activity FY 2010 FY 2011 FY 2012 Jun 2012 Accumulated June 2012 % Trade Fairs 11 19 8 38 70% Business Roundtables 2 6 3 11 21% Trade Missions 2 2 1 5 9% TOTAL 15 27 12 54 100% Effectiveness of this activity is defined by the project’s overarching development objective “To contribute to increase MSMEs sales and productivity in urban and peri-urban areas so as to create employment and increase their profitability”. FINDINGS: MSMEs’ Sales, profits, productivity and competitiveness BPC’s major interventions to improve productivity include technical assistance, training, and workforce placement assistance. Before June 30, 2012 BPC provided technical assistance to 298 MSMEs and conducted 256 training events (for MSMEs, workforce insertion and institutional strengthening) taken by more than 6,600 people. BPC defines and measures productivity as the ratio between sales and number of workers before and after its intervention16. Measured it this way; BPC has helped to increase productivity in 29% of the assisted firms. Table 19 presents productivity increases due to BPC among selected value chain sectors. Table 19: BPC: Increased Productivity in MSMEs assisted Value Chain Δ Productivity Processed Foods 41% Textiles and Apparel 28% Wood manufacturing 4% Likewise BPC reports that it has helped MSMEs to reach sales’ increments in the amount of USD 9,705,064. Table 20 breaks down how the sales component behaved in selected production sectors. Table 20: BPC: Increased Sales in MSMEs assisted (USD) Value Chain Δ Sales Processed Foods 1.754.260 Textiles and apparel 4.729.374 Wood manufacturing 2.556.977 Other Chains 586.434 With respect to profits, BPC argues that because sales and productivity have increased, MSMEs profits have also increased. BPC however, recognizes these increases as indirect evidence of profits’ increments due to the project. No systematic survey or other kind of study supports this assertion and profitability is not an indicator regularly monitored by the project. 16 This measurement was proposed by Chemonics and accepted by USAID as valid measure of productivity. 30 A similar outcome occurs in regards to competitiveness in all MSMEs intervened by the BPC because the concept assumes that by increasing sales contributes automatically to improve MSMEs competitiveness. However, measuring competitiveness needs to consider additional variables, most of which are beyond each firms’ outreach. Again there is no survey or any other kind of study that supports any claims in terms of effects in competitiveness and it is not an indicator routinely monitored by the project17. Finally in terms of job generation, BPC reports that its interventions have allowed MSMEs to create 790 direct jobs and 3,478 indirect jobs. Jobs creation is distributed by value chain as follows: Table 21: BPC: Employment generation by MSMEs assisted Value Chain New direct jobs Processed foods 114 Textiles and Apparel 594 Wood manufacturing 52 Other Chains 30 This evaluation team wanted to determine respondents’ perceptions on the effects of BPC on their business in terms of sales, productivity and profitability, and new job generation. Likewise, it intended to ask MSMEs about their satisfaction with both the project and the service providers. Annex 10 shows the results of those interviews with 46 enterprises located in La Paz/El Alto, Santa Cruz de la Sierra, Cochabamba and Tarija. There were MSME interviews in which business owners were not willing to provide specific information about sales or productivity. In other cases, they gladly admitted an improvement but did not know the exact figures. Likewise, there are a few cases in which respondents clearly thought that BPC made no difference. 5.2 - FINDINGS - Activity: Institutional Strengthening to Develop MSMEs Objective No. 2, of the BPC project was to strengthen local institutions, which supplied technical support for MSMEs. Given that the BPC chose to apply an approach to work through local institutions to reach MSMEs and assist their needs, the strengthening of this platform of service-providing institutions was vital to accomplishing a good level of sustainability. Institutions Strengthened through Direct Mechanism (ISDM) The evaluation found three of the ISDMs (CNI, IDEPRO and CPTS) as large, well established institutions with good reputations and permanent staff. Only COTEXBO, an association of textile producers, can be considered as a relatively new institution with no permanent staff. The table below shows where the BPC project was effective in its interventions pursuing to strengthen the capabilities of those four ISDMs. 17 Competitiveness is a concept that goes beyond firm’s limits to determine its relative position to competitors at all times, sectors and locations. It is a product not only of the firm’s performance but, more importantly, the result of a myriad of factors of the business environment, e.g. as infrastructure, workforce, social conditions, education, technological advancement, trust, regulatory and policy frameworks, among many others. There are even positions that argue that there are not competitive firms or sectors whenever the country itself is not competitive. 31 Table 22: Institutions Strengthened through Direct Mechanism Areas of intervention Institutions strengthened through direct mechanism COTEXBO CNI IDEPRO/ CEDETEX CPTS Equipment* X X X X Process improvement X X Training of personnel X X X (*) In COTEXBO, CNI and IDEPRO/CEDETEX office equipment - In the case of CPTS the equipment is a wood￾dryer oven. All four institutions strongly agreed that the effect was very positive and the project helped them increase their capacities to provide technical services to MSMEs because of the equipment provided. The project also led to improvements in their processes by adjusting and increasing their own universe of clients through connections with a broader number of MSMEs. From the information and data gathered about these four institutions, the evaluation team was able to compare the benefits of the direct and indirect channels of strengthening. Results show that all of them have favorable opinions of the indirect mechanism, which is a clear sign of the validity of the approach. For these institutions this approach became very relevant, as observed by this evaluation and reported by the four ISDMs. Strengthened Institution’s Capacities to Manage USAID Concepts and Vision For Institutions Strengthened through Direct Mechanism (ISDM): Two questions in the survey were meant to assess the issue of relevance. First, when asked directly about their impression of how well the institutions assimilated BPC’s concepts and vision, a recurrent answer was that there was only some involvement of BPC personnel in determining what types of technical services should be provided by these institutions. When asked about this specific issue, the two ISDMs (IDEPRO and CNI) answered that it was due to the minimal involvement of BPC’s personnel in technical assistance and training provided (only CNI considered that involvement was higher in the case of commercial promotion services). On the other hand, COTEXBO´s view of the involvement of BPC’s personnel in the technical contents of the training and commercial promotion provided by the former was high, allowing the assimilation of a few aspects of the concepts and vision. Only the CPTS reported that there was no involvement of BPC’s personnel in the contents of the training provided by the institution. Among the sub-objectives of the BPC is the dissemination of how benefited institutions improved in managing USAID’s concepts and vision. To measure such heterogeneous variables the evaluators decided to find out through a test held in a focus group, which revealed that despite BPC’s effort to spread both issues, administrative challenges within the BPC were more relevant. Much time and effort were spent for the process of granting of any services to ISDMs and/or ISIs. Even though this activity is of limited relevance and moderate effectiveness, the capabilities gained by the institutions in the management of USAID concepts and vision may be sustainable because of the low rotation of technical personnel in all four bigger ISDMs and ISIs. Photo 6: shows a community benefited by PPA - Tarija District, Bolivia – Nov. 2012 32 For Indirectly Strengthened Institutions (ISI) - According to the information provided by the institutions in the survey, 59% declared that they assimilated BPC’s concepts and vision, 34% considered that only a few assimilations took place, and 7% responded that there was no assimilation at all (see Table 32). The assimilation was due to the involvement of BPC’s personnel in the services provided by ISIs (either technical assistance, training, commercial promotion, or the facilitation of public-private alliances). As shown in Table 33, the vast majority of the institutions expressed that there was at least some involvement of BPC’s personnel in supervising the quality of all mentioned services. During the interviews, some of the ISIs also expressed that discussions of administrative issues absorbed a lot of their time during contacts with BPC’s personnel. However, small institutions also recognized that they didn’t expect a high level of involvement of BPC’s personnel in their technical activities. That wasn’t the case in larger institutions where the involvement of BPC’s personnel was perceived sometimes even as an intrusion in fields were they think they have enough expertise. Added to the above, the issue related to low rotation of technical personnel (average of 9% rate of rotation a year). But interviews revealed that in the case of small institutions, these rates did not take into consideration the temporal recruitment of specialized consultants who in reality are in charge of most of the technical part of the services provided. Therefore sustainability as a whole can be considered as medium risk. 5.3 FINDINGS: Leveraging Funds through Public-Private Alliances (PPAs) One of the project objectives – was structured as a full-fledged activity of the project to leverage funds from the private sector, other donors, and public institutions using a public-private alliance (PPA) fund. The PPA was considered one of the key vehicles for project implementation with an assigned a budget of US$ 1 Million. The main reason for this approach was that the PPA model puts the market as the driver of the potential solutions for MSMEs’ development and expansion. But in Bolivia, the model places emphasis on building alliances between the public and private sectors to jointly address development objectives. In the specific case of BPC the Project, these alliances were structured between the MSMEs searching for assistance on the private side and USAID providing support on the public side of the equation. This model seemed to better fit the overall complex circumstances in the country and it was meant to solve critical technical and productivity bottlenecks at the plant and administrative levels of the MSMEs, which was common to all the value-chains chosen by the project. At the outset the BPC had determined that more than 95% of all MSMEs’ were in need of risk capital investment for starting-up or graduating from their embryonic stage. The financing or capital participation should normally come from private parties, which is not a common business practice in Bolivia. This factor was determined to be the Achilles’ heel for the MSMEs’ sector based on information from the BPC project and the evaluation team’s findings. PPA activities were implemented directly with local MSMEs using public media and related strategies: (a) participating in competitive processes announced or published in local and national newspapers, and (b) participating in direct proposals announced by the BPC in areas of innovative products and/or production methods or bio￾PML-EE-based industrial production by MSMEs. Moreover the project decided to have a larger and deeper reach through producers’ associations (e.g. Bolivia Institute for furniture Producers, Valles Foundation), Chambers of Industry (e.g. La Paz, Santa Cruz, Tarija), and consulting institutions that could directly offer assistance and services to small communities and MSMEs (e.g. IDEPRO, UNEC). Picture No. 7 apparel MSME received TA from specialized contractor YESMAR in Santa Cruz – Nov. 2012 33 As mentioned by the project staff, the conditions of the working environment for the BPC have not changed since October 2009 when the project’s activities were initiated. The political and social environment remained unchanged from implementation up to 2012 and the platform of strained, semi-neutral diplomatic/political relations (USG-GOB) remained. But the methodology of promoting and helping the incubation and growth of MSMEs through PPAs seemed to become the optimal and most effective tool to accomplish objective number four. But what is more relevant is the effectiveness of this activity and that it can be replicated in similar projects. Integral approach for PPA’s implementation considerations: According to the BPC Project´s Manual, PPAs were meant to solve critical technical, productive or institutional bottlenecks within the various value chains where the project worked. This condition is clearly verified because all MSMEs visited had received donations or machinery, which help solve important technical or administrative bottlenecks. Special efforts were placed in the creation and support of start-ups and new productive enterprises of participating rural and urban producers. The BPC was also very keen on finding and supporting MSMEs which have adopted natural products (bio-products), innovative technologies of production, solutions to existing specialized market opportunities, and solutions to production-level linkages with financial and economic opportunity. Perhaps the second most important condition in qualifying prospects by the PPA activity was how it can be replicable in other circumstances and/or projects across Bolivia and across other strategic sectors. Although the BPC project technical staff and management had insisted on an integral support and approach to PPA development, the reality was different. These objectives were not met and although a large proportion were given technical support (80% of PPAs), the results demonstrated evidence that something failed in this process. More than 90% of the supported MSMEs were well in their way to increased productivity but there was a lack of management and marketing capabilities. Other relevant aspects include:  Two out of 27 MSMEs (7% of PPAs), which chose to introduce PML cleaner production or energy efficient practices, received some but limited technical assistance to complement their implementation and investments. This was also true for the few alliances that had decided to adopt practices for better environmental management of their production processes;  Five out of (5/27) MSMEs (19%) receiving PPAs needed additional guidance for an adequate process for the procurement and purchase of new adequate industrial machinery or equipment in order to identify the proper size and the best suppliers. They also needed assistance in negotiating better prices to meet current legal and tax provisions under USAID and domestic guidelines;  BPC promoted agreements to develop projects in an integral manner, however the business-environment context only allowed a limited application of an “integral development process” because of: (a) Lack of preparedness by businesses which each PPA required, (b) certain level of certainty of the existence and size of a market for the additional production of products as intended by the BPC in order to support any MSMEs; (c) the time required to prepare and implement a PPA with an integral framework, which included pre￾investment stages from the identification of MSME-partner, followed by the signature and implementation of the grant agreement and all other stages of this endeavor; and (d) application of accelerated processes for attraction marketing and then the approval process, and specially to attend those MSMEs-linked to chosen value chains or with innovative ideas. However the comprehensive approach to strengthen these partnerships had been implemented only partially, placing greater emphasis on resolving production bottlenecks and increased productivity. But improvement in sales and other competitiveness factors (the product mix and the market mix of each MSME) felt short on all cases. Special mention should have been placed on issues to improve MSME administration processes and 34 procedures, opening of new markets, developing a new mix of products, and cross-cutting issues such as gender and environmental consideration as part of an improved social conscience. 5.4 BPC and workforce development The BPC project had conducted workforce training events to improve workforce competencies not only to manufacturing workers, but also to middle-management, owners, personnel working for those organizations providing business development services to MSMEs, as well as other stakeholders such as students and potential entrepreneurs. Training has been provided in subjects such as costs calculation, design and fashion trends, pattern making, tailoring, “polleras tailoring”, manual weaving, import-export, fashion cycles, fairs participation, access to European markets, requirements for exporting processed foods, generating business ideas, hygiene practices in dairy products, handmade certification, business plans, technical training for wood processing, apparel and food manufacturing, machines’ operation, wood carving, furniture finishing, bakery and pastry, etc. With the variety in courses, subjects, and participants, positive outcomes were expected for MSMEs in general and the targeted value chains in particular, so the number of people trained was a total of 8.998, considering the detail of trainings provided in the Table 23. 5.5 BPC work in the improvement of the Business Environment BPC has delivered what seems to have been its contractual commitment in this “business environment improvement” initiative. Under the project contract’s definition this was a minor objective due to the limited interaction with the GOB. Furthermore the BPC instead diverted its efforts and resources to the area of Trade Promotion, which seems to have been somewhat successful as it is shown in Table No. 24 (BPC MSMEs Supported Trade Promotion Activities) 6. CONCLUSIONS The results and lessons learned within this evaluation process carry us to a logical, or at less practical conclusions to be answered in the framework of the “evaluation questions”: To what extent have project activities been effective in achieving their intended objectives?: The evaluation of this BPC project for MSMEs development activity, in terms of performance shows very positive effects, reasonable level of adequate outputs, completion of key tasks, although 1 out 4 was results- or limited , and executing components. Its contribution to productivity improvements is well regarded by MSMEs and other stakeholders. Quantifying the economic extent to which it has improved MSMEs and their value chains’ competitiveness remains a challenge, as well as its effect in actually improving the overall business enabling environment; BPC: Workforce training for MSMEs and MSMEs Associations Categoríes Trainees General training courses 5.205 Training for workforce insertion 1.259 Training in the workplace 2.079 Training in the workplace for PPAs 317 Training for institutional strengthening 138 TOTAL 8.998 Table 23: BPC: Workforce training for MSMEs and MSMEs Associations Table 24: BPC MSMEs Supported Trade Promotion Activities Table BPC: Trade Promotion Activities Type of Activity FY 2010 FY 2011 FY 2012 Jun 2012 Accumulated June 2012 % Trade Fairs 11 19 8 38 70% Business Roundtables 2 6 3 11 21% Trade Missions 2 2 1 5 9% TOTAL 15 27 12 54 100% 35 To what extent did the BPC Project contribute to the current SEGE program objectives?: The PPA mechanism allowed the BPC project to meet its objectives to leverage private funds and thus contribute significantly to the fulfillment of project objectives. However, this mechanism could remain only as a model for Bolivian local conditions What are the most relevant elements that contributed to/impeded increase productivity and sales of MSMEs? The most outstanding results were obtained by the implementation and application, limitedly, of the PPA mechanisms because they were the most effective tool in leveraging funds from the private MSME-sector that were also the counterpart partners. However other donors were not interested or were not approach by none of the interested parties. Although the general results of most BPC activities and actions were adequate considering there were a few conditions that impeded its normal development and full success counting among them the current economic and political context in Bolivia and because the public institutions have not been benevolent or collaborative toward this project. How can they be incorporated into future mission strategies?: Business owners placed high value on BPC’s commercial promotion efforts; however they suggested that a long term support program would be more appropriate and less circumstantial. Trade fairs in Bolivia are still a very local promotional mechanism, which could become a very effective tool for commercial promotion; however professional improvement is required. Of course the basis for a sustainable increase in the export market will only be supported by an adequate and effective Quality Assurance System (QAS) and availability of trained workers Over the course of implementing the project, what new best practices and lessons learned were identified or revalidated that should be continued in the Mission’s future programs?: The granting of funds was very effective for the formation of partnerships and also as effective for attracting new and innovative ideas into higher risk endeavors, and also for the introduction of new technologies. Because the Bolivia-PPA-model is extremely flexible and effective to attract the interest of any type of endeavor and lends itself to be applied in new USAID projects How have investments made in strengthening the current value chains contributed to improved competitiveness and productivity of supported sectors? Were they strengthened in an integral approach? Are there other priority sectors that should be supported in the future? BPC should have insisted on the concept of “Integral Technical Assistance” as a necessary practice when delivering technical support, to ensure a balanced and competitive development between the supply and demand sides of the equation, as experienced by the BPC. To what extent have BPC activities strengthened local institutions capacities in being able to provide adequate services to MSMEs, compete for resources, address MSMEs’ issues, and manage USAID funds or concepts? BPC contributed to improve the availability of business services for MSMEs; moreover it contributed to increase the demand from MSMEs for such services and beyond by supporting “institutional strengthening” initiatives of organizations providing such services, which were forced to become more competitive because of increased competition and demand from the MSME-market. In consideration to the fact that conclusions should be linked to each evaluation questions, the evaluation team thinks that it would like to present a table designed to represent or expressed our opinion “in-good-faith” or opinion based on “best-practices” of all and each member of the Evaluation team. The x-axes show the five questions and 11 sub-questions as “variables” to measure. On the y-axes as parameters or indicators it is shown the three evaluation criteria, as well as in the last column the final grade “qualification” calculated for each individual variable versus each criteria. The team used a grading system from one to five: grade 1-poor to 5-Excellent (Table 25, below) 36 Table 25: USAID-BPC Overall Performance Evaluation by Evaluation Technical Team USAID - Bolivian Productivity and Competitiveness Project – M&E overall grading based on findings Grading range: 0 (not complied) to 5 (Excellent) Chart No. 2: BPC EVALUATION: USAID QUESTIONS EVALUATION CRITERIA Evaluators Relevance Effectiveness Sustainability Final Grading To what extent project activities have been effective in achieving their intended objectives 4 4 3 Good To what extent did the BPC Project contribute to the current SEGE (Sustainable Economic Growth and Environment) program objectives? 4 4 3 Good What are the most relevant elements that contributed/impeded to increase productivity and sales of MSMEs. 5 4 3 Good To what extent the GDAs/PPAs mechanisms were effective in leveraging funds from the private sector and other donors? 5 5 4 Excellent Were the GDAs/PPAs mechanisms adequate considering the current economic and political context in Bolivia? 4 5 4 Excellent How can GDAs/PPAs mechanisms be incorporated into future mission strategy 5 5 4 Excellent Over the course of implementing the project, what new best practices and lessons learned were identified or revalidated that should be continued in the Mission’s future programs 5 4 4 Excellent How investments made in strengthening the current value chains contributed to improve the competitiveness and productivity of supported sectors 4 3 3 Acceptable+ Were value chains strengthened in an integral approach? 3 3 3 Acceptable Are there other priority sectors that should be supported in the future? 4 4 4 Good To what extent BCP activities strengthened local institutions capacities in being able to provide adequate services to MSMEs 4 3 3 Acceptable+ To what extent BCP activities strengthened local institutions capacities in being able to address MSMEs issues 3 3 2 - Acceptable To what extent BCP activities strengthened local institutions capacities in being able to compete for resources 4 3 3 Acceptable+ To what extent BCP activities strengthened local institutions capacities in being able to manage USAID funds or concepts. 2 3 2 Weak As Per USAID Development Policies: Environmental Concerns Environment issues were not measured As per USAID Evaluation policies: Gender Concerns Women were benefited indirectly although not planned As a general conclusion the Evaluation team has graded the overall performance of the BPC, as shown in the previous table, is that the BPC answered at a relative acceptable level. And that it is interesting to discover that against all odds it generated a few “Best-Practices”, for cases similar to the circumstances in Bolivia. 7. RECOMMENDATIONS Since the project has been cut short and knowing that there will be a continuation of its activities, at least for some time and different socio-political circumstance, our recommendations will focus on what could be carried or transfer to similar existing projects by USAID. Much knowledge, experience, and best practices have been accumulated by the BPC, therefore, it would be important to ensure that the knowledge gained by the PPA personnel of operations could systematized, and transform into manuals and normal documented practice. Since the BPC attained very positive performance results and outputs making its contribution to industrial MSME￾sector productivity, added to the fact that it created an efficient and well-organized network of institutions and clients, the GOB and USAID will be short-sided not to find ways to extend the survival of this program. Our suggestion is that taking advantage of the political and financial clout the GOB and USAID enjoy they should find a way to collaborate. We recommend that the private sector must convert itself into the buffer (and participate) to facilitate this partnership. If trade fairs are working only at a local level of promotion, the GOB and the Bolivian Private sectors should coordinate to improve in the reach, scope and specialization of several trade fairs, to make them more specialized and more effective, through the professionalization of sectoral staff (Chambers of Commerce); 37 It is time that Bolivia should have a real Quality Assurance System (QAS), and most of all a strong oversight institution that will force the improvement and availability of QAS services, certification, tradability, and other services, hopefully supplied by private firms. This QAS can only be put in place with the full collaboration of all interested parties and, perhaps financed by one of the international development institutions. 8. LESSONS LEARNED Lessons Learned in Promoting MSME-Activities: Value-Chain development as well as any program intended to strengthen them, will require a few long-term costly conditions from any GOB or donor projects. These conditions must include aspects dealing with policies, supporting institutions, improved technical and business environment, etc. However, real possibilities and resources for its implementation are hard to come by. Nationwide sectorial improvement was beyond BPCs possibilities, from its onset, due to the limitations of time and resources. There BPC should have been smaller, but much focused in its area of influence and sector￾dedicated to no more than two or three MSMEs groups. Productivity and competitiveness were the center piece of BPC. Definitely BPC actually contribute to elevate sales of MSMEs. This event had a spill-over effect in each related value-chains. The indicator for this variable was defined and recorded as “increase in sales”. However, many other key variables of competitiveness, e.g. such as public policy, infrastructure, and changes in the workforce market were rarely monitored. In a weak business environment such as the one surrounding MSMEs in Bolivia, any activities in the field of business promotion has no sustainable effect in the near future. The lesson here is that commercial promotion takes much more resources and know how than just an occasional sales opportunity. Lessons Learned from the Institutional Strengthening Component: The evidence shows that some capacities of the implementing institutions (not only administrative ones) were actually strengthened as a consequence of working with the BPC project. This could mean that indirect mechanisms work well. However, indirect mechanisms need a higher level of personal, technical and public relations contact in order to be effective. In terms of institutional size, the BPC experience shows that almost always bigger institutions may be better placed to answer more sophisticated demands to take or supply technical support either to improve its own services or their portfolio of clients. Smaller business support institutions have more needs and are proportionally weaker. Therefore, initial differentiation is important at the outset of the project in order to build equilibrium between using the services of larger institutions and simultaneously strengthening smaller ones to improve competition and supply side variety and quality of business services. Best practices should be applied in the process/procedures to choose institutions for the delivery of business support services. Integral evaluations, analysis and standards must be applied in order to increase the probability of success. However the time and conditions required for approval and implementing PPAs and any support mechanism for MSMEs are inversely proportional to the level of interest and the number of interested beneficiaries for any. In the implementation of PPAs activities a few lessons also emerge: The most evident weakness in any MSME assisted by the BPC is the lack of MARKET research and intelligence of conditions, needs, demand size, how to define it, etc. Concomitantly MSMEs lack the knowledge of MARKETING practices: thinking, planning, strategizing, research, product-mix, marketing-mix, and similar activities; Public-Private-Alliances (PPAs) according to the traditional model were possible in Bolivia only after a special model of PPA arrangements and few adjustments were approved by the contracting institution, where the public counterpart was supplied by the USG; 38 Business people and BPC staff were consulted about the possibility of creating investment funds and/or angel funds; they had the opinion that in Bolivia there were small possibilities of finding public or private institutions that would be willing and interested to invest in these types of funds; The only alliance that seems to fit the original PPA model was established with the Catholic University of Bolivia to support a Master’s Program to help teachers from the Public University of El Alto (UPEA) in obtaining MBAs for promoting the creation and development of MSMEs. This training seems to have included incubation and development concepts for MSMEs. The evaluation showed a high level of performance results. Furthermore, it generated a strong spill-over effect and an apparent high MSME-multiplication-factor, plus a reduction in the risk of failure for the program itself. It should be mentioned that the development stage of small businesses supported by the project through PPAs seemed to have placed the majority of them on solid ground for future real success, or at least, to be viable and strong potential for surviving. The coordination with other USAID projects, particularly “Pro Lago Project,” was effective in identifying future partners. These agreements led to synergies, increased the probability of success in a specific geographic zone (south shores of Lake Titicaca) plus, it generated increased interest from other groups. The most effective and practical source of communication for MSMEs’ was the daily press especially, especially because it attracted a larger audience and competition for grant funds. In second place of effectiveness were any electronic means (e-mail) and/or sectorial institutions (e.g. regional chambers of commerce or industry) were also valuable tools, but not as effective. PPA grants funds seemed to have been the most effective and practical tool to attract new innovative ideas ( e.g. “Ideas Emprendedoras” contest) into business endeavors and also for the introduction of innovation and new technologies, which most likely will shy away from formal financial systems, due to their intrinsic high risk level. Graph 3 below tells us that the Bolivia-PPA-model is extremely flexible and effective to attract the interest of any type of endeavor and lends itself to be applied in any circumstance or stage of MSME development. However it must be properly assigned, adequately structured and fitted the characteristics of the beneficiary and the objectives pursued, and furthermore, well monitored. (e.g. variety of MSMEs in the PPA portfolio) A general lesson is that there is great potential for MSME businesses and commercial opportunities for bio-products, ecology-minded, innovation and new technologies; especially in the food - cosmetics - pharmaceutical industry where Bolivia enjoys comparative advantages, due to its natural diversity, which are not fully developed BPC’s subcontracting strategy for the delivery of specialized services and/or financial evaluations, and/or training services has been a demonstration and tested positive for effective results and delivery of specialized technical assistance. This strategy seems to have several corners of indirect benefits; a) scares available resources were utilized with greater efficiency and effectiveness, b) subcontracting allowed a myriad of large and small institutions dedicated to business services supply (technical, IT, administrative, market development) to strengthen internally and improve the quality of their services thanks to an increased in the demand of services. Figure 3: PPAs Portfolio according to Productive Chain 39 The lesson and implications from the BPC - and PPAs - experiences is that a new project could run a bigger SOW, carry a larger number of alliances, over a longer time period with a smaller size of resources. 9. BEST PRACTICES The BPC project was able to develop a few processes/ procedures that can be defined as “Best Practices” in different areas of MSME, PPA, and institutional development activities. Or, these, at least could qualify as improved ways to achieve its declared goals; especially if applied under the practical realities of Bolivia, including those that have been improved within the project’s own learning curve, and could qualified as “best practices” due to their scalability and transferability potential to other contexts: In the process of trying to overcoming a few challenges and bottle-necks common to most programs for the proliferation and promotion of MSMEs, the PPAs stand-out because they work only if there is pre-existing collaborative attitude, by all interested parties, which motivates them to think beyond their own limitations in terms not only of production technology, but also their markets. The key that unlocks the most difficult “door” or challenge, the start-up stage of a business endeavor is the availability o seed funds provided by donors (or governments). In the BPC the option or model of intervention applied was through the formation of public-private-partnerships. These PPAs provided a reasonable proportion of the initial investment as a grant to solve any bottleneck or weakness of the MSME. The seed funds prompted the beneficiaries to put-up a larger portion from their own resources to be part of the solution. Other areas from the experience of BPC can also be acknowledge as “Best Practices” such as PPAs for innovative proposals and bio-products; the design of plans for “workforce inclusion”; and the application of an integral technical assistance for MSMEs, which, it could at least constitute, as a good lessons to be learned in the Bolivian context The installation of the so called “Little Schools” for workforce insertion: this initiative was implemented as a training model aimed to create qualified employees for when the MSMEs were ready to demand additional labor. They were installed as a theoretical-practical training model for a period of three months. Trainers, MSMEs’ owners and trainees work together to improve workforce, production and real job opportunities according to market demands. In the practical side a sense of commitment developed between all parties involved, which resulted in an improvement of working environments and overall results. To ensure MSMEs proper development to increase its stability any technical assistance delivered must be of Integral Nature. Although usually MSMEs focus their problems in the production area, the reality is usually more complex, therefore sooner or later they will demand further assistance in the areas of management, organization, finances and marketing/sales. Furthermore, although almost repetitive, the area where the evaluation team found major traces of best￾practices was in the Public-Private-Alliance activity. However in this section it will be mentioned for the interesting and, somewhat lean set of procedures that were demanded from all parties, in order to arrive to a PPA agreement, which is shown in Graph 4. However this PPA process also includes a list of procedures that Figure 4: PPA Grant Implementation Process 40 MSMEs have found too burdensome, it showed that many MSMEs were able to cope with it with little guidance. These beneficiaries also felt that they had learned a lesson on how to comply with international requirements and therefore became project-ready for any donor institution. Despite a few shortcomings, the PPA-Bolivia model could easily work for any number of projects and could be replicated in other countries when USAID is displaying efforts to generate work opportunities through the creation and multiplication of MSMEs. 10. FUTURE DIRECTIONS Bolivia seems to be a country of important economic potential due to plentiful natural resources and large numbers of trained professionals and workers. Perhaps as an answer to the numerous attempts by several international governments and development institutions to generate sustainable economic growth in Bolivia, the GOB has adopted a similar systemic approach to the country's competitiveness by introducing legal tools and policies and programs to strengthen the productive sector. The GOB has also adopted the concept of strengthening and developing productive value-chains, because this approach has been considered as the most efficient way to stimulate the productive sector. Since the BPC Project was implemented by prioritizing several key productive value-chains, perhaps a new window of opportunity could be opened if the GOB would accept to place a renovated emphasis on new sectors such as Quinoa, camelid-family textiles (Alpaca-like) (wool and meat), chestnut, banana, palm-hearts, grapes, wines and Singanis (alcoholic spirits) considering their productive, export potential and product added value In terms of sustainability, as suggested by the evaluation both issues should be addressed: (a) since a few individual MSMEs showed measurable market strengths, they should be guided and supported to procure additional financial and technical support. (b) Because the MSME program of the BPC lacked support from the GOB a sequel program should be promoted among donor development institutions and large private sector firms, which must be convinced that it would help to transform their respective value-chains to increase their own competitiveness. Agreement among different key business sectors for these kinds of solutions may prompt the GOB to collaborate in the development of local private sector. Since the BPC is coming to an early closeout, many activities have not achieved completion level, yet. It may be a good idea that few successful activities to be assimilated by other USAID projects. Perhaps BPC-institutional partners could track them and support their completion to improve their corporate image After completion of the review and analysis of PPAs as a financing mechanism for MSMEs, it was observed that this method could be a very good option for some sort of direct cooperation from the people of the United States to final beneficiaries in Bolivia. (Picture No. 8, MSME Pharmaceutical Laboratory – Montero –Santa Cruz – District - Nov. / 2012) 41 Annex 1: Evaluation Statement of Work OBJECTIVE The Bolivian Productivity and Competitiveness (BPC) Project is planned to undergo the final performance evaluation as defined in USAID evaluation policy. The purposes of evaluating the BPC Project is to: 1) evaluate the performance of project results and deliverables from September 30, 2009 through June 30, 2012; 2) highlight lessons learned and best practices developed for micro, small and medium- sized enterprises (MSMEs) and Public Private Alliances (PPAs) that could be replicated; and 3) provide recommendations on strategies to promote PPAs in new programs under the Country Development Cooperative Strategy (CDCS) for the period 2012 through 2017 and in the design of new projects. The evaluation will provide important information of activities that should be taken into account in future projects, methodologies and strategies to be replicated for technical assistance, training and developing PPAs. The evaluation team will carry out the BCP project final performance evaluation, following guidelines of USAID Evaluation Policy: http://transition.usaid.gov/policy/ads/200/220mab.pdf 2. BACKGROUND The main purpose of the Bolivian Productivity and Competitiveness (BPC) Project is to increase productivity and sales of MSMEs in order to generate/sustain employment and increase incomes in urban and peri-urban areas. The project is implemented by Chemonics International Inc, through Task Order No. 8 EEM-I-00-07-00008 for a Total Estimated Cost (TEC) of $US 9.771.913 for a period of four years. The project started in October, 2009 and will end by August 30, 2013. The Contracting Officer’s Representative (COR) for this project is Denise Fernandez. Due to the latest significant budget reductions for the Sustainable Economic Growth and Environment (SEGE) programs, the BPC Project will have an early closeout. The TEC for the project will be reduced by 18% as well as the implementation period from August 2013 to January 2013. Therefore, this performance evaluation is key for capturing best practices and lessons learned during the implementation of the project. The BPC Project`s development hypothesis is that by increasing the productivity and sales of urban and peri-urban MSMEs employment will be generated and/or sustained and incomes would increase. The project by strengthening value chains and developing new and existing market linkages domestically and internationally will generate gains in market transactions and cash sales for MSME’s. In the process, high-value labor intensive products and new market niches will be identified and interventions will improve profitability and increase value at all stages of the chain from producer to consumer. In addition, the BPC Project will strengthen the capacity of local organizations by working through them to provide services to the productive sector, which will also promote initiatives that will improve the business enabling environment and workforce development to support competitiveness. The BPC project specific objectives are as follows: 1) Improve the competitiveness and productivity of selected value chains; 2) Strengthen local institutions that support the development of MSMEs; 3) Promote dialogue between key stakeholders (e.g. private sector organizations, public sector, social organizations) to improve the business environment and competitiveness of value chains and 42 MSMEs; and, 4) Leverage funding from the private sector, other donors and public institutions using public￾private alliance (PPA) funds and other sources of funds. The project works in five value chains (textile & apparel, wood manufactures, processed foods, bio￾products and handcrafts). The main areas of assistance are in the departments of La Paz, Cochabamba, Santa Cruz, Tarija and Chuquisaca. Main areas of project assistance include: technical support to improve productivity and management practices; good manufacturing systems for processed food; support MSMEs to attend trade fairs and, business roundtables; and, training to strengthen the labor force and improve management skills. In addition, the implementation of training courses followed by labor placement of trainees is highly important to respond to businesses' demands of better and improved labor skills. The strengthening of local institutions improves their capacities and methodologies to better assist firms according to their needs. The BPC project also works in collaboration with other USAID projects to increase impact in value chains assisted. The BPC Project responds to the difficulties people in the urban and peri-urban areas face in finding sustainable employment opportunities under current economic conditions, and the negative impact this can have in terms of migration patterns. It is designed as a complement to the more rural-based SEGE Office programs. The BPC Project supports the SEGE Strategic Objective “Improved livelihoods through promotion of sustainable agriculture and natural resources based business and productive opportunities”. The project supports directly the Intermediate Result 3 (IR 3); “Improve competitiveness and productivity of micro, small and medium-sized businesses and their services providers to generate sustainable employment opportunities and increased sales”. In regional terms, Bolivia’s economy has historically been highly vulnerable to international commodity price fluctuations due to the high concentration of non-renewable resource exports. One current challenge for Bolivia, concentrated mainly in private sector is how to diversify the country’s exports into labor-intensive, value-added goods. In December 2008, Bolivia was suspended from receiving U.S. Andean Trade Preference and Drug Enforcement Agreement (ATPDEA) preferences. This suspension has mainly affected the textile/apparel sector by reducing the levels of exports in the sector significantly. 3. STATEMENT OF WORK 3.a. USES OF THE EVALUATION The final performance evaluation of the BPC Project has the following objectives: 1) to learn to what extent the project´s objectives and goals have been achieved 2) to learn to what extent has the development hypothesis being achieved, and if it fed the SEGE, currently Sustainable Development and Environment Office (SDE) results framework; and finally 3) to explain lessons learned and best 43 practices that can be shared throughout the Agency to improve future programs that could be conducted in Bolivia or in other Latin-American countries. The main audience of the evaluation report will be USAID/Bolivia, especially the SEGE team, LAC Bureau and the implementing partner. This evaluation will provide an additional report sanitized in order to share it with the Bolivian Government officials, private sector representatives and other international donors as needed. 3.b. EVALUATION QUESTIONS Please note that Evaluation questions are listed in priority order for USAID/Bolivia: 1. To what extent have project activities been effective in achieving their intended objectives? To what extent did the BPC Project contribute to the current SEGE program objectives? 2. What are the most relevant elements that contributed to/impeded increase productivity and sales of MSMEs. To what extent the PPA mechanisms were effective in leveraging funds from the private sector and other donors? Were they adequate considering the current economic and political context in Bolivia? How can they be incorporated into future mission strategies? 3. Over the course of implementing the project, what new best practices and lessons learned were identified or revalidated that should be continued in the Mission’s future programs? 4. How have investments made in strengthening the current value chains contributed to improved competitiveness and productivity of supported sectors? Were they strengthened in an integral approach? Are there other priority sectors that should be supported in the future? 5. To what extent have BCP activities strengthened local institutions capacities in being able to provide adequate services to MSMEs, compete for resources, address MSMEs’ issues, and manage USAID funds or concepts? 3.c. CURRENT SOURCES OF INFORMATION The evaluation team will be expected to meet with USAID representatives and the BPC technical team to learn about the project achievements and challenges. They will review the contract terms, annual work plans, project PMP, performance reports and other documents as needed. Interviews with key actors in promoting the MSMEs and beneficiaries are essential to capture a complete perspective of the performance of the project. The main project indicators by objective are as follows: Objective 1; Improve competitiveness and productivity of selected value chains. 1.1 No. of trained persons (labor, management and other at technical level). 1.2 No. of students that obtained a job after training. 1.3 No. of firms receiving USG assistance to improve their management, administrative and Technical practices. 1.4 No. of enterprises created with USG assistance. 1.5 No. of sustainable direct and indirect employment generated with USG assistance. 1.6 Amount of sales generated (disaggregated by local or international). 1.7 Increase level of productivity. 1.8 Economic impact on selected value chains. 44 Objective 2; Strengthen local institutions that support the development of MSMEs. 2.1 No. of institutions strengthened with USG assistance. 2.2 No. of persons trained in the institutions strengthened. Objective 3; Promote dialogue between key stakeholders to improve the business environment and competitiveness of value chains and MSMEs. 3.1 No. of events (e.g. workshops, fairs, business round tables and forums) sponsored to promote strategic commercial alliances for BPC Project. 3.2 Value of sales intentions generated from business round tables and other similar events for BPC Project beneficiaries. 3.3 No. of institutions receiving USG assistance. Objective 4; Leverage funding from private sector and other donors and public institutions using GDA and other sources of funding. 4.1 No. of PPAs (Public Private Alliances) implemented with USG assistance. 4.2 Amount of funds leveraged thorough alliances (PPP) with USG assistance. Indicators have a baseline. 3.1 METHODOLOGICAL APPROACH To obtain reliable and scientifically rigorous data that will provide the evaluation of the BPC project with accurate and actionable findings, the evaluation team will apply a combination of quantitative and qualitative approaches in the evaluation. Using a mixed methods approach will allow the team to design and conduct targeted evaluation instruments to answer each evaluation question listed before. 3.1.1. QUANTITATIVE METHODS 3.1.1.1. SAMPLE PLAN Leveraging the monitoring data collected by the BPC Project (to be provided by USAID and/or the implementing partner), the evaluation team will obtain a list of project beneficiaries from USAID/Bolivia or the technical team to have an overall understanding of the beneficiary population. With this knowledge, a sampling plan will be developed to randomly select participants. The evaluation team will apply the necessary techniques such as geographic clustering or proportional sampling by value chain sector if needed to ensure that the obtained sample is representative of the beneficiary population. The evaluation team will also inform and work with USAID/Bolivia in determining the appropriate sample size based on the beneficiary population and acceptable sampling parameters (i.e. confidence levels and confidence intervals). To have an adequate number of observations for the methods presented in the Analysis section below, the evaluation team will conduct at least 200 surveys. The target sample size is the most convenient for determining the significance of outcomes (e.g. inference testing) based on the proposed methods below. The evaluation team will develop a sampling plan, applying the necessary procedures of randomization, clustering, and stratifying if needed, and will work with USAID on feasible approaches given the sampling parameters. 45 3.1.1.2. SURVEYS The evaluation team will build on existing instruments used by implementers to collect monitoring data for the main indicators of the BPC project. Indicators on improved competitiveness and productivity (Objective 1), local institutions strengthening (Objective 2), dialogue promotion (Objective 3), and PPPs (Objective 4) are collected and the evaluation team will develop an instrument collecting the same information to conduct a survey but on sub-sample of beneficiaries to be surveyed, as detailed in the sample plan above. Given that the evaluation is also expected to commence in September 2012, the planned survey can also be a proxy or snapshot of preliminary third quarter performance for 2012. The survey will also have a module to collect information on beneficiary perceptions, beliefs, and attitudes towards various project components including, but not limited to, management, implementation, technical support, and forging local partnerships. As specified in the RFQ, the module will also explore business constraints and its sources, if any, identified by MSMEs and other stakeholders. 3.1.1.3. ANALYSIS The integration of BPC Project monitoring data will allow the evaluation team to conduct analyses with an expanded time frame beyond the period of the final evaluation. The project collects data on a quarterly basis since its implementation in September 2009. The most recent round of collection is from the March 2012 quarterly report but data up to June may already be available by the time the evaluation starts. Thus, a set with up to 12 rounds of collected (cross-sectional) data will be constructed. 46 Table 1: Illustrative Time Series Data of BPC Project Indicators Period Time Amount of local No. of trained No. of trained X Sector period sales, USD persons (labor) persons variable (management) 4Q 2009 1 36,000 105 33 1 1Q 2010 2 37,080 108 34 1 2Q 2010 3 38,192 111 35 1 3Q 2010 4 39,338 115 36 1 4Q 2010 5 40,518 118 37 1 1Q 2011 6 41,734 122 38 1 2Q 2011 7 42,986 125 39 1 3Q 2011 8 44,275 129 41 1 4Q 2011 9 45,604 133 42 1 1Q 2012 10 46,972 137 43 1 2Q 2012 11 48,381 141 44 1 3Q 2012* 12 48,381 141 44 1 4Q 2009 1 39,832 101 32 2 1Q 2010 2 41,027 104 33 2 2Q 2010 3 42,258 107 34 2 3Q 2010 4 43,526 110 35 2 4Q 2010 5 44,831 114 36 2 1Q 2011 6 46,176 117 37 2 2Q 2011 7 47,561 121 38 2 3Q 2011 8 48,988 124 39 2 4Q 2011 9 50,458 128 41 2 1Q 2012 10 51,972 132 42 2 2Q 2012 11 53,531 136 43 2 3Q 2012* 12 55,137 140 44 2 Etc. Legend: Sector 1: Textile, Sector 2: wood manufactures, etc. X: A vector of other variables (e.g. No. of enterprises created with USG assistance) *Evaluation survey data The availability of data on multiple periods will allow the evaluation team to apply effective and relevant methods for analysis. One example is time-series analysis, which examines data measured in regular periods and over fixed intervals. Table 1 provides a hypothetical example of time-series data using BPC project indicators. It can clearly be seen that factoring in data from previous rounds yields a richer data set with more observations compared to using only more recent data for a final performance evaluation. This expanded data set will provide the evaluation with more precise measures for statistical analysis and more degrees of freedom for testing the significance of outcomes (e.g. hypothesis or inference testing). The evaluation team will also be able to conduct more targeted time-series analyses to answer specific evaluation questions. To explore the effectiveness of project activities (Evaluation Question 1), the evaluation team will conduct trend analyses, which will examine the different project indicators over time to identify patterns of convergence (or divergence) of activity outcomes towards objectives. To identify the most relevant elements that contributed or impeded productivity increases (Evaluation 47 Question 2), the evaluation team will apply methods that can identify attribution or possibly even causality. An illustrative approach is to conduct correlation tests between identified elements, such as the number of PPAs implemented with USG assistance, and a productivity indicator, such as amount of generated sales. Resulting correlation coefficients will determine if that element has a positive or negative relationship with productivity. The same approach can be applied to evaluate how investments made in value chains contributed to competitiveness and productivity (Evaluation Question 4). Where possible, data collected will be disaggregated by gender, enterprise size (e.g. by sales or employment), and geography (e.g. department). 3.1.2 QUANTITATIVE METHODS 3.1.2.1. DOCUMENTARY REVIEW Upon award, the evaluation team will conduct a review of relevant BPC Project documents including, but not limited to, contract terms, annual work plans, the performance management plan (PMP), PPA reports, and performance reports. This review will provide the team with a comprehensive understanding of the project as well as the various local and contextual elements that affect it. This task will ensure that the team’s approach to the evaluation is relevant to local conditions and responsive to the scope of work. 3.1.2.2. KEY INFORMANTS INTERVIEWS The evaluation team will conduct key informant interviews with USAID and the project staff or technical team to learn about project achievements and challenges. Interviews will also be conducted with subcontractors, and grantees of the BPC project to get insights on project design, implementation, management, and other areas of operations (Evaluation Question 3). A third set of interviews will be conducted with a randomly selected group of MSMEs and stakeholders to understand the beneficiaries’ perspectives on the project’s implementation, particularly in activities aimed to improve productivity and competitiveness (Evaluation Question 4). In all interviews, questions on how partnerships, alliances, and dialogues were formed under the project’s PPA mechanisms to assess how effective they were in leveraging funds from the private sector and other donors. Interview protocols will be designed to collect relevant and verifiable information. Each interview is expected to take one hour to complete. 3.1.2.3. FOCUS GROUP DISCUSSION Focus group discussions, targeting up to 20 participants in each, will also be held among a group of beneficiaries and stakeholders. Discussion agendas will be developed to gather information on perceptions and opinions related to project activities, performance, and results. Attention will be given on how activities strengthened local institutions’ capacities in providing adequate services to MSMEs and providing support in addressing MSME issues (Evaluation Question 5). The data will be collected and analyzed to identify trends in responses and any significant outcomes. Participants on this task will include MSME beneficiaries in the production sectors of textile and apparel, wood manufactures, processed foods, bio-products, and handicrafts to ensure representation along the value chain dimension. Discussions will take two hours to complete and will be conducted in La Paz, Cochabamba, and Santa Cruz to achieve programmatic and geographic representation but balancing it with local conditions on travel and logistics. The evaluation team will coordinate with USAID/Bolivia on areas for selection taking into consideration which specific areas are most suitable in terms of logistics, travel requirements, local willingness to participate, and other relevant factors. 48 3.2 SCHEDULE OF WORK Table 2 below details the proposed number of days needed for each evaluation team member to complete all tasks of the approach. Team Leader MSME Specialist Institutional Strengthening Specialist Phase 1: Document review 4 3 3 Phase 1: Data collection 21 21 21 Phase 3: Data analysis 5 5 5 Phase 4: Develop Draft Report 3 3 3 Phase 5: Develop Final Report 2 1 1 Total LOE 35 33 33 The following is an illustrative timeline of how BPC evaluation will be conducted, detailing the tasks to be performed. 49 3.3 THE EVALUATION TEAM COMPOSITION The team will be conformed of at least one international consultant and two local consultants to work on the evaluation team.  Team Leader: (Mr. Milton Nunez Garcez) o Independent external consultant with a Phd or Master’s degree in Business Administration, Economics or related field, is highly recommended that has postgraduate education on evaluation, evaluation methodologies or related field. o At least five years of senior level experience working in programs in the LAC region, related to: Competitiveness, support of MSMEs dedicated to the production of value added products and local institutional strengthening. o At least five of extensive experience performing performance evaluations of USAID and/or other donor programs. o Excellent oral and writing skills in English and Spanish (level 3/3) o At least three years of experience leading evaluation teams and preparing high quality evaluation documents, demonstrated by past performance evaluation letters. o Should have wide experience in the implementation of USAID- funded programs and good understanding of project management and financing.  MSME Specialist (Alejandro Bernal) o Graduate degree in Business Administration, Economics, Commercial Engineering or other similar degree. o Ten years of experience working with MSMEs in developing countries in LAC, preferably in Bolivia. o S/he should be knowledgeable in program assessment and evaluation methodologies in organizational and capacity building. o Experience in developing public private alliances in Bolivia or other LAC counties.  Institutional Strengthening Specialist (Sergio Villarroel Bohrt) o Graduate degree in Business Administration, Economics, Commercial Engineer or other similar degree o Seven years of extensive experience working with local organizations to strengthen their service capacities to the productive sector in Bolivia and/or other LAC countries. 3.4 DELIVERABLES AND TIMEFRAMES The following deliverables will be delivered: • Work plan: During the evaluation process the team will prepare a detailed work plan. The work plan will be submitted to the COR after four days of signing the contract. • Written Evaluation Design: A written evaluation design will be discussed with USAID prior to implementation; this evaluation design will be shared with USAID COR prior to implementation of evaluation by evaluation team within country. Also tools to gather data will be presented and discussed as necessary with USAID COR. 50 • Discussion of Preliminary Draft Evaluation Report: The team will submit a draft report on findings and recommendations to the USAID COR, who will provide preliminary comments prior to the Mission debriefing and before departure of the consultant’s team leader. • Draft Evaluation Report: A draft written report of the findings and recommendations should be submitted to the USAID COR, prior to the team leader exiting Bolivia. The draft report should clearly include findings, conclusions and recommendations. • Final Report: The team will submit a final written report that incorporates the Mission’s comments and suggestions no later than five days after USAID/Bolivia provides written comments on the draft evaluation report. 3.5 LOGISTICAL SUPPORT AND CONTENTS The evaluation team is responsible for arranging all logistical support for the evaluation. 3.5.1. FINAL EVALUATION REPORT The draft and final report should be presented both; in English and Spanish and must contain at a minimum the following: 1) Executive summary – Concisely state the most relevant findings and recommendations. 2) Table of contents. 3) Introduction – Purpose, audience and synopsis of task. 4) Background – Brief overview of the SEGE strategic objective and the SDE program in Bolivia, the BPC project results framework, development hypothesis, USAID program strategy and activities implemented in response to the problem, and brief description of the evaluation and its purpose. 5) Methodology – Describe the applied evaluation methodology, including constraints and gaps. 6) Answers to the key questions. 7) Best practices found – From a technical perspective describe the best practices identified that contributed to the sustainability of the BPC project. 8) Lessons learned - Describe the lessons learned about what contributed to/impeded the effective implementation of the BPC project. 9) Findings/conclusions and recommendations – All findings should be documented including spot checks verification, data quality and reporting systems. 10) Issues – Provide a list of key technical and/or administrative issues, if any. 11) Future directions. 12) Attachments 13) References – Bibliographical documentation consulted. 14) Required annexes are: “a statement of differences” (regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team). 51 15) Annexes -- Evaluation methods, evaluation tools: questionnaire templates, schedules, interview list, focus group’s discussion. The final report will be submitted to USAID in three hard copies as well as electronically. The report should be restricted to Microsoft products and 12 point – type. While preparing the final evaluation report, the contractor must consider the principles of the document “Criteria to Ensure the Quality of the Evaluation Report” (Attachment No.1) 52 Annex 2: General Conceptualization of Evaluation Approach vs. Project Objectives Project Objective Project Indicators Evaluation Question Methods of Data Collection (Performance indicators) Quantitative Qualitative Objective 1: Improve competitiveness and productivity of selected value chains. Better stated as: Improve competitiveness and productivity of MSMEs of five value chains. (In agreement with USAID and Chemonics only three value-chains will be tested for practical purposes) (This objective may be construed to refer to value chains, as well as individual MSMEs) 1.1- No. of trained persons (labor, management and other at technical level). 1.2- No. of students that obtained a job after training. 1.3- - No. of firms receiving USG assistance to improve their management, administrative and technical practices. 1.4- No. of enterprises created with USG assistance. 1.5- No. of sustainable direct and indirect employment generated with USG assistance. 1.6- Amount of sales generated (By local or international). 1.7- Increase level of productivity. 1.8- Economic impact on selected To what extent project activities have: Been effective in achieving their intended objectives? Contributed to the current USAID￾SEGE (Sustainable Economic Growth and Environment) program objectives? Review of project reporting and data: - Indicators of performance, de Activities and Context (See M&E Manual for the BPC project) Project Performance Reports MSME’s interviews MSME’s surveys Project baseline At least 2 visits to MSMEs of each sector Business Associations interviews * MSME’s surveys of a relevant sample - Indicators of performance, de Activities and Context (See M&E Manual for the BPC project) Project Performance Reports Focus Groups with MSME’s of the five sectors in the program) Focus Groups with students Focus Groups with local institutions 53 value chains. Objective 2: Strengthening local institutions that support the development of MSMEs 2.1- No. of institutions strengthened with USG assistance 2.2- No. of persons trained in the institutions strengthened To what extent BPC activities strengthened local institutions capacities in being able to: 1- Adequate services for MSMEs 2- Compete for resources 3- Address MSMEs issues 4- manage USAID funds or concepts Review and test: - Indicators of performance, of Activities and Context (See M&E Manual for the BPC project): Project performance reports Test sample of institutions for integrity of reporting Institutional Reports on events, training, staffing, etc. * Surveys sample to be defined, mix, location, etc. with USAID and Chemonics, in-situ Review and test: - Indicators of performance, of Activities and Context (See M&E Manual for the BPC project): Interviews and focus-groups with management of benefited institutions Institutional Reports on events, training, staffing, etc. Objective No. 3: Promote dialogue between key stakeholders (e.g. private sector organizations, public sector, social organizations) to improve the business environment and competitiveness of value chains and MSMEs 3.1- No. of events (e.g. workshops, fairs, business round tables and forums) sponsored to promote strategic commercial alliances for BPC Project. 3.2- Value of sales intentions generated from business round tables and other similar events for BPC Project beneficiaries. 3.3- No. of institutions receiving USG assistance Answers to USAID question No. 2. - To what extent the PPAs mechanisms were effective in leveraging funds from the private sector and other donors? 5.- The extent BPC activities and investment strengthened local institutions capacities in being able to provide adequate services to MSMEs Review and test: - Indicators of performance, of Activities and Context (See M&E Manual for the BPC project): Project performance reports Test sample of institutions for integrity of reporting Institutional Reports on events, training, staffing, etc. * Surveys sample to be defined, mix, location, etc. with USAID and Chemonics, in-situ Review and test: - Indicators of performance, of Activities and Context (See M&E Manual for the BPC project): Interviews and focused groups with management of benefited institutions Institutional Reports on events, training, staffing, etc. 54 Objective No. 4 Leverage funding from the private sector, other donors and public institutions using public-private alliance (PPA) funds and other sources of funds Indicators: 4.1 - No. of GDAs (Public Private Partnership) implemented with USG assistance. 4.2 Amount of funds leveraged thorough alliances (PPA) with USG assistance. Answers to USAID question No. 2. - To what extent the PPAs mechanisms were effective in leveraging funds from the private sector and other donors? 5.- The extent BPC activities and investment strengthened local institutions capacities in being able to provide adequate services to MSMEs Review and test: - Indicators of performance, of Activities and Context (See M&E Manual for the BPC project): Project performance reports Test sample of institutions for integrity of reporting by Surveys Institutional Reports on events, training, staffing, etc. * Surveys sample to be defined, mix, location, etc. with USAID and Chemonics, in-situ Review and test: - Indicators of performance, of Activities and Context (See M&E Manual for the BPC project): Institutional Reports on events, training, staffing, etc. 55 Annex 3: BPC Organizational Structure ORGANIGRAMA   PROYECTO DE PRODUCTIVIDAD Y COMPETITIVIDAD ‐ BOLIVIA PROYECTO PC‐B ESPECIALISTA EN MONITOREO Y EVALUACIÓN José Montaño MONITOREO Y EVALUACIÓN, SEGUIMIENTO CADENA DE VALOR ALIMENTOS Carla Cáceres ESPECIALISTA EN COMUNICACIÓN INSTITUCIONAL, SEGUIMIENTO CADENA DE VALOR ARTESANIAS Ximena Jauregui ASISTENTE DE COMUNICACIÓN INSTITUCIONAL Marianela Escobar ESPECIALISTA EN CONTRATOS Y FORTALECIMIENTO INSTITUCIONAL Silvana Lara ASISTENTE DE ADMINISTRACIÓN, FINANZAS Y CONTRATOS Eliana Roca JEFE DE EQUIPO Walter Nú ez GERENTE TÉCNICO/ESPECIALISTA EN CADENAS DE VALOR Y MIPYMES L is Pard SUB-JEFE DE EQUIPO/GERENTE DE FINANZAS Y OPERACIONES J h Carrasc ESPECIALISTA EN CAPACITACIÓN Y FORTALECIMIENTO LABORAL, SEGUIMIENTO CADENA DE VALOR TEXTILES Cecilia Segovia ASISTENCIA TÉCNICA A LA CADENA DE VALOR DE MADERA CADEFOR + OTROS SUBCONTRATISTAS ASISTENCIA TÉCNICA A LA CADENA DE VALOR DE TEXTILES IDEPRO + OTROS SUBCONTRATISTAS ASISTENCIA TÉCNICA EN PRODUCCIÓN MAS LIMPIA CPTS ASISTENCIA TÉCNICA A LA CADENA DE VALOR DE ALIMENTOS/BIOPRODUCTOS VARIOS SUBCONTRATISTAS Y DONATARIOS CAPACITACIÓN ICAPS, ePC—Gerencia CONTADORA Carmen Arnez ASISTENTE DE CONTABILIDAD Adhemar Aparicio SOPORTE TICs Peter Luna ASISTENTE ADMINISTRATIVA Shirley Cayoja RECEPCIONISTA Ingrid Lundgren CHOFER José Luis Quinteros SERENO Leonardo Cori 56 Annex 4: MSMEs Interviewed List BPC’s Evaluation: MSMEs Visited and Interviewed Name Service Received Sector City Date 1 ALTEXBOL TA Apparel El Alto 11/6/2012 2 COMBITEX TA Apparel El Alto 11/6/2012 3 JHOSELINE TA Apparel El Alto 11/6/2012 4 ALIANZA TEXTIL TA Apparel El Alto 11/6/2012 5 CALZART BOLIVIA Training Leather La Paz 11/6/2012 6 TEJIDOS MILANI Training Apparel La Paz 11/7/2012 7 PUCARA AT Processed Foods El Alto 11/7/2012 8 PICOS BILLAR AT, CP Furniture El Alto 11/7/2012 9 CARPINTERIA TECTO AT, CP Furniture El Alto 11/7/2012 10 AMASC - CHILUYO PPA Bio-products El Alto 11/7/2012 11 SILLONES CORONA TA Furniture La Paz 11/8/2012 12 MAESTRANZA HERCAR Training Furniture La Paz 11/8/2012 13 ALIMENTOS SALUDABLES TA Processed Foods La Paz 11/8/2012 14 DAYLAYTEX TA, WI Apparel El Alto 11/8/2012 15 ECOLOGIZATE TA, CP Processed Foods La Paz 11/9/2012 16 JOHN PACHECO Training, CP Apparel La Paz 11/9/2012 17 DECO OFFICE CP Furniture El Alto 11/9/2012 18 MUEBLES SARMIENTO TA, CP Furniture La Paz 11/10/2012 19 MUEBLES JEBEMA (Assoc) PPA Furniture El Alto 11/10/2012 20 MADECOR WI Furniture El Alto 11/10/2012 21 COTEXCO TA Apparel Cochabamba 11/12/2012 22 SPORT BOYS TA Apparel Cochabamba 11/12/2012 23 WILLIAM'S JEANS TA Apparel Cochabamba 11/12/2012 24 BENEGY CP Furniture Cochabamba 11/13/2012 25 COOP. FOTRAMA CP Alpaca Textile Cochabamba 11/13/2012 26 VANER Q CP Furniture Cochabamba 11/13/2012 27 MUEBLES ARCADIA TA Furniture Cochabamba 11/14/2012 28 DANATELI CONFECCIONES Training, CP Apparel Cochabamba 11/14/2012 29 VIVE SANO CP Processed Foods Cochabamba 11/15/2012 30 G-7 TA Apparel Santa Cruz 11/12/2012 31 PATRA TA Apparel Santa Cruz 11/12/2012 32 ALTIMAD TA Furniture Santa Cruz 11/12/2012 33 HERLOZ TA Furniture Santa Cruz 11/12/2012 34 TECNOALIMENTOS TA Processed Foods Santa Cruz 11/12/2012 35 IDELCUSI PPA Bio-products Santa Cruz 11/12/2012 36 LIDERSPORT TA Apparel Santa Cruz 11/13/2012 37 ICOMTEX TA Apparel Santa Cruz 11/13/2012 38 MUEBLES OSINAGA TA Furniture Santa Cruz 11/13/2012 39 UNIARTE (Artisans' Assoc.) Training, TA Handicrafts Santa Cruz 11/14/2012 40 MUEBLES CARVAJAL CP Furniture Santa Cruz 11/14/2012 41 INDUSTRIAS ALIMENTA TA, CP Processed Foods Santa Cruz 11/15/2012 42 MANJAR DE ORO TA Processed Foods Santa Cruz 11/15/2012 57 43 ARTE HOGAR (Artisans' Assoc.) CP Handicrafts Tarija 11/12/2012 44 MONTANA CP Furniture Tarija 11/12/2012 45 IMTAR R&J CP Furniture Tarija 11/12/2012 46 TAJSARA (Artisans Assoc) TA Handcrafts Tarija 11/12/2012 Source of information: BPC and IBTCI – M&E Team from field work, Nov/2012 58 Annex 5: List of People Individually Interviewed BPC’s Evaluation: Persons interviewed Name Position 1 Walter Núñez BPC’s Team Leader 2 José Montaño BPC’s Technical Manager 3 Luis Prado BPC’s Monitoring & Evaluation Specialist 4 Ximena Jáuregui Paz BPC’s Institutional Communications Specialist 5 Denise Castrillo de Fernández USAID/Bolivia BPC’s Program Manager 6 Lourdes Ximena Rodríguez USAID/Bolivia Monitoring and Evaluation Specialist 7 Giovana Chavez Nuñez Cámara Nacional de Industrias: Responsible for MSMEs 8 Fernando Hinojosa García Cámara Nacional de Industrias: General Manager 9 Emilio Gutiérrez Apóstrofe SRL: Team Leader 10 Henry Casana Apóstrofe SRL: Marketing and Exports Specialist 11 Martín Claure Pantoja Yesmar Consultores: Team Leader 12 German Tarqui Zelada Milani Tejidos: Owner 13 Olga Zerda Renjífo Muebles Santos: Owner 14 Willan Zambrana Muebles Benegy: General Manager 15 Vania Felipez Vaner Q (Muebles): Owner 17 Gustavo Sejas Fotrama: Production Engineer 18 Simón Pedro Padilla COTEXCO: Commercial Manager 19 Nicolás Carvajal Muebles Carvajal: Owner 20 Mario Osinaga Mueblería Osinaga: Owner 21 Eduardo Dabdoud Paz Indelcusi: Owner 22 Jorge López M. Herlios Muebles: Owner 23 Amelia Solórzano de Herrera Patra Ropa Deportiva: Owner 24 Katia Ribera Tecnoalimentos: Owner 25 Mirko Aramayo HiQ Alimentos Saludables: Owner 26 Félix Rubén Sarmiento Muebles Sarmiento: Owner 27 Marcelo Vásquez Torrez Piccos Fábrica de Billares: Owner 28 Rolando Saire Gómez Calzart Bolivia: Owner 29 Miguel Charupá Tamacoine UNIART, Artisans Union: General Coordinator 30 Filiberto Condori Ecologízate: Owner 31 Steve Nacif Manjar de Oro: Owner 32 David Choque Altexbol: Owner 33 Freddy Pajsi Combitex: Owner 34 Juan Carlos Jerónimo Jhoseline: Owner 35 Miguel Mamani Alianza Textil: Owner 36 Celestina Cora Pucara: Owner 37 Alejandro Calatayud Carpintería Tecto: Owner 38 Miguel Condori Calisaya Asociación Multiactiva Agropecuaria San Carlos: Manager 39 Hernán Illatarco Sillones Corona: Owner 40 Iblin Carvallo Maestranza Hercar: General Manager 41 Franz Siñani Daylatex: Owner 42 John Pacheco Atelier John Pacheco: Owner 43 Juan José Munguía Deco Office: Owner 59 BPC’s Evaluation: Persons interviewed Name Position 44 Félix Quispe Sánchez Madecor: Owner 45 Jesús Tarqui Muebles Jebema: Owner 46 Carlos Alcalá Sport Boys:General Manager 47 William Nina Quispaya William’s Jeans: Owner 48 Rolando Codima Saavedra Muebles Arcadia: Owner 49 Mónica Jaldín Coimbra Danateli: Owner 50 Jaime Sánchez Pérez Vida Sano: General Manager 51 Sergio Escalante G7: Owner 52 Felipe Algarañaz Altimad: Owner 53 Clemente Churata Lidersport: Gerente 54 Pablo Meneses Icomtex: Owner 55 Jorge Taborga Industrias Alimenta: Owner 56 Sofia Mayu ArteHogar Handcrafts: Employee 57 José Antonio Arce Montana: Owner 58 Roberto Castañón IMTAR R&J: Owner 59 Yeny Ayllon TAJSARA: General Manager Source: BPC and IBTCI Evaluation Team-from field work instruments (Nov.2012) 60 Annex 6: Summary of Evaluation Findings in Directly Strengthened Institutions (ISDMs) Assessment Justification Assessment Justification Assessment Justification Environment Gender - Improvement in institution's capacities to provide adequate services to MSMEs High Mainly due to equipment acquired Also thanks to experience gained and adjustment of own techniques while working as implementing agencies Low Only 4 institutions strengthened, 6 lived behind and 5 didn’t considered to reach the goal of 14 High 3 are big institutions with capacities to raise funds and charge for services - Improvement in institution's capacities to compete for resources Medium Only if we take into account the institution s curriculum-enrichment (new equipment and more experience, the later as a consequence of working as implementing agency), because they already had some capacities to compete for resources Unknown Hard to measure in absense of credible and realistic baseline and goal of funds to be raised on a competitive basis High Low rates of rotation of personnel and well instaled capacities give a durable advantage in competing for resources - Improvement in institution's capacities to address MSMEs issues High Thanks to more contact with MSMEs while providing services as implementing agencies, not due to the strengthening component per se High The institutions gained the capacity even though it wasn’t planned as part of the strengthening component Medium Industrial changes are very dynamic Therefore, in absence of continuous contact with MSMEs, strengthening components alone (equipment and processes) cannot guaranty the sustainability of the capacities gained - Improvement in institution's capacities to manage USAID funds High Training provided to the administrative personnel of the institutions and direct contact through telephone calls, e-mails, meetings and visits, were very effective High 92% achievement in Indicator 2 However, the percentage has been obtained considering all implementing institutions, not only the strengthened ones through direct channel High Rotation of administrative personnel is low in the three biggest institutions Besides, they considered that the administrative requirements of the BPC were normal, showing a good cpacity to handle resources provided by foreign donors - Improvement in institution's capacities to manage USAID concepts and vision Medium Only some involvement of BCP personnel in technical characteristics of services provided by the institutions Low Discussion of administrative issues absorbed a lot of time during contacts between BCP and the implementing institutions High In the three biggest institutions, the rotation of technical personnel is very low Effectiveness (planned vs. accomplished) Sustainability Always a priority for BCP and clearly established as a prerequisite from the very beginning Depending on each institution’s policies Institutions Cross cutting issues Evaluation results according to predefined criteria Relevance (cause-effect) 61 Annex 7: Summary of Evaluation Findings in Indirectly Strengthened Institutions (IDIs) Assessment Justification Assessment Justification Assessment Justification Environment Gender - Improvement in institution's capacities to provide adequate services to MSMEs High Thanks to experience gained and adjustment of own techniques while working as implementing agencies Conditional probability corroborates this conclusion High Statistics shows that the results were better than planned for a specific group of institutions that worked intensively with BCP, signing many SBK, grants or MoU Medium The effect varies according to the size of the implementing institution (the bigger the institution the higher the chance for more sustainability) - Improvement in institution's capacities to compete for resources Medium Curriculum-enrichment in the case of small instittuions Unknown Hard to measure in absense of credible and realistic baseline and goal of funds to be raised on a competitive basis Low In the long run, investment in equipment and more permanent personnel is needed - Improvement in institution's capacities to address MSMEs issues High A high percentage of institutions responded in the questionnaire that their abilities to identify MSME’s needs and understand the functioning of the value chains had improved Contact￾intensity exists but is less evident High Considering that initial expectations were not so high given the indirect channel used Medium New capacities need to be constantly updated trough more contact with MSMEs Institutions that had medium to high contact￾intensity with the BPC would be better equipped to gain certain level of sustainability, but only if they take advantage of their abilities to compete for resources - Improvement in institution's capacities to manage USAID funds High Training provided to the administrative personnel of the institutions and direct contact through telephone calls, e-mails, meetings and visits, were very effective High 92% achievement in Indicator 2 (considering all implementing institutions) High In general, the rotation of administrative personnel in the institutions is very low - Improvement in institution's capacities to manage USAID concepts High Only some involvement of BCP personnel in technical characteristics of services provided by the institutions Medium Small institutions recognized that they didn’t expect a high level of involvement of BPC’s personnel in their technical activities, so the positive collateral effect was seen as good Medium Technical personnel in small institutions are mostly consultants hired on a temporary basis Always a priority for BCP and clearly established as a prerequisite from the very beginning Depending on each institution’s policies Institutions Evaluation results according to predefined criteria Relevance (cause-effect) Effectiveness (planned vs. accomplished) Sustainability Cross cutting issues Source: Annex prepared by IBTCI- Evaluation Team, Nov. 2012 62 Annex 8: Institutional Strengthening Methodology Plan (Source: IBTCI Evaluation Team, Nov. 2012) Project Objective Project Indicators Evaluation Question Methods of Data Collection Quantitative Qualitative Objective 2: Strengthening local institutions that support the development of MSMEs. Indicator 1 No. of institutions strengthened with USG assistance Indicator 2 No. of persons trained in the institutions strengthened To what extent BPC activities strengthened local institutions capacities in being able to:  provide adequate services to MSMEs  surveys in the four institutions strengthened through direct mechanism and in a sample of indirectly strengthened ones  focus group with representative from the institutions strengthened through direct mechanism  project document review  interviews with local institutions representatives and project personnel  compete for resources  address MSMEs’ issues  manage USAID funds or concepts 63 Annex 9: Institutional Strengthening Coverage (Source: IBTCI- Evaluation Team, Nov. 2012) No. Coverage No. Coverage No. Coverage Direct strengthened 4 3 75% 4 100% 4 100% - CPTS 1 1 1 1 - CNI 1 1 1 1 - IDEPRO/CEDETEX 1 1 1 - COTEXBO 1 1 1 1 Indirect strengthened 58 42 72% 26 45% - Implementing Institutions 56 n/a n/a 40 71% 24 43% - Institutions that received APP 2 n/a n/a 2 100% 2 100% TOTAL 62 Survey Interviews Methods of data collection Institutions Total Focus Group 64 Annex 10: BPC: MSMEs Experience with BPC Objectives Bolivian Productivity and Competitiveness (BPC) – MSMEs’ Opinion in reference to BPC’s Services Company Size Service Received Sales Change % Costs Reduction % Profits Change % + Productivity Change % New Jobs Results Satisfa ction with BPC Satisfac tion with Impleme nter 1 ALTEXBOL Med TA, WI ND YES/ND ND 50 NA Management improvement, formalization Don't know HS 2 COMBITEX Small TA 30 25 ND 35 15- 20 Ind Better quality, sustained market Don't know HS 3 JHOSELINE Small TA 25 ND 40 ND NA Quality improved Don't know HS 4 ALIANZA TEXTIL Small TA 15 5 15 5 NA Quality improved for Maquila Don't know HS 5 CALZART BOLIVIA Med Training NA NA NA NA NA High but not attributable to project Don't know Neutral 6 TEJIDOS MILANI Micro Training YES/ND ND 100 ND NA High but not attributable to project HS Neutral 7 PUKARA Micro AT No No No lowered NA Better controls S Don't know 8 PICOS BILLAR Micro AT, CP 300 ND 300 YES/ND NA Management improvement, formalization HS Neutral 9 CARPINTERIA TECTO Micro AT, CP YES/ND ND 300 YES/ND NA Management improvement, formalization, wider vision Don't know HS 10 AMASC - CHILUYO Asocc PPA NA NA NA NA NA Hope will bring more money, better environment for our associates HS HS 11 SILLONES CORONA Micro TA No No No No NA Management improvement, formalization, wider vision Don't know S 12 MAESTRANZA HERCAR Micro Training No No No No NA Nothing implemented NS NS 13 ALIMENTOS SALUDABLES Small TA ND YES/ND ND YES/ND NA Not implementable yet Don't know S 14 DAYLAYTEX Small TA, WI ND ND ND 20 14 Better personnel, improved work environment S S 15 ECOLOGIZATE Micro TA, CP ND ND ND ND 100 Ind Sales effects do not last NS NS 16 JOHN PACHECO Micro Training, CP 25 ND 25 ND NA High but not attributable to project HS Neutral 17 DECO OFFICE CP 3 ND ND 3 NA High but not attributable to project Neutral Neutral 18 MUEBLES SARMIENTO Micro TA, CP YES/ND ND 15 ND NA Fairs' contacts led to sales S NS 19 MUEBLES JEBEMA (Assoc) Assoc PPA, WI NA NA NA NA NA Nothing related to this company. It is actually a PPA NA NA 65 Bolivian Productivity and Competitiveness (BPC) – MSMEs’ Opinion in reference to BPC’s Services Company Size Service Received Sales Change % Costs Reduction % Profits Change % + Productivity Change % New Jobs Results Satisfa ction with BPC Satisfac tion with Impleme nter 20 MADECOR WI ND ND ND 200 4 Good workers, two remain Don't know HS 21 COTEXCO Micro TA No No No No NA Inadequate, irrelevant intervention Neutral Neutral 22 SPORT BOYS Micro TA No No No 10 NA Poorly executed intervention NS NS 23 WILLIAM'S JEANS Micro TA No 10 No No NA TA good, but poor commercial effect ND HS 24 BENEGY Small CP, WI 20 YES/ND 10 ND NA Commercial assistance not very effective S S 25 COOP. FOTRAMA CP No No No No NA Commercial assistance not very effective ND S 26 VANER Q CP, WI 20 No No No 4 Transitory commercial effect HS S 27 MUEBLES ARCADIA Small TA No No No 30 3 Poorly executed intervention S Neutral 28 DANATELI CONFECCIONES Training, CP, WI 40 30 YES/ND 60 4 I felt no discrimination against me S HS 29 VIVE SANO CP 20 No 20 No NA Expectations met HS S 30 G-7 Small TA ND ND ND YES/ND NA Wider vision S S 31 PATRA Med TA ND ND ND ND NA Company seems successful with excellent perspectives Don't know NS 32 ALTIMAD Med TA YES/ND No ND 50 NA Management, organization improved S HS 33 HERLOZ Small TA ND ND ND ND ND Mgt. improved formalization, wider vision S S 34 TECNOALIMENTOS Small TA ND 15 ND YES/ND Higher efficiency S S 35 IDELCUSI Small PPA NA NA NA NA NA Business sustainability difficult due to raw material supply Neutral S 36 LIDERSPORT TA ND YES/ND 30 100 NA Management, organization improvement, sustained growth S S 37 ICOMTEX Micro TA NA NA NA NA NA Company no prepared to properly receive assistance. It's been semi￾paralized Neutral S 38 MUEBLES OSINAGA Micro TA 50 YES/ND 50 100 NA Management, organization improvement, sustained growth S HS 39 UNIARTE (Artisans' Assoc.) Assoc Training, TA NA NA NA NA NA Poorly executed intervention, irrelevant Neutral NS 40 MUEBLES CARVAJAL CP NA NA NA NA NA Fairs are not always effective Neutral Neutral 66 Bolivian Productivity and Competitiveness (BPC) – MSMEs’ Opinion in reference to BPC’s Services Company Size Service Received Sales Change % Costs Reduction % Profits Change % + Productivity Change % New Jobs Results Satisfa ction with BPC Satisfac tion with Impleme nter 41 TECNOALIMENTOS Small TA, CP ND ND ND YES/ND NA Intervention reinforced ongoing progress Neutral S 42 MANJAR DE ORO Small TA ND ND ND YES/ND NA Reinforced technical processes, but Integral Assistance is needless for us S S 43 ARTE HOGAR (Artisans' Assoc.) Assoc CP ND ND ND ND NA Product development has been institutionalized Neutral S 44 MONTANA CP ND ND YES/ND ND 6 Transitory commercial effect HS NA 45 IMTAR R&J CP YES/ND YES/ND YES/ND YES/ND NA Transitory commercial effect HS Neutral 46 TAJSARA (Artisans' Assoc.) Assoc TA YES/ND YES/ND 50 YES/ND 6 Mgt. organization improvement, sustained growth HS HS Source: BPC files and IBTCI Evaluation ) Evaluation Team -Nov. /2012 67 Annex 11: Guides for Conducting Surveys, Focus-Groups, Key Personal Interviews Sampling The evaluation team based all data and information gathering specially by leveraging the monitoring data collected by the BPC Project –provided to/by USAID and/or the implementing partner, then the team must obtain a list of project beneficiaries from USAID/Bolivia and/or the BPC technical team to have an overall understanding of the beneficiary population. With that knowledge a simpler sample plan should be designed for surveys, personal interviews and focus-groups events. To select key groups, randomness and testing independence principals must be maintained in all process, especially for selecting individual beneficiaries. To ensure randomness and testingv independence different parameters can also be applied i.e. geographic clustering or proportional sampling by value chain sector if needed to ensure that the sample is representative of the beneficiary population. The appropriate sample size will be based on beneficiary population and acceptable sampling parameters (i.e. confidence levels and confidence intervals) Guide to Qualitative methodology Procedures: A. Procedures for Interviews to Key informants.- The team must do its best to find-out answers to all five evaluation questions18. These will be tested through interviews with representatives benefited MSMEs, Grantees, institutions strengthened and its trained personnel, NGOs, MSMEs, and other key stakeholders like subcontractors and grantees, making sure to have been randomly selected. Also and most importantly USAID and project personnel (Chemonics). Make sure to hold in your hands (or bag) all the proper material that will be required for the full process to be successful, e.g. questionnaires used in the surveys, utensils and if possible a watch and a tape recorder (only if the interviewee allows its use). Once the interviewer meets the interviewee, they should sit in an environment free of bothersome noise or distractions. Make sure to hold in a visible place the form or questionnaire or interview guide. If the interviewee has already responded to the survey, make sure to cross-check his/hers answers as part of this instrument. In all interviews, pre-elaborated questions on how partnerships, alliances, and dialogues were formed under the project’s PPA mechanisms were very effective to assess how effective they were in leveraging funds from the private sector and other donors. Interview protocols will be designed to collect relevant and verifiable information. Each interview is expected to take one hour to complete  Documentary review.- The evaluation team will review all documents produced within the project, i.e. contract terms, annual work plans, the performance management plan (PMP), PPA reports, and performance reports; only when necessary or tasked by USAID documents related to internal functioning of strengthened institutions (balance sheets, legal statutes, strategic plans, payrolls only of personnel benefited by BPC, etc.) which must have been provided to USAID in order to receive any donations. 18  Variables of question-5: To what extent BCP activities strengthened local institutions capacities in being able to provide adequate services to MSMEs, compete for resources, address MSMEs’ issues, manage USAID funds or concepts? 68 This review will provide the team with comprehensive understanding of the project as well as the various local and contextual elements that affect it. This task will ensure that the team’s approach to the evaluation is relevant to local conditions and responsive to the scope of work. Focus group.- Focus group discussions will target no more than 20 participants in each, which will be held with groups of beneficiaries and stakeholders. Discussion agendas will be developed on arrival to Bolivia in coordination with USAID. These exercises will allow the team to gather information about perceptions and opinions related to project activities, performance, and results. Closed attention will be given on how activities strengthened local institutions’ capacities in providing adequate services to MSMEs and providing support in addressing MSME issues (Evaluation Question 5). The data will be collected and analyzed to identify trends in responses and any significant outcomes. Participants on this task will include MSMEs and other beneficiary sectors (value-chains) previously decided with USAID (i.e. textile and apparel, wood manufactures, processed foods, bio-products, and handicrafts) to ensure representation along the value chain dimension. Discussions will take two hours to complete and will be conducted in La Paz, Cochabamba, and Santa Cruz to achieve programmatic and geographic representation but balancing it with local conditions on travel and logistics (an additional city could be considered under special written request from USAID). In the case of results gathered with regard to capacities to provide adequate services for MSMEs and efforts to address MSMEs’ issues, the evaluation team will conduct one focus group specifically in the area of institutional development because of the reduced number of strengthened institutions (4). Added to these improved institutions the team will invite the participation of Sub-contractors and other institutions, in order to test the validity of capacity strengthening, which have been classified as stakeholders allowing to promote self-disclosure among participants and receive fruitful perceptions and opinions about project activities and performance. Conclusions of the focus group will be disaggregated by strengthening area (governance, management practices, human resources, financial issues, service delivery, and external relations), type of institution and purpose of the organization The evaluation team will coordinate with USAID/Bolivia on areas for selection taking into consideration which specific areas are most suitable in terms of logistics, travel requirements, local willingness to participate, and other relevant factors Quantitative methodology  Survey.- The evaluation team will build on existing instruments used by implementers to collect monitoring data for the main indicators of the BPC project. Review of indicators collected by the BPC on improved competitiveness and productivity (Objective 1), local institutions strengthening (Objective 2), dialogue promotion (Objective 3), and PPPs (Objective 4). If possible, and if time allows, the evaluation team will develop an sampling instrument for collecting the same information in the form of a sub-sample of beneficiaries to be surveyed. Given that the evaluation is also expected to commence at the end of Oct.-201219. The survey may also have a module to collect information on beneficiary perceptions, beliefs, and attitudes towards various project components including, but not limited to, management, implementation, technical support, and forging local partnerships. As 19 If BPC collects data on the beneficiary population, the evaluation team will explore applying statistical weights to the evaluation survey so results can be compared to prior data collected 69 specified in the RFQ, the module will also explore business constraints and its sources, if any, identified by MSMEs and other stakeholders In the case of results produced by Indicator 2, specifically with regard to capacities related to provide adequate services to MSMEs and (more broadly) address MSMEs’ issues, the evaluation team will conduct surveys with the personnel, which were trained and remain working for the beneficiary institution. The survey questionnaires will include multiple-choice questions, instead of open-ended ones. Where possible, the surveys will be conducted by electronic means by sending questionnaires to several beneficiaries by email. Additionally important beneficiaries that have not answered will be surveyed through face-to-face interviews. These channels will be used as tools in order to reach the proposed sample size. The main tools that are going to be used to analyze the collected data are descriptive statistics and cross tables. 70 Annex 12: PROTOCOLS (INSTRUMENTS) APPLIED IN SURVEYS, FOCUS￾GROUPS AND PERSONAL-INTERVIEWS A EMPRESAS USUARIAS MSME – Protocolo Fecha:________ 1. INFORMACION DEL EMPRESARIO 1) Nombre: ____________________________________ 2) Genero: M F 3) Funcionario Socio Gerente 4) Edad de la empresa____ años 2. INFORMACION DE LA EMPRESA 5) Nombre de la Empresa: ______________________________ 6) Ciudad: _____________________ 7) No de Empleados: ______ 8) Rama o Actividad: _____________________________ 9) Mercado: Local Regional Nacional Exportación 3. INFORMACION DEL SERVICIO RECIBIDO 10) Nombre o tipo de Proyecto realizado en la empresa: ________________________________________ 11) Qué tan satisfecho está con el servicio que le brindó USAID? Muy satisfecho Satisfecho Indiferente Insatisfecho Muy Insatisfecho 12) Qué beneficio obtuvo al trabajar con el USAID? Capacitación para mi Capacitación empleados Asistencia Técnica Promoción comercial Un asesor adecuado Seguimiento al trabajo del asesor Una trabajo más completo por parte del asesor Una donación para capital 13) Qué tan satisfecho está con el servicio que le brindó USAID? Muy satisfecho Satisfecho Neutro Insatisfecho Muy Insatisfecho Comentario: ______________________________________________________________________ 14). Ha mejorado su empresa en algo a causa de la asesoría/capacitación recibida? Sí No 15) En cuáles funciones o procesos ha mejorado la empresa a causa de la asesoría recibida? Formulación de la Estrategia del Negocio Investigación del Mercado Aumento en las Ventas Métodos de Producción Adquisición de Insumos y Materiales Calidad Seguridad y Salud en el Trabajo Procesos Contables Organización General de la empresa Sistemas de Promoción o Mercadeo Canales de Distribución Diseño del Producto Administración de Inventarios Manejo de Costos Gestión de Personal Gestión Financiera 71 Sistemas de Información Gestión Ambiental 16) Qué tan satisfecho está con el servicio que le brindó el asesor/capacitador? Muy satisfecho Satisfecho Neutro Insatisfecho Muy Insatisfecho Comentario: ______________________________________________________________________ 17) Tales cambios tuvieron un impacto en su empresa en alguna de las siguientes maneras? Aumento en la Productividad (unidades/hora/hombre) Si % Reducción de Costos Si % Creación de nuevos empleos Si, cuántos Mayor rentabilidad Si 18) Ha mejorado su competitividad como resultado directo de la asesoría recibida? Sí No Aun no 19) El aumento de su competitividad le ha significado: Incremento en ventas? No Si % No se Aumentó en utilidades? No Si % No se EVALUACION PROYECTO BPC-USAID BOLIVIA ENCUESTA A EMPRESAS QUE ESTABLECIERON APPS Ciudad: Fecha: 1. INFORMACION DEL ENTREVISTADO 1) Nombre del Entrevistado: ____________________________________ 2) Genero: M F 3) Empleado Socio Gerente 4) Tiempo trabajado en empresa ___ años 5) Correo electrónico: 6) Teléfono: 2. INFORMACION DE LA EMPRESA 7) Nombre de la Empresa: 8) Dirección Planta : 9) Nombre dueño de la Empresa: 10) Años en el mercado: 11) Capital total (Aprox.): 12) Número total actual empleados: 13) Actividad/Sector: 14) Ventas (Promedio) último año en US$: 15) Mercado: (último año del total % de ventas a nivel local, regional, nacional o de exportación) Local: % Nacional: % Regional: % Exportación: % 3. INFORMACION SOBRE ALIANZA PUBLICO-PRIVADA Información general 16) Cuál fue el objetivo de la APP realizada: 17) Cómo se enteró del programa de ayuda por Proyecto BPC y del mecanismo de Alianzas Público Privadas? A través de la prensa Gremio de empresarios Correo electrónico Otros Trabajo anterior realizado con BPC Especificar: 72 18) ¿El aporte fue monetario y/o en especie (equipo o maquinaria) que recibió: 19) ¿Cuál fue el valor de su aporte al proyecto o %? Relevancia 20) ¿Su proyecto se ejecutó en forma total? Si No Parcialmente Porcentaje: Le permitió/permitirá seguir en el futuro: 21) ¿Qué tan satisfecho está con el apoyo que le brindó el Proyecto BPC – USAID/Bolivia? Muy satisfecho Satisfecho Insatisfecho Muy Insatisfecho Opine por qué?: ……………………..………………………………………………………………………… 22) ¿Estaría Ud. dispuesto a hacer otra alianza (AAP) en el futuro? Si. ……. No… Tal vez…… 23) ¿De las condiciones que Ud. acordó cuales no son aceptables?: ¿Por qué si/no? 24) ¿De las condiciones que Ud. acordó que mejoraría? 25) ¿Si la contribución a su proyecto APP fue mayor a 50%, le interesaría hacer una nueva ALIANZA APP aunque su aporte fuera de una proporción mayor y el aporte del programa menor? Si… No… Tal vez … ¿Por qué? 26) De su experiencia con esta APP ¿Qué otros efectos positivos o negativos le causó el proyecto BPC? POSITIVOS: NEGATIVOS: Efectividad 27) ¿Ha mejorado su productividad (mayor producción/día) como efecto de APP? Sí No Aún no 28) ¿Ha mejorado su competitividad (vende más) como efecto de APP? Sí No Aún no (Si su respuesta es sí, pasar a la pregunta 29, si es no pasar a la pregunta 30) 29) El aumento de su productividad/competitividad le ha significado: ¿Incremento en ventas? No Si …% No sé Explique………………………………………………………………………………………………………………... ¿Aumento en utilidades? No Si …% No sé Explique………………………………………………………………………………………………………………... 31) ¿Ha incrementado su demanda de productos y servicios (ofrecida por otras empresas) como efecto indirecto de la APP? Sí No Aun no Si es si: ¿bienes/productos?.... o ¿Servicios? …. o ¿personal?….. ……………………………………………………………………………………………………………………………. 32) ¿Se incrementó la demanda por mano de obra adicional como un efecto del incremento en ventas y la ayuda recibida de la APP? Sí No Aún no ¿Cuánto aumentó en 2011..…. empleados - ¿Cuánto aumentó en 2012 ….. Empleados? Sostenibilidad 33) El plan o proyecto convenido con el Proyecto BPC está funcionando al momento? Si No Parcial ¿Porcentaje de lo planeado?: % - ¿En este momento está saludable su negocio? …… 34) ¿Si está funcionando su negocio, considera Ud. que la APP le brindo SOLVENCIA para mantenerse en el tiempo? Mucho Aceptable Muy poco Ninguno ¿Por qué? ………………………………………………………….. 35) Ventas totales desde (…..año) hasta junio 30 de 2012 atribuibles a la Alianza? (Se aumentaron en): US$ 1,000- 5,000 6,000 – 10,000 11,000 – 15,000 16,000 – 25,000 Mas de 26,000 ¿Por qué? Definir una o dos razones: 73 36) ¿Incremento de trabajadores y cooperados atribuido como efecto de la alianza (…..año) hasta junio 30 de 2012? De 1 a 5 De 6 - 10 Incremento de 11 a 25 ….. De 26 a 50 …… Más de 50 trabajadores……. ¿Cuantos cooperados o aliados nuevos ocurrieron desde (…..año) hasta junio 30 de 2012?: Género (Como se benefició a las mujeres) 37) ¿En base al incremento de trabajadores mencionado en numeral 26, desde el inicio de la alianza (…..año) hasta junio 30 -2012 ¿Proporción de mujeres? …. % ¿Proporción de varones? …… % Medio ambiente 38) ¿En el convenio de aporte de capital o equipo, se contempló el llenado de la ficha ambiental? Si No No necesitó uno ¿Si la naturaleza del negocio es beneficioso o amigable al medio ambiente, no necesita ficha ambiental Explique…………………………………………………………………………………………………………………… …………………………………………………………………………………………………………………………… 39) ¿Como resultado de la APP, su empresa implementó medidas o prácticas para reducir el impacto ambiental? Esa sola vez Nunca Mantiene sistemas de monitoreo hasta ahora EVALUACION PROYECTO BPC-USAID/Bolivia Formulario de Entrevista sobre Alianzas Público-Privadas / aplicable con ejecutivos del BPC              Relevancia.  Según su experiencia y criterio personal, exprese su opinión sobre el Proyecto BPC (no institucional), cuál es su percepción sobre:                         1. La importancia que tuvo el Proyecto BPC para mejorar la productividad y competitividad de las MiPyMEs, como ejecutores?                   Muy Importante     Neutro     Poco Importante                       2. La importancia que tuvo el Proyecto BPC para mejorar la productividad y competitividad de las MiPyMEs, a través de Asistencia Técnica?                Muy Importante     Neutro     Poco Importante                       3. La importancia que tuvo el Proyecto BPC para mejorar la productividad y competitividad de las MiPyMEs, a través de Alianzas Público‐Privadas?                Muy Importante     Neutro     Poco Importante                       74 4. Cúal fue el efecto del trabajo en Promoción Comercial realizado por el Proyecto BPC, en las instituciones que ofrecen este servicio?                Muy Importante     Neutro     Poco Importante                       5. Cuál fue el efecto del trabajo realizado por el Proyecto BPC en el mejoramiento y fortalecimiento de instituciones que ofrecen servicios a las MiPyMEs                Muy Importante     Neutro     Poco Importante                          6. La importancia que tuvo para los beneficiarios (MiPyMEs y gente desempleada)?                         Muy Importante     Neutro     Poco Importante                       7. Cuáles son las lecciones aprendidas identificadas o revalidadas de prácticas anteriores que deberían continuarse implementando en futuros programas de USAID?  (P7. Pregs. USAID y Criterios Eval.)                a) …………………………………………………………………………………………………………………………………………………………………… b) …………………………………………………………………………………………………………………………………………………………………… c) …………………………………………………………………………………………………………………………………………………………………… d) …………………………………………………………………………………………………………………………………………………………………                8. El mecanismo de Alianzas Público‐Privadas fue adecuado considerando el contexto económico y político actual en Bolivia? (P5. Pregs. USAID y Criterios Eval.) Muy adecuado     Neutro     Poco Adecuado                       8. Cómo podría ser incorporado el mecanismo de Alianzas Público‐Privadas, dentro la estrategia de USAID, en futuros proyectos? (P6. Pregs. USAID y Criterios Eval.) ………………………………………………………………………………………………………………………………………………………………………………… …………………………………………………………………………………………………………………………………………………………………………………                9. En qué magnitud el Proyecto BPC contribuyó a los objetivos del programa SDE (Programa de Desarrollo Sostenible y Medio Ambiente)? (P2. Pregs. USAID y Criterios Eval.) ………………………………………………………………………………………………………………………………………………………………………………… …………………………………………………………………………………………………………………………………………………………………………………                         Efectividad.  Cuál es su opinión de conjunto (considerando el trabajo realizado) y percepción sobre:                   10. En qué grado se cumplieron los objetivos de las Alianzas Público‐Privadas?      Alto     Medio     Bajo Porcentaje? …………                   11. En qué grado se cumplió con el financiamiento planificado y convenido por parte del donante? 75    Alto     Medio     Bajo Porcentaje? …………                   12. En qué grado la empresa privada cumplió con la contribución de contraparte (inversión privada) comprometida? (P4. Pregs. USAID y Criterios Eval.)    Alto     Medio     Bajo Porcentaje? …………                   13. Cúal fue la importancia del proyecto BPC en la creación de nuevas plazas de trabajo en las MiPyMEs?    Alta     Media     Baja Porcentaje? …………                         Sostenibilidad.  Según su percepción y opinión, como miembro del Proyecto BPC (según los resultados acumulados):                   14. La inversión de la Alianza Público‐Privada ayudó o mejoró la expansión de las diferentes APPs?    Alto     Medio     Parcial     Se redujer on                   Porcentaje?.................                                 15. Las ventas de las Empresas que participaron del mecanismo de APPs se incrementaron como efecto de este Proyecto?      Alto     Medio     Parcial     Se redujer on                   Porcentaje?.................                                 16. El número de empleados se incrementó?               Alto     Medio     Parcial     Se redujer on                   Porcentaje?.................                                 17. La tecnología o equipos donados por efecto de la APP mejoraron el nivel de de producción y/o productividad?      Alto     Medio     Parcial     Se redujer on                   Porcentaje?.................                                       Género.  De acuerdo a su percepción y/u opinión.  El efecto acumulado del Proyecto BPC fue:                      76 18. Cuántas empresas privadas que utilizaron el mecanismo de Alianzas Público‐Privadas pertenecen a mujeres?    Alto     Medio     Bajo Porcentaje? …………                   19. En cuántas Alianzas Público‐Privadas se contrata o ayuda a mujeres?      Alto     Medio     Bajo Porcentaje? …………                         Medio Ambiente. De acuerdo a su percepción y/u opinión. El efecto acumulado del Proyecto BPC fue:                         20. Qué proporción de Alianzas Público‐Privadas ofrecieron/entregaron un Estudio de Impacto Ambiental (EIA)?    Muchas     Pocas     Ninguna % del total Intervenido? …………                   21. Cuántas APPs son directamente amigables al medio ambiente (operaciones agrícolas, productos naturales y bio‐ambientales)?    Muchas     Pocas     Ninguna % del total Intervenido? …………                   22. Cuántas APPs mejoraron  sus prácticas de respecto al medio ambiente?          Muchas     Pocas     Ninguna % del total Intervenido? …………                   23. Cuántas APPs están aplicando procesos para reducir la contaminación en su producción?        Muchas     Pocas     Ninguna % del total Intervenido? …………                   24. Cuántas APPs introdujeron debido al instrumento, procesos para el ahorro de energía?        Muchas     Pocas     Ninguna % del total Intervenido? …………                         77 EVALUACION PROYECTO BPC-USAID BOLIVIA PROTOCOLO ENCUESTA A INTITUCIONES DE APOYO TECNICO A MSMES Fecha: Cargo: Ene/Feb-2012 Dic.-2010 Poco útil Útil Muy útil Poco útil Útil Muy útil SI NO Mucho Poco Nada SI NO SI NO Administrativ Técnica Administrativ Técnica 14. Si respondió afirmativamente a la pregunta 13 indique en qué área. Caso contrario pase a la pregunta 15 15. Proporcione por favor una tasa aproximada de rotación de personal en su institución (anual) de acuerdo a las áreas mostradas. "Muchas gracias por su cooperación" 16. Si tiene algún comentario sobre el fortalecimiento institucional de su institución gracias a la relación con el proyecto PC-Bolivia, favor redáctelo abajo. ……………………………………………………………………………………………………………………………………………………………………. ……………………………………………………………………………………………………………………………………………………………………. Alianza Público Privada Nada de Involucramiento Poco Involucramiento Mucho Involucramiento 12. Identifique en la lista de abajo el servicio que usted brindó (puede ser más de 1) y evalúe cual fue el involucramiento del proyecto PC-Bolivia, no desde la perspectiva administrativa sino de orientación sobre los contenidos técnicos del servicio. Asistencia Técnica Capacitación Promoción Comercial NO 10. Evalúe en qué medida su institución ha asimilado los conceptos y la visión entregada por el proyecto PC-Bolivia 11. Considera usted que su institución está en mejores condiciones de competir en la captación de fondos gracias al proyecto? 8. Considera usted que su institución puede entender mejor el funcionamiento de las cadenas productivas con las que trabaja gracias a su relación con el proyecto PC-Bolivia? 9. Considera usted que su institución está en mejores condiciones de identificar las verdaderas necesidades de las MIPYMEs gracias al proyecto? NO SI SI NO 6. Considera usted que, luego de haber trabajado con el proyecto PC-Bolivia, la capacidad para proveer los servicios técnicos en los que su institución se especializa, han mejorado y son más adecuadas a los requerimientos de las MIPYMEs? 7. Si respondió afirmativamente a la pregunta 6, indique las principales razones (marque una o dos opciones máximo). Caso contrario pase a la pregunta 8 5. Cómo considera usted que son los requisitos administrativos del proyecto PC-Bolivia Normales Muy Burocráticos Poco Burocráticos FORTALECIMIENTO INSTITUCIONAL PARA BRINDAR MEJORES SERVICIOS A LAS MIPYMEs 1. Participó algun miembro de su institución en los talleres de capacitación brindados por el proyecto PC-Bolivia? Otra (especifique abajo) Porque ganó mayor experiencia FORTALECIMIENTO INSTITUCIONAL PARA MANEJAR FONDOS DEL PROYECTO OTROS 13. Ha recibido su institución apoyo en fortalecimiento institucional de otros cooperantes en los últimos dos años? SI 3. Si respondió afirmativamente a la pregunta 1, evalúe cuan útil fue el taller para facilitar su relación administrativa con el proyecto PC-Bolivia 4. Evalúe cuan útil fue el contacto directo (vía consultas telefónicas, visitas o correos electrónicos) entre su institución y personal del proyecto PC-Bolivia para facilitar su relación administrativa con el mismo 2. Si respondió afirmativamente a la pregunta 1 indique en que taller. Caso contrario pase a la pregunta 4 Por la orientación técnica (no administrativa) dada por el proyecto PC-Bolivia Porque ajustó sus propias técnicas al poder llegar a más MIPYMEs FORMULARIO - PROTOCOLO DE ENCUESTA (M&E del PC-B proyecto de USAID-Bolivia) "Se agradece la colaboración brindada y se garantiza la confidencialidad de toda la información proporcionada" Nombre y Apellido: Nombre de la Institución: DATOS GENERALES DE LA INSTITUCIÓN EVALUACIÓN DEL PROYECTO DE PRODUCTIVIDAD Y COMPETITIVIDAD 78 FOCUS – GROUP (FG) Questions and aspects to be addressed by moderator: (La Paz – Cochabamba – Santa Cruz) DESCUBRIR LA RELEVANCIA Y OPINION SOBRE PRÁCTICAS PARA EL DESARROLLO PRODUCTIVO Y EFECTOS EN LAS MIPYMES / APPs: Los objetivos principal de los FG son:  Descubrir las opiniones, sentimientos y percepcion por parte de instituciones invitadas sobre la ayuda y soporte tecnico que ellas han ofrecido o necesitan las MIPYMEs/ APPs.  Lecciones aprendidas para que USAID disponga de elementos de juicio para perfeccionar futuras intervenciones en otras areas de apoyo. Para facilitar las reflexiones de la audiencia durante el evento (y manejadas por el moderador), los temas a considerar y extraer opinions seran los siguiientes: AGENDA DE TEMAS A TRATAR Y MODERAR a) Opinan uds que la labor o apoyo del proyecto USAID-BPC ayudo para que vuestra institucion haya mejorado/empeorado el entorno del desarrollo productivo y su relación con las MIPYMEs/ APPs 1. ¿Cuáles debieran ser las prioridades para apoyar el desarrollo productivo de las MIPYMEs/ APPs? o Apoyo financiero y/o con equipos de mayor tecnologia Prioridad: _______ o Reformar el entorno legal del sector de la empresa privada Prioridad: _______ o Ayuda tecnologica directa (en planta) Prioridad:_______ o Capital y/o Fuentes de financiamiento Prioridad: _______ o Mejorar y/o disminuir los procedimientos burocraticos: Prioridad: _______ o Otros: _________________________________________________________ 2. ¿Con la ayuda del proyecto BPC vuestra institucion mejoro la oferta servicios y otros mecanismos de apoyo al desarrollo productivo de las MIPYMEs/ APPs? o Servicios burocraticos y/o para facilitar procesos OP o Nuevos Servicios creados por la demanda o necesidad de MIPYMEs/ APPs o Servicios especiales para MIPYMEs/ APPs innovadoras (Bio-tec / ITC) b) Mejorar productividad de las MIPYMEs/ APPs aplicando/promoviendo Alianzas Publico – Privadas: 3. ¿En su experiencia, el mecanismo de la APP es una herramienta útil para el desarrollo de la productividad de la MIPYME? 4. ¿Considera que ha sido efectivo el mecanismo de la APP para incrementar las oportunidades de demanda de la mano de obra y así apoyar la productividad de las MIPYMEs/ APPs? 5. ¿La inyección de capital y tecnologías innovativas a través de co-inversión es posible y eficiente para las MIPYMEs/ APPs? c) Sobre las lecciones aprendidas y recomendaciones 6. ¿Según su experiencia, qué instrumento de desarrollo empresarial es el más demandado por las MIPYMEs/ APPs y por qué? 79 7. ¿Qué recomendaciones tiene para nuevas intervenciones de USAID en el ámbito del desarrollo productivo para apoyar a las MIPYMEs? MODERADOR: Nombre y apellido: ……………………...... Firma: ………………………………… Lugar y Fecha: ……………………………………………………………….. 80 Annex 13: Bibliography and documentary References Manuals BPC Project (November, 2009). Monitoring and Evaluation Manual. Prepared by Chemonics International Inc. by Bolivian Productivity and Competitiveness Project. BPC Project (February 11, 2010). Deliverable I. Branding Strategy and Marking Plan. Prepared by Chemonics International Inc. by Bolivian Productivity and Competitiveness Project. BPC Project (March 10, 2010). Deliverable 3. Diagnostics and Intervention Plans in Production Chains. Prepared by Chemonics International Inc. BPC Project (March 10, 2010). Deliverable 4. Institutional Strengthening Plan. Prepared by Chemonics International Inc. BPC Project (March 10, 2010). Deliverable 7. Strategic Plan for Four Years. Prepared by Chemonics International Inc. BPC Project. Services Delivery Fund/Grants Manual. Task Order No. 8 UNDER THE GBTI II IQC No. EEM-I-00-07-00008. Prepared by Chemonics International Inc. BPC Project. Annexes to Services Delivery Fund/Grants Manual. Task Order No. 8 UNDER THE GBTI II IQC No. EEM-I-00-07-00008. Prepared by Chemonics International Inc. BPC Project (December, 2010). Performance Monitoring Plan. Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project, Task Order No. 8, GBTI II IQC No. EEM-I￾00-07-00008-00. BPC Project (February 14, 2011). Global Development Alliance (GDA)/Public-Private Alliances (PPA) Activities Manual. Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project, Task Order No. 8, GBTI II IQC No. EEM-I-00-07-00008-00. Reports BPC Project (April, 2010). Deliverable 8. GDA Report Q02 Final USAID. Prepared by Chemonics International Inc. BPC Project (June, 2010). Deliverable 8. GDA Report Q03 Final USAID. Prepared by Chemonics International Inc. BPC Project (September, 2010). Deliverable 8. GDA Report Q04 Final USAID. Prepared by Chemonics International Inc. BPC Project (December, 2010). Deliverable 8. GDA Report Q05 Final USAID. Prepared by Chemonics International Inc. BPC Project (March, 2011). Deliverable 8. GDA Report Q06 Final USAID. Prepared by Chemonics International Inc. BPC Project (June, 2011). Deliverable 8. GDA Report Q07 Final USAID. Prepared by Chemonics International Inc. 81 BPC Project (September, 2011). Deliverable 8. GDA Report Q08 Final USAID. Prepared by Chemonics International Inc. BPC Project (December, 2011). Deliverable 8. GDA Report Q09 Final USAID. Prepared by Chemonics International Inc. BPC Project (March, 2012). Deliverable 8. GDA Report Q10 Final USAID. Prepared by Chemonics International Inc. BPC Project (June, 2012). Deliverable 8. GDA Report Q11 Final USAID. Prepared by Chemonics International Inc. BPC Project (March 10, 2010). Deliverable 10. Institutional Communications Plan. Prepared by Chemonics International Inc. by Bolivian Productivity and Competitiveness Project. BPC Project (May, 2011). Performance Monitoring Plan Figures FY2010. Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project. BPC Project (April, 2012). Performance Monitoring Plan Figures FY2011. Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project. BPC Project (January, 2010). Performance Monitoring Plan Figures Q1 (October-December, 2009). Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project. BPC Project (April, 2010). Performance Monitoring Plan Figures Q2 (January-March, 2010). Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project. BPC Project (July, 2010). Performance Monitoring Plan Figures Q3 (April-June, 2010). Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project. BPC Project (October, 2010). Performance Monitoring Plan Figures Q4 - FY2010 Report (October 2009- September, 2010). Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project. BPC Project (January, 2011). Performance Monitoring Plan Figures Q5 (October-December, 2010). Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project. BPC Project (April, 2011). Performance Monitoring Plan Figures Q6 (January-March, 2011). Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project. BPC Project (July, 2011). Performance Monitoring Plan Figures Q7 (April-June, 2011). Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project. BPC Project (October, 2011). Performance Monitoring Plan Figures Q8 - FY2011 Report (October 2010- September, 2011). Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project. BPC Project (April, 2012). Performance Monitoring Plan Figures FY2011. Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project. BPC Project (July, 2012). Performance Monitoring Plan Figures Q11 (April-June, 2012). Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project. BPC Project (July, 2012). Project Report Q11 (April-June, 2012). Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project. 82 BPC Project (October, 2012). Project Report Q12 (July-September, 2012). Prepared by Chemonics International under the Bolivian Productivity and Competitiveness Project. Tables and Databases Bolivian Productivity and Competitiveness Initiative (BPC Project). Work Plan Budget FY 2012. Contract No. EE-I-00-07-00008, Task Order No. 8. Chemonics International Inc. BPC Project (November, 2012). Chart of information of leveraged resources by Quarter and by Year up to June 2012. BPC Project (November, 2012). Chart of information of PPA Beneficiaries by Departments up to September 2012. BPC Project (November, 2012). Chart of information of TA Beneficiaries by Departments up to September 2012. BPC Project (November, 2012). Chart of information of TA and PPA by Grants and Subcontracts up to September 2012. BPC Project (November, 2012). BPC Project Database of PPA investments up to June 2012. BPC Project (November, 2012). BPC Project Database of Training courses and Work Force Insertion up to September 2012. PowerPoint Presentations BPC Project (October 29, 2012). BPC Project PowerPoint Presentation of Project Information up to September 2012. 83 Annex 14: Description of Technical abilities Personnel of IBTCI – Evaluation￾Team Milton G. Núñez-Garcés: IBTCI – Evaluation - Technical Team Leader M&E Specialist with more than 15 years of international experience in the evaluation and monitoring of projects and programs for USAID, Inter-American development Bank, World Bank, EU/EC. In the same manner his experience extends to a wide geographic variety of countries and continents: USA, Latin￾America, Africa, Middle-East, and Southern Asia. A firm practitioner of high standards and international best-practices, and a well-known professional in Regional and Municipal Economic Development, with the philosophy that a strong private sector, especially if growth is led by competitive MSME value-chains, it is sure to be successful and sustainable. He holds several Masters Degrees in Economic Development and International Trade; Project Formulation, Monitoring and Evaluation; Project Management; as well as in Higher (university) Education; and a BSC in Engineering from prestigious universities in USA, Ecuador, and other international programs. More than 20 years as university professor in the fields of Project Evaluation (IDB & USAID methodologies), Micro-Economics, International Business, Economic development and International Economic Relations. ----------------------------------------------------------------------- Luis Alejandro Bernal: (MSME Specialist) Luis Bernal is an international consultant specialized in private sector development with emphasis in micro, small and medium enterprises. With strong academic background in Business, Project Management, Monitoring & Evaluation, as well as Public Policy, he has devoted more than fifteen years exclusively to evaluation of projects, programs and public policy, mostly in the USA, Latin America and the Caribbean, working for donors such as USAID, IDB and FOMIN. ----------------------------------------------------------------------- Sergio G. Villarroel Böhrt: (Institutional Development Specialist) Mr. Villaroel holds a degree in Industrial Engineering, a two masters’ degrees in applied economics (Georgetown University) and public policy (Universidad Católica Boliviana–MpD). He is an alumnus of the macroeconomic policy executive programs at Harvard University, and he also has been trained in Local Governance (Regional Autonomy – Tokyo College). His professional consulting experience has been concentrated with international development agencies (USAID, GTZ, AECID, SNV, CIDA), in fields like fiscal decentralization, public management and economic development. Since 2010 he is an associate expert in local governance and decentralization at the UNDP Regional Service Center for Latin America and the Caribbean (Headquarters in Panama). He has also invested some time in public sector as director of industrial development in the Ministry of Economic Development and as deputy director of decentralization and regional development in the program for Democratic Institutions Strengthening, a USAID-program. He has published several articles, domestically and abroad, about decentralization and macroeconomics; and has co-authored two books in the same fields. He is also a part-time teacher at the Catholic 84 University of Bolivia. In 2011 he founded the consultant firm Enklave SRL, and has been the main partner and CEO since then. -------------------------------------------------------------------- Claudia Roca: BPC Project Final Evaluation - Logistics Coordinator Ms Roca holds an MBA in Finance, with more than 10 years of experience in the cooperation field in different areas such as grants administration and finance, project monitoring and evaluation, pre-award assessments, fundraising, proposal development and microfinance. She also worked in multinational companies of the services sector. Personal strengths are planning, organization, data analysis, management, multi-task oriented and risk analysis 85 Annex 15: Conflict of Interest Forms 86 87 DISCLOSURE OF CONFLICT OF INTEREST FORM Name SERGIO VILLARROEL B. Title Mr. Organization IBTCI Evaluation Position? Team member Evaluation Award Number (contract or other instrument, if applicable) AID-511-O-12-00042 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Bolivian Productivity and Competitiveness (BPC) I have real or potential conflicts of interest to disclose. Yes No X If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. Signature: Date: 12/05/2012