Submitted to: Under: The Evaluation IQC AEP-I-00-00-00023-00 Submit fed by: Development Associates, Inc. 1730 North Lynn St. Arlington. VA, 22209-2023 May 2003 DEVELOPMENT ASSOCIATES, INC. MANAGEMENT AND GOVERNMENTAL CONSULT-4KTS I7;O XORTH LYNX STREET ARLINGTON. VIRGINIA 22209-2025. USA TEL: (703) 2ib-06-i FAX: liO5) 1760132 E-MAIL: DE~ASSOC(iiDEV.ASSOC1.COX1 WEB SITE: \V\V\\'.DEVASS~CI CO\l May 6,2003 A. John Wright USAID/Kingston, SO# 1 2 Haining Road Kingston 5, Jamaica RE: AEP-1-00-00-00023-00, Task Order # 828 Evaluation of USAID/Jamaica's SO UNEP Dear Dr. Wright: Development Associates is pleased to transmit the final version of its report. "Evaluation of the New Economy Project in Jamaica." Per the Task Order requirements, five hard copies will follow by mail. Also enclosed is a short summary of our response to AID'S April 24, 2003 comments. We have addressed a number of these comments in the report. We would like to thank you and your staff for the assistance provided to the team while it was in Jamaica and to Development Associates during the report process. It is our hope that this report will assist USAID/Jamaica in carrying out its important efforts to assist the people of Jamaica. Sincerely, Peter B. Davis President WALRT CREEK - 1175 NURTH BROADWAY. Scm 200. WALSLT CREEK. CA 945% (931 935-971 1 FAX (9251 935-5413 R~ERTO Rlco - P.O. Bos 3968. GLAYS.ABO. PR 00970 (787) 281 -01 Fax KO. (787) 281-01 20 D~MISICAS REPL'BLIC - EGYPI - EL S~LVADOR - JALI.-\ICA - MALI - NICAR.~GVA - WRI' - SOLTH AFRlC.4 - UMBARXE - - - - - DEVELOPMENT ASSOCIATE'S RESPONSE TO OEG COMMENTS ON THE NEW ECONOMY PROJECT EVALUATION REPORT Comment No. 1: "In light of USAID'S request "that the evaluation be conducted in the confeu of its future task to develop its 2005-2009 strategic plan" (see page S), it would be heZpful if the evaluators were to provide specific recommendations for the new strategy. In particular we would appreciate comments and advice regarding the appropriateness or otherwise of the NEP model for the new strategv period and recommendations for any modij2ations that may be necessary to the NEP model. " Comment No. 2: "Overall, the evaluators were silent on a number of issues including the following which would assist rhe Mission in designing its new strategv and in determining the relevance of the lessons learnt from the NEP experience: Has the project been effective in terms of overall sustainability and if not how can this be achieved? Does the model have long-term impact and if not what could be done to achieve this? What other lessons have been learned about project design, implementation, modus operandi that could be used to enhance USAID's demand-led platform during its new strategy period? Does the model represent a suitable return on the level of the investment? What are the strengths and weaknesses of this approach? Could this model be replicated globally?" For both comments above, we have consolidated observations made in the initial dmfi on these subjects into a new section in the Conclusions & Recommendations chapter, and we have added observations on the project's model in terms of its appropriateness. replicabilit); and the related questions. Related conclusions were incorporated into the Executive Summary. We have not addressed the question of whether the model represents a suitable return on investment. The need to compare this project with other AID projects to determine "suitable return," the level of effort afforded by the contract, and the guidance on looking at budget data received from OEG all conspired to preclude such analysis. New Economy Project I .Wzy, 2003 Final Evaluation Repori USAID/Jamaica Develo~menl Associates, Inc. Comment No. 3: "The suggestion in the Executive Summary @age 1) that "overall achievement to date has been limited" is inconsistent with several other statements pertaining to the impact and achievements cited in various sections of the report. For example, the evaluators acknowledged (in the same paragraph of the Executive Summary) that NEP is exceeding its targets. In fact, NEP performance has resulted in OEG exceeding Mission targets for the SO. The suggestion that "achievement is limited" dues not take into consideration that activities are ongoing and could lead the reader to infer that implementation of the various activities is flawed." We have revised the "overall achievement" statement to address this point. Comment No. 4: "Overall, the evaluators appear to have had some difficulfy with accepting the fact that this project is entirely demand-led This is apparent as evidenced by several comments such as the following: "NEP has not addressed in a significant way the transportation issues and the public debi?$wal management issues as suggested in its contract" (see page 6). The fact is that these issues were not addressed because there was no buy-in from any public sector entity that would have had to be the ones to drive the process." This comment does not accurately portray the paragraph in question. The point expressed in this paragraph is that NEP has done an excellent job in meeting the terms of the contract's agenda. The point about transportation and other issues not being addressed - a small set of issues in the context of the whole agenda -- is a caveat. On the broader point of understanding that the project is "demand-led," we observe that a "demand-led" project cannot excuse a contractor fiom meeting the terms of a contract. Objectives and contract terms still need to be met. But overall, as stated in the report, NEP has done an excellent job of meeting the terms of the contract. Comment No. 5: "The comment (on page 10) in relation to the Briefing Room regarding "concern that the forum will be utilized by those more interested in protectionism than expanding free trade" ignores the fact that the facility is open topersons of widely ranging views which is in keeping with the democratic approach that is necessary for such a forum." The statement doesn't ignore the "democratic approach" of Briefing Room, but notes a possible manifestation of this approach that worries some of the people the evaluation team interviewed. The forum may be open to wide-ranging views, but some fear that those who most fear fiee trade will use it to promote a protectionist agenda. New Economy Project 2 May, 2003 Find Evaluation Report USAID/Jarnaica Development Associutes, Znc Comment No. 6: "The statement made (on page 11) regarding denying NEP credit for the contribution in dra9ing and reviewing legislation on the institution of credit bureaus is not accurate, as the criterion for the meeting of IR I Indicators is not that restrictiva We agree with the statement made further on in the report (on page 21), that "credit should be given for activities on which NEP was Q crucial interlocutor". Hence although the legislation has not yet been put into effect, NEP will receive credit for the contribution towards the establishmenr of a credit bureau upon SUCC~SS~U~ completion of the activip and acceptance of the deliverable by the Ministry of Finance. As the evaluators opined (on page 21) also, "it is possible and appropriate to give IVEP credit for legislation that has been draped and accepted by the ministry with which NEP worked". This comment was made in relation to the work on e-government legislutiort. " We have revised the statement on page 1 1 to address this inaccuracy. Comment No. 7: "Reference (on page 11) to the fact that LETS Investments and Capital and Credit Merchant Bank (CCMB) were approached by NEP does not in our opinion defeat the purpose of the demand-driven approach as these entities were approached as part of the IVEP marketing effort and the activities pursued with these entities were not supply-driven. " We have removed the referenced clause on page 11 without diminishing the point of the paragraph. Comment No. 8: "Contrary to the observation made at footnote I1 on page 15, the FTC activi~ wiil contribute to the reduction of two constraints, viz: 1) the achievement of a 40% reduction in the time it takes to resolve cases of anti-competitive practices and 2) an improvement in the Commission's case management system." The observation in the footnote is that there is a contradiction between NEP's February 2003 status report (of "(0)" IR Count) and the Task Order. In the initial draft's text we chose to utilize the task order's information and the comment above confirms our choice. We have removed the footnote as it is no longer needed. New Econom-v Project 3 .Vaj; 2003 Final Evaluation Reporr USAfD/Jarnaica Dmeiopment Associates, Inc. Comment No. 9: "The observation on page 18 that '>eople are willing to pay full price for the training because they know its benefits" has attracted a comment from NEP personnel that the more significant issue is whether the activities resulted in an upsurge in training among SMMEs. " This section of the report addresses factors influencing achievement. One of those factors is the use of subsidies to support training programs. Over the medium- and long-term, subsidies can hurt sustainability of programs for the reasons provided in the paragraph in question. We have dealt with the results of the training programs in other pms of the report (in two prior parts of Section IV and in Section V). We would not characterize the impact of IR2 programs as an "upsurge in training among SMMEs." Comment No. 10: "We had expected that the evaluators would have been more helpful in recommending a more suitable Indicator for the overall SO than the current Indicator of Number of New Businesses Registered which everyone agrees is not the most suitable. The discourse on pages 18-19 therefore did not point us in any particular direction in terms of identicfjring a suitable Indicator that more closely measures the SMME business environment and SO 1's impact on it." We have added a recommendation to this section. Comment No. 11: "NEP personnel haw indicated that all training institutions conduct post￾course evaluations agreed with NEP at the outset of the program, which is contrary to the statement made in paragraph 2 on page 22 that some training vendors do not survey their trainees for satisfaction. NEP has indicated that assessment techniques are not limited to observation and review of instructors' credentials but also involve random telephone interviews and personal interactions with 'beneficiaries. In addition, Marion Blake has conducted surveys to support her evaluation of the SMME Business Skills program." We revised the paragraph to correct this inaccuracy. Comment No. 12: "The team has not provided supporting evidence to justijj the point made (on page 23) that "the internal organization of the NEP team could be reviewed in order to lower the administrative and management carried by porfjolio managers. We revised the section to make the recommendation more precise and to provide additional support for it. New Economy Project 4 May, 2003 Final Evaluation Report Development Associates, inc. Comment No. 13: "lf would be helpful ifthe team were to elaborate in a bit more detail on their suggestion that "the NEP project would tremendous& benefit fiom the institution of a simple monitoring and evaluation (M & E) system. " We provide more detail on the recommendation. Comment 14: "We believe it is important to make the point that there are really no activities carried out by NEP that have no impact on SMMEs since in the Jamaican context virtually all business entities fall under this classification, hclving less than 250 employees. The comments recorded at several places in the report and in particular on page 25 in relation to impact on SMMEs are a bit confusing, suggesting a possible lack of understanding of the fact that there are very fa? Jamaican companies that are nof classified as SMMEs based on the hurdle rate of 250 employees. Also, the recommendation that WEP activity proposals should continue to focus on the potential impact of the activity on SMMEs ^(on page 29) is therefore superfluous. " The "hurdle rate" should be considerably tower. The statistical definition of SMMEs mries by country and is usually based on the number of employees or the value of assets. The lower limit for small-scale enterprises is usually set at 5 to 10 workers and the upper limit at 50 to 100 workers. The upper Iirnit for "medium-scale" enterprises is usually set between 100 and 250 employees. Because Jamaica is a relatively small country, the upper limit in defining a medium￾scale enterprise should be set closer to 100 employees than 250 employees. Also. the point in the report on SMME targeting applies to several specific activities. The managers of these activities themselves acknowledged that their products did not target SMMEs - under their own definition of SMME -- and some did not even realize that they were supposed to target SMMEs. The point and recommendation in question on page 29 deal with the lack of focus on this objective by some activity managers. Additionally, the report also notes that the activities in question have helped few companies thus far and are unlikely to help significant numbers of companies in the future, regardless of how SMME is defined. Finally, the objective of the project is to improve the business environment for developing SMMEs. There is little reason to specify SMMEs in the objective if it were AID'S intention to help virtually all businesses in Jamaica. Overall, the comment validates our recommendation for a tishter focus on SMMEs. Rew Economy Project 5 .Ww. 2003 Final Evaluation Report USAID/Jamaica Page No. - - Acronyms .................... .... .... ........................................................................................ JI Executive Summary ............................................................................................................. iv I. 11. 111. IV. v. VI. Introduction ................................................................................................................ 1 Context of the New Economy Project ........................ .. ...................................-... 1 Background 9 of the NEP ....................................... .....................................................- "7 Purpose of the Evaluation ............................. .. ......................................................... - -. Scope & Background of NEP Activities .......................................................... .... ..... 2 -, Evolution from Design to Implementation ................................. .. ........................... 3 Achievement & Impact Of The Project .................................................................. 4 Achievements 5 to Date ................................................................................................ Activity Status & Indicator Matrix ............................................................................ 6 Much Larger Reach Likely in 2003 - 2004 ................................................................ 8 NEP's Impact ............................................................................................................. 1 1 Factors Influencing Achievement ............................................................................. 13 Indicators & Performance Targets .................................................................... 14 SO1 Indicator for Improving the Business Environment for SMMEs: Number of New Businesses Registered ............................ .. ..................................... 14 IR1 Key Business Processes Reduced While Fostering Competition: Number of Key Business Processes Reduced ............................. .... .................... 16 IR2: Business Skills of Companies Improved: Number of Companies Achieving Technological Improvements ............................ 17 Management & Operation Of The New Economy Project ..... ... ................ ......... 18 Management of Approved Activities ........................................................................ 18 Monitoring and Evaluation System ........................................................................... 20 Workplans and Reporting .......................................................................................... 20 AID Oversight ............................................................................................................ 2 1 33 Contracting & Task Orders ................ .. ................................................................. -- Conclusions & Recommendations 33 ........................................................................... -- Recommendations.. ................................................................................................. 23 ANNEXES Annex 1 Persons Interviewed ............................................................................................... 1 - 1 Amex 2 Documents Reviewed ............................................................................................. 2- 1 Annex 3 Joint World Bank-IFC ............................ .. ........................................................... 3- 1 ~Vew Economy Project Final Evaluaiion Report USA ID/Jornaicu ACS ADSC BDS CARICOM CCMB CEO COP C-Trade EBRD FIAS FNSAC FTAA FTC GDP GoJ IDB IFC IMD IR ISP IT JCCP JCSE JEA JNMCC M&E MCC MCS MIND MIS NEP NLA OEG ORC PCA PMP PSO PSOJ PWC SAJ SMA SMME SO 1 sow STTA Activity Criteria Selection Anti-Dumping Subsidies Commission business development services Caribbean Community & Common Market Capital and Credit Merchant Bank Chief Executive Officer Chief of Party Caribbean Trade and Competitiveness Development Pro-aam European Bank for Reconstruction & Development Foreign Investment Advisory Service Financiai Sector Adjustment Company Free Trade Agreement of the Americas Fair Trade Commission Gross Domestic Product Government of Jamaica Inter-American Development Bank International Finance Corporation International Institute for Management Development intermediate result internet service provider information technology Jamaica Cluster Competitiveness Project Jamaica Central Securities Exchange Jamaica Exporters' Association Jamaica National Microcredit Company monitoring and evaluation Manchester Chamber of Commerce Management Control Systems, Ltd. Management Institute for National Development management information system New Economy Project National Land Agency Office of Economic Growth (AID) Office of Registrar of Companies Pesticide Control Authority Performance monitoring plan private sector organization Private Sector Organization of Jamaica Price Waterhouse Coopers Shipping Association of Jamaica Spectrum Management Authority smaf f medium and micro-enterprise strategic objective Number I Statement of Work short-term technical assistance :Vat! Economy Projec'r I I May. 2003 Fit~al Evalmion Report USA IDVamaicn Development Associates, Inc. 'u TA TOT WDI technical assistance train' the trainers World Development Index ... New Economy Project 111 May, 2003 Final Evaluation Repoft USAILWumaica Jamaica has experienced relatively poor economic performance over the past 25 years. with slow growth, high inflation, high unemployment, a financiai crisis, and heavy public debt. But despite these problems, the incidence of poverty has decreased, probably due to the growth of the informal sector and increasing real wages amidst the decline of inflation. Additionally. over the past three years the Government of Jamaica (GoJ) has implemented a strateg>* to stabilize Jamaica rnacreconomically and to reduce public debt, which has improved the o~erall econorn>-. AID-Jamaica is supporting GoJ's strategy with its own Economic Growth Strategy for the FY99- 04 period (Strategy Objective I), which seeks 'Yo improve the business environment for developing the small, medium and micro-sectors." To achieve this objective. AID has initiated several projects to meet three Intermediate Results (IR) objectives -- reduce key business processes while fostering competition, improve business skills at the company level. and increase the availability of private credit for small and micro-businesses. The New Economy Project (NEP) is the "engine" of Sol. Over the past 2 $5 years the project has been harnessing the demand of private and public-sector clients to meet the strategy's objectives. It has completed, initiated or is now preparing about 50 activities (eg sub-projects) to meet the three IR objectives. Private and public-sector recipients of assistance are usually required to contribute significant resources to their activities to encouwe ownership and sustainability. The project initially received $6 million for the FY00-03 period. and will receive an additional $2 million for FY04. AID-Jamaica has asked the Development Associates Evaluation Team to examine NEP's operation, quality of assistance, quality of management. achievement of performance targets, and, if necessary, recommend modifications to the project to be implemented in the current project period as well as in the context of the 2005 - 2009 strategic plan. During the February 20 - March 19,2003 period, the Evaluation Team conducted interviews in Kingston, Mandeville and Montego Bay with the managers and key personnel of AID-Jamaica, NEP, and 31 of NEP's activities, as well as held a focus goup and several interviews of activity end-users. This report documents the evaluation. NEP has done an outstanding job in squarely addressing almost all of the items on the \-ei-y wide￾ranging agenda presented in its current contract. It is seen by many beneficiaries as an important catalyst to initiate SMME-friendly projects more quickly with higher quality. It appears likely that NEP will exceed its IRl targets over the next 18 months and continue to exceed its IR2 targets as well. IR3 targets are likely to be met through the Jamaica National Micro-Credit Company (JNMCC). But achievement to date has been limited by slower-than-expected implementation of IRI activities. the area that has the most potential for impacting the SMME sector. Additionally, in some cases the targeting of the SMME sector and the sustainability of activities remain challenges. In general, the model!concept of the project is appropriate and. when it is implemented as intended. works well and is likely to be sustainable and have long￾term impact. The approach is also used successfully by other donors for a variety of applications and, thus, has proven that it can be replicated. Most activities support 1R1 objectives and the potential reach of these activities is much broader in scope than the activities supporting the other IR objectives - reaching tens of thousands rather than hundreds. Further, they are very important TO the agendas of CARlCOM regional economic integration and the advent of the FTAA. Some can serve as models in other parts of the world. lVew Economy Project Final Evaluation Repon USA KVJamaica Development Associates, Inc. irJ But due to their ambition and complexity, they take longer to implement. IR2 activities have limited reach but most of them squarely target the SMME sector and they can be implemented 'w more quickly than some IR1 activities. Under IR3, there are no more activities in the pipeline, their results have been mixed, and NEP has registered some lack of enthusiasm for them. NEP should focus more of its resources on IRl and IR2 activities, and vacate the IR3 space. To facilitate clarity and focus on IR1 activities, AID should create two "sub-IRs" - one addressing legislative and regulatory constraints to business, and a second to address the businessiagency process re-engineering activities. For the legislative/regulatory sub-IR, NEP may want to consider hiring a full-time legal expert. NEP should also receive credit for its work on legislative and regulatory matters when it results in a law being drafted and accepted by a Ministry or agency as a possible bill for parliamentary consideration. Nurturing the Regs & Legs activity should be a top priority and sustainable hnding needs to be developed, preferably through the private sector, but cost-sharing between the private sector and government if necessary. Under IR1, NEP should also develop additional linkages between activities, including greater involvement of PSOJ in the Legs & Regs effort and between NEP activities and the C-Trade and Jamaica Cluster Competitiveness Project (JCCP). AID should encourage NEP's vision to help improve the administrative processes of Parliament. NEP should also continue its focus on IR2 activities, particularly those supporting training. NEP should make the programs more sustainable by ending or quickly phasing out the subsidies and by expanding their reach through a "train the trainers" (TOT) approach. NEP has efficiently and responsively managed its portfolio but there is room for improvement, Project Officers do not spend sufficient time providing technical assistance. The internal organization of NEP should be reviewed to lower the administrative and management burden carried by the portfolio managers. Monitoring of results will become increasingly important as more activities become operational and needs improvement. A Monitoring and Evaluation system should be implemented by administrative staff under the supervision of project officers. The reporting to OEG does not adequately focus on results and impact. Consideration of NEP activity proposals should continue to focus on the potential impact of the activity on SMMEs and the degree the activity manager is likely to focus on the SMME sector. NEP should provide written reports to OEG no less fiequently than quarterly and such reports. should include performance data on current activities. Semiannual reviews should focus on the status of performance. AID should assume responsibility for conducting surveysifocus groups and checking the results of all NEP activities. The current SO1 indicator should be revised for the new strategy period (FY05-09) to more closely measure the SMME business environment and Sol's impact on it. The indicator should comprise a composite of a number of indicators that measure the SMME business environment, particularly in those areas of the strategy's focus (reducing constraints on business, improving business skills, and increasing access to credit). The current IR1 indicator is appropriate but it should be applied more precisely. The current IR2 indicator should be revised to read, "Number of business entities achieving business skill improvements," IR2 performance targets should increase significantly to reflect the potential of current and planned activities. New Economy Project Final Evaluation Report USA ID/Jamaica Y May, 2003 CONTEXT OF THE NEW ECONOMY PROJECT Per capita GDP growth has been negative for most of the last 25 years and average incomes in Jamaica have not improved for the last three decades. This occurred despite large amounts of investment as a proportion of GDP (30%), implying very inefficient investment. Throush the early 1990s Jamaica's economy experienced slow growth and continuing high inflation (abo1.e 20%) and high unemployment (16% in 2001). The Government of Jamaica's (GOJ) intenention in the 1996 financial crisis precluded a run on banks and averted collapse of the financial system. But it increased public sector debt to about 144 percent of GDP (2000) and has caused continuing high real interest rates, crowded out investment, and reduced export competitiveness. Additionally, problems in infrastructure, public security, and public administration are compounded by lack of government resources. Despite these difficulties, the incidence of poverty has declined in the 1990s, particularly since 1997. The proportion of people falling below the poverty line fell fiom 3 1 % in 1989 to about 17% in 1999. One cause is that real wages have increased as high inflation has fallen in the late 1990s. Another cause is that the informal sector, perhaps accounting for 40% of GDP. has been growing steadiIy, and this has escaped calculations of GDP growth. The World Bank estimates that the informal sector could account for 30 percent of the increase in per capita consumption expenditure in the late 1990s. With the financial crisis behind it, GOJ in 2000 began strong corrective steps. including fiscal tightening and irnpiementation of a strategy to reduce public debt- Its goals are to restore economic growth, ensure that growth is inclusive and that the poor are adequately protected, improve governance, efficiency and effectiveness in the public sector, and ensure sustainable development. Inflation has declined to single digits and real interest rates have fallen. The economy has grown slightly in each of the past two years, despite the negative impact on the tourism sector of the September 1 1,200 1 attack. GOJ's strategy to reduce its debt and increase its efficiency while sustainably reducing poverty through economic growth are key targets of AID-Jamaica's development pro+pm. To address these problems, in I999 AID-Jamaica developed a new Economic Growth Strategy for the FY99-04 period (Strategic Objective 1 (Sol)). The goal of the SO1 strategy is "to improve the business environment for developing the small, medium and micro-sectors." AID-Jamaica intends to achieve this objective by reducing key business processes while fostering competition, improving business skills at the company level, and increasing the availability of private financial for small and micro-businesses. Kw Economy Project 1 .\far. 2003 Final Evaluation Repori USAILMJamaica Development Associates, Inc. BACKGROUND OF THE NEP AID-Jamaica desired to tackle these objectives with an approach that differed from many traditional AID projects. It wanted a "demand-driven" approach where AID harnesses the demand of private and public sector clients in Jamaica to achieve its objectives for the SMME sector. Through this "merchant banking" approach, assistance is provided to programs/projects of local companies and public agencies that have the best chance to contribute to AID-Jamaica's SO1 objectives for the SMME sector. h 2000, AID contracted with the Carana Corporation to implement the New Economy Project (NEP), which AID-Jamaica calls the "engine" of its SO1 Economic Growth Strategy, Under the $6 million contracted, NEP has over the past 2 '/2 years completed, initiated, or is now preparing about 50 "activities" (e.g. sub-projects). AID anticipates providing Carana $2 million more to extend the project another year from August 2003 to August 2004. Each of these activities is designed to support one or more of SO 1's three Intermediate Results (IR) objectives: b Reduce key business processes while fostering competition b Improve business skills at the company level, and b Increase the availability of private financing for the micro and small business sectors Recipients of assistance are usually required to contribute significant resources to their activity, so AID/NEP assistance is designed to leverage additional resources, which should encourage ownership and sustainability. Under this demand-driven approach, NEP doesn't conduct project "identification," "appraisal," and "preparation," but "markets" its services, applies "selection criteria," and manages "deal-flow." For example, in Year 1 it considered over 100 activities, short-listed and reviewed 60 of them, and initiated support on 20 of them. PURPOSE OF THE EVALUATION In January 2003 AID contracted Development Associates, Inc. to evaluate NEP and its contribution to the overall SO1 objective (as opposed to other activities under SO1 being managed directly by AID-Jamaica under Cooperative Agreements). AID-Jamaica asked the two￾person Evaluation Team to examine NEP's "operation and impact," "the quality of NEP project management," and the "quality and impact of assistance being provided by NEP." This includes the extent to which the activities being implemented by SO1 management are likely to achieve the performance targets under the three Intermediate Results. AID-Jamaica also asked that the Evaluation Team make "recommendations for modifications in SO 1 activitieslproject design and implementation for the remaining life of the SO," including adjustments in "the level of performance targets for the indicators." Additionally, during the initial meeting in Kingston to organize this evaluation, AID-Jamaica asked that the evaluation be conducted in the context of its future task to develop its 2005 - 2009 strategic plan. To conduct the evaluation, AID-Jamaica specified that the team should review SO1 documents and interview key managers of Sol, NEP, and activities funded or sponsored by NEP. The Evaluation Team comprised two people who spent four weeks (February 20 - March 19) in Jamaica. During this period, the team reviewed project documents, conducted interviews with the top managers or key people for 3 1 NEP activities, held a focus group of people who received training under NEP's IR-2 activities, interviewed several other trainees, conducted numerous interviews with managers and key staff at AID-Jamaica and NEP, and drafted this evaluation New Economy Project Final Evaluation Report USAII)/Jamaicu 2 May, 2003 Developmenf Associates, Inc. report. The interviews and the focus group were conducted in Kingston. Monteo Bay and Mandeville, the three geographic areas of focus for NEP's activities. 11. SCOPE & BACKGROUND OF NEP ACTIVITIES EVOLUTION FROM DESIGN TO IMPLEMENTATION The AID contract with Carana specifies a demand-driven approach, as described above. and sets out a wide-ranging agenda for NEP to address. In keeping with a demand-driven approach. the Statement of Work (SOW), rather than specify activities to complete. identified under each IR objective "opportunities for intervention" and "activities" to pursue. The opportunities and activities to support development of the SMME sector include: Reducing redundant bureaucratic requirements and streamlining approval processes (business registration, business licensing, customs procedures, documentation requirements, and phytosanitary requirements), Establishing a business roundtable to consider constraints on business (to include trade negotiations, FINSAC debt, the GOJ budget, privatization, and customs modernization) Addressing privatization procedures and utilities' regulation (the telecommunications sector), Encouraging quality services at competitive prices in the telecommunications sector. including helping to establish the regulatory authority, advising the new Spectrum Management Authority (SMA), and other requests hm AID Working with the public and private sectors to encourage the growth of internet service providers (ISPs) and alternative voiceldata carriers Intervening in the transportation sector to reduce congestion of people and goods Improving company skills (market research, management tools, plant layout. product quality, and develop sustainable follow-on activities for the Jamaica Esporters' Association (JEA) programs) Help the financial sector to expand lending to micro- and small businesses. including developing new financial products, expanding geographically, and attracting capital for greater on-lending, and Achieving "buy-in" from the public and private sectors for this agenda Over the contract period NEP has squarely addressed most of this wide-ranging agenda. As will be detailed below, it has focused most of its efforts on reducing the constrainrs posed both by government "red-tape" and lack of business capacity in the prjvate sector, improvin~ business skills, and improving access to credit. In the first year it addressed part of the teIecommunications deregulation agenda specified above. NEP has not addressed in a significant way the transportation issues and the public debt/fiscal management issues as - suggested in its contract. Given the 2 !4 year period it has operated, the wide breadth of its agenda, and the demand-driven nature of the project, NEP has been very successful in initiating action as suggested by its contract. NEP's initial strategy was to work with the private sector. But over the life of the project the GOJ has become a large client. This is due to a number of factors: ~Vew Economy Project Final Evaluation Repoll USAID/Jamaica Development Associates, Inc. b GOJ "red-tape" is a major constraint on tens of thousands of SMMEs b GO3 has persistently demanded support b Government-related activities tend to be larger and, therefore, their potential impact is larger while posing fewer activities to manage, and b The capacity of private sector organizations (PSOs), particularly associations of businesses, has proven to be relatively weak NEP also deliberately attempted to target for assistance stronger organizations which could "deliver" benefit to the SMME sector. As a result, NEP does not target at the firm level those businesses that need the most help. Rather it attempts to target its assistance on larger, more capable agencies and businesses that can best deliver improved conditions for SMMEs. In this sense, project beneficiaries are not those with the most need, but those that can provide the most help to the needy. According to data provided by NEP, it has completed, is implementing, or is considering supporting 50 activities. In targeting assistance, NEP has not focused on any particular sector but on those public and private organizations that help SMMEs across all sectors. Projects have been spread fairly evenly among the private sector, non-profit organizations representing the private sector, and government agencies. NEP has focused most of its effort on IR1 (reducing key business processes) and IR2 (improving business skills) objectives. IR3 (improving access to credit) has decreased in importance over time. Technology, particularly information technology (IT), has been a strong focus due to its perceived capacity to streamline business processes, reduce constraints on business, and to facilitate access to credit. Although IT and finance figure prominently in NEP's activities, they do not represent a focus on the IT or financial sectors. The average size of NEP's contribution to activities has increased over time, rising from about $60,000 per activity for those activities already completed, to about $168,000 for those projects now being implemented. The increase is due in part to the lengthier periods required to implement larger projects and the fact that "Phase 11" follow-on activities implement recommendations and solutions developed in the shorter, smaller, Phase I planning activities. Nevertheless, the increase in size and length of activities also reflects thinking at NEP and OEG that implementation of fewer, larger projects is easier to manage and will provide more impact per dollar. Additionally, NEP over time has learned "who can deliver and who cannot," which further limits the number of prospective partners with which it works. This section assesses the success in implementing activities under the project as well as their ultimate impact on SMMEs once they are operational. It also assesses the likely future results of activities now being implemented but not yet operational. In doing so, the section address the likelihood that NEP will achieve its performance targets, beneficiaries' views of NEP's assistance, and the factors influencing NEP's progress. It also assesses the relative merits of progress made towards each of the IR objectives. New Economy Project 4 May, 2003 Final Evaluation Report USAID/Jomaica Development Associates. Inc ACHIEVEMENTS TO DATE Table 1 below illustrates the achievements NEP has made to date.' The table indicates that most activities are still being developed andor tested. It also indicates that several activities with potentially very large impact will move from pilot testing to active operation 01-er the nest several months. Results are likely to increase greatly during 2003. Eight activities in the table are now currently achieving results. primarily in suppon of the XR2 objective. The cohort of training programs under IR2 - Manchester Chamber of Commerce (MCC), SMME Business Skills, and BDO -- have either been completed and awaiting approval on renewal, or are ongoing. Collectively. they are far exceeding the IR2 performance target. The MCC activity ended in late 2002. It trained people fiom 130 SMME "business entities." falling short of the activity-specific target of 1 70.' Heavy flooding and MCC market ins challenges contributed to the shortfall. With one month left in the activity, the SMME Business Skills activity has trained people fiom 304 business entities, currently falling short of its activity￾specific target of 500.' Given the time remaining left on the schedule, this activity also appears unlikely to gain enough commitments for training before March 3 1 to meet its activity-specific target. %DO started its 3-year program in July 2002. As of the end of February, it had trained 120 entities, out of a 3-year cumulative target of 600 - 1,200 SMMEs. The wide range in the target is due to a revision of the BDO program, cutting class sizes from 60 to 30 and cutting training time from two days to one day. At its current pace, it is on track to train people fiom about 700 business entities. Overall, these training programs are meeting their SO1 IR2 targets but are missing their activity-specific targets. Response to the training has been positive. A focus group and several interview of those people receiving the training indicate that it is of high quality and great use to them. This finding is consistent with those found in an audit of the software training components of the SMME Business Skills Improvement ~rog-ram." Our focus group and inteniew participants rated the training, based on the type of courses, the structure and size of the classes. the instructional materials, and the instructors, very positively. Additionally, the MCS Bizpay.com activity under IR2 is also being implemented and is achieving results. It is supposed to help 500 companies over three years (March 2002 - Febmaq" 2005) to automate their payroll functions through an online application (BizPayCentral.com) service and support. The service costs US520 per month, which is affordable for most SMMEs. It was tested in January 2003 and was launched in February. As of February 2003 the BizPay program has 93 customers. MCS has also won a GoJ contract to provide the sen-ice to government agencies. 1 TWO activities NEP supported are not listed because they have become directly managed by AID and. therefore. - are outside of the scope of this evaluation. These activities are Automated Clearinghouse and the Jamaica Conference Board. Jamaica National Micro-Credit Company is also an activity directly managed by .AID under a cooperative agreement. Because of its importance to the IR3 objective. we cover it in this evaluation. "Business Entity" is defined as a company. a partnership. a start-up or a soie proprietor. The data for MCC. SMME Business Skills and BDO was provided by NEP on March 13. The performance target set for the IR2 indicator differs from the combined total set the three training programs. The combined ac~ivity￾specific targets are more ambitious than the IR2 target. See "SMME Business Skills Improved Program: Interim Evaluation Report - Software Training Component." Marion E. Blake & Associates. March 6.2003. rVm Economy Project 5 .%fay, 2003 Final Evaluation Report USAID/Jnmaicu Developnlent Associates, Inc. Table 1 Type of Measure Numerical Measures Mixed Mcasurcs Non-Numeric Measures* Legislation No Indicator Selected Suspended, Stalled or Completed with No Result Paymaster CCM R SBAJ JCSE Approval Process 1 Startup Manley CIB MCS-2 JMA Bus. Skills JlPO IRAE Implementation I Pre￾Results NLA# Customs EPayment# SAJ PWC Exordia MIND IBM - ." FTC PSOJ ECommerce Leg. Credit Bureau Safeguard Leg. Regs & Legs ADSC I..- I # These activities are now conducting pilot testing with a smaH number of users. They are scheduled for Custotns EPavment in Summer 2003. Implementation I Achieving Results SMME Bus. Skills BDO 11 implementation soon -- NLI Completed / Achieved Results ORC@ MCC MCS- I JNMCC LETS Investments Mona School Briefing Room ICT Sector Assessment ICT Intec l l -- in Spring, 2003 and @ These activities have just been launched -- ORC on March 17, 2003 and PCA in February 2003 -- and it will take about six months to understand the rcsults they will achieve. * Use of surveys could measure results, as it was for Briefing Room New Economj? Project Final Evaluation Report US,4/D/Jamaicu Development Associates, Inc. A smaller number of IRl programs are currently achieving results. The Briefing Room is making some progress in illurninatin8 trade-related issues. It is providinz trade briefs to a number of CEOs, has a website with trade information. and led a delegation of 11 prominent CEO and public officials to Quito, Ecuador to learn more abut the FTAA. Our meetings indicate that reach and impact have been limited thus far but is improving. There is some concern that the forum will be utilized by those more interested in protectionism than expanding fi-ee trade. .A survey completed in September 2002 indicates that Briefing Room has succeeded thus far in raising awareness about trade agreements, but it has not yet significantly improl-ed the knowledge about specific issues that is required to help CEOs make decisions.' The Mona School seminar on the comercialization of science and technology reached over 100 attendees. but it was a relatively small, "one-off' delivery of information and networking opportunities. With respect to performance targets, Briefing Room represents one business constraint tqeted by AID. NEP has also initiated reduction of one business process under PCA and four more business processes starting March 17,2003 (ORC; a fifth will be added once the Companies Act is enacted). NEP staff indicates the full impact of the ORC and PCA will take about six months to be seen. Thus, six business constraints are currently being addressed, but the resulrs for most of them will not be apparent until later this year. NEP's IR1 activities have thus far achieved limited res~lts.~ Under IR3, NEP's assistance to the JNMCC has been valuabie. NEP has helped with the development of a training manual for loan oficers along with the development of a loan management system. MFIs sustainability is highly dependent on the quality of its loan officers and on loan management systems, given the high administrative cost of small loan management. loan assessment, and collection efforts, JNMCC has been able to provide a better semice to its clients and increase its outreach using the products supplied by NEP. NEP's credit bureau activity addresses one of the main bottlenecks in SMME financins by enhancing the quality and accuracy of information on potential clients. This allows commercial banks to better assess loan requests and to avoid inflating the perceived risk because of lack of client information. NEP has assisted JETS with the drafting and review of a new le_eislation on the institution of credit bureaus nationwide. The draft legislation is currently in the Ministr). of Finance, awaiting submission to Parliament. GOJ feedback has revealed some differences of opinion that are currently being rnanased by JETS. The advent of credit bureaus in Jamaica will certainly have a significant impact on the SMME sector if their services are valued by the commercial banking sector. Although the legislation has not yet been put into effect, XEP's input has contributed to developing the proposal. The impact of NEP's support of securities dealers LETS Investments and CCMB on the SMME sector is not clear. These two institutions were approached by NEP and asked to participate in the NEP program. Representatives of these two institutions confessed that their pro-ms were not necessarily geared towards SMMEs and did not particularly consider the SMME sector as a target market. The subsidies they have received from NEP for their marketing efforts were welcomed by the LETS and CCMB. But both institutions informed us that the medium "A Survey of the importance of Briefing Room." Market Research Sewtces Lirniled. September 2002. 6 This assessment does not apply to the ICT and IC lntec I1 Desip work NEP conducted for GOJ and the Inter￾American Development Bank ( IDB) because the team was unable to cover these activities. New Economy Project 7 Maj, 2U03 Final Evaluation Repon USAID/Jamaica Developmenr Associates, Inc. enterprises existing in their current client portfolios were firms that were already able to access funds through the commercial banking sector. These firms could have sought the use of securities products if necessary. It is not clear that the injection of AID funds had any impact on attracting new SMMEs to securities products. SMMEs that do not have access to the financial sector, constituting the main target segment of IR3 as currently defined, would not have been eligible for LETS and CCMB services. In terms of performance targets, of the IR3 activities, only JNMCC is meeting its target each year. The CCMB activity was abandoned with no genuine impact. Under LETS Investments, the number of SMMEs receiving financial services is unclear, but it may fall well short of the 500 targeted over a two-year period. MUCH LARGER REACH LIKELY IN 2003 - 2004 Table 1 indicates that the most significant payoff from NEP's efforts will likely be achieved this year. Fifteen activities are now in implementation and, of these, four are now testing or have just launched systems that will reach a large number of SMMEs within the calendar year. The Office of the Registrar of Companies (ORC) has just completed its testing of the system and launched it to the general public on March 17, 2003. NEP staff indicate it will take about six months for the full impact of the ORC activity to be felt. With the launch, ORC is hoping to significantly reduce (fkorn weeks to a day) the time SMMEs and other registrants spend on four business processes associated with company registration (one more process will be improved when the Companies Act legislation is enacted, which is expected this year). Zt also hopes to improve compliance with registration laws by 20%. The new system will help the existing 35,000 SMMEs that are currently registered and the additional 6,000+ entities that are likely to register annually in the future. The activity will likely result in more SMMEs paying taxes and accessing credit from banks and credit unions. It will also help those who require information about individual companies, such as banks, law enforcement personnel, market researchers and others. The potential impact is limited by the large number of SMMEs who associate registration of their company with tax collection authorities. It is also limited by the accessibility of SMMEs to the internet. However, the improvements in business processes that are inherent in the system will enable the ORC to provide faster service for those who visit ORC offices directly. The E-Commerce Legislation, which will enable credit card payment for services, will eliminate the mail-in process for credit-card holders. The PCA activity involved two phases. The first was to identify constraints to pesticide-related businesses and to improve the approval process. The inability to complete labels for pesticide products and access to testing services were identified. Processes for labeling and testing were changed. The target was to reduce the time required to register and license new pesticides by 30% and increase the transparency of the process.7 The baseline monitoring exercise indicated that 19% of cases are processed in less than six months, 44% take six months to a year, 21% take one to two years, and 12.5% take more than two years. The target was to reduce this to under six months for 50% of cases, six months to a year for 40% of cases, and one to two years for 10% of cases. PCA indicates that they do not yet know how impact of their process improvements. A significant impediment to reaching the target in testing involves the relatively poor capacity of 7 This fibwre 1s from the Task Order and the Francis-hurse memo lo AID on PCA Basellne data Februarr 1 1 2002 Thc K~pstrar gmc us a target of 50',, reducrlon In llme New Economy Project 8 May, 2003 Final Evaluation Report USAID/Jomaica Development Associates, Inc. local testing services. The domestic testers can handle perhaps 50% of the reylated chemicals. The remaining 50% must be tested abroad - a UK company is the primary vendor at present. Thus, all samples must be sent via DHL and must clear Jamaican customs. Phase 2 involves automating PCA's workflow. The new MIS system was installed in December. tested in January and has just become operational after "debug@$' and staff training. NEP staft' indicate it will likely take some time to see the full impact of the implementation. In addition to significantly reducing the time required to process applications. it will enable PCA to monitor the time its takes to process cases. There are about 200 to 300 chemical-related stores ("conservatively"), about 20 local manufacturers of pesticides: and about 25 major importers. There are also about 20,000 end-users, mostlyfarmers. The project will directly impact the 300- stakeholders that are involved in the distribution of the chemicals and will impact end-users indirectly. There is also a demonstration effect. Three other government agencies have come to look at the effort. In conclusion, about 300 SMMEs are likely to receive significantIy improved processing of applications. Farmers and other end-users will see a wider variety of pesticides from more suppliers at lower cost, which will likely improve agricultural productivity. However. domestic testing capacity will limit the overall reductions in time to process approvals. The National Land Agency @LA)'$ automation of its E-Land system for land search, titling, mapping and transfer processes began testing in January 2003 with about 30 law firms who represent important users of the system. Once operational, the NLA will support its existing 150,000 registrants and those interested in Jamaican real-estate. The system is targeted to reduce the time to obtain a title (50%), real estate sales data (75%), the geodetic survey (50°b) and the geographic survey (1 5%). In actuality, NLA expects these processes to be reduced fiom days to hours or minutes, Land-use planning will also be facilitated by the system's search capabilities. benefiting real estate developers and foreign investors. The potential impact is limited by the accessibility of SMMEs to the internet. It is also limited by the fact that the current system will not enable NLA to accept credit card payment (even after enactment of E-Commerce legislation). Users mail in payments, a user account is set up, and the expense of senices is debited fiom the account. However, the improvements in business processes that are inherent in the system wili enable the NLA to provide faster service for those who visit %LA offices directly. The Customs E-Payment system started pilot testing in February and testing is scheduled for completion March 3 1, 2003. The pilot involves 16 customs brokers and four sites-"%en the system is made public, targeted for June 2003. it will cover both Bustamante-Kingston and Montego, which handle nearly 100 percent of international trade volume in Jamaica. About 350 customs brokers and 1,000 major commercial importers are major beneficiaries. as are smaller importers who help drive 3,500 - 4,000 imports per week. It is expected to reduce the time to pay customs from 24 hours to two hours.9 Overall, it is also hoped the activity will improve customs compliance and increase the tax base. However. the impact on the overall importdexport process is contingent on a number of other processes being automated and inte_g-ated together - requirements of the Ministry of Health. the Bureau of Standards and others. Without improvements in these other processes. the overall irnp~~export experience will remain 'They have transacted as of March 13. 2003 43 electronic paymenls from two banks roraling about J973.000. This figure was provided by a senior Customs Department official and varies with the TO target of reducing the process from two to four hours down to five to I0 minutes. New Economy Project 9 Mq,. 2003 Find Evaluation Repon USAID/Jamaica Development Associates, Inc. challenging, despite the very significant improvements about to be instituted by the Customs Department. When these four activities become fully operational this year, their reach will be quite broad - improving government services and the capacity of wholesale suppliers to provide service to their customers - for tens of thousands of SMMEs. Compliance with laws and regulations will likely improve, thereby expanding the tax-base. Other industries such as banks and real-estate, and other agencies, such as law enforcement, will likewise benefit. Collectively, they are targeting the reduction of 12 business processes, which represents most of the constraints targeted under the TR1 objective.'* Consequently, most of the impact represented by AID'S targets will come on line in 2003. Thispotentially broad impact of several of these activities is mitigated somewhat by some important constraints that remain - access to the internet, inability to pay with credit cards (legally andor technically), or existing deficiencies of other stakeholders (other government agencies and local chemical testing services). Nonetheless, the impact of these activities will become quite significant this year. As indicated in Table 1, the potential achievements and impacts of the remaining IRl activities that are now in implementation or, as is the case with legislation, waiting to be implemented, are not clear. Some have potentially large impact, but the timing of their completion is difficult to predict. Others may well be implemented this year, but their potential impact compared to those described above is not as significant. Some activities are likely to achieve significant results, but their implementation to full operational status will likely wait until next year. NEP has assisted in some capacity to develop legislation - the E-Commerce legislation, Safeguard legislation and Credit Bureau legislation. All three can potentially play an important role in significantly improving the business environment - respectively, dramatically facilitating internet-use and commensurate business reach, giving confidence to the country in moving on its free-trade agenda, and improving access to credit. For a variety of reasons completely outside of NEP's control, primarily the political process, these proposals have not yet been acted upon. Additionally, the Regs & Legs activity is now becoming operational. It currently is organizing its staff and steering committee. Its goal is to tackle "low-hanging fruit" in terms of reducing constraints through improved regulations, business processes and legislation. Its agenda is now being determined. The ultimate impact of these activities on reducing business processes is still in question. It appears likely that NEP will exceed its IR1 targets over the next two years. In addition to the full operation of ORC, PCA and PSOJ (seven business processes), NLA and Customs E￾Payment will begin full operation this year, reducing six more business processes. When added to Briefing Room, a total of 14 business processes will likely be reduced by the end of FY 2003. Further, FTC and SAJ are now being implemented, and it is possible they will begin reducing business processes in FY2004, adding five more constraints being reduced as a result of NEP action. So while NEP has thus far fallen short of its XRI performance target, it will likely substantially exceed them over the next 18 months. The technical assistance offered on the Fair Trading Commission (FTC), the Anti-Dumping Subsidies Commission, the Shipping Association (SAJ), and the Private Sector Organization of '* ACH and JCB represent the remaining targets attained. New Economy Project 10 May, 2003 Final EvaIuation Report USAID/Jamaica Deveh~ment Associates. Inc. Jamaica (PSOJ) is designed to improve business process flow and enhance IT capacit\-. In the case of the ADSC, the technicai assistance is desiped to enhance public education programs and. in the case of the PSOJ, to reduce the time to achieve consensus among private sector organizations and government. PSOJ accepted a human resource management plan last year and has just accepted its new accounting software in January. but it reports it is continuing to u-ork through its "bugs". PSOJ has just begun using the software operationally. The connecrion of these tasks to improving consensus between private sector organizations (PSOs) and sovemrnent agencies is unclear at best. The FTC is to achieve a 40% reduction in the time it rakes to resolve cases of anti-competitive practices. The ADSC indicator has not yet been selected. S.AJ continues to identify business processes to improve and is starting to tackle "quick-hits" process improvements before automation is underway. As these activities are still in implementation. it is not yet clear when they may start to achieve results. The remaining programs in Table 1 under pre-results implementation are IR2 activities - IBM ASP, Price Waterhouse Coopers (PWC) Exordia ERP, and the Management Institute for National Development (MIND). IBM ASP is now in a pilot through May 2003. Vendor relationships for support of the PWC Exordia ERP are still being arranged. MIND is now recruiting short-term consultants to support its development of its website, online curricula and marketing plan, alI scheduled for completion in August of this year. The impact of the IBM and PWC Exordia activities is questionable. PWC Exordia managers did not realize the purpose of the activity is to benefit SMMEs, and the pricing - up to USS6,OOO a month for the full package - is not conducive to marketing to SMMEs. IBM ASP is supposed to target 400 to 600 SMMEs over the next three years, but this seems unlikely given the USS1,000/month price for the service. This price is already a 50% reduction fiom the original USS2,000/month offering. IBM managers blame high overhead for the current price, despite NEP'S procurement of the back￾office software for them. They concede that the product is more suited to large or high-end medium-sized firms. The MIND program will target 50 managers initially with three online training programs. It hopes to attract 100 participants jn Year 1 and 2,000 total over eight years. Its ability to target SMMEs will depend on development of the curricula and the marketing plan. particularly the setting of course tuition, The collective impact of the IBM and PWC activities is likely to be low, while the impact of the MIND activity is uncertain. IR2 is already exceeding its SO1 IR2 targets by a wide margin. Provided that existing training vendors continue their work through the next 18 months, and provided that planned activities such as MCS BizPay and JMA Business Skills activities are approved and implemented effectively, it has the potential of even further exceeding the current performance tarzets. No 3R3 activities are now moving through the pipeline. JNMCC is lending to SMMEs and meeting its targets but, as noted earlier, CCMB is closed and LETS Investments is not likely to yield much new activity. The Credit Bureau legislation is awaiting Ministry of Finance decisions. NEP'S IMPACT Speed. Catalyst. Bridge. Mindshare. Quality. The Evaluation Team heard these types of adjectives repeatedly from those who have received assistance fiom NEP. Many beneficiaries indicated that, if NEP did not exist. their activities would have been initiated at some point, but they would be much later and the results would not be as good. This observation was made by New Economy Project Final Evaluation Repor? USAID/Jamaicu Development Associates, Inc. both private and public sector stakeholders. NEP is seen as a catalyst for innovation, bringing together critical stakeholders and resources and facilitating mutual understanding towards a common goal. Some used the word "passion" to describe the support provided by some NEP staff. More than a few observers applauded AID'S idea to contract out this project to NEP. In the words of one private sector stakeholder, "NEP has been very good at reducing the transactional friction" in starting and implementing projects with AID. There was disappointment registered by some, but this was usually confined to specific instances, such as the quality of a particular consultant or problems with an RFP process. Overall, the beneficiaries rate NEP and its assistance quite strongly. FACTORS INFLUENCING ACHIEVEMENT The findings presented in previous sections raise several factors that influence achievement. First, implementation of some activities has been slower than anticipated. A number of activities have taken more than two years from initial idea/discussion to actual implementation and achievement of results. Customs E-Payments was initiated in June, 2000. The ORC task order was approved in March 2001; Phase 1 began immediately and lasted three months; Phase 2 followed a month later and lasted 13 months. The launch occurred seven months after Phase 2 was completed. NLA, PCA and LETS Investments took similar lengths of time to develop, roll￾out, or close. PSOJ took about a year and a half to launch. Second, several activities have been suspended or abandoned -- CCMB, Paymaster and JSCD have been ended or suspended before completion. There are several causes of these lengthy implementation periods and the abandonment or suspension of activities. First, it is in part the "nature of the beast". Working with government agencies and struggling PSOs to implement significant change inevitably results in delays. Additionally, NEP has identified several other causes of slower-than-anticipated implementation: b Some activities were implemented without the client committing a full-time project manager to the activity. Building support and sustaining initiative on the activity within the partner's organizatibn was therefore difficult. b The assistance provided by NEP in some cases was "front-loaded." Once NEP completed its assistance in the design and planning of an activity, it had less influence over the client during implementation. This has been particularly a problem if the activity was highly leveraged and NEP's share of the activity's cost was relativeIy low. b It took some time to organize the appropriate skills to fit the activities NEP was pursuing. Because of the demand-driven approach of the project, it was more difficult to predict the types of skills that would be demanded. Many of the activities involved IT work, but early in the project there was inadequate IT experience on NEP's staff. Also, the initial plan was to match one project officer for each IR objective, but NEP learned it was better to fit the project officer's skills with the specific type of assistance required, regardless of the IR objective being pursued. It took some time to determine who should focus on marketing NEP's services and who should manage the clients. Further, NEP experienced staff turnover early in the project, as did OEG. New Economy Project 12 May, 2003 Final Evaluation Report USA ID/Jumaica Development Assocktes. Inc. b It took some time to build consensus among NEP and OEG staff on the use of the "mezzanine approach" to helping SMMEs. NEP has adjusted its approach to address many of these factors. NEP's staffing has stabilized and the OEG-NEP relationship is more clear. In terms of client commitment, KEP requires a partner to have a full-time project manager work on the activity. Almost all activities in KEP's portfolio include a cost-sharing component. NEP believes it will often be a minorit>- investor in most projects as it seeks to leverage funds from the beneficiary and other interested contributors. Therefore, its goal is to earmark half of its contribution for consultant time and half for implementation, thereby "holding the keys" to the activity all the way through its implementation. But closer attention should be paid to activities conducted through large and well-established firms such as IBM and PWC. For example, it is difficult to see how esisting hardware within IBM can constitute a direct participation and investment in the activity. There are additional causes to difficulties that NEP still needs to address. Some acri\.ities ha\-e not targeted SMMEs adequately. Prior pages have detailed the insufficient targeting of SMMEs of JCSE, CCMB, LETS Investments, IBM ASP and PWC Exordia activities. These organizations are not natural suppliers of products to the SMME community. Reorienting at least part of their operation toward the SMME sector is a difficult undertaking. IBM and PUFC are not sufficiently aware that the purpose under the NEP activity is to target SMNEs. The CCMB and LETS Investment activities were initiated early in the projecr's life and thus are part of the experience base NEP is using to adjust its approach and techniques. But IBM ASP and PWC Exordia were initiated in 2002. Part of the problem can be attributed to supplydriven project selection. Following the demand driven approach. most of the beneficiaries initiated activities by initially approaching NEP and submitting their proposal for consideration. But some organizations such as LETS investment, PWC, and CCMB mentioned that they were approached by NEP with the proposition to help them with their ongoing activities. The demand driven principle is crucial beneficiary ownership and commitment to an activity. The principle should be applied for all activities. Other activities are being subsidized, which could hamper their sustainability. The training programs under IR2 have been well-received by the people receiving training. However, the training has been subsidized at rates of 30% to 50%. The training providers are using the subsidy as a marketing tool, which is helping to boost numbers but can be counterproductive in the medium-term. As potential trainees become accustomed to a reduced price, they will be reluctant to sign up for similar programs at the full market price. This is especially true in small markets such as Montego Bay or Mandeville where IT trainers have a limited client base. Further, the limited number of trainees with whom we spoke in the focus group and personal interviews indicate that people are willing to pay full price for the training because they know its benefits. The existing training programs appear to be sustainable without a subsidy. R;ew Economy Project 13 .tfa"r, 2003 Final Evaluation Repm LrSd ILXJamaica Development Associates, Inc. IV. INDICATORS & PERFORMANCE TARGETS This section assesses the efficacy of the indicators for each IR objective and in some cases recommends adjustments. SO1 INDICATOR FOR IMPROVING THE BUSINESS ENVIRONMENT FOR SMMES: NUMBER OF NEW BUSINESSES REGISTERED The number of new businesses registered with the Office of the Registrar of Companies (ORC) attempts to indicate the health of the business environment for SMMEs. It has been called "the best proxy available" for this objective. However, it is fraught with problems. It neither indicates the health of the business environment for SMMEs, nor shows the impact of AIDNEP￾supported interventions. The number of new businesses registered is affected by many other factors than the health of the business environment for SMMEs. First, the number of new business registered is driven in part by the number of people becoming unemployed, particularly as GOJ downsizes. In this case, the number of new businesses registered is a result of a poor economy more than a sound economy. Future business registrations are likely to increase as GOJ continues its down-sizing. Second, remittances may represent 10 - 12% of Jamaica's GDP. An unknown number of new business registrations are fueled by an influx of remittances. In these cases, the overall ebb and flow of remittances drives some new business registrations. Consequently, difficult social and economic conditions in Jamaica that drive emigratiordremittances and the health of conditions in foreign countries, primarily the United States, are impacting the number of new business registrations. Third, some GOJ tax laws exempt certain types of debt from taxation. As a result, a number of people establish new businesses and register them so they can loan capital from the first company to a second company, thereby reducing the tax paid on earnings of the second company. An unknown number of new businesses are represented twice in ORC's database. For these thee reasons, the linkage between new business registered and the health of the SMME business environment is seriously diluted. On the other side, the indicator also does not accurately capture the impact of AID/NEP interventions. To be sure, meeting the objectives of reducing the constraints on business, improving business skills (of sole proprietors and unregistered business people), and making credit more widely available will positively impact the number of new business registered. But much of these efforts help businesses that are already registered. Those that probably focus more on unregistered businesses - such as the JN Microcredit project - do not require businesses to register with ORC to receive loans. So the indicator misses much of the impact of AID/NEP activities. Measuring the health of the SMME business environment is beyond the capacity of any single indicator. Over the past decade, a number of institutes have focused significant resources to develop measures of business and investment environments.'' These indices utilize a large I I The International Institute for Management Development (IMD) World Competitiveness Yearbook, World Economic Forum's Global Cornpetltiveness Report, Doing Business Index, PRS Group's International Country Risk Guide, and Price Waterhouse Cooper's Opacity lndex are some examples. New Economy Project 14 May, 2003 Final Evaluation Repoi? USAID/Jamaica Develovmeni Associates. Inc￾number of individual indicators to present an overall picture of a business environment. Some emphasize some aspects of business environments over others (e.g. access to credit, friendliness of government regulations to business, infrastructure. political risks or risks of public securit).. etc.). Additionally. over the past several years. international donors such as the World Bank. the International Finance Corporation (IFC). the joint Bank-IFC Foreign lnvestmenr Advisory Service (FIAS), and the European Bank for Reconstruction and Development (EBRD) have committed significant resources to assessing the business and investment climares in ~arious countries. They typically utilize surveys with between 50 and 100 questions to deril-e their assessments. Measuring "the improvement of the business environment for developing the small, medium and micro-enterprise sectors" will require a composite "basket" of indicators. For consistency in reporting results. we recommend retaining for the current contract period the existing indicator for measuring the health of the business environment. But for the next strategic planning period, we recommend that AID develop a composite of indicators that: 1) are most relevant to the SMME sector in Jamaica; 2) emphasize those IR objectives under SO1 that AID will be pursuing; 3) enable measurement of progress over time; and 4) are objective measures of Jamaica's business environment (as opposed to the opinions of businessmemexperts or a measure of Jamaica in comparison to other countries). There are two basic approaches for developing a basket of indicators: 1) conduct annual surveys of the SMME business environment in Jamaica and develop a composite indicator based on the survey; or 2) develop a composite indicator based on existing data being collected by other organizations. The first option would be tailored to AID'S specific needs and therefore would likely be a better measure, but would obviously consume resources. In pursuing this option. AID could adapt existing tools for measuring the SMME business environment. gearing these tools toward the three IR objectives - reducing constraints, improving business skills, and increasins access to credit. Annex 3 presents the SMME investment environment component of a larger business environment survey that is currently utilized by the World Bank & IFC. They intend to conduct annual surveys for all countries participating in their investment climate assessment. A number of questions in the sections on Government-Business Relations, Administrative Costs, CapacityILeaming, Finance, and Employment could be utilized or adapted for developing a composite indicator for SO]. The World BankIIFC have plans to conduct a survey on Jamaica. Therefore, a potential alternative is to partner with them in conducting this suney so that it can be modified to suit Sol's needs. It may also be possible to incorporate such a joint sunTey as part of the Conference Board's agenda in conjunction with the Indices of Business Confidence. The second option is to utilize existing measures of business environments to develop a composite indicator. Several of these tools exist but pose one or more difficulties - they don't cover Jamaica, they utilize opinions of businessmen rather than objective measures. they are not conducted annually or regularly, or they do not contain data important to OEG's SO1 objective (e.g. reducing government constraints on business), The World Bank's Development Indicators report contains more than 500 indicators, many of them relevant to the SMME business environment.'' But developing a composite indicator out of some of this data may take " Types of indicators in the WDl indude Investment & Risk: Financial Depth: Financial Flo\vs: Investment and Trade; Labor & Employment; Education: Information and Technology: Transponarion, Po\\-er 22 Communicarions: Travel & Tourism: Land-use and Agricultural Production. IWW Economy Project 15 .\fay. 2003 Final Evaludion Reporr (iSAID/Jamaica Development Associates. Inc. significant time and resources while not yielding a good measure. In conclusion, despite the likelihood that it will require more resources, AID may want to give more favorable consideration to the first option. IRI KEY BUSINESS PROCESSES REDUCED WHILE FOSTERING COMPETITION: NUMBER OF KEY BUSINESS PROCESSES REDUCED The number of key business processes reduced is a straight forward and appropriate indicator for the IR 1 objective. There have been some difficulties mentioned by both AID and NEP in applying it to activities. The first difficulty arises when defining "business process." As indicated in Table 1, most IRl activities have indicators that can be measured quantitatively. They set activity-level targets such as the reduction of time to complete a registration, accuracy of the data managed, or some other quantifiable objective. For most IRl activities, the definition of "business process" is not a significant issue. However, it is an issue with some activities where objectives that are more difficult or time-consuming to measure are set. One example is increasing the transparency of a business process. PSOJ, PCA, FTC and others have this objective. Some people are interpreting it as meaning transparency to the end-user, while others feel it means increasing the transparency of the process to the service deliverers. There is also general concern about how to measure cctransparency." Other examples are "increasing awareness" or "improving public knowledge". ADSC and Briefing Room have these types of objectives. All of these activities except Briefing Room have other objectives that are readily measurable, usually the time required to complete a business process on behalf of the customer. They also assume correctly that people will be satisfied by taking less time to receive a service so there is no need to measure the attitude of the customer. For these activities, we recommend excluding such hard-to-measure objectives and retain the measurable objectives of reducing the time taken to complete a business process andor improving the accuracy of records data. For activities where "transparency" has been a specific problem and is a key goal, the specific process that needs to be more transparent should be defined. Then the indicator will be more readily defined and be measurable. Many of the IRI activities involve establishment of an MIS system where the process is tightly defined, convenient to follow, and therefore is readily transparent. If transparency of the process is an issue where an MIS system is not a solution, then the process should be defined, followed and include notifying customers of the status/completion of their application, registration, et aI. In this case, the indicators will be the number or percentage of cases where the process waslwas not followed. The performance target would be a minimum percentage of cases adhering to the defined process correctly. Then records can be sampled to measure performance against this minimum target. Briefing Room and the public education elements of the ADSC activity pose a different issue involving the objective of public education and awareness of trade-related issues. With this objective, one cannot measure the process and assume that the customer will be satisfied with it, one must measure the customer's knowledge itself. Therefore, it is not readily measurable without a survey. NEP has addressed this issue with a survey of the Briefing Room activity, which was completed in September 2002. We recommend a continuation of such survey work on Briefing Room so that progress towards the public education objective will be measured. For the remainder of this strategic plan period, the same survey and sample frame should be used to New Economy Project Final Evalualion Report USAID/Jamaica 16 May, 2003 Development Associates, Inc. facilitate a consistent time-series measure. Surveys should also be conducted on other activities with a public education objective, such'as ADSC. PSOJ seeks four results - reduce the time to develop a unified position amonp public sector organizations consolidated under PSOJ. streamline the interface with GOJ. eliminate duplicate services, and introduce new services. The latter two objectives are reasonable and measurable. The first two objectives are not reliably measurable. They also do not match NEP-s support of PSOJ -- providing technical assistance to develop a human resource management plan and providing technicai assistance to implement improved accounting sofhvare. Ordinad>-. the first two objectives of the PSOJ activity should be dropped and replaced with measures of the progress in implementing the human resource plan and on improved efficiencies offered by the new accounting system. However, with no baseline measurement conducted for this activity prior to their implementation, the usefblness of such an exercise is much diluted. Credit should be given for activities on which NEP was a crucial interlocutor. Some legislation is critical for progress on business processes and NEP assistance may be hizhly appropriate to support its development. NEP should be given credit for helping to develop such legislation when it is done in conjunction with, and under the auspices of, a GOJ agency. The E-Commerce legislation is the best example. In NEP's February 2003 Status Report, it does not appear that NEP is receiving credit for its work (the "IR Count" reads "TBD). It is not possible to hold NEP accountable for the actions of parliament or GOJ agency. But is possible and appropriate to give it credit for legislation that has been drafted and accepted by the ministry with which NEP worked. In reality, activities under IRl belong to two major categories that necessitate different sets of skills -- regulatory and legislative activities; and improvements in administrative processes. To more adequately assess the accomplishments under this IR, increase the focus of mana, Dement efforts, and optimize the use of NEP and OEG resources, it would be helpful to create two sub￾intermediate results and measure achievements under each using different indicators. OEG and NEP can use the Legs & Regs forum to identify and prioritize legislative and regulator). items that will require NEP's assistance. The Legs and Regs activity will also take on the advocacy role necessary to expedite the revision and approval of draft laws and acts by the legislative power. In summary, most IRI activities have reasonable and appropriate indicators to measure. Most require measuring quantifiable improvements in business processes and some require surveys of attitudes. Exceptions for a few activities involve measuring "transparency." attainment of "consensus" or "streamlining" interaction with governments. These types of objectives should be dropped in these cases and avoided in future activities, particularly when they are accompanied by more direct and measurable indicators. IR2: BUSINESS SKILLS OF COMPANIES IMPROVED: NUMBER OF COMPANIES ACHIEVING TECHNOLOGICAL IMPROVEMENTS This indicator does not appear to match the IR objective. The IR objective is to improve the business skills of companies. but the indicator targets the number of companies achieving L'technological impmwrnmt.~". The activities being implemented under IR2 match the IR objective more than the indicator. BDO and IMP are currently training people in business ;Uw Economy Project 17 .tla~, 2003 Final Evaluation Repon LSAID/Jamaica Develovment Associates. Inc. management courses and these are being counted against the indicator despite its specification of "technological improveme'nts." Likewise, training programs now being developed will also target business management curricula (ManleyiMCC Executive Training, IMP Business Training et al.). Also, companies are now counted in "business entity" units, where '"ousiness entity" is defined as a company, partnership, start-up business or a sole proprietor. For these reasons, we recommend changing the indicator for the IR2 objective to read, "Number of business entities achieving business skill improvements." Currently, assessment of the training is being done through in-class observation, reviews of instructor credentials and post-classiin-class surveys of trainees. But the existing quality control techniques do not sufficiently address the impact of the training received. in addition to measuring the quality of training via post-classhn-class surveys, the impact of the training in the workplace also needs to be measured. We recommend that NEP training programs add an annual random sample survey of trainees who have received training in the prior fiscal year. This survey would measure the use and impact of the training in the trainee's workplace. AID and NEP should agree on a minimum target for both trainee satisfaction and trainee perceptions of the relevance of the training to hislher work or job-search. Falling below these minimum performance targets would prompt NEP to adjust training activities and/or trainee screening criteria. The current IR2 performance targets do not match the performance targets set at the activity level. The IR2 performance targets call for the employees of 450 companies to be trained in FY01-03. This cumulative target has been exceeded quite substantially by NEP. The individual activity-level targets of training programs approved under IR2 are 500 for SMME Business Skills, 170 for MCC, and 600 - 1,200 for BDO. New programs now being planned will add to the number of business entities targeted for training. In addition, we are recommending adjustments to IR2 programs to enable them to reach more trainees. For these three reasons, the IR2 target should be increased substantially. The exact targets should be developed based on hture funding levels. MANAGEMENT OF APPROVED ACTIVITIES Given its demand driven nature and multi-component focus, the NEP project requires close and well-organized management. The organization of the NEP team, staffed with a COP and 3 portfolio managers, has allowed for an efficient and responsive management of the different ongoing activities. A number of beneficiaries commented positively on the technical assistance they receive from NEP, both from the permanent staff and the short-term consultants. Additionally, some in NEP expressed the desire to provide more TA to projects where possible. Further, over time understanding of the requirements of technical assistance has improved because NEP has more experience in the types of activities that are supported and the staffing of NEP has been adjusted to fit these requirements more closely. The main infusion of technical assistance by the NEP team takes place in the inception phase of an activity during the proposal design stage. Portfolio managers review and fine-tune the proposals generated by the beneficiaries in order to enhance their feasibility and increase the impact of the assistance provided. Most, if not all, of the technical assistance provided at this New Economy Project Final Evaluation Report USAID/Jarnaica Development Associares, inc. stage is conducted through short-term local or expatriate experts. While the use of STTX is crucial given the nature of specialized assistance that is often required. it will be more cost effective for portfolio managers to be in a position where they can allocate more of their time to direct TA. Each portfolio manager, depending on hisher expertise. should be in a position to dedicate a significant portion of their level of effort to assist with the development and implementation of specific tasks identified on their projects. The original responsibilities of NEP's project officers included the delivery of direct technical assistance to the beneficiary partners, as well as marketing to initiate project activities. Over time, the need for project officers to market NEP's services has decreased. The decrease in marketing efforts began about one year into the project. Recently, OEG and NEP have decided to freeze all marketing efforts as a result of their concern over the level of funds remainins in the budget to cover ongoing activities and other projects in the pipeline. as well as the timing of the availability of additional project extension funds. Additionally, over time NEP has developed a modus operandi for the Chief of Party to focus on marketing and the project officers to focus on project management and technical assistance. The decreased demand for marketin3 and the shifi of marketing responsibilities to the Chief of Party free up more time for the project officers to deliver TA. The project officers can gain additional time for providing TA through adjustments to the amount of administrative tasks they perform. Designating an NEP administrator (assisted by the rest of the administrative staff members) - to perform information collection, logistics, and budgetary management -- will allow portfolio managers to increase the time spent on the technical facets of their projects. NEP currently has seven permanent staff members. and three of them perform administrative functions. The ratio of administrative staff is high. There is room to shifi administrative tasks to the administrative staK freeing up additional time for the project officers. The delivery of direct technical assistance by the portfolio managers assumes that the mix of expertise within the team follows the nature of projects carried on through NEP. The mandate under $01 covers the main three components of private sector development: enabling environment (IRl), business development services (BDS) (IR2), and access to finance (IR3). The current composition of the NEP team follows this organization. It is important to point out. however, that ttR1 activities seem to fall within two categories: b Regulatory and legislative tasks: revision and modernization of existing acts and introduction of new laws pertaining to the business environment, and b Business facilitation: simplification arid streamlining of administrative procedures using new models and information technology The business facilitation component is well covered by NEP IT experts. Given the importance of IR1 and the size and complexity of the task at hand, the NEP team and project identification and implementation would gain by the presence of a long-term law expert who can provide direct technical assistance to the beneficiaries and design well targeted tasks. New Economy Projecf 19 .\fa,: 2003 Final Evaluation Repon GSAID/Jamaica Development Associates, Inc. MONITORING AND EVALUATION SYSTEM In the area of project information management, the NEP project would tremendously benefit from the institution of a simple monitoring and evaluation (M&E) system fed by project administrators. Several requests from the evaluation team to prepare simple tables of basic information on the project and its activities were met with a lot of resistance. The evaluation team was first asked to comb through the contractors newly created project files to gather the information itself. This is indicative of the need to establish an M&E system that can be used on a regular, timely basis to make sound program decisions. Such a system would comprise one report that would present all basic project data required to manage the activities, objectives and portfolio. The data would be updated regularly by the administrative staff under the supervision of the project officers. For each activity, data could include budget data, expenditure data, indicators used to measure progress, the performance target for each indicator, the current status of performance, deadline to achieve the performance target, the number of SMMEs benefiting from the activity, a cost-benefit indicator(s)", inputs to the activity, milestones for the activity inputs, progress toward achieving the milestones, resources being leveraged, sector being assisted, and project classification. This list is not exhaustive and will need to be tailored to the project and its individual activities. Maintaining it in an MSExcel spreadsheet format would facilitate management analysis of the portfolio. An M&E system would enable the NEP to ameliorate the quality of management at the levels of activity, IR objective and overall portfolio. It would also help AID to improve the quality and efficiency of its supervision of the project and help it provide strategic direction. In doing so, an M&E system could also reduce the amount of time and effort to produce reports. Overall, it would enable OEG and NEP to focus more on results of the activities and make decisions accordingly. This will become increasingly important as more of the portfolio progresses from implementationlpre-results phase to the completed/achieving results phase. WORKPLANS AND REPORTING The quality of semiannual reports prepared by NEP is satisfactory but could be significantly improved. The workplans tend to be largely descriptive and redundant. While containing some figures on achievements to date and ongoing activities, there is little information on the impact (actual, potential, or perceived) of implemented activities on the SMME sector. Given the demand-driven nature of NEP, it is difficult to develop accurate workpiam. It is however important to view the reports as a management tool presenting analytical information of ongoing activities that allows both OEG and NEP to better steer the project and prioritize its sectors of focus. The reports should reflect the opinion of the contractor on the strategic orientation of the project. In addition to the workplan, NEP presents its ongoing activities to the OEG team during monthly meetings. Reports should be designed to include information on possible impact on the SMME sector. i 3 For example, with training programs the cost-benefit mdicator could be the number of people per dollar spent on the activity.. With IRI activities, -it could be the number of end-users benefiting per dollar spent on the activity. New Economy Project 20 May, 2003 Final Evaluation Report USAID/Jamaica AID OVERSIGHT OEG's supervision of the NEP project has significantly improved over the past year. OEG ream members have been assigned to specific portfolios to avoid redundancy and ameliorate the quality of oversight and the speed of decision-making. The proposal assessment and apprmaal time has been reduced to acceptable levels. Each OEG portfolio manaser works closely with designated NEP managers. This model is justified given the importance of NEP to SO1 activities and its broad mandate. This close proximity to contractor activities. in addition to the reports regularly submitted to OEG, should allow for a high degee of visibility. In addition to receiving information and following implementation activities. OEG's oversight should also consist of a careful assessment of the proposed activities to the overall SO I objecti\-e and to the specific IR goals. Targets, as defined currently defined under each IR. do nor necessarily measure the degree of achievement of overall SMME development goals identified in the PMP. While achieving set targets is important, OEG is in a position to pilot the project in order to meet its programmatic priorities and to leverage linkage possibilities with other current or future AID activities. As detailed earlier, NEP's current overall portfolio includes projects that will have little or no impact on the SMME sector. OEG management should analpe submitted proposals with the goal of understanding their potential (not necessarily direct) impact on SMMEs. The fact that the NEP project is demand-driven does not constitute a reason for a diminished focus on the achievement of overall AID programmatic objectives in the field of private sector development. The ACS form should include a specific question that direct15 addresses the potential impact on proposed tasks on the SMME sector. Systematically including a brief section in each proposal on the potential or perceived impact, going beyond a general description, will also help in assessing proposed activities. Supervision of NEP would also gain by the institution of systematic feedback mechanisms allowing OEG members to regularly assess and evaluate the value added by project deliverables to the SMME sector. This is particularly true as higher impact activities become operational in 2003. Through regular surveys and visits of final beneficiaries, OEG members can have a better sense of the actual value and meaning of reported target figures. Surveys of some IRl activities and training programs under IR2 were recommended earlier in the report. In keeping with its oversight responsibilities, we recommend that AID manage the conduct of these suneys rather than NEP. Sustainability equals leverage and efficient expenditure. Several activities can be made more sustainable by creating a plan, during the inception phase. to gradually phase out subsidies and direct funding. The Briefing Room is a perfect example of a highly successful activity, embodying a high impact potential, that would not be able to survive without AID'S assistance. Creating direct cooperative agreements to allow certain activities to continue does not address the sustainability issue. Focusing on sustainability in the early stages of an activity also enhances the sense of ownership of local partners and increases their awareness of the importance of identifying other sources of funds. Dependency on AID funds to undertake and continue certain activities is counterproductive in the medium term. Nm Economj Project 2 1 May. 2003 Final Evaluation Report LSAID/Jamaica Deveio~menf Associates. hc. CONTRACTING & TASK ORDERS Several interviewed organizations did not seem aware of the details of their respective activities, especially when it comes to targets, deadlines, and overall goals. As an example, both IBM and PWC did not see providing their products, subsidized by AID funds, to the SMME sector as being their primary goal. The New Horizon training institute marketed subsidized training programs to its existing clients that already have access to IT training and did not see increasing their outreach to SMMEs as an objective. LETS investment offers its securities management services to firms who already had access to finance from the commercial banking sector. Some activities were implemented without the drafting of a written agreement between NEP and the beneficiary organization. While this seems to be the exception in some cases rather than the rule, drafting detailed task orders, spelling out the overall goal of positively impacting the SMME sector as well as including detailed workplans with every partner is important. NEP's approach to identifying and selection consultants to provide short-term technical assistance is efficient. By including the beneficiaries in the process, NEP has better chances to hire consultants with a profile and experience that matches the exact needs of the final recipients of the TA. Beneficiaries, in most cases, get to directly interview the short list of possible consultants. This approach greatly eliminates the risks of fielding experts who do not respond to the requirements of the partners. It also significantly reduces the possibility of complaints on the quality of TA, increases the sense of ownership from the partners, and shortens the start up phase of the assignment given the fact that the beneficiaries and the potential consultants discuss specific issues before the fielding process is initiated. Most of the interviewed partners expressed their satisfaction with the quality of the TA received. However, several beneficiaries expressed less than full satisfaction with one or more of the short-term consultants working on their activities, in part because of their perception that the consultant was partly responsible for delays andlor shortcomings in the activity. But one beneficiary selected its own consultant and wished it had gone through NEP. The Evaluation Team notes the delicate nature of selecting short-term consultants. VI. CONCLUSIONS & RECOMMENDATIONS NEP is squarely addressing almost all of the wide-ranging agenda items suggested in its current contract. It is seen by many beneficiaries as an important catalyst to initiate SMME-friendly projects more quickly with higher quality. Overall achievement to date has been limited because many important activities are still being developed or tested. But in 2003 a number of activities will become operational and have significant impact on the SMME sector. It appears likely that NEP will exceed its IR1 targets over the next 18 months and continue to exceed its IR2 targets as well. IR3 targets are likely to be met through the JNMCC. Progress to date has been impeded by a number of factors and NEP has moved to address many of them. But the targeting of the SMME sector with activities and the sustainability of activities remain two challenges. Most activities support IRl objectives and the potential reach of these activities is much broader in scope than the activities supporting the other IR objectives - reaching tens of thousands rather than hundreds. Further, they are very important to the agendas of CARICOM regional economic integration and the advent of the FTAA. Some can serve as models in other parts of the world. But due to their ambition and complexity, they take longer to implement. IR2 activities have limited reach but most of them squarely target the SMME sector and they can be implemented New Economy Project 22 May, 2003 Final EvaIuation Report USA ID/Jamaica more quickly than some IRI activities. Under IR3, there are no more acti~~ities in the pipeIine. their results have been mixed, and NEP has registered some lack of enthusiasm for them. NEP has efficiently and responsively managed its portfolio but there is room for improvement. Project Officers do not spend sufficient time providing technical assistance. The reporting to OEG does not adequately focus on results and impact. Monitoring of results will become increasingly important as more activities become operational and needs impro\@ernent. RECOMMENDATIONS The Agenda NEP should focus more of its resources on IRI and IR2 activities, and vacate the IR3 space. To facilitate clarity and focus on IRL activities, AID should create two "sub-IRs" - one addressing legislative and regulatory constraints to business, and a second to address the business~a~ency process re-engineering activities. Each requires a separate set of skills and acknowledging their differences will enable NEP to address each type of activity more effectively. For the legislative/regulatory sub-IR, NEP may want to consider hiring a full-time legal expert. NEP should also receive credit for its work on legislative and regulatory matters when it results in a law being drafted and accepted by a ministry or agency as a possible bill for parliamentap consideration. Nurturing the Regs & Legs activity should be a top priority. Regs & Legs encompasses the entire panoply of IRI activities. In effect, it represents a potential "exit strategf'for NEP. enabling it to leave behind a public-private Jamaican partnership to carp on its work in perpetuity once it disbands. Legs & Regs is just starting, but one can see a sustainability issue immediately. NEP is hnding the entire cost of the two permanent staff positions working under the Steering Committed housed in the Office of the Prime Minister. AID and NEP acknowledge that this is not sustainable, but the activity's potential warranted the support. Wow that it is launched, a sustainable funding for the staff needs to be developed. There are four options - GOJ funding, AIDNEP fbndifig, private sector funding, or a combination of the three. The best solution is for the private sector to fund the staff itself because then the private sector can suficiently promote the effort to reduce unnecessary legislative and regulatory constraints to business. GOJ fbnding is not as desirable because the staff will be more beholdened to political pressures. AID/NEP funding is not sustainable. A second-best solution is for a sharing of expenses, enabling the private sector to increase its share over time. Under JRI, NEP should also develop additional linkages between activities. Currently. the same project officer is supporting both the SAJ and Customs &Payment projects, thereby serving coordination between the two. One important linkage that should be explored is the PSOJ and Regs & Legs activities. If private sector organization capacity is to improve, it will require PSOJ's development. NEP's past efforts with PSOJ are now just starting to bear hit. NEP should consider additional assistance to PSOJ with the goal of helping it contribute more to the Legs & Regs activity. Linkages between NEP's activities and other donor-supported programs. such as C-Trade and the Jamaica Cluster Competitiveness Project (JCCP), should also be encouraged. !Vm Economy Project 23 M~J, 2003 Final Evaluarion Repon L?'SAID/Jamaica 33 D~elopment Associates, hc. W Several private sector leaders noted the slow pace of legislation in Jamaica's parliament and attributed much of it to the overwhelmed staff of the Parliamentary Counsel. The slow pace of lsnl legislative accomplishment has been quite relevant to a number of NEP's efforts to reduce business processes. It may be possible for NEP to help improve the Parliamentary Counsel's administrative functions with technical assistance. NEP understands this potential and AID should encourage any business-process activity that has a reasonable chance to improve Parliament's hnctioning. NEP should also continue its focus on IR2 activities, particularly those supporting training. NEP should make the programs more sustainable by ending or quickly phasing out subsidies. To expand the reach of the training, IR2 resources should facilitate a SMME focus among more training companies and implement a "training of trainers" (TOT) approach. The implementation of a TOT program should follow the demand-driven approach of the project. The NEP team WI could market its assistance to existing business development services (BDS) providers and provide its assistance to interested organizations. The quality of BDS provided to SMMEs by BDS service suppliers should be assessed and NEP should help the supplier on: bw b How to develop adequate training products targeting the SMME sector b Now to appropriately price their services b How to correctly package their products b How to assess their client base and determine the profile of their potential clients, and b How to adequately market their products The injection of these sets of skills to selected BDS providers wiIl greatly enhance the sustainability of their services while eliminating the need for subsidies. Only BDS providers who iuw' are able to develop the right products and to correctly and adequately market them will survive. Similar to any other businesses, business-training organizations must compete and show the value of their products to SMMEs to enlist clients. Putting the burden of product development lisi and marketing on the BDS providers pushes them to view services to the SMME sector as valuable products worthy of their investment in human and financial resources. 'nw Management, Reporting & Monitoring The internal organization of NEP should be reviewed to lower the administrative and Y management burden carried by the portfolio managers. Administrative staff should be empowered with a monitoring & evaluation system. 'r+t Consideration of NEP activity proposals should continue to focus on the potential impact of the activity on SMMEs and the degree the activity manager is likely to focus on the SMME sector. The ACS form should include a specific question that directly addresses the potential impact of u the proposed task on the SMME sector. Sustainability should also be a focus early on in the process. Task Orders should emphasize the activity partner's commitment to serving the SMME sector. k? NEP should provide written reports to OEG no less frequent than quarterly and such reports should include performance data on current activities. Semiannual reviews should focus on the h$ status of performance, and process should be considered in the context of performance status. New Economy Project Final Evaluation RepoH USAID/Jarnaica May, 2003 AID should assume responsibility for conductins surveys/focus groups and checking - the results of all NEP activities. Due to the increasing number of activities becoming operational. AID should adequately budget for such work. Performance Targets & Indicators For the SO1 indicator, AID-Jamaica should implement a composite indicator that more closely measures the SMME business environment and Sol's impact on it. We recommend developing a composite indicator from an annual survey of the SMME business environment in Jamaica that focuses on the reduction of business constraints, improved business skills, and increased access to credit. AID-Jamaica should consider working with the World Bank and the Jamaica Conference Board to develop the sun7ey. The current IRI indicator is appropriate but it is sometimes mis-applied. For specific activities where appropriate measurable objectives can be found, do not add redundant or secondary objectives that cannot be measured. If transparency is a specific problem and an IT system is not the solution, then the process causing the transparency problem should be defined and foliowed. and the indicator should be the percentage of cases the process is not followed correctly. Two "sub-indicators" -- one for legislation/regulation activities, and a second for agencybusiness processes to reduce - should be created and measured for this IR. The current IR2 indicator should be revised to reflect current reality and read, "Number of business entities achieving business skill improvements." IR2 performance tarsets should increase significantly to reflect the potential of current and planned activities. The Model's Appropriateness, Prospects & Replicability The Evaluation Team believes the overall model/concept is appropriate and should be continued in the next planning period and replicated in other jurisdictions. As detailed earlier. the concept is to delegate provision of AID assistance to a contractor which meets AID'S strategic objectives by providing assistance on a demand-driven basis to those organizations that can best help achieve the objectives. This concept channels assistance not to those who are most in need, but to those who can provide the most help to the needy. Project recipients demonstrate their own commitment to an activity by providing significant resources of their o\m. The approach also changes AID'S role from execution to supervision. To a large extent, the overall project is being implemented according to the model and thus will likely achieve significant results. As detailed in this evaluation, the implementation of the model is improving the delivery of development assistance in several ways. First, it is reducing "the transactional friction" between AID and private organizations and government agencies. Under this model, both government and private organizations are more likely to learn about the availability of AID assistance and receive aid in a way that can achieve results more quickly. Second, the allocation of the project's overall budget on specific activities championed by specific organizations is enabling assistance to be targeted more precisely to organizations that demonstrate through the application process that they are more likely to succeed. Additionally. the approach is accommodatjng enhanced flexibility to respond to new assumptions or chansing conditions "on the ground," thus enabling project managers to reallocate resources more quickly to activities that are most effective. NEP's increasing emphasis on larger IRI activities and its !Vew Economy Project 25 .Way. 2503 Final Evaluaiion RepoH L'SAID/Jnmaica Developrnenl Associates, he. effort tci identify those Jamaican organizations that can "deliver" and those which cannot illustrate this advantage. Thus, project activities are more likely to achieve results and the impact of the overall project is likely to be more significant. Third, the model provides the possibility of leveraging aid to attract significantly more financial resources than is budgeted by AID. Leverage rates varied widely among the activities funded by NEP, but beneficiary resources devoted to individual activities frequently exceed the resources provided by AID. Finally, sound implementation of a demand-driven process better supports the sustainability and long-term impact of AID'S assistance. Under many activities in this project, Jamaicans propose their own initiatives, contribute their own resources, and manage the overall activities themselves with the support of outside technical assistance. As a result, these activities are more likely to be sustained by Jamaicans and, therefore, achieve long-term results. This evaluation has noted exceptions and recommended adjustments. The IR2 training activities are subsidized and, therefore, are not sustainable. We have recommended adjusting the program by ending or phasing out subsidies quickly and expanding its reach by targeting more training companies with a "train the trainers" approach. Any new programs that help service companies provide productivity-enhancing IT tools need to be better targeted to support SMMEs, and beneficiaries need to contribute more to the overall activity. Under IR1, the Legs & Regs activity is both promising and very important, and we have recommended that it should be made sustainable with contributions from the private sector and the government. Sustainability and targeting issues under some noted IR3 programs are obviated by the recommendation to vacate this space. These exceptions generally result from divergences from the model. In general, when the model is implemented as intended, it works well. But the model has several potential vulnerabilities that must be managed carefidly during project implementation. The approach accommodates flexibility in identifying activities to fund and, therefore, increases the potential for a project veering away from its objectives. Another issue is the degree to which NEP helps an organization to develop a concept and proposal. Too little help risks abandoning a potentially strong activity prematurely. Too much help risks creating a supply-driven project that proves unsustainable or ineffective due to lack of beneficiary "ownership". A related issue is knowing how long to work with an organization to develop a promising proposal before determining that it just doesn't have the capacity to succeed. Another corollary is knowing the quantity, type and timing of assistance to provide to an organization. Poor decisions in this context, particularly with private-sector beneficiaries, can result in providing assistance to organizations which will pursue an activity regardless of whether it receives AID support. Thus, the assistance becomes a mere subsidy rather than an enabler. Or alternatively, a poor decision can result in abandonment of the activity with no effect. The project model presents one additional issue, particularly relevant to the IR2 and IR3 space - does AID support of a single organization in a market niche constitute an unfair advantage for that organization, or is AID simply rewarding a properly aggressive organization which has a sound idea? Is AID picking "winners" over "losers"? As noted in previous sections, the overall experience of the project is usually consistent with the intent of the model - fund proposals that target one or more of Sol's three objectives, that originate predominantly from a beneficiary? and that are implemented with sufficient beneficiary "ownership". Those projects that originated with NEP approaching other organizations, do not - New Economy Project Final Evaluation Report USAILhVamaica May, 2003 Developmenr Associates, Inc sufficiently focus on helping SMMEs, and/or do not include suficient beneficiac-pro\-ided resources have been relatively less successful. To address the model's shortcomings idenrified above, OEG/NEP have developed criteria embodied in the ACS form for considerins activit>- proposals. Activities generally should meet a clear development need. be demanded by a Jamaican organization, be supponed by one or more organizations with the capacity to implement the activity successfully (with the support of technicai and other assistance). and require assistance consistent with NEP's core capabilities. Additionally. as noted earlier in the report, NEP has also changed its modus operandi to address these issues. The report recommends additionaf measures such as strengthening the ACS form and ensuring that beneficiary proposals and task orders strengthen the emphasis on targeting SMMEs. Regarding the issue of "picking winners," in general, NEP is on solid ground when an organization approaches it with its own idea that supports an SO1 objective. The issue arises if a clear need is identified by NEP and it seeks a specific organization to develop an activity to address the need. Then it is open to the charge of creating an unfair advantase. This is one more reason to be cautious in initiating a proposal. If OEG/NEP identifies a clear need that is most appropriately addressed with a private sector solution, then all organizations servicing the market niche should be publicly notified that assistance is available to those offering the best solution(sf. The winner(s) of the assistance should be publicly disclosed. In addition to sound measures and criteria to select activities, the model also requires strong oversight by AID. The project's approach has necessitated a change in AID'S role from one of project execution to one of oversight and supervision. The report has noted AID'S progress in adjusting to this new role. But it also recommends that supervision be strengthened in several ways - implement an M&E system, assign AID responsibility for evaluating results of activities with surveys/focus groups, improve the reporting of M&E information, and utilize the M&E system and improved reporting to focus management more tightly on the results of individual activities. The model can be replicated in other sectors and other nations. It requires clear objectives. contracting out to a capable implementing agency, clear understanding of roles between the donor and the implementing agency, flexibility in activities to meet objectives. beneficiary "ownership," and sound oversight and supervision. The model is being used with success by various donors in a number of applications across the globe. SMME sector development is a frequent application, as is development of social services and smaller-scale infrastructure through "social investment funds" and "community-driven deveiopmenf' projects. Consequently, the model as implemented in the New Economy Project can be successfdly replicated elsewhere. New Economy Project 27 .Way, 2053 Final Evaluation Repon CfSAID/Jamaica Page No. Annex I Persons Interviewed .............................................................................................. 1 - 1 Annex 2 Documents Reviewed .......................................................................................... 2- I I Annex 3 Joint World Bank-1FC ............................................................................................ 3- I ;Yew Economy Projecr Final Evaluation Reporr O'SA/R/Jamaica USAID Mosina Jordan, Mission Director Robin J. Brinkiey, Director, Program Development & Management Charles Clayton, Office of Program Development & Management Jim Watson, OEG Director John Wright, SO1 Team LeaderJCTO Jimmy Burrowes, SO1 Project Officer Donna Service, Project Management Assistant Shenil Thompson, Project Management Assistant Peter Klosky, Controller Avril Britton, Financial Support Advisor NEW ECONOMY PROJECT Robert Otto, Executive Vice President, Carana Corporation Michael Julien, Chief of Party Michael duQuesnay, Case Manager Cheryl Francis-Nurse, Case Manager Nigel Hall, Case Manager Counterpart Agencies / Service Providers Tony Chang, former President of Jamaica Chamber of Commerce Graham Duddey, Coffee Industry Board Clive Nicholas, Tax Administration Garfield Knight, National Land Agency Elizabeth A. Stair, National Land Agency Camella Rhone, Ministry of Commerce, Science & Technology Hycinth Chin Sue, Pesticide Control Authority Carlton Davis, Office of the Prime Minister Lorna Simonds, Office of the Prime Minister Judith Ramlogan, Office of Registrar of Companies Wilford Monison, Office of Registrar of Companies Windelf Smith, MCS Ltd. Robert Davis, MCS, Ltd. Trevor Fearon, Jamaica Conference Board & Jamaica Chamber of Commerce Rosalea Hamilton, Ministry of Foreign Affairs & Foreign Trade Frederick G. Betty, Jamaica Central Securities & Depository Limited Frank WhyIie, Jamaica National Microcredit Company Stephen Meghoo, IBM Barbara Lee, Fair Trading Commission Sara Ruth-Allen, Anti-Dumping Subsidies Commission ,Vew Econom\. Projed 1 -I .\fay. 2053 Find Evaluu~ion Reporr L'SAID/Jarnaica 37 Development Associates, Inc. Andrea Marie Brown, Anti-Dumping Subsidies Commission Loreen Walker, Jamaica Intellectual Property Office Ruby Robinson, Jamaica Advanced Computer Systems John Bayles, LETS Investments Donovan Cover, Manchester Chamber of Commerce Sophia Dunkley-Wright, World-Wide Technologies Carol Miller, Briefing Room Cynthia Mullins, Paymaster Athenia Campbell, Paymaster Michelle Williams, Jamaica Customs Department Anna Kay Lee, New Horizons, Ltd. Denise Williams, New Horizons, Ltd. Richard Dower, Price Waterhouse Coopers Mark Golding, Credit BureadJETS Pauline Grey, Shipping Association of Jamaica William Washington-Welch, BDO Patricia Francis, JAMPRO Greta Bogues, Private Sector Organization of Jamaica Andrew Cocking, Capital 62 Credit Financial Group Hugh Campbell, Emoquad Chad Cunningham, Carib Exchange 7 participants in training programs under IR2 Improving Business Skills New Economy Project 1-2 May, 2003 Final Evaluation Reuorr New Economy Project USAID/ Carana Contract ff OUT-PCE-1-801-98-000 14 New Economy Project Draft Workplan, October 2,2000 New Economy Project Draft Workplan, April - September 2001 New Economy Project Year 1 Review & Year 2 Workplan, September 2001 R4 Document April 200 1 USAIDLTamaica 200 1 Semi-muaI Review USAIDLTamaica FY2002 Annual Report Bilateral Program Performance monitoring PlanfSO 1, January 2002, US AID Jamaica New Economy Project Workplan, April - September, 2002 SAR 2002 Numbers from NEP, October 200 f - September 2002 2002 Annual Report Data for Jamaica New Economy Project Semi-Annual Workplans, October 2002- March 2003 SO1 USAD Semi-Annual Review, December 2002 NEP Status Report for February 2003 USAID SO1 Financial Status Report, Project No. 532-01 83, February 2003 USAIDNEP Activity task orders files USAIDMEP Close-out reports New Economy Project Proposals Sent to NEP Economic and Social Survey Jamaica 2001, Planning Institute of Jamaica World Bank Jamaica Country Assessment Strategy, November 2000 WTO Jamaica Trade Policy Review, 1998 Jamaica Business Roadmap - Reducing Business Constraints, Booz-Allen & Hamilton, February 200 1 BDO Small Business Training Guide Jamaican Cluster Competitiveness Program (JCCP) Program Description C-Trade Scope of Work New Economy Project SMME Business Skills Improved Program, Interim Evaluation Report - Software Training Component, Marion E. Blake, March 6,2003 Proposal for An Evaluation of Various Projects. Market Research Services, Ltd. September 2002 Survey of the Importance of the Briefing Room, Market Research Services Ltd, September 2002 Consumer & Business Confidence Report, 3Id Quarter 2002 Background of the Regs & Legs Survey, Market Research Services Ltd.. January 12, 2003 Jamaica Regulations, Legislation, and Process Improvement Project, Project Plan Phase 2, February 10,2003 New Economy Project Analysis of Small Business Association of Jamaica, Beverley Hall, December 2002 Office of the Registrar of Companies report on new businesses registered Activity Criteria Sefection form iVac Econorn.~ Projeci Final Evahratiort Report USA ID/Jumaicu Development Associaies, Inc. Update of Performance Targets for IR2 Projects, February 2003, Cheryl Francis￾Nurse Manchester Chamber of Commerce activity reports and data New Economy Project 2-2 May, 2003 Final Evaluafion Report USAfO/Jnmaica Investment Climate Assessment (ICA) SME Module (supplemental to core ICA survey) SALES AND SUPPLIES I. For the domestic inputs, what % comes from the same region where your plant is located? 90 2. For the domestic sales, what % went to the same region where your plant is located? ? o RELATIONS WITH SUPPLIERS k?iD CUSTOMERS 3. a. Over the past year, what percentage of your output (in value) is subcontracting work you do for other firms? % (if 0, skip to question 4) b.For what activities do firms subcontract to you? Identie no more than three (3) of the most important activities. i) Production of intermediate goods ii) Research and Development iii) Training iv) Accounting& nancial management 4 Marketing advertising vi) Repairsimaintenance vii) Personnel managemenu' organizational services viii) Legal services jx) Public relations x) Design services xi) Data processing xii) Cleaning. laundry xiii) Security xiv) Other (specify) - BUSINESS ASSOCIATIONS 4. Do you cooperate with other local producers in your industry in any of the following ways ? a) Lendingiborrowing machinery b) Product development c) MarketingIMarket Research d) Training of workers e) Purchase of inputs f) Attracting investments g) Exchanse of infomarion h) Subcontracting i) Other (specify) USUALLY FREQUESTL\- SOMETIMES SELWM NEVER 3 4 5 6 iVew Economj Pro~ect 3- I .Way* 2003 Final Evaluatiot~ Repor! USA JLMJan?aica Development Associates, Inc. w 5. a. Is your establishmentifirm a member of a business association or chamber of commerce? Is it voluntary or non-voluntary ? i) Business association Yes No Voluntary Non-Voluntary ii) Chamber of Commerce Yes No Voluntary Non-Voluntary b. Over the past 12 months have you used the services of or attended an event sponsored by the business association or chamber of commerce of which you are a member? Yes No FINANCE 6. If you have not applied for a bank loan what is the reason? i) Do not need loans Yes No ii) Application procedures for bank loans are too cumbersome Yes No iii) Collateral requirements of bank loans are too stringent Yes No iv) Interest rates are too high Yes No v ) Corruption in the allocation of bank credit Yes No vi) May create complications with tax authorities Yes No vij) Other Yes No 7. We've heard that establishments are sometimes required to make gifts or informal payments to bank officials to receive loans or credit. In your dealings with bank officials, was a gift or informal payment expected? YES NO BUSINESS-GOVERNMENT RELATIONS 8. Assuming total time spent dealing with government regulatory and administrative requirements to be loo%, what is the share (in percent) of management time spent on National, State, Municipal regulatory requirements?(must total 100%) 1 National % I StateProvince % [ Municipal % IDK I 9. Whether or not you currently do business with the government, could you please comment on the ease or difficulty of the following: 0 = No obstacle I = Minor obstacle 2 = Moderate obstacle 3 = Major obstacle 4 = Very Severe Obstacle Demee of Obstacle a) learning about public (government or parastatal) tenders. 0 123 4 b) pre-qualifying to bid for public tenders. 0 0 123 4 c) the bidding process itself: openness, fairness, complexity 0 12 3 4 d) the size of contracts as opposed to your capacity. 0 123 4 e) award of contracts: timeliness, fairness. 0 123 4 f) supervision of contracts -- competence, fairness. 0 12 3 4 g) timeliness and adequacy of payment for goods and services. 0 123 4 h) competition for contracts: w/ private firms, parastatals. 0 123 4 i) requirement of unofficial payments 0 123 4 10. (add to core qn. 44) b. Did your firm lobby the regional/local government or otherwise influence the content of laws or regulations affectjng your establishment? bud New Economy Project 3-2 May, 2003 Final Evaluation Report USA IDIJamaica Develoament Associates. Inc. 1 1. Has your establishment received any of the following forms of assistance from a national, regional or local government business support body ? If so how would you rate the value of such assistance? I = minor vahie; 2= moderate valzre;3 = niajor- value 4= critical valzie to ?mu* fh~i Not Provided Value to sour firm a) Financial assistance e.g. loan Suaranlee, venture capital XP 1234 b) Training, advisory or information services YP 1274 c) Technology development. R&D support NP 1234 d) Export marketing assistance or advice N P 1234 e) Special Tax Treatment (ie. exemptions) N P I2;J f) Other. specify NP 1234 ADMINISTRATIW COST~ 12. If the company was registered or re-registered during the last 36 months. please answer the questions in the table below for the most recent registration experience. Did you hire an outside company to help you complete the registration procedures? Yes=l, No=O If you have received any of the licenses listed below in the last 36 months. please answer the 1. TOTAL (MCLUDIXG ALL REGISTMTION PROCEDURES). OF Total Time (calendar days 15. How did the number of inspections change compared to the previous year at this time? [ Increased 1 Decreased I Stayed the 1 Do not know f N A I Value of Unofficial payments. gifts in local currenq Total Registration Cost cukstions in the table. 16. (add to core $5 1) g) If you resolved cases through the couns. Official Fees at Registration Agency h)was an unofficial payment to the judge required to resolve the dispute? (i) Each Time (ii) Sometimes (3) NEVER (8) DK (9) NA i) On average, how many weeks did those court cases take to resolve? weeks Unofticial payments. @fis j) In the most recently resolved case, if you won. were you able to enforce judpent (collect payment)? YES NO Oficial Fees ai asency k) If yes. how much did you spend for the enforcement fee? (%of settlement value) (8) DK (9) NA 1. BASIC ACTIVITY LICENSEPERMIT 2. CONSTRUCTION ~Vew Economy Project 3 -3 May. 2003 Final Evuiuation Reporl USA ID/Jamaica 3. WHOLESALVXETAIL (Calendar Days) Total Time (days) 4. RESTAURANT 5. OTHER No. of Ofices Visited 1 Development Associates, Inc. 'r 17.During the past three years, did you resolve an important disagreement with a customer or another business using any of the following private or informal methods of conflict resolution (for example, CAPACITY, INNQVATION, LEARNING 18. In considering how much to invest in training your workers, how important are each of the following factors? 0 = Not important I = Minimally important 2 = Moderately important 3 = Important 4 = Ve y Important using a mediator, etc.) other than through the courts? a) Training is not affordable due to my firm's limited resources b) Training is costly because of high labor turnover c) We lack knowledge about training techniques and the management of training programs d) The firm uses a mature technology, and new workers become proficient in the job through learning by doing e) Skilled workers can be readily hired from other firms t) Skills that workers learn in school are adequate to our needs g) We are skeptical about the benefits of training h) In-house informal training is adequate EMPLOYMENT Lawyer (without going to court) Formal Mediator1 Arbitrator Government Official Respected member of business community Degree of Importance 0 12 3 4 YES YES YES YES NO NO NO NO 19. a) If the employment level is higher than optimal then why do you not reduce workforce? (check one) Jamaica NEP Final Report.doc/EvalQC-2X NO NO -- YES We believe it is wrong to fire people; We are prevented from firing people according to the contract; Due to regulations, this is associated with high expenses; b) If the employment level is lower than optimal then why do you not increase workforce?(check one) New Economy Project 3 -4 May, 2003 Final Evaluation Report USAID/Jamaica Member of my business association or chamber (1) (2) (3 Can not find skilled workers for the position Labor taxes make it too expensive Due to regulations, new workers might be difficult to fire once they are hired Mandatory minimum wage is too high Mandatory Benefits for workers are too costly Other (specify) Not Applicable YES Other (specify) (1) (2) (3 (4) (5) (6) (7) YES I NO We prefer not to have troubles caused by local officials; We worry about social consequences of firing workers. We worry about violent reaction from workers Other (specify) Not Applicable (4) (5) (6) (7) . (8)