USAID COMMUNITY STABILIZATION PROGRAM (CSP) COUNTERINSURGENCY (COIN): SPECIAL STUDY ON BUSINESS DEVELOPMENT COMPONENT January 5, 2009 This publication was produced for review by the United States Agency for International Development. It was prepared by Harvey Herr and Rich Mason. COMMUNITY STABILIZATION PROGRAM (CSP) COUNTERINSURGENCY: SPECIAL STUDY ON BUSINESS DEVELOPMENT COMPONENT International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Program, Phase II (MEPP II) 8614 Westwood Center Drive Suite 400 Vienna, VA 22182 Contracted under 267-C-00-05-00508-00 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) i MEPP II CSP BDP Study Table of Contents LIST OF ACRONYMS ..................................................................................................................II EXECUTIVE SUMMARY ..............................................................................................................1 I. INTRODUCTION ....................................................................................................................................5 II. CSP PROJECT IMPLEMENTATION TO SUPPORT MICRO/SMALL/MEDIUM ENTERPRISES.........5 III. PURPOSE OF THE BDP SPECIAL STUDY.........................................................................................6 IV. RESEARCH DESIGN AND STUDY METHODOLOGY........................................................................6 V. FINDINGS.............................................................................................................................................8 VI. ADDITIONAL FINDINGS AND RECOMMENDATIONS ....................................................................11 VII. CONCLUSIONS ................................................................................................................................14 ANNEXES ANNEX I. SCOPE OF WORK (SOW) FOR BUSINESS DEVELOPMENT PROGRAM STUDY, JULY 29, 2008 ..............................................................................................................................1 ANNEX II. A RESEARCH METHOD AND SAMPLE DESIGN TO ANSWER BDP STUDY QUESTIONS.................................................................................................................................4 ANNEX III. SPECIAL BDP STUDY SURVEY INSTRUMENT.....................................................14 ANNEX IV. SURVEY IMPLEMENTATION SCHEDULE.............................................................19 ANNEX V. ANALYSIS OF THE BDP SURVEY DATA................................................................20 ANNEX VI. GLOSSARY ............................................................................................................58 ANNEX VII. BIBLIOGRAPHY .....................................................................................................61 ANNEX VIII. SO TEAM COMMENTS AND IBTCI RESPONSES...............................................62 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) ii MEPP II CSP BDP Study List of Acronyms AoR Area of Responsibility (of IPs) BOQ Bill of Quantity CERP Commander’s Emergency Relief Program CMT Community Mobilization Team (of IPs) COP Chief of Party CSO Civil Society Organization CSP Community Stabilization Program CTO Cognizant Technical Officer DAC District Advisory Council DCOP Deputy Chief of Party EGY Employment Generation and Youth EOP End of Project or Program FY Fiscal Year FSO Focused Stabilization Office GOI Government of Iraq HQ Headquarters HR Human Resources IBTCI International Business & Technical Consultants, Inc. ICAP Iraq Community Action Program ILO International Labor Organization IP Implementing Partner IR Intermediate Result IT Information Technology LG Local Government LGP Local Governance Program LOE Level of Effort LOP Life of Project or Program M&E Monitoring and Evaluation MEPP II Monitoring and Evaluation Performance Program, Phase II MIS Management Information System MOU Memorandum of Understanding MSME Micro, Small or Medium Enterprise NAC Neighborhood Advisory Council NGO Non-Governmental Organization OJT On-the-job Training PMP Performance Management Plan PC Provincial Council PRS Project Reporting System PRT Provincial Reconstruction Team RF Results Framework RFA Request for Application RIG Regional Inspector General SME Small or Medium Enterprise SO Strategic Objective SOW Scope of Work SPSS SPSS predictive analytics software www.spss.com TA Technical Assistance UN United Nations USAID United States Agency for International Development USG United States Government International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 1 MEPP II CSP BDP Study EXECUTIVE SUMMARY The Community Stabilization Program (CSP) is currently USAID/Iraq’s most heavily funded project. Awarded to International Relief and Development (IRD) in May 2006, CSP is responsible for achieving USAID/Iraq’s Strategic Objective 7 “Focused Stabilization: Reduce the Incentives for Participation in Violent Conflict.” The CSP project includes: 1) creation of jobs and development of employable skills with a focus on unemployed youth, 2) revitalization of community infrastructure and essential services, 3) support for established businesses and development of new sustainable businesses, and 4) help to mitigate conflict in selected communities. By carrying out these activities the CSP was expected to achieve measurable progress towards achieving the strategic objective. A large part of CSP’s focus has been to provide business development opportunities through grants that will lead to sustained employment and employment growth in the private sector. The development hypothesis is that creating employment opportunities will reduce incentives for youth to join the insurgency. In July 2008, International Business and Technical Consultants Inc’s (IBTCI) Monitoring and Evaluation Performance Program II (MEPP II) team received a Scope of Work (SOW) from USAID/Iraq’s Focused Stabilization Office requesting a special study “to quantitatively measure the effectiveness of the program in creating jobs 12 months after the grant was completed.” This report focuses on one aspect within the SOW (Annex 1) addressing the Business Development Program (BDP). Upon review of CSP’s databases, 352 grants met the criteria of having been established for at least 12 months. From this universe, MEPP II pulled a sample totaling 60 for review. This sample size was determined to be adequate enough in size to draw reliable conclusions. (Survey Methodology Annex II) The SOW required the study to answer a variety of questions related to job creation and sustainability over a 12-month period. The findings are noted below. Business success: ™ Overall 98% of grantees were still in business after one-year – a strong indication of BDP program success and the durability of USAID investments in this area. ™ Micro grants have the highest rate of business success across the four business sectors (agriculture, industrial/manufacturing, trade and service). Ninety-nine percent of the micro grantees, which are predominantly family-run informal businesses, were still operational. ™ Small grants have a success rate of 97%, while medium grants were successful 85% of the time. ™ While the least successful, medium grants to the industry/manufacturing sector according to the study definition, still had an almost 85% rate of success. Initial job creation (start-up jobs): ™ Of the total number of start-up jobs created, small grants produced more jobs than the other two grant types. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 2 MEPP II CSP BDP Study ™ While the total number of jobs created was lower than that of small and micro grants, medium size grants produced the highest average number of start-up jobs with eleven jobs created per grant. ™ The trade sector produced the highest total number of start-up jobs (most grants were awarded in this sector) while the highest average number of new jobs was created in the agricultural sector. Cost per employee: ™ The average program cost not including business owner or local cost share per employee was highest for medium grants at an estimated $6180 per new job created. Small grants averaged $2550 per new job, and micro grants $1380 per new job created. ™ The average program cost not including cost share per employee was highest for medium grants to the industrial/manufacturing sector and lowest for micro grants across all business sectors. Post-grant job growth: ™ Overall, the number of jobs grew by 2% in the period at least 12 months after the grant was awarded. The total number of additional jobs created beyond the initial grant requirements was less than 60. ™ All job growth came from grants to pre-existing businesses. ™ On average, job growth per grantee was highest for medium size grants to pre-existing businesses in the industrial/manufacturing sector (4.3 jobs per award). Regional differences: ™ There were too few qualifying grantees in regions outside Baghdad to make a determination of statistical differences between them. Baghdad grantees were marginally less likely to be successful when compared to combined regions outside Baghdad. Recommendations and Additional Findings for the Way Forward Determining the “sweet spot” for grant size and sector is complicated and dependent on the criteria utilized to evaluate the grants The CSP BDP grants strategy has been very successful at providing durable employment for grant recipients, their families and additional workers. As evidenced by the results of this study, micro grants provide the highest rate of business success and the lowest project cost per job in direct grant support. All other factors being equal, including costs of grant development and administration, speed of implementation, salaries from created jobs, etc., micro grants may be the best way forward in order to provide quick employment gains in insecure areas. If other priorities are considered that may be outside the immediate goals of CSP, such as the desire to create more formal positions beyond “mom and pop” shops and/or to encourage long￾term job growth by supporting established larger businesses, small and medium size grants produce different types of important job gains although at higher costs per individual position. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 3 MEPP II CSP BDP Study On average, job growth per grantee was highest for medium size grants to existing businesses in the industrial/manufacturing sector, adding an estimated 4.3 jobs per grant award. Depending on the most pressing priorities of a given area and the desired immediate, medium and long￾term results, the grant size and type of business supported may be very focused in one area or may require a mix of grants. Clarify the objectives of BDP job creation and tailor grants and business support to the specific needs of different business types. The types of businesses supported by BDP can be classified into two broad categories as either formal or informal. The International Labor Organization (ILO) characterizes the informal sector as “"household enterprises", where fixed and other assets do not belong to the business unit but to the owner; units cannot engage in transactions or enter into contracts nor incur liabilities on their own behalf; expenditure for production and capital goods are often indistinguishable from household purposes.” This definition appears to characterize many of the BDP grantees, especially micro grant recipients. It is likely that the strategy for informal businesses differs from that for formal grantee businesses. One of the reasons that the smaller size grantees have a higher business “success” as defined for this study is because the business is not separated from the family – it is more like asking is the family still operating one-year on. The more formal enterprises are at greater risk of not being successful because they are not household enterprises. Conversely, informal business owners may face greater challenges in operating and building their business and expanding their payrolls, while more formal enterprises may have the capital to generate larger revenues and profits and access credit when required. Informal sector grant intervention has more to do with livelihoods than it does with job growth, but it represents as much as one-third of the labor force in the Middle East according to the ILO. Depending on the type and mix of businesses and employment generation required, CSP may choose to emphasize one business type over the other and should tailor its programming accordingly. Moving forward, CSP might consider providing additional business training to targeted businesses that show capacity for growth and long-term job creation and encourage linkages between BDP grantees and other USAID economic growth programs. It is clear that CSP has met their immediate objective to put people to work at all levels within their AOR. While sustainability was not an initial goal of CSP, as it starts to draw down, it might consider instituting a robust business skills training program targeted to the grantee. In other words, the micro-grantee recipient operating an informal business out of the home would likely require much more basic business training than would a business owner who received a medium-sized grant for an existing business who already has the basic skills necessary to operate a business. This additional training might increase the sustainability of the business and at the same time generate more formal jobs and thus improve overall economic growth in the private sector. Additionally, formal businesses may be ready to seek commercial finance with the role of the CSP to help them by providing information to entities that may be better equipped to fill this role. Liaison with private banks and Micro Financing Institutions is recommended in particular when the business request is a medium grants to the industrial/manufacturing sector. USAID’s Tijara project has programs designed to provide micro-finance assistance and the Iraqi government administers MSME programs through MOLSA and other ministries. CSP staff is best positioned to provide information and referrals to these and other programs due to their existing relationships with grant recipients. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 4 MEPP II CSP BDP Study It does not appear that BDP grants specifically targeted youth that might otherwise be involved in the insurgency. This could be due to the fact that the target population does not have the financial means to provide matching funds or in-kind contributions necessary to receive a grant. CSP might want to consider, as one of the criterion for grants, requiring grantees to employ youth in the age category most at risk to join the insurgency. In the near-term, it is recommended that CSP amend the definitions used in business creation and employment generation by: • Classifying businesses by whether they are formal or informal businesses using roughly the ILO definition and whether they are registered as formal businesses. • Improve how expansion grants and grants to start-up enterprises are categorized in the database, or identified in the grant screening. There was a wide discrepancy between the survey findings and what is in the database with respect to start-up or existing businesses. Based on the characterization from the survey it is important to understand this distinction as a means of implementing program management. Little is known about the income generated from the grants in being able to fully support families and/or employees. The fact that businesses were still operating one year after the grant is a good indication of success, but may not provide enough information to evaluate grant impacts and produce the most informed programming decisions. Little is known of the livelihoods generated by the grants. The CSP might consider undertaking a special study to determine whether livelihood levels are improving for grantees and determine if grantees have other sources of earnings beyond the business to which the grant was awarded (it is difficult to conceive that a grant of less than $3,000 is sufficient to sustain a business and a large number of dependents). Such a study could lead to other strategies that might bring sustained community stabilization. Conclusions In the short term, BDP has delivered a successful job creation program that may have helped to contain the insurgency as the primary focus of CSP is to provide business development opportunities through grants that will lead to sustained employment growth in the private sector based on the development hypothesis that creating employment opportunities will reduce incentives for youth to join the insurgency. Though training and outreach, CSP staff should help BDP grantees establish linkages with finance projects, organizations and ministries to cement short-term stabilization gains into long-term robust economic development. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 5 MEPP II CSP BDP Study I. INTRODUCTION1 On May 29, 2006, USAID/Iraq awarded cooperative agreement number 267-A-00-06-00503-00 to International Relief and Development (IRD) to implement. The primary objective of the CSP falls within USAID/Iraq’s Transition Strategic Objective 7 “Focused Stabilization: Reduce the incentives for participation in violent conflict.” Initial funding under the CSP award limited activities to areas of significant insurgent activity in Baghdad. CSP was later expanded to cover similar areas in Tameem, Ninewa/Tel Afar, Ramadi, Felluja, North Babil and Diyala. Initially designed for two years, the project has been extended to 30 September 2009. The CSP is seen as a key element to transition Iraq to a stable, democratic and prosperous state. As defined in the Request for Application (RFA) the purpose of CSP is to complement military security efforts, and civilian local government development, with economic and social stabilization efforts. The design of the CSP project includes: 1) creation of jobs and development of employable skills with a focus on unemployed youth, 2) revitalization of community infrastructure and essential services, 3) support for established businesses and development of new sustainable businesses, and 4) help to mitigate conflict in selected communities. By carrying out these activities the CSP is expected to achieve measurable progress towards achieving the strategic objective. The CSP Performance Management Plan (PMP) identifies the measurable indicators that will evidence the achievement of the strategic objective. This special study focuses on one aspect of the project, the Business Development Program, which supports established businesses and develops new sustainable businesses. The BDP advertised for and identified micro, small and medium enterprises in the affected communities that could become platforms for sustainable employment and employment growth. II. CSP PROJECT IMPLEMENTATION TO SUPPORT MICRO/SMALL/MEDIUM ENTERPRISES CSP interventions in the Micro, Small and Medium Enterprises (MSME) business sector included grants and training. The BDP advertises for and identifies MSMEs within communities that could become platforms for sustainable employment and employment growth. Grants provided are based on the preparation of business plans, adherence to the principles in the grants manual and the oversight of technical IRD staff. Grants are not provided as cash, but as equipment or supplies. Selected businesses are mentored and visited by IRD technical staff to monitor use of the grant materials and to resolve problems that the grantee might face. Records are kept of the materials provided through Bill of Quantity (BOQs) as well as employment that the enterprise generated and sustained. IRD maintains a CSP Grants database used for tracking the activities of the grantee businesses. IRD’s experienced Employment Generation and Youth EGY field staff work with the grantees to identify problems and monitor the progress of the grants. 1 Names of some organizations and people have been removed for security reasons. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 6 MEPP II CSP BDP Study III. PURPOSE OF THE BDP SPECIAL STUDY In June 2008, the cognizant technical officer (CTO) for the CSP in USAID/Iraq’s Focused Stabilization Office requested the Monitoring and Evaluation Performance Program (MEPP) II implementing partner, International Business and Technical Consultants, Inc. (IBTCI) to conduct a special study of the business development component of CSP in order “to quantitatively measure the effectiveness of the program in creating jobs, 12 months after the close out of the grant.” The results of this study will enable CSP to better target businesses with appropriate grant sizes, within specific geographic areas and according to economic sector in order to produce the most employment opportunities. The study was designed to determine whether BDP grants resulted in sustainable business operations and job growth one-year post grant award. The study drills-down to see what evidence there is that certain aspects of project implementation (business sector and grant size) yielded differential benefits in terms of employment generated and business sustainability. Specific questions extracted from the SOW are listed below: Study Questions from the SOW 1. At least 12 months after the grant was awarded, has the BDP accomplished the objective of facilitating job growth? • What percentage of the grantees is still in business after one year? • Are there quantitatively measured trends in business success (still in operation) or failure? • Which grant size demonstrated the highest rate of grantees still in business? • Which grant size demonstrated the greatest number of new jobs created (estimate how many grant dollars were needed to create new jobs within the various grant windows)? • Which business sectors proved to be the most successful in creating new jobs? • Were additional jobs created by the grantee 12 months after the initial grant? If so, how many? Which grant size and type experienced the most post-award job creation? 2. Are there different challenges in respective regions? • Does the location of the business in regards to city size or region affect business success? IV. RESEARCH DESIGN AND STUDY METHODOLOGY The research design was prepared to ensure, within the limits of time and resources that the questions posed above have a high probability of being answered with a statistically valid response. IBTCI and the Focused Stabilization Office (FSO) identified and focused on two major factors: business sector and size of grant. The two major factors are further subdivided into four business sector subdivisions and three grant size subdivisions. (The Design and Study Methodology is appended to the report as Annex II) International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 7 MEPP II CSP BDP Study Business sectors were derived from the CSP BDP grants database. Grant size subdivisions were taken from the CSP grants manual and are identified as a data item in the CSP BDP grants database. The four business sector subdivisions agreed to between IBTCI and the CSP for this study are: Agriculture, Manufacturing/ Industry, Trade and Services. These business sector categories were collapsed from more detailed categories found in the database. The three grant sizes are micro ($100 – $3,000), small ($3,000 – $25,000) and medium ($25,000 – $100,000). The combination of grant size and business sector resulted in 12 possible study groups. Using IRD’s databases the number of qualifying BDP grants completed or closed before 1 September 2007 was found to be 2122. The 2122 grants were place appropriately within the 12 study groups and the grants to be studies were randomly selected from each group. The study selected a total sample of 326 grantees. Implementation of the BDP survey and survey non-response Implementing a field survey that will yield statistically valid results is a rigorous and pains-taking undertaking. The steps and timings for the BDP survey are detailed in Annex IV. Every effort was made to determine the disposition of the grantees selected in the sample so that non-responses were kept to a minimum. Field monitors were asked to make at least four attempts to contact the grantee or neighbors who might know the whereabouts of the grantees. MEPP II contractor was careful not to confuse non-responses to be confused with enterprises that were no longer in business. A non-response for purposes of this survey means that IBTCI was unable to locate the grantee. Non-respondents are not included in the data analysis. It is known that approximately one in ten persons has been displaced (International Organization for Migration (IOM) estimations of Iraq IDPs) during the war and insurgency so that movement of grantees from their previously known address is not considered unusual. This sample design provides for statistical comparisons between the relevant study groups provided that the assumptions about effect size (The Glossary appended as Annex VI defines this and other statistical terms used) were accurate and non-response was limited. A detailed description of the data analysis conducted that produced the findings is provided in Annex V. Actual survey response was 86% (280 out of 326) distributed evenly across the factorial groups as shown in the table below. The response rate was sufficient to proceed to attempt the comparisons between the study groups. Survey Response by Factorial Group Grant Size/ Business Sector Medium Small Micro Total Agriculture 5 34 0 39 Industrial/Manufacturing 27 35 36 98 Trade 5 33 36 74 Service 5 31 33 69 Total 42 133 105 280 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 8 MEPP II CSP BDP Study V. FINDINGS All the findings presented here have been tested for their statistical validity. Before responding directly to the questions posed for the special study some definitions of key elements (variables) in the responses are presented. Business success: A business was considered successful if it remained in operation at the time of the survey interview. Success does not include an assessment of business earnings or profit. Job growth: Job growth is the increase in employment both full time and part time that has occurred in the period since the grant was awarded (12 or more months since the award). Job growth was determined by comparing current employment (at the time of the survey interview) with employment when the grant was awarded. Start-up employment: For start-up businesses this is employment created when the grant was awarded (in the survey this is the date when grantees said they started using the grant to make money). A new enterprise was started. New employment was determined at the time the grant was awarded. Existing employment: Existing employment is underwritten by the grant award in the hopes of business expansion. Grant awards to existing businesses are an example of grants that sustain existing employment. Existing employment was not considered part of job creation. Existing employment was determined at the time of the grant award. An existing business is one that reported pre-grant employment. An estimated 37% of grants were awarded to existing businesses. Grant sizes: BDP awards three categories of grant. Micro grants $100 - $3000; Small grants $3,000 - $25,000; and Medium grants $25,000 - $100,000 Business sectors: Four business sectors were identified for this study. These are agriculture, industrial/manufacturing, trade and services. These were condensed from a wider range of business types available on the CSP BDP database. In addition, respondents were asked to describe their business. Responses to the questions posed in the SOW: What percentage of the grantees is still in business after one year? 98% of grantees were still in business after one-year. Are there quantitatively measured trends in business success (still in operation) or failure? The survey findings include the following: • Micro grants in all sectors had the greatest success in all sectors. Ninety-nine percent of the micro grantees are still in business. • Medium grants to the industry/manufacturing sector are least likely to be successful according to the study definition, but even here failure rate is less than 15%. • Small grants to the service sector have estimated 6 to 1 odds for success. • Small grants to the industry/manufacturing sector have 17 to 1 odds for success. Which grant size demonstrated the highest rate of grantees still in business? International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 9 MEPP II CSP BDP Study • Micro grants have a ninety-nine percent record of business success (micro grants go primarily to informal sector businesses linked to families); small grants have a success rate of 97%, while medium grants were successful 85% of the time. Which grant size demonstrated the greatest number of new jobs created (estimate how many grant dollars were needed to create new jobs within the various grant windows)? 2 The highest average number of start-up jobs created per grant award was 11 and came from medium grants, small grants averaged 5 and micro grants just 2. In total, most start-up jobs have been created through small grant awards. The average cost per employee was highest for medium grants at an estimated $6,180 per new job. Small grants averaged $2,550 per new job, and micro grants $1,380 per new job. Grant size and business sector combinations show that the cost per employee was highest for medium grants to the industrial/manufacturing sector, and was lowest for micro grants across all business sectors. Which business sectors proved to be the most successful in creating new jobs? The trade sector produced the most number of new jobs from start-ups (most grants were awarded in this sector), but the average number of new jobs created per grant was highest in the agricultural sector. Were additional jobs created by the grantee 12 months after the initial grant? If so, how many? Which grant size and type experienced the most post-award job creation? Job growth created by the grantees after the initial grant award did occur, but not often. Overall job growth in the period at least 12 months after the grant was awarded was limited to less than 2. The total number of jobs created from the 280 grants was less than 60. All job growth came from existing businesses. On average, job growth per grantee was highest for medium size grants to existing businesses in the industrial/manufacturing sector. These added an estimated 4.3 jobs per grant award in the period after the initial award. 2 Employment was first segmented into start-up jobs created by the grant award, existing jobs, and job growth created by the grantee stimulated by the grant award. This study question relates to start-up jobs created by the grant award. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 10 MEPP II CSP BDP Study Summary of Grant Size and Sector by Odds of Business success, Odds of Job Growth in 12 Months since Award, Estimated Number of Start-up Employees, Cost per Start-Up Employee, and Cost per Existing Employee: Grant Size and Sector Odds of Business Success Odds of Job Growth in 12 Months since Award Estimated Number of Start-Up Employees (weighted estimates) Cost per Start-Up Employee (USD) Cost per Existing Employee (USD) Medium grants 5 to 1* 1 in 5* 226 6087 7006 Small grants to agriculture >17 to 1 nil 772 3195 2590 Small grants to industry/ manufacturing 14 to 1* nil 401 1949** 3311** Small grants to trade >17 to 1 1 in 15 1178 2599** 2903** Small grants to service 6 to 1* 1 in 8 338 2019** 2822** Micro grant to industry/ manufacturing >17 to 1 nil 276 875** 988** Micro grant to trade >17 to 1 nil 916 1538** 626** Micro grant to service >17 to 1 nil 273 1057** 672** Overall 17 to 1 limited 4380 2288 2783 * statistically significant at the 0.1 level ** Not statistically different within the grant size class. Micro grants are different from small grants, but not different statistically from each other. The same is true for small grants. Overall business success has been extraordinarily high. However, grant size and business sectors where business success is less likely are the same ones where job growth is most likely, e.g. medium grants to the industrial/manufacturing sector. In addition, job growth has come entirely from existing businesses, and from businesses that had more than two employees prior to the issuance of the grant. Employment or Job Creation The study made a distinction between employment creation and job growth (additional jobs). The creation of new employment from a grant occurred when the grant award was made to a start-up. By definition, employment created in this way happened with start-ups. Sixty-three per cent of grant awards went to start-ups. Overall new employment created in this way totaled an estimated 4,300 jobs. The remaining 37% of grants were awarded to existing firms where employment was already in place. The number of existing businesses found in IBTCI’s analysis of the survey data (37%) far exceeded the number classified as expansion or existing businesses according to the CSP BDP database (5%). International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 11 MEPP II CSP BDP Study For purposes of this study, job growth is classified as additional employment created in excess of what existed at the time of the grant award. This is established by comparing employment at 12 or more months ago (at the time of the grant award) with employment during the time of the survey. Job growth was limited. Very few grants exhibited growth, and some job loss was identified. All job growth came from businesses that existed before the grant award, while losses came from businesses that were newly created. Both losses and gains were small. It is noteworthy that the numbers of jobs reported under this study might well differ from what is noted in the CSP data base given that the study did not look at the entire universe within the CSP data bases but rather only those who met the selection criteria Business Sectors and Grant Sizes Grantees are predominantly in the trade and service sectors (nearly 75%). While the overall allocation of grant funding across business sectors favors the trade sector (more overall grant funding went to the trade sector than to any other), on a per grant basis the industrial/manufacturing sector received the greatest average grant. Grantee contributions were highest in the industrial/manufacturing sector and lowest in the trade sector. Grants to the trade sector encouraged limited grantee contributions. Numerically, the largest number of grants was micro grants, but in terms of total value small grants exceeded the others by a wide margin. Contributions by grant size show that micro grants had proportionately the lowest grantee contributions and are the least well leveraged. The use of logistic regression allowed the analysis to pinpoint differences in business success by grant size and sector. Medium size grants to the industrial/manufacturing sector had the least chance of business success; however, as evidenced by an analysis of variance that same group had the best chance for job growth. VI. ADDITIONAL FINDINGS AND RECOMMENDATIONS IBTCI’s data analysis explored additional questions intended to enlighten further the findings about CSP BDP grantees. These are presented here in brief, but further developed in Annex V. Grantee Characteristics • Most grantees support a large number of dependents; there are 8 people in the average grantee household. The average household has 3 persons under age 15. • Eighty-six percent of the respondents are married. Eleven percent is single having never married; the remainder is widowed or divorced. • Approximately half the grantees did not finish high school, although there were some college graduates. In the analysis of variance used to estimate job growth the level of education is significant and probably has more influence than grant size or business sector in determining job growth. About half of the grantees said that they had attended vocational or technical training related to their business. This training however did not prove to have an effect on job creation. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 12 MEPP II CSP BDP Study • Sixty percent of the grantee businesses are not registered and are considered non￾formal businesses. Sixty-five of the businesses are sole proprietorships or family owned businesses and do not employ outside labor. • Businesses that were characterized as employing outside labor or as partnerships and cooperatives were more frequently registered and can be thought of as formal businesses. These businesses appear to be the source of job growth. • Gender of the grantee appears to have no effect on predicting job growth. • One quarter of the grantees noted they sought to borrow money for continued business operations. Of these, 4% said they went to a private bank or micro-finance institute; most went to friends or relatives. • Ninety-three percent of the grantees are optimistic that they will see increased revenues in the coming year. Eighty-two percent say they will add employees in the coming year but this is not supported by the survey findings on job growth. Determining the “sweet spot” for grant size and sector is complicated and dependent on the criteria utilized to evaluate the grants The CSP BDP grants strategy has been very successful at providing durable employment for grant recipients, their families and additional workers. As evidenced by the results of this study, micro grants provide the highest rate of business success and the lowest project cost per job in direct grant support. All other factors being equal, including costs of grant development and administration, speed of implementation, salaries from created jobs, etc., micro grants may be the best way forward in order to provide quick employment gains in insecure areas. If other priorities are considered that may be outside the immediate goals of CSP, such as the desire to create more formal positions beyond “mom and pop” shops and/or to encourage long-term job growth by supporting established larger businesses, small and medium size grants produce different types of important job gains at higher costs per individual position. On average, job growth per grantee was highest for medium size grants to existing businesses in the industrial/manufacturing sector. They added an estimated 4.3 jobs per grant award. Depending on the most pressing priorities of a given area and the desired immediate, medium and long￾term results, the grant size and type of business supported may be very focused in one area or may require a mix of grants. Clarify the objectives of BDP job creation and tailor grants and business support to the specific needs of different business types. The types of businesses supported by BDP can be classified into two broad categories as either formal or informal. The International Labor Organization (ILO) characterizes the informal sector as “"household enterprises", where fixed and other assets do not belong to the business unit but to the owner; units cannot engage in transactions or enter into contracts nor incur liabilities on their own behalf; expenditure for production and capital goods are often indistinguishable from household purposes.” This definition appears to characterize many of the BDP grantees, especially micro grant recipients. It is likely that the strategy for informal businesses differs from that for formal grantee businesses. One of the reasons that the smaller size grantees have a International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 13 MEPP II CSP BDP Study higher business “success” as defined for this study is because the business is not separated from the family – it is more like asking is the family still operating one-year on. The more formal enterprises are at greater risk of not being successful because they are not household enterprises. Conversely, informal business owners may face greater challenges in operating and building their business and expanding their payrolls, while more formal enterprises may have the capital to generate larger revenues and profits and access credit when required. Informal sector grant intervention has more to do with livelihoods than it does with job growth, but it represents as much as one-third of the labor force in the Middle East according to the ILO. Depending on the type and mix of businesses and employment generation required, CSP may choose to emphasize one business type over the other and should tailor its programming accordingly. CSP should provide additional business skills and finance training to businesses and encourage linkages between BDP grantees and other programs to maximize longer-term gains While long-term sustainability was not an initial factor in the design of CSP, this study shows that businesses who received grants more than one year ago are by and large still operational and providing income to the grant recipients. Additional business training might well not only increase the sustainability of these businesses in the long-term and also increase the number of durable jobs created. Should a robust training program be initiated, it will have to be designed with the grantee in mind in terms of their ability to understand and implement the training. In other words, training provided to an illiterate grantee operating an informal business would be very different than training provided to a college graduate who is successfully operating a formal business. Both types of training however, could be important if beneficiary selection is focused on those who want to show a willingness to undertake the training and want to see their businesses grow. Formal businesses may be ready to seek commercial finance with the role of the CSP to help them by providing information to entities that may be better equipped to fill this role. Liaison with private banks and Micro Financing Institutions is recommended in particular when the business request is a medium grants to the industrial/manufacturing sector. USAID’s Tijara project has programs designed to provide micro-finance assistance and the Iraqi government administers MSME programs through MOLSA and other ministries. CSP staff persons are best positioned to provide information and referrals to these and other programs due to their existing relationships with grant recipients. It does not appear that BDP grants specifically targeted youth that might otherwise be involved in the insurgency. This could be due to the fact that the target population does not have the financial means to provide matching funds or in-kind contributions necessary to receive a grant. Moving forward, CSP might want to consider, as one of the criterion for grants, requiring grantees to employ youth in the age category most at risk to join the insurgency. In the near-term, it is recommended that CSP amend the definitions used in business creation and employment generation by: • Classifying businesses by whether they are formal or informal businesses using roughly the ILO definition and whether they are registered as formal businesses. • Improve how expansion grants and grants to start-up enterprises are categorized in the database, or identified in the grant screening. There was a wide discrepancy between International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 14 MEPP II CSP BDP Study the survey findings and what is in the database with respect to start-up or existing businesses. Based on the characterization from the survey it is important to understand this distinction as a means of implementing program management. Little is known about the income generated from the grants in being able to fully support families and/or employees. The fact that businesses were still operating one year after the grant is a good indication of success, but may not provide enough information to evaluate grant impacts and produce the most informed programming decisions. Little is known of the livelihoods generated by the grants. The CSP might consider undertaking a special study to determine whether livelihood levels are improving for grantees and determine if grantees have other sources of earnings beyond the business to which the grant was awarded (it is difficult to conceive that a grant of less than $3,000 is sufficient to sustain a business and a large number of dependents). Such a study could lead to other strategies that might bring sustained community stabilization. VII. CONCLUSIONS BDP grants have accomplished the goal of providing sources of income and employment in areas of instability through the creation and expansion of successful businesses i.e., businesses that still exist one year or more after the grant award. Grants to start-ups and existing businesses have created and sustained substantial new employment while job growth beyond the original grant delivery occurred exclusively in pre-existing enterprises. The findings suggest that CSP BDP did create income generation for the grantees of start-up enterprises and one-year- on most of these businesses are still in operation. By and large however, these enterprises remain small and have not been able to create new jobs. Analysis of the cost of employment is based entirely on the CSP BDP grant contribution. It could be argued that this should be offset by grantee contribution. Grantee contribution leverages the grant value. When grantee contributions were considered the adjusted cost per employee results in a revised estimate of which grant-types are economically efficient. When adjusted the cost of a job in the small trade sector increases, while small industrial grants become attractive options. A primary focus in the CSP is to provide business development opportunities through grants that will lead to sustained employment growth in the private sector based on the development hypothesis that creating employment opportunities will reduce incentives for youth to join the insurgency. In the short term, BDP has delivered successful job creation programs that may have helped to contain the insurgency. Though training and outreach, CSP staff should help BDP grantees establish linkages with finance projects, organizations and ministries to cement short-term stabilization gains into long-term robust economic development. These findings and the need to prioritize job creation strategies in light of the various results of this study are important factors to be considered in the coming months for the CSP project as well as other stabilization and economic growth initiatives. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 1 MEPP II CSP BDP Study Annex I. Scope of Work (SOW) for Business Development Program Study, July 29, 2008 I. Objective of the Study The purpose of this SOW is to conduct a quantitative study of USAID/Iraq’s Community Stabilization Program’s (CSP) Business Development Program (BDP). Since CSP’s start-up in May 2006, the BDP has approved grants to over 6,700 businesses worth over $47 Million. Within USAID/Iraq’s Performance Management Plan, the BDP supports Strategic Objective 7, Reduced Incentives for Participation in Violent conflict, and Intermediate Result 7.1, Unemployment Decreased with a Focus on Young Men. This study will strive to quantitatively measure the effectiveness of the program in creating jobs 12 months after the grant was completed. The study will investigate: a) which grant types (according to business sector) and size (in terms of dollar value) produced the most jobs (on a job per grant dollar basis); b) what percentage of businesses have survived after one year; and c) Does the location of the business in regards to city size or region affect business success Answering these questions will enable CSP to better target businesses with appropriate grant sizes, within specific geographic areas and according to business sector in order to produce the most employment opportunities. In addition, the results of this study will potentially help inform other USAID/Iraq programs and similar initiatives at other USAID missions. II. Background The CSP works nationwide to create economic and social stability in Iraqi communities. CSP is a key element in the United States’ Mission in Iraq to help defeat the insurgency. The objective of the program is to provide access to capital for business start-up or expansion. The criteria for expansion are that businesses increase production and create new jobs. The BDP awards grants ranging from US$500 to US$100,000 to individuals and businesses in CSP focused areas (see Annex 1 for a BDP grant manual). Grants are awarded in the following categories they are only in the form of equipment and materials: 1. Micro $100 – $3,000 2. Smal $3,000 – $25,000 3. Medium $25,000 – $100,000 Another portion of the BDP is training component helps develop essential managerial skills to successfully operate a private business. The training program is linked with to the grant program and trainees are encouraged to apply for grants if they meet the guidelines. The BDP training program will not be studied in this SOW. Beneficiaries are Iraqis who submit an application which is studied based on the potential: (a) number of jobs created; (b) income generated; (c) measurable increases in production; (d) ability of the applicant to provide a minimum of 25 percent in-kind contribution to the project; and (d) impact the activity has on the community. The objective of the program is to provide financial assistance to vulnerable entrepreneurs engaged in micro, small and medium sized business activities. All business ideas are eligible for funding whether it is for business start-up or expansion. However, businesses that seek International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 2 MEPP II CSP BDP Study assistance to expand their operations are required to increase production and create new jobs to meet the demand of increased production. In March 11, 2008, International Business & Technical Consultants, Inc. (IBTCI) completed an independent study to monitor BDP projects and verify implementation. The study looked at grants over $1,000 that closed by the start of the survey on July 8, 2007. Of the 1,660 approved grants, 352 met the stated criteria for inclusion into the study. Sixty of these grants were randomly selected and 49 grants were actually monitored. Eleven of the selected projects were not accessible for monitoring and were classified as non-respondents. The report found that of the 49 businesses selected, 228 employment opportunities were created. This is a ratio of 4.65 jobs per grant. This is in contrast to the 13,070 jobs created from 4,049 completed grants reported in the IRD’s weekly CSP Activity Report for 4-10 May 2008. This demonstrates a job per grant ratio of 3.2. Although the IBTCI evaluation only surveyed 49 out of 1,660 grantees (2.95%), evidence indicates there may be positive growth in jobs after the grant is awarded. IV. Study Questions 1. At least 12 months after the grant was awarded, has the BDP accomplished the objective of facilitating job growth? • What percentage of the grantees is still in business after one year? • Are there quantitatively measured trends in business success (still in operation) or failure? • Which grant size demonstrated the highest rate of grantees still in business? • Which grant size demonstrated the greatest number of new jobs created (estimate how many grant dollars were needed to create new jobs within the various grant windows)? • Which business sectors proved to be the most successful in creating new jobs? • Were additional jobs created by the grantee 12 months after the initial grant? If so, how many? Which grant size and type experienced the most post-award job creation? 2. Are there different challenges in respective regions? • Does the location of the business in regards to city size or region affect business success? V. Study Methods: Design Methodology The contractor should employ the most appropriate combination of research methods necessary to answer these questions. Methods may include but are not limited to desk research, surveys, focus groups and key informant interviews. The contractor should propose a methodology to USAID that is the most technically effective and cost efficient. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 3 MEPP II CSP BDP Study VII. Team Composition The contractor should propose an appropriate team composition to USAID. VIII. Schedule and Logistics Task Notional Timeline Pre-study research and planning: review background documentation, design data analysis, design BDP survey instruments, prepare report template. 2 weeks Train Field Monitors, field test survey instruments, conduct surveys. 3 weeks Consolidate data and perform high-cut analysis. 2 weeks Develop preliminary findings and provide briefing to USAID. 2 weeks Draft report (in USA) 2 weeks Total LOE 11 weeks IX. Reports A final, full briefing and written report on findings and recommendations will be provided not later than November 1, 2008. Study report will be the following format: 1. One page summary of report findings and recommendations 2. Executive summary of not more than 6 pages. 3. Main body of the report findings and recommendations of not more than 40 pages. 4. Report will be formatted in accordance with USAID publication, “Constructing an Evaluation Report,” dated April 14, 2006. The final report will be provided to USAID no later than 7 working days after receipt of comments from USAID on the draft. It is anticipated that USAID review of the draft will require up to two weeks, with comments to be returned to the team for final editing of the report. Deliverables: Report findings and recommendations on improving BDP program effectiveness and efficiency. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 4 MEPP II CSP BDP Study Annex II. A Research Method and Sample Design to Answer BDP Study Questions This sample is designed to answer the questions posed in the “Scope of Work (SOW) for Business Development Program Study” dated 29 July 2008. In the SOW the CTO for the CSP project defined seven questions that are the basis for the study. Providing quantitative answers to these questions imposes a framework on the sample design. The questions asked imply or state study domains3 that include: grant size, business sector and city (or region). In particular, one question asks the study to determine which grant size and type (business sector) was most successful. This combination of these two determinants of the study domains sets the limits for the sample design. The desire to compare different aspects of a program lends itself to the use of factorial design to answer the questions about what aspects of the program work best. In general a factorial design is used to look at a variety of program variations to see which works best. In factorial design groups are defined by the major variables of interest: in this case the main variables of interest are business sector and grant size. The purpose of the design is to compare the main effects and the interactions effects of the factors. In pure experimental design subjects would be randomly assigned to the different groups in the factorial design before the treatments were applied, here we use the technique a priori on data that have already been collected. In order to make statistical inferences about the comparative differences between the factors we need to determine how large the sample should be in each of the groups. In the analysis of the data we use multiple comparison post hoc statistical analysis. The first task is to breakdown the posed questions into “answerable” questions. To do this we gathered information about business sectors and grant sizes. Conveniently there was a ready source of information on grant sizes and business sectors: information on both of these variables is contained in IRDs CSP BDP Grant tracking system (hereinafter referred to as the BDP database). The BDP database provides a comprehensive list of BDP grantees under the IRD CSP program.4 Grant size is specified in the database as is shown in the three categories defined in the SOW, so no manipulation or recoding of the grant size data were needed to segment the grantees into study domains. Business sector also occurs as a variable in the BDP database, but consolidation of business sector categories was needed. In the original BDP database (July 2008 version) 15 business sector categories are identified. Fifteen categories were deemed to be too many to make the questions “answerable.” To leave the number at 15 categories would mean that answering the question about which grant-size and type would mean comparing 45 combinations and that some of the combinations would have very few subjects. Business type was therefore collapsed into just four business categories: Agriculture, Manufacturing/Industry, Trade and Services. Identification of business type on the BDP database is subjective and may be a source of classification error (to assess this we added a question in the survey instrument asking the respondent to describe their business). 3 UN definition: “In the course of tabulation, data may actually be provided for many population segments; however, a study domain would be a segment identified in the overall statistical plan as one for which a certain level of detail and certain data reliability were required. The study domains chosen may coincide with the strata adopted for stratified sampling or may cut across them. “ 4 IRD also administered a BDP program under USAID/Iraq’s the CAP I and CAP II programs. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 5 MEPP II CSP BDP Study Following reclassification of business types there were three grant sizes and four sector types. When these are combined there are twelve possible study domains for comparison. These twelve study domains (or cells in the table) are shown in Table 1 below. The numbers in the table are the total number of grants one or more year old that occur for each grant size and sector combination. These numbers originate in the CSP Grant database. In preparing this table only completed projects with completion dates prior to 1 September 2007 were included. These projects also are the sample frame used to select the sample for the study. The sample design has to support the estimation of the number of successful businesses (defined in the data analysis as a ratio estimate of the percent of BDP projects currently in operation) for each study domain. In addition, the sample design has to support estimation of the number of new jobs created (defined in the data analysis as the average number of new jobs created) for each study domain. For this design the target confidence interval has been set at a modest 90%. The ratio estimate was adopted here as the test statistic that determines the sample size calculation. Table 1 Study Domains Defined All Medium Small Micro Agriculture 5 250 2 Industrial/Manufacturing 32 187 91 Trade 5 350 891 Service 5 154 150 From Table 1 it can be observed that the numbers of projects are unevenly distributed among the study domains. This reality influences the study design. There are relatively few medium size grants (medium grants are actually the highest value grants) and most of these are in the Industrial/Manufacturing category, with very few in Agriculture, Trade or Service categories. It was therefore decided to treat the medium size grants as one study domain without reference to sector categories. In the sample design all medium size grant projects are to be scheduled for interview. Small grants are the relatively common and there is a viable number in each sector category to select a sample from each of the four sectors. Micro grants are the most common with sufficient numbers in all sector types except for agriculture for a sample to be selected. In summary the distribution found in the sample frame will permit sample selection in seven out of the twelve study domains. These are highlighted in Table 1 above. Ideally we would like to be able to compare each of the study domains with any of the others. Comparison of study domains is stressed in the design rather than the need to make estimates of population characteristics. In a fully randomized experiment businesses within each of the study domains would be randomly assigned to receive a grant or not to receive a grant. Receiving a grant would be the “treatment” in the experimental design. In this study we propose a quasi-experimental design that will compare study domains where the “treatment” is a combination of grant size and grant sector. A successful outcome for this quasi-experimental design will indicate which grant size/sector are the most likely to be successful. Such an outcome can guide program resource allocation. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 6 MEPP II CSP BDP Study Calculating Sample Size For the purpose of calculating sample size a null hypothesis is proposed within grant-size category. The null hypothesis states that there is no difference in business success across sector categories (e.g., between study domains) but within a grant size category. Business success is indicated if the business is still in operation at the time of the interview. This is a binary outcome. Differences between grant-size categories in business success are referred to below as effect size. The study anticipates rejecting the null hypothesis as we believe that manufacturing and agricultural projects have a higher prospect of success than do trade and service businesses. The study design model is described below for each grant size. Traditionally, data collected in a research study is submitted to a significance test5 to assess the viability of the null hypothesis. The p-value provided by the significance test, and used to reject the null hypothesis, is a function of three factors: 1. The larger the observed effect (proposed differences in business success between sectors), 2. The larger the sample size, and/or 3. The more liberal the criterion required for significance (alpha), the more likely it is that the test will yield a significant p-value. A power analysis was executed in the planning of this study. It is used to anticipate the likelihood that the study will yield a significant effect and is based on the same factors as the significance test itself. Specifically, the larger the effect size used in the power analysis, the larger the sample size, and/or the more liberal the criterion required for significance (alpha), the higher the expectation that the study will yield a statistically significant effect. The power analysis was conducted to find an appropriate balance among these factors by taking into account the goals of the study. For this study the significance level is set at 10% (a 90% confidence interval). This means that there will be a 10% chance of a Type I error. A Type I error is committed when the true effect is null but the study yields a significant p-value and leads the researcher (in error) to reject the null. The effect size for our study is the anticipated difference in the proportion of successful businesses between sector categories and is explained below. The sample size will be adjusted until the “power” of the study design reaches 80%. Power is the proportion of studies that will yield a statistically significant effect (assuming the effect size, sample size, and criterion alpha specified in the study design).6 5 Leonard Mlodinow in “The Drunkard’s Walk: How Randomness Rules Our Lives” expresses the concept of significance using a non-technical example “…suppose that a student in a research study on extrasensory perception predicts the result of some coin tosses. If in our observations we find that she is almost always right, we might hypothesize that she is somehow skilled at it, for instance, through psychic powers. On the other hand, if she is right about half the time, the data support the hypothesis that she was just guessing. But what if the data fall somewhere in between or if there isn’t much data? Where do we draw the line between accepting and rejecting the competing hypotheses? This is what significance testing does: it is a formal procedure for calculating the probability of our having observed what we observed if the hypothesis we are testing is true. If the probability is low, we reject the hypothesis. If it is high, we accept it.” Page 172 6 The statistical software Sample Power, version 2, distributed by SPSS, Inc. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 7 MEPP II CSP BDP Study Medium size grants: There were forty-seven medium sized grants and 32 of these were in the Industrial/Manufacturing sector. For the study all medium sized grants were included in the survey. Small Grant Sample: Nine hundred and forty-one small grants qualified for the sample frame. Table 1 shows that small grants are well distributed across the four sector categories allowing a sample to be drawn from each of the categories. To determine the appropriate sample size a null hypothesis is proposed and a study design prepared. Hypothesis to be tested One goal of the proposed study is to test the null hypothesis that the event rate (rate of business sustainability) is identical in the Agriculture, Industry/Manufacturing, and Trade and Service sectors. Effect size Power is computed to reject the null under the following alternate hypothesis: For the agriculture sector the event rate is 0.82, for industrial/manufacturing sector the event rate is 0.95, for trade sector the event rate is 0.60, and for the service sector the event rate is 0.70. These event rates are hypothesized informed by business survival noted in other studies.7 The trade sector was set lowest as most of these grants had a lower value and were used to re-supply a retail establishment with new inventory (trade establishments also have a low entry threshold and unemployed persons may attempt to establish a small retail kiosk while looking for longer term employment). Once that inventory is sold a shop may find it more difficult to generate additional revenue than a manufacturer who was provided with new equipment to produce some product. Industrial /Manufacturing requires specialized skills and equipment; Agriculture needs land, and Services need a marketable skill. The overall employment rate in Iraq is approximately 72% and this was used as an indicative guidepost. Sample size Through trial and error the study design overall sample will include a total of 120 grantees assigned to the four business sectors as follows: 30 in Agriculture, 30 in Manufacturing, 30 in Trade, 30 in Service. To accommodate possible survey non-response 10 additional grantees were included in each business sector. 7 The MEPP II study of IRD’s BDP grants under the CAP II program had a survival rate that exceeded 90% after nine-months. However, these were not categorized by sector type. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 8 MEPP II CSP BDP Study Alpha and tails The criterion for significance (alpha) has been set at 0.10. The test is 2-tailed, which means that an effect in either direction will be interpreted. Power For this distribution, effect size (event rates of 0.82 (agriculture), 0.95 (manufacturing), 0.60 (trade), 0.70 (services)) sample size 120, and alpha (0.10, 2-tailed), the calculated power is 0.82. This means that 82% of studies would be expected to yield a significant effect, possibly rejecting the null hypothesis that the event rates are identical. The power table below shows the relationship of sample size to power for small-size grants. Power as a Function of Sample size Alpha = 0.100, Tails = 2, Event rates for factorial group 1 (0.82), group 2 (0.95), group 3 (0.60), Group 4 ( 0.70), with the sample distributed in equal proportions of 25.0% to each factorial group Micro-Grant Sample: One thousand one hundred and thirty-four micro grants qualified for the sample frame. Table 1 shows that micro grants are not well distributed across the four sector categories. There were only two qualifying micro grants in the Agricultural sector. There however were sufficient qualifying grants to allow a sample to be drawn from each of the remaining three categories. To determine the appropriate sample size a null hypothesis is proposed and a study design prepared. Hypothesis to be tested International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 9 MEPP II CSP BDP Study The CSP Business Development Study is to test the null hypothesis that the event rate (the rate of business sustainability) is identical in the Industrial/ Manufacturing, Trade and Service sectors. Effect size Power is computed to reject the null under the following alternate hypothesis. For the Industrial/Manufacturing sector the event rate is 0.95, for the Trade sector the event rate is 0.60, and for the Service sector the event rate is 0.70. These event rates are estimates informed by previous monitoring and national statistics (see the section above on Small grants). Sample size The study will include a total of 100 grantees, with 33 assigned to each group. To adjust for possible non-response the sample size was increased to 40 grantees for each factorial group. Alpha and tails The criterion for significance (alpha) has been set at 0.10. The test is 2-tailed, which means that an effect in either direction will be interpreted. Power For this distribution, effect size (event rates of 0.95, 0.60, 0.70) sample size (100), and alpha (0.10, 2-tailed), power is 0.84. This means that 84% of studies would be expected to yield a significant effect, rejecting the null hypothesis that the event rates are identical. The power table below shows the relationship of sample size to power for micro-size grants. Power as a Function of Sample Size Alpha = 0.100, Tails = 2, event rates for factorial group 1 (0.95), Group 2 (0.60), Group 3 (0.70), with the sample distributed in equal proportions of 1/3 to each factorial group International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 10 MEPP II CSP BDP Study Regional Estimates: Regional estimates were configured as an artifact of the design for the study domains. The study assumed that region on its own was a domain of study and not in combination with sector type or grant size. It was not used as a primary determinant in the sample design Table 2. Grant Size by Sector within Region Ninewa Medium Small Micro Agriculture 0 3 0 Industrial/Manufacturing 0 10 5 Trade 3 87 79 Service 0 19 23 Ramadi/Felluja Agriculture 0 10 0 Industrial/Manufacturing 1 16 2 Trade 2 134 6 Service 0 15 13 Tameem Agriculture 0 52 2 Industrial/Manufacturing 3 19 26 Trade 0 32 234 Service 0 29 36 Baghdad Agriculture 5 185 0 Industrial/Manufacturing 28 142 58 Trade 0 97 572 Service 5 91 78 A power analysis shows that an overall sample size of 160 distributed 25% in each region yields a power of .82 given estimated event rates. Estimated event rates were: Ninewa 0.82, Ramadi/Felluja 0.74, Tameem 0.66 and Baghdad 0.95. Tameem was ranked lowest as most grants were micro-grants to small scale retail traders thought to be most at risk of failure. Baghdad ranked highest as the market that has more customers and more wealth; generally more tolerant against business failure. Should these effect size estimates and response by region hold then statistical comparisons by region may be possible. Grant Size Estimates: Grant size estimates were configured as an artifact of the design for the study domains. A test was performed to confirm that the selected sample size for the study domains (above) would be of adequate size to test a null hypothesis based on grant size alone. Hypothesis to be tested International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 11 MEPP II CSP BDP Study To test the null hypothesis that the event rate (the rate of business sustainability) is identical for Medium size, Small size and Micro size grants. Effect size Power is computed to reject the null under the following alternate hypothesis. For Medium size grants the event rate is 0.95, for Small size grants the event rate is 0.85, for Micro size grants the event rate is 0.70. These estimates for event rates are based on the relative risk of failure and assumptions about the difficulty of entering the market. Sample size The study assumed a sample total of 120 subjects, assigned as follow: 33.3% in Medium, 33.3% in Small, and 33.3% in Micro. Alpha and tails The criterion for significance (alpha) has been set at 0.10. The test is 2-tailed, which means that an effect in either direction will be interpreted. Power For this distribution, effect size (event rates of 0.95, 0.85, 0.70) sample size (120), and alpha (0.10, 2-tailed), power is 0.80. This means that 80% of studies would be expected to yield a significant effect, rejecting the null hypothesis that the event rates are identical. It is therefore likely that statistical tests across grant size will be possible. Summarizing Sample Sizes Table 3 summarizes the sample size for the CSP Business Development Study. All medium size grants are to be included in the study. For each of the other study domains (except Micro grants in Agriculture) a sample size of 40 will be drawn from the sample frame (see Table 1). A sample size of 40 includes provision for non-response. The non-response rate in a related CAP II study of BDP projects was approximately 23%. That non-response rate informed this sample size. Table 3. Actual Sample Size for Study Domains (factorial groups) Business Sector of Grant Type by Size Grant Medium Small Micro Total Agriculture 5 40 0 45 Industrial/Manufacturing 32 40 40 112 Trade 5 40 40 85 Service 5 40 40 85 Total 47 160 120 327 Regional comparisons were not directly included in the sample design. The actual drawn sample yields more than sufficient power, using the marginal distributions shown in Table 4, to International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 12 MEPP II CSP BDP Study test the null hypothesis that there is no difference in the rate of business sustainability between cities. So indirectly the sample design is proven sufficient to address whether or not there are regional differences. Table 4. Sample Coverage by City and Grant Size Grant Type by Size City Medium Small Micro Total Ninewa 3 17 14 34 Ramadi/Felluja 3 27 4 34 Tameem 6 24 32 62 Baghdad 38 92 70 200 Total 50 160 120 330 Similarly, using the marginal distributions at the bottom of Table 4 and the event rates in our assumptions, the sample size is sufficient to test for a null hypothesis across overall grant size categories. This holds true for testing the null hypothesis across grant sectors. Table 5. Sample Coverage by City and Business Sector Business Sector of Grant City Agriculture Industrial/ Manufacturing Trade Service Total Ninewa 0 6 16 12 34 Ramadi/Felluja 2 5 21 6 34 Tameem 8 20 13 18 59 Baghdad 35 81 35 49 200 Total 45 112 85 85 327 Making population estimates from the sample data. Grantee population estimates (when making estimates of grantee characteristics as a whole) cannot be made directly from the survey data. It is necessary to weight the sample data by the inverse of the grantee’s probability of selection. These weights are given in the table below. Table 6. Sample Weights to be used for Grantee Population Estimates All Grant Size / Business Sector Medium Small Micro Agriculture 1 6.25 0 Industrial/Manufacturing 1 4.68 2.28 Trade 1 8.75 22.28 Service 1 3.85 3.75 Implementation of Design Implementation of the design meant drawing 7 separate random samples of 40 each (all of the medium size grants were included) and then recombining them into one BDP sample file that International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 13 MEPP II CSP BDP Study contained all 327 businesses that were to be interviewed. This file was then sorted according to the location of IRD offices. IRD field staff worked with ----- to locate grantees that were in the sample. IBTCI provided oversight to the fieldwork and designed the data dictionary. Prior to implementation of the field work a pretest of the sample questionnaire and field procedure was completed and adjustments to the questionnaire made in advance of final study implementation. A copy of the final survey instrument is provided here in Annex III. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 14 MEPP II CSP BDP Study Annex III. Special BDP Study Survey Instrument CSP LONG-TERM JOB CREATION FROM BUSINESS DEVELOPMENT PROJECTS -SPECIAL STUDY INSTRUMENT￾INSTRUCTION TO THE MONITOR: COMPLETE ITEM 1 TO 10 OF THIS FIRST PAGE OF THE MONITORING FORM BEFORE PROCEEDING TO THE PROJECT SITE. GRANTS INFORMATION PANEL (1-10 ARE TO BE COMPLETED FROM IRD DATABASE) 1 GRANT PROJECT CODE: 2 GRANT TITLE 3 GRANTEE NAME 4 SITE VISIT DATE (DAY 0-31/MONTH 01-12/YEAR 2008) 5 GOVERNORATE: 6 DISTRICT (QADA1): 7 SUB-DISTRICT (NAHIYA): 8 MAHALLA/ZUQAQ: 9 MONITOR’S NAME 10 MOBILIZER’S NAME 11 RESULT OF INTERVIEW: COMPLETED ............................................1 REFUSED ................................................2 NOT AT HOME/BUSINESS..........................3 INVALID ADDRESS.....................................4 SECURITY PREVENTED ACCESS ................5 BUSINESS HAS CLOSED............................6 BUSINESS HAS RELOCATED......................7 OTHER (SPECIFY) ....................................9 12 DATA ENTRY CLERK: NAME: ________________________________ DATE OF ENTRY _________________________________ MONITOR NOTES: IN THIS SPACE THE MONITOR RECORDS NOTES ABOUT WHY THE SITE VISIT WAS NOT COMPLETED. MONITOR NOTES: IN THIS SPACE THE MONITOR RECORDS HIS OBSERVATIONS ABOUT THE MONITORING VISIT. RECORD HERE IMPRESSIONS ABOUT THE VISIT. WHO WAS PRESENT. WERE THEY WELCOMING OR DISTANT... International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 15 MEPP II CSP BDP Study PART I: GRANT UTILIZATION AND COMPLIANCE # QUESTION RESPONSE SKIP HERE PREPARE THE RESPONDENT FOR WHAT IT IS YOU WANT TO TALK TO HIM ABOUT. TELL HIM HOW LONG IT WILL TAKE AND WHAT THE INFORMATION WILL BE USED FOR. ASK PERMISSION TO PROCEED. FIRST I WOULD LIKE TO ASK ABOUT YOU AND THE PEOPLE IN YOUR HOUSEHOLD 1 NAME AND TITLE OF RESPONDENT 2 SEX OF RESPONDENT MALE .........................................1 FEMALE......................................2 3 HOW MANY PERSONS LIVE IN YOUR HOUSEHOLD INCLUDING YOURSELF? 4 HOW MANY PERSONS IN YOUR HOUSEHOLD ARE BELOW 15 YEARS OF AGE? 5 HOW MANY PERSONS IN YOUR HOUSEHOLD ARE AGE 65 OR ABOVE? 6 HOW MANY PERSONS IN YOUR HOUSEHOLD DEPEND ON THE INCOME YOU EARN FROM THIS BUSINESS? 7 WHAT IS YOUR MARITAL STATUS? SINGLE, NEVER MARRIED ............1 MARRIED....................................2 DIVORCED..................................3 WIDOWED ..................................4 8 WHAT IS THE HIGHEST LEVEL OF EDUCATION YOU HAVE OBTAINED? (CATEGORIES FROM COSIT) ILLITERATE.................................1 READ AND WRITE........................2 ELEMENTARY .............................3 INTERMEDIATE ...........................4 SECONDARY...............................5 VOCATIONAL SCHOOLS ...............6 VOCATIONAL CENTERS ...............7 DIPLOMA ....................................8 BACHELOR .................................9 HIGH DIPLOMA..........................10 MASTER ...................................11 DOCTORATE.............................12 OTHER .....................................99 9 HAVE YOU ATTENDED ANY TECHNICAL OR VOCATIONAL EDUCATION RELATED TO YOUR BUSINESS? YES ...........................................1 NO.............................................2 NOW I WOULD LIKE TO ASK YOU ABOUT YOUR BUSINESS AND ANY GRANTS THAT YOU HAVE RECEIVED THAT SUPPORT YOUR BUSINESS 10 WHEN DID YOU FIRST OPEN YOUR BUSINESS? (MONTH 01-12/YEAR) ENTER MONTH AND YEAR ONLY; USE 4 DIGIT REFERENCE FOR THE YEAR / International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 16 MEPP II CSP BDP Study 11 PLEASE CHARACTERIZE HOW YOUR BUSINESS IS RUN (CONFIRM WITH OTHER RESEARCH) SOLE PROPRIETOR .....................1 FAMILY OWNED BUSINESS EMPLOYING ONLY FAMILY MEMBERS ..................................2 FAMILY OWNED BUSINESS WITH NON-FAMILY EMPLOYEES............3 PARTNERSHIP/COOPERATIVE .....4 OTHER........................................5 12 PLEASE IDENTIFY THE KIND OF BUSINESS YOU CURRENTLY OPERATE AGRICULTURE ............................1 LIVESTOCK.................................2 CONSTRUCTION..........................3 MANUFACTURING .......................4 HANDICRAFT ..............................5 WHOLESALE/RETAIL SHOP...........6 REPAIR SERVICES.......................7 PERSONAL SERVICES..................8 OTHER .......................................9 13 DESCRIBE THE NATURE OF YOUR BUSINESS AND HOW IT OPERATES WRITE ALL RESPONSES 14 HAVE YOU REGISTERED YOUR BUSINESS WITH THE GOVERNMENT? YES ...........................................1 NO.............................................2 15 DESCRIBE WHERE THE BUSINESS IS REGISTERED (FOR EXAMPLE, THE MINISTRY OF TRADE), OR IF NOT REGISTERED WHY YOU HAVE DECIDED NOT TO REGISTER. WRITE ALL RESPONSES 16 DID YOU RECEIVE A GRANT FROM THIS PROGRAM TO HELP SUPPORT OR OPEN YOUR BUSINESS? YES ...........................................1 NO.............................................2 END 17 DID YOUR BUSINESS RESTART DUE TO THIS GRANT? YES ...........................................1 NO.............................................2 18 WHEN DID YOU FIRST START USING THE GRANT TO MAKE MONEY? (MONTH 01-12/YEAR) ENTER MONTH AND YEAR ONLY; USE 4 DIGIT REFERENCE FOR THE YEAR 19 WHAT MATERIALS AND EQUIPMENT DID YOU RECEIVE? 20 ARE YOU STILL OPERATING THE SAME BUSINESS THAT WAS ASSISTED WITH THE GRANT? YES ...........................................1 NO.............................................2 GOTO PART II 21 WHEN DID YOU STOP USING MATERIALS OR / International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 17 MEPP II CSP BDP Study EQUIPMENT PROVIDED BY THE GRANT TO EARN REVENUE FOR YOUR BUSINESS? (MONTH 01-12/YEAR) ENTER MONTH AND YEAR ONLY; USE 4 DIGIT REFERENCE FOR THE YEAR 22 WHY DID YOU STOP OPERATING USING THE MATERIALS OR OTHER SUPPORT PROVIDED BY THE GRANT? WRITE ALL RESPONSES PART II: EMPLOYMENT CREATED AS A RESULT OF THE GRANT # QUESTION RESPONSE SKIP NOW I WOULD LIKE TO ASK YOU SOME QUESTIONS ABOUT PAID EMPLOYEES IN YOUR BUSINESS MALE FEMALE PART￾TIME FULL￾TIME 1 HOW MANY PAID EMPLOYEES (INCLUDING THE GRANTEE AND ANY FAMILY MEMBERS) DID YOU HAVE IN THE FOLLOWING CATEGORIES BEFORE YOU RECEIVED THE GRANT? ENTER 00 IF NONE TOTAL MALE FEMALE PART￾TIME FULL￾TIME 2 HOW MANY PAID EMPLOYEES (INCLUDING THE GRANTEE AND ANY FAMILY MEMBERS) DO YOU HAVE IN THE FOLLOWING CATEGORIES AT THE PRESENT TIME? ENTER 00 IF NONE TOTAL MALE FEMALE PART￾TIME FULL￾TIME 3 HOW MANY PAID EMPLOYEES (INCLUDING THE GRANTEE AND ANY FAMILY MEMBERS) DID YOU HAVE IN THE FOLLOWING CATEGORIES WHEN THE GRANT ENDED? ENTER 00 IF NONE TOTAL IF 2 AND 3 ARE THE SAME GOTO 6 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 18 MEPP II CSP BDP Study # QUESTION RESPONSE SKIP 4 WHY DID THE NUMBER OF JOBS IN YOUR BUSINESS CHANGE BETWEEN NOW AND THE TIME YOUR GRANT ENDED? WRITE ALL RESPONSES 5 EXPLAIN THE MOST CRITICAL FACTOR THAT HAS PERMITTED THE BUSINESS TO ADD NEW EMPLOYEES? WRITE ALL RESPONSES MALE FEMALE PART￾TIME FULL￾TIME 6 HOW MANY UNPAID FAMILY WORKERS DO YOU HAVE IN THE FOLLOWING CATEGORIES AT THE PRESENT TIME? ENTER 00 IF NONE TOTAL NOW I WANT TO ASK ABOUT YOUR BUSINESS PROSPECTS 7 WHAT OBSTACLES ARE YOU FACING IN YOUR BUSINESS? WRITE ALL RESPONSES 8 HAVE YOU ATTEMPTED TO BORROW MONEY TO EXPAND YOUR BUSINESS SINCE YOUR GRANT AWARD? YES........................................... 1 NO ............................................ 2 GO TO 11 9 WHICH OF THE FOLLOWING DID YOU APPROACH FOR FINANCING FRIENDS AND RELATIVES............ 1 PRIVATE BANK ........................... 2 MICRO-FINANCE INSTITUTE......... 3 INTERNATIONAL DONOR.............. 4 OTHER....................................... 5 NOT SURE ................................. 9 10 EXPLAIN THE PROCESS USED TO OBTAIN THESE ADDITIONAL FUNDS WRITE ALL RESPONSES 11 DO YOU EXPECT YOUR BUSINESS TO GROW (INCREASE REVENUE EARNED) IN THE COMING YEAR? YES........................................... 1 NO ............................................ 2 12 DO YOU EXPECT YOUR BUSINESS TO HIRE ADDITIONAL NEW EMPLOYEES IN THE COMING YEAR? YES........................................... 1 NO ............................................ 2 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 19 MEPP II CSP BDP Study Annex IV. Survey Implementation Schedule BDP Study Implementation Date 1 SOW received, reviewed and study method formulated 6/23/2008 2 Sample design for factorial design drafted for the study 7/23/2008 3 BDP grant database segmented into factorial groups 7/23/2008 4 BDP grantees sampled from each factorial group 7/23/2008 5 BDP grantee samples merged 7/23/2008 6 SOW study methodology confirmed with CTO 7/27/2008 7 Draft questionnaire completed 7/27/2008 8 Purchase Order to ----- with SOW 7/31/2008 9 Pre-test sample selected 8/3/2008 10 IRD notified of startup through CTO 8/3/2008 11 Pre-test sample distributed to IRD and ----- 8/4/2008 12 Objective of BDP special study clarified with ----- 8/5/2008 13 BDP grant samples to ----- (Baghdad,Tameem, Ninewa, and Anbar). 8/7/2008 14 Draft questionnaire translated 8/11/2008 15 ----- contacts IRD field and pre-test is fielded 8/11/2008 16 Points of contact identified 8/12/2008 17 Questionnaire revised, interview procedures clarified as needed 8/25/2008 18 Survey fielded (all but Baghdad). 8/27/2008 19 Baghdad sample redone excluding IACCI grantees 9/5/2008 20 Baghdad new sample fielded 9/15/2008 21 Data dictionary prepared for ----- based on revised questionnaire 9/23/2008 22 Data entry and validation begins as interviews are completed 9/24/2008 23 Sample data received from ----- 10/1/2008 24 Data entry completed 10/19/2008 25 Trial data analysis begins 10/20/2008 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 20 MEPP II CSP BDP Study Annex V. Analysis of the BDP Survey Data Survey non-response Fieldwork for the survey concluded on 20 October 2008. At that time 8 out of 327 interviewees had not been accounted for although field monitors continue to trace them. The disposition of respondents as of 20 October 2008 is shown in the table below. Result of Interview Count Completed 269 Refused 1 Not at home or business 14 Invalid address 3 Security prevented access 12 Business has closed 9 Business has relocated 4 Other 7 Total 319 Converting some non-response codes into businesses assumed closed. Field monitors were asked to explain why they were unable to complete an interview when they had entered a non-response code. In this survey it was important to differentiate between a grantee whose business had failed and he couldn’t be located, and a grantee who simply could not be located or was not available to complete an interview. In the latter occurrence the respondent would be excluded from the data analysis. In the former the respondent could be included as a failed business. In all of the instances where a non-response code occurred the monitor’s explanation was reviewed and the non-response code edited accordingly. When there is evidence that the business has closed8 , survey question 20 in Part 1 of the questionnaire should also show a response of “no.” In the following table the overall success rate of businesses is estimated along with survey non-response that is shown here as “unknown status.” Those with unknown status are excluded from further consideration in the data analysis. Business status Count Column   % Business is operational 269 84.3% Business has closed 11 3.4% Unknown status 39 12.2% Total 319 100.0% 8 Part of this survey took place during Ramadan and the Eid al Fitr holiday. As Ramadan is a period of fasting some of the small shops like bakeries could have been closed temporarily. We did not treat these businesses as closed, but gave them the status of unknown. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 21 MEPP II CSP BDP Study When those with unknown status are excluded from the data analysis the estimated overall percentage of sustainable businesses is 96.1% (the weighted estimate is 98.5%, see Sample weights below for a discussion of sample weights). Merging Sample data with Data from the BDP Grants Database The study design envisioned merging sample survey data with data from the CSP BDP Grants database. Administrative data not collected during the survey such as value of the grant, value of grantee participation, business sector, grant size are available from the database and including these data are critical to answering some of the questions posed in the SOW. The sample data were merged with the database using the grant project code (the sample was selected initially from the grants database and these codes were used to identify survey respondents). The Grants Information Panel of the survey questionnaire includes the grant project code that was pre-entered on the questionnaire and then entered as data along with survey responses. Some minor difficulty was encountered in merging the data when project codes were not correctly entered, however simple data entry corrections corrected the problem and all sample data were successfully merged with the database information. Sample Weights Sample weights are the reciprocal of the probability of a grantee’s selection. This is explained in Annex II on sample design. Here the weights include adjustment for non-response. These weights have been added to the sample survey data file. Weights are used when estimates are made of an overall population characteristic. Weights inclusive of non‐ Business Sector of Grant response Medium Small Micro Agriculture 1 7.35 N/A Industrial/Manufacturing 1.19 5.34 2.53 Trade 1 10.61 24.75 Service 1 4.97 4.55 Statistical Outliers and Data Validation All of the variables collected during the survey were subjected to a frequency analysis that included the identification of outliers. Outliers are values that are more than two standard deviations from the mean. Outliers bear further investigation to ensure that they are not the result of data entry or recording errors. This was done for all variables included in this analysis. When outliers were detected they were either corrected or declared as missing values and not included in the analysis. provides IBTCI with a data file in SPSS format. The data entry is done according to a data dictionary prepared by IBTCI. In addition provide IBTCI with scanned copies of all the original questionnaires in Arabic and translated into English. When during the analysis anomalies are found IBTCI refers initially to the questionnaires. If the original questionnaire International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 22 MEPP II CSP BDP Study does not explain the anomaly, then ----- may be asked to return to the field to verify a finding. This action was taken when data anomalies occurred. Data Anomalies Grantees either had businesses that were currently operating, had operated in the past or were start-up businesses. We wanted to include in the study a description of whether the grantees were restarting an existing or former business, or were starting out in a new venture. To do this, survey respondents were asked to respond to questions about when the business started. This was compared with information on the database classifying a business as “new” or “expansion.” To further the triangulation of whether the business was new or existed before the grant respondents were asked whether the grant was used to “restart” their business. The reason for this attention to detail is to help in the assessment of how much “new” employment has actually been generated by the grants. As a means of triangulating how long the grantee’s business has been in operation respondents were asked for the month and year when they started making money from the grant (this date is also referred to as the date the grant was initiated). This data was then compared to the grant close out or the date of completion so that we had a better understanding of how long businesses had been operating under the benefit of the grant. Comparing close out/completion date with the respondent’s stated date for beginning to use the grant revealed wide differences in Tameem, and to a much less extent in Ninewa. Investigations showed that the problem arose in preparing the sample frame of grantees. In preparing the sample frame individual grantee tracking databases from Ninewa, Tameem, Anbar and Baghdad were merged. Unfortunately these tracking databases are not all in the same format and considerable data manipulation is necessary to create a common format. An error was made when the Tameem data were merged with the others. This error occurred in the dates recorded for project completion or closeout (called the Status Date in the database). The result of the error was that most of the grantees selected from Tameem were of a more recent date that those from other cities and did not meet the criteria of grant completion before 1 September 2007. Unfortunately this did not come to light in time to correct the problem and resample. This has a possible impact on the data analysis and therefore the analysis shows results with and without Tameem data when that is appropriate. In general, more recent grantees will show higher sustainability levels, but less new employment generation. There is a possible influence on the findings and these are expressed in the analysis. Grant values and Grantee Contributions Grant values are an important part of this special study. Grant value represents the investment made by the CSP through the grant mechanism with the intention of achieving a positive result in terms of employment and business sustainability. The CSP BDP database records grant value. Grants are never made in cash and always represent the value of equipment or supplies provided to the grantee. Field monitors were asked to verify the equipment or supplies received by the grantee. Coupled with the grant value is the value of contributions made by the grantee to the business project that is the object of the BDP grant. Grantee contributions are auditable values that should provide a minimum of 25% in-kind contribution to the project in the form of operational costs, rental fees, purchase of materials, equipment or raw materials, as well as salaries of new employees. Grant value and grantee contributions were merged with the survey data to permit the calculation of grant cost per employee estimates and the cost of job growth. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 23 MEPP II CSP BDP Study Characterizing Businesses and Grants Business Size The size of business is tallied from Part 2 Questions 1 to 3 and refers to the number of paid employees at the enterprise before the grant, when the grant was initiated, and at the current time. Using full time employment at the time of the survey the following chart characterizes the size of businesses that have been awarded grants. Two is the median size for businesses that have been awarded grants (50% of businesses have more than two employees; 50% have two or one). The largest business encountered by the survey had 30 full time employees; less than 3% of grantees had 10 or more full time employees. The table below shows that smaller businesses of just one or two employees are associated with the trade and service businesses sectors (as might be expected). Many of the small grantees are individual cart sellers operating in areas that were previously closed for security reasons. The distributions below correspond to that characterization. The agricultural sector and the industry/manufacturing sectors are more labor intensive (and tend to be more formal businesses. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 24 MEPP II CSP BDP Study Business Sector of Grant Size of Business (FTE) Agriculture Industrial/ Manufacturing Trade Service Total 1 1.5% 2.3% 95.1% 1.1% 100.0% 2 9.5% 68.8% 21.7% 100.0% 3-4 16.0% 17.4% 46.0% 20.6% 100.0% 5 or more 34.0% 33.0% 20.5% 12.5% 100.0% Total 11.8% 14.7% 59.1% 14.4% 100.0% Chi-square test confirms that these differences are statistically significant Business sectors The trade and service sectors dominate the grantees; nearly three-quarters of grantees are in these two sectors. Grant allocation by business sector shows that the trade sector received the largest proportion of grant funding. Agriculture and industry/manufacturing has larger allocations than the number of grants would suggest. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 25 MEPP II CSP BDP Study Grantee contributions by business sector reveal that all but the trade sector have contributed in proportion to what they have received. The trade sector grantee contribution is approximately one-third of the grant value. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 26 MEPP II CSP BDP Study Grant Size There are three categories of grant: medium, small and micro described in the study design Annex II. Each grant category attracts a different degree of grant application due diligence. Naturally, awarding micro grants asks for less information from the would-be grantee than it does for an enterprise seeking a small or medium grant. Many more enterprises are able to qualify for a micro grant than they would for a higher value grant and the distribution is as expected. More than half the grants are micro grants. The next chart shows how the grant money was allocated within the grant size categories. Small grants, in money terms, received more overall grant allocation than the other grant sizes. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 27 MEPP II CSP BDP Study Micro grantees contributed a comparatively smaller proportion to the overall grant (grant plus grantee contribution) than did medium and small grantees (not calculated here, but seen by comparing the two charts). Business Administration We knew little about how the businesses were organized from the CSP BDP database so grantees were asked about how their business was administered/organized and whether the International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 28 MEPP II CSP BDP Study business was registered. Unregistered businesses might be thought of as informal businesses. Informal businesses would be less likely to obtain commercial debt, and may be less likely to consider job growth or experience job growth. This view is substantiated in the following tables. Size of Business (FTE) Part 1 Q14: Have you registered your business with the government Yes No Total 1 13.7% 86.3% 100.0% 2 27.4% 72.6% 100.0% 3-4 27.3% 72.7% 100.0% 5 or more 57.3% 42.7% 100.0% Total 29.9% 70.1% 100.0% Chi-square test confirms that these differences are statistically significant Smaller size businesses rarely attempted to borrow money, while about one-third of the larger businesses did. This relates to the smaller businesses being informal and not registered. The relationship shown below is statistically significant. Percentage of Business Atempted to Borrow by Size of Business: Size of Business by Full Time Employees (FTE) Size of Business (FTE) Part 2 Q8: Have you attempted to borrow money to expand your business since your grant award Yes No Total 1 6.3% 93.7% 100.0% 2 16.0% 84.0% 100.0% 3-4 26.7% 73.3% 100.0% 5 or more 29.2% 70.8% 100.0% Total 19.4% 80.6% 100.0% Chi-square test confirms that these differences are statistically significant Business registration relates to attempts to borrow money and this is statistically significant. In a formal commercial relationship business registration is a likely prerequisite. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 29 MEPP II CSP BDP Study Attempt to Borrow Money by Business Registration % within Part 1 Q14: Have you registered your business with the government Part 1 Q14: Have you registered your business with the government Part 2 Q8: Have you attempted to borrow money to expand your business since your grant award Yes No Total Yes 26.2% 73.8% 100.0% No 14.9% 85.1% 100.0% Total 18.0% 82.0% 100.0% Chi-square test confirms that these differences are statistically significant Most businesses characterized themselves as sole proprietors (60%) and this is obviously related to the small sizes of most grantee business, but also the informal nature of most grant supported businesses. Larger size businesses hire non-family members and tend to be registered as partnerships or cooperatives. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 30 MEPP II CSP BDP Study The likelihood of attempting to borrow money to expand the business is clearly related to how the business is organized, or simply whether it is a formal or informal business. Attempt to Borrow Money by how the Business is Run (in percentage) Part 1 Q11: Characterize how your business is run (Type of Business) Part 2 Q8: Have you attempted to borrow money to expand your business since your grant award? Type of Business Yes No Total Sole proprietor 14.3 85.7 100 Family owned business with no non-family employees 13.8 86.2 100 Family owned business with non-family employees 15.3 84.7 100 Partnership/Cooperative 69.2 30.8 100 Total 18.0 82.0 100 Chi-square test confirms that these differences are statistically significant The distribution for business registration (below) closely follows that for attempts to borrow money (above). Percentage of Businesses Registered by type of Business Part 1 Q11: Characterize how your business is run Part 2 Q8: Have you registered your business with the government? Type of Business Yes No Total Sole proprietor 27.7 85.7 100 Family owned business with no non-family employees 21.1 86.2 100 Family owned business with non-family employees 38.9 84.7 100 Partnership/Cooperative 63.9 30.8 100 Total 27.7 72.3 100 Chi-wquare test confirms that these differences are statistically significant International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 31 MEPP II CSP BDP Study The table below is concerned with grantees whose grants became active 12 months or more ago and whether they had created any new employment since the grant award. This is a sub￾set of the entire population of CSP BDP grantees. Overall less than 2% of grantees managed to achieve any job growth. We might have stopped the analysis at this point. However, as we will see below, this job growth can be traced to specific business sectors and grant sizes. The following two tables illustrate that most job growth occurred in what we think of as a the more formal business sector, and that businesses with just one or two employees at the outset did not increase their employment in the 12 months since the grant was awarded. Part 1 Q11: Characterize how your business is run Did employment growth occur in the 12-month period since the grant? Type of busines Yes No Total Sole Proprietor 1.1% 98.9% 100.0% Family-owned Business (no non￾family employees) 100.0% 100.0% Family-owned Business (non-family employees) 5.8% 94.2% 100.0% Partnership/Cooperative 2.4% 97.6% 100.0% Total 1.8% 98.2% 100.0% Chi-square test confirms that these differences are statistically significant Size of Business Did employment growth occur in the 12-month period since the grant? Number of Full Time Employees Yes No Total 1 100.0% 100.0% 2 100.0% 100.0% 3-4 3.8% 96.2% 100.0% 5 or more 2.3% 97.7% 100.0% Total 1.7% 98.3% 100.0% Chi-square test confirms that these differences are statistically significant The analysis now moves to assessing the design study model of whether the grant size/business sector factors have any effect on the chances of business success. The use of these more complicated statistical procedures is a tool for exploring relationships in the data that may not be obvious on the surface. What we have seen above in the tables and the Chi￾square tests shows the distributions associated with each category of a variable are statistically different from the marginal distribution. For example, the distribution of grantees between whether or not growth occurred is influenced by the size of the business and is statistically different from the marginal distribution (this is what the Chi-square measures). It does not tell us whether the individual distributions say between businesses of 5 or more employees compared with the marginal distribution is statistically different. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 32 MEPP II CSP BDP Study Effect sizes (aka event rates) A determination of the sample size needed for the study hinged on what was the assumed to be the effect size (also known as event rates). Event rates were the percentage of sustainable businesses that we estimated were associated with each factor. These assumptions were listed in the study design and are repeated here: the agriculture sector event rate is 0.82, the industrial/ manufacturing sector event rate is 0.95, the trade sector event rate is 0.60, and for the service sector event rate is 0.70. These assumed event rates were used across the grant size categories. From the sample data we can now estimate the true event rates and we found that our assumptions were incorrect. Event rates estimated from the sample data are systematically higher than our assumptions and there are only small differences between the factors. This has two effects. The first is that the power of the test is reduced, and the second is that analysis of comparisons may show more broad confidence intervals. The reduction in power means that a much larger sample would have been needed to detect significant differences between the factors. The good news is that grantees in terms of business success seem to be doing better than we assumed. In terms of decision making about what grant size/ business sector factor has the best chance of success, the answer is that there is little differences between the factors. Nevertheless it is shown below that the null hypothesis can be rejected and that individual factors are found that do not perform as well as others. Logistic regression, testing the null hypothesis The basis for the sample design was to have a large enough sample so that the null hypothesis could be tested. As stated above, the calculated sample size depended in part on assumptions about effect size in a factorial model. Effect sizes proved to be much less than assumed and there is little difference between the eight business sectors and size of grant groups that make up the factors in the design. As we have seen above the overall business sustainability was 96.1%. Despite small differences between the factor groups a test of the null hypothesis using logistic regression9 did permit rejecting the null hypothesis that all the factor effect sizes are statistically the same. The basis for rejecting the null used all eight of the factors as independent variables that predicted the dependent variable of business success or failure. Results of the logistic regression are explained next. Logistic regression is an appropriate statistical tool to use when assessing the probability of business success or failure (or any other dichotomous variable). Logistic regression predicts the natural log of the odds or logit (see the glossary). Fortunately the relatively unfamiliar term logit can be transformed back into a more familiar reference of the odds of an event occurring. The survey results were subjected to a test of the null hypothesis using logistic regression to estimate the dependent dichotomous outcome variable captured in Part 1 Question 20 of the questionnaire (adjusted as described above). To do this each of the business sector/grant size groups were first transformed into dummy variables. A dummy variable is one that assigns “1” for membership in the group and “0” for lack of membership. There are eight factor dummy 9 Logistic regression using categorical independent variables was the analytical model used to develop the sample size requirements. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 33 MEPP II CSP BDP Study variables entered as independent variables in the logistic regression model. What we look for is whether our ability to predict the outcome dependent variable is influenced by the independent factor variables. Logistic regression works most reliably when the dependent variable has a probability of between 0.2 and 0.8; here the overall probability of success is 0.9610 so some caution with these results is advised. In the first model tested (results shown below) all eight of the dummy variables representing the factor groups were entered concurrently. The Chi-square result in the standard Omnibus Tests table below shows that the introduction of the eight variables improved the model and that the improvement was statistically significant. There was a 35.6% percent improvement in the goodness of fit resulting from this first model (i.e., the model is potentially a better predictor of business success with the independent dummy variables than without them). The significance level for the Chi-square is below .1 (the p-value in the glossary) so that we can reject the null hypothesis that poses that there is no relationship between the dependent and independent variables in the model. Omnibus Tests of Model Coefficients Chi-square df Sig. Step 25.962 8 .001 Block 25.962 8 .001 Step 1 Model 25.962 8 .001 The Nagelkerke R-square below shows that the model explained about 30% of the variance. Model Summary Step -2 Log likelihood Cox & Snell R Square Nagelkerke R Square 1 72.938a .089 .298 a. Estimation terminated at iteration number 20 because maximum iterations have been reached. However when we wish to assess the contribution of each of the independent dummy variables to the model we find that none of them are significant (see the column labeled “Sig.” in the table below; none of the values are below our p-value). We want to do this in order to answer the question posed regarding which business sectors and grant sizes show the highest rate of business success. Thus with respect to the influence of any particular factor in the regression the results are inconclusive. In exploratory analysis such as this a stepwise regression where each of the dummy variables are entered in turn, but entered conditionally depending on whether a statistical criteria for inclusion is met, is recommended. In this way variables that may be “suppressing” other variables can be excluded from the model, and those variables that are most influential in contributing to the probability of success (or failure) of the business are 10 Hosmer, D.W. and Lemeshow, S. (1989), Applied logistic regression, New York, Wiley, page 168. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 34 MEPP II CSP BDP Study retained. The conditional stepwise inclusion of the factor dummy variables was used as an alternative logistic regression model. Variables in the Equation (all variables included) B S.E. Wald df Sig. Exp (B) small_agricultural(1) 19.331 6893.037 .000 1 .998 2.485E8 small_industrial(1) .932 1.052 .784 1 .376 2.538 small_trade(1) 19.331 6996.701 .000 1 .998 2.485E8 small_service(1) .038 .929 .002 1 .968 1.038 micro_industrial(1) 19.331 6698.828 .000 1 .998 2.485E8 micro_trade(1) 19.331 6698.828 .000 1 .998 2.485E8 micro_service(1) 19.331 6996.698 .000 1 .998 2.485E8 medium_industrial(1) -.214 .935 .052 1 .819 .808 Ste p 1a Constant -99.283 15335.261 .000 1 .995 .000 When forward conditional stepwise entry of the dummy variables is done the resulting best model reached on step 3 (additional steps did not improve the fit of the model) identifies three of the variables as significant. These are the dummy variables representing 1) small grants to industrial sector businesses, 2) small grants to service sector businesses and 3) medium grants to industrial sector businesses. Each of the B values is negative meaning that being a member of any one of these categories reduces chances of business sustainability. In logistic regression B represents natural log of the odds. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 35 MEPP II CSP BDP Study B S.E. Wald df Sig. Exp(B) Step 1 medium_industrial(1) -1.764 .653 7.289 1 .007 .171 a Constant -1.658 .546 9.239 1 .002 .190 small_service(1) -2.089 .736 8.054 1 .005 .124 medium_industrial(1) -2.340 .743 9.916 1 .002 .096 Step 2b Constant .430 .916 .221 1 .638 1.538 small_industrial(1) -1.724 1.018 2.869 1 .090 .178 small_service(1) -2.618 .890 8.647 1 .003 .073 medium_industrial(1) -2.869 .896 10.250 1 .001 .057 Step 3c Constant 2.683 1.771 2.296 1 .130 14.633 These results are intuitively acceptable as all of the occurrences of business failure are seen in these three highlighted factors. The odds of success depend on the grant size/business sector that you are in. The table below shows the estimates of the odds of success (success means the grantee is still in business after one-year). From the table it is seen that the overall chance of success is 22 to 1, small grants to the industrial sector have less chance of success at 17 to 1 odds, small grants to the service sector have lesser odds still at 7 to 1, while the lowest odds of success are with medium grants to the industrial sector. All of the other grants size and business sector combinations were not significantly different from the overall odds or had no failed cases. However, as we will see below, CSP should not rush to curtail medium grants as they seem to provide the best opportunity for job growth in the 12-month period since the grant award. Odds by Factorial Group Group Odds of success Overall 22 to 1 Factor Group    Small grant to industrial sector 17 to 1 Small grant to service sector 7 to 1 Medium grant to industrial sector 5 to 1 Other factor groups >22 to 1 Excluding Tameem (and the effect of more recent grants) brings the odds of success down. This makes sense because business sustainability is obviously not well tested with more recently started businesses. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 36 MEPP II CSP BDP Study Odds by Factorial Group (excluding Tameem) Group Odds of success Overall 17 to 1 Factor Group    Small grant to industrial sector 14 to 1 Small grant to service sector 6 to 1 Medium grant to industrial sector 5 to 1 Other factor groups >17 to 1 Looking at the odds of success sector by sector reveals that trade and agriculture are near sure winners with the industrial/manufacturing sector following and the service sector last. The data were weighted to reflect the overall number and type of grantees that are in the statistical universe. This adds considerable influence to the trade and micro grantees and improves the overall estimate of success. Medium grants are the most risky, although they may be the only ‘bankable’ projects. Odds by Business Sector Group Group Odds of success Weighted odds of success Overall 22 to 1 58 to 1 Sector Group       Agriculture no failures no failures Industrial/Manufacturing 15 to 1 19 to 1 Trade 70 to 1 near to no failures Service 13 to 1 14 to 1 When Tameem is excluded from the calculations the results are marginally worse. Odds by Business Sector Group (excluding Tameem) Group Odds of success Weighted odds of success Overall 17 to 1 47 to 1 Sector Group       Agriculture no failures no failures Industrial/Manufacturing 10 to 1 13 to 1 Trade 58 to 1 near to no failures Service 10 to 1 11 to 1 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 37 MEPP II CSP BDP Study Odds by Grant Size Group Odds of success Weighted odds of success Overall 22 to 1 58 to 1 Size Group       Medium $25K to $100K 6 to 1 5 to 1 Small $3K to $25K 21 to 1 29 to 1 Micro $100 to $3K no failures no failures Odds by Grant Size (excluding Tameem) Group Odds of success Weighted odds of success Overall 17 to 1 44 to 1 Size Group       Medium $25K to $100K 6 to 1 5 to 1 Small $3K to $25K 20 to 1 28 to 1 Micro $100 to $3K no failures no failures The final question posed in the SOW asks whether city size or region affects business success. There are many fewer qualifying grantees in regions other than Baghdad. In addition, the regions have a different mix of business sector and grant sizes. Overall, Baghdad experienced a lower success rate than the other regions. “Lower” is relative against a background of a very high success rate in general. With the information above we can begin to provide answers to part of the questions posed in the SOW: 1.1. What percentage of the grantees is still in business after one year? 1.1.1. 96 % (94% without Tameem) of grantees were still in business after one-year (when weighted this rises to 98%). 1.2. Are there quantitatively measured trends in business success (still in operation) or failure? 1.2.1. Micro grants in all sectors are the best bet for business success. Nearly all of the micro grantees are still in business. 1.2.2. Medium grants to the industry/manufacturing sector are least likely to succeed, but even here the odds of success are good at 5 to 1 of success. 1.2.3. Small grants to the service sector have an estimated 6 to 1 chance of success. 1.2.4. Small grants to the industry/manufacturing sector have a 17 to 1 chance of succeeding. 1.3. Which grant size demonstrated the highest rate of grantees still in business? International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 38 MEPP II CSP BDP Study 1.3.1. Micro grants have a nearly perfect record of business success, small grants have a success rate of 96%, while medium grants were successful 86% of the time. 2. Are there different challenges in respective regions? There were too few qualifying grantees in regions outside Baghdad to make a determination of statistical differences. 2.1. Does the location of the business in regards to city size or region affect business success? 2.1.1. Baghdad experiences a marginally lower rate of success than the other regions, but this is a reflection of the mix of grant sizes and business sectors that have been awarded grants. Also in the data Baghdad has been awarding grants for much longer than the other regions so the survival rate is bound to be less simply because the businesses have been around longer. To answer the remaining SOW questions about employment and job creation we need first to define the different concepts intended by “create new jobs” and “post award job creation.” And finally we need to estimate the cost of creating new jobs Employment The remaining questions posed in this special study need an estimate of employment that resulted from BDP grant interventions. Gauging BDP grant inspired employment is complex. It is not clear from the employment data on the CSP BDP database if new employment was gained or existing employment maintained. Existing employment here means that grants were given with the objective of enhancing an existing business that may already have had employment. In other words these types of grant did not create new employment at the time the grant was awarded (although new employment may be generated after the grant award as the business prospers). “New” employment is defined here as employment generated at the time of the grant award. The assumption is that no previous business existed and grantees are embarking on a new venture (a start-up). The difference between “new” and “existing” is not reliably recorded in the CSP Grants database.11 For this reason the special study questionnaire asks grantee respondents about the business employment history. The source of data on employment comes from the survey questionnaire Part 2 Questions 1 through 3 as well as Question 6. When pre-grant employment is compared with employment at the time of the grant award (literally when the grant was first used to make money) 37% of grantees reported that there was no difference between pre-grant employment and employment at the time of the grant. The conclusion is that 37% of the grant awards were to existing businesses that added no “new” employment. The reason for this distinction is to classify BDP grant employment generation as either “new” or “existing.” The cost for generating new and existing employment is discussed below. 11 The CSP BDP database classifies grantees as “new” or “expansion.” In the database only 5% of businesses were nominally “expansion” grants. Based on an analysis of pre and post grant employment figures the study estimates that about 37% of the grants were awarded to existing firms. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 39 MEPP II CSP BDP Study In addition, grantee respondents were asked for the number of employees they currently had (observed by the field monitors). These employment figures were compared with employment at the time when the grant was used to make money as a means of gauging employment growth since the grant award (typically over the past one-year or more). It is this employment growth figure that is used to answer the question “Were additional jobs created by the grantee 12 months after the initial grant?” The table below is the weighted estimate of total employment in new or existing businesses by size of the grant. The tables show summary estimates as well as the median and mean numbers of employees per grantee. On average a medium grant to a new business produces 11 new jobs or sustains 8 jobs when the business is a continuing enterprise. New jobs are created in relative proportion to the size of the grant. Small grants were the favored vehicle for new job creation and account for the largest number of new jobs created. Total Estimated New and Existing Employment by Size of Grant Grant Type by Size Total employment at grant end Existing businesses New businesses Total Medium Mean 8 11 10    Median 7 8 8    Sum 154 226 380    Row Sum % 41% 59% 100% Small Mean 4 5 5    Median 4 4 4    Sum 1577 2688 4265    Row Sum % 37% 63% 100% Micro Mean 2 2 2    Median 1 2 2    Sum 863 1465 2327    Row Sum % 37% 63% 100% Total Mean 3 4 3    Median 2 3 3    Sum 2594 4378 6972    Row Sum % 37% 63% 100% The table below is the weighted estimate of total employment created or sustained by CSP BDP grants to MSME by business sector. On average a grants to agriculture or industry/manufacturing produce more new jobs that grants to the trade and service sectors. Most new jobs were created in the trade sector followed by industry/manufacturing and agriculture. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 40 MEPP II CSP BDP Study Total Estimated New and Existing Employment by Business Sector Business Sector of Grant Total employment at grant end Existing businesses New businesses Total Agriculture Mean 6 6 6    Median 5 5 5    Sum 714 799 1512    Row Sum % 47% 53% 100% Industrial/Manufacturing Mean 5 5 5    Median 5 4 4    Sum 614 838 1452    Row Sum % 42% 58% 100% Trade Mean 2 3 2    Median 1 2 2    Sum 897 2108 3005    Row Sum % 30% 70% 100% Service Mean 4 3 3    Median 4 3 3    Sum 370 633 1003    Row Sum % 37% 63% 100% Total Mean 3 4 3    Median 2 3 3    Sum 2594 4378 6972    Row Sum % 37% 63% 100% The final employment table below is the weighted estimate of total employment created or supported in existing businesses by CSP BDP grants cross tabulated by the factors identified in the study design. Here we identify the sector size and grant combinations that, in summary, have produced the most summary employment and the greatest average employment per grantee. Medium size grants are lumped together as the numbers are sparse. Small grants to the trade sector produced, in summary, the most new employment. Looking at the mean values it can be seen that a small grant to a business in the agricultural sector produces more new employment than any of the other combinations of grant size and business sector. The next step in the analysis is to determine whether the differences in the mean values are statistically significant. To do this we use the general linear model to do an analysis of variance discussed in the next section. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 41 MEPP II CSP BDP Study Total Estimated New and Existing Employment by Grant Size and Business Sector Grant size/ business sector factors Total employment at grant end Existing businesses New businesses Total medium grant Mean 8 11 10    Median 7 8 8    Sum 154 226 380    Row Sum % 41% 59% 100% small agricultural Mean 6 6 6    Median 5 5 5    Sum 706 772 1477    Row Sum % 48% 52% 100% small industrial Mean 5 5 5    Median 5 4 5    Sum 449 401 849    Row Sum % 53% 47% 100% small trade Mean 3 4 4    Median 3 3 3    Sum 244 1178 1422    Row Sum % 17% 83% 100% small service Mean 4 4 4    Median 4 4 4    Sum 179 338 517    Row Sum % 35% 65% 100% micro industrial Mean 2 4 3    Median 2 4 4    Sum 33 276 309    Row Sum % 11% 89% 100% micro trade Mean 1 2 2    Median 1 2 2    Sum 644 916 1559    Row Sum % 41% 59% 100% micro service Mean 5 3 3    Median 4 2 2    Sum 187 273 460    Row Sum % 41% 59% 100% Total     2596 4380 6973 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 42 MEPP II CSP BDP Study Job Growth Additional jobs created by the grantees in the 12 months after the initial grant (job growth) are determined by comparing current employment (Part 2 Question 2 of the questionnaire) with employment at the end of the grant (jobs when the grant equipment and supplies were delivered and the grant closed that was recorded in Part 2 Question 3 of the questionnaire). In the study all grantees selected were to have had a grant closure date of before 1 September 2007. Due to a processing error some of the selected grants did not meet these criteria and they were excluded from this estimation. Exclusion was done by using the date provided by the respondent in Part 1 Question 18. Here the respondent stated when he started using the grant to make money. If this date was after December 2007 the grantee response was excluded from this analysis. Overall the generation of new employment since the grant was initiated was very small amounting to just less than 2% or an estimated increase of just 59 jobs (weighted estimate). The odds of a grantee creating new jobs in the 12 months since the grant started are about 50 to 1 against. However, drilling down analytically shows that most of this limited job creation came from just a few sources. The table below shows these estimates and categorizes them by grant size and whether the business in a new business or an existing one. All of the new employment created by the grantee since the grant was initiated is found in existing businesses (also referred to as expansion businesses). Existing business grantees experienced a 5.6% growth in employment over the year. Growth was measured by using the number of jobs when the grant was initiated as the denominator and the number of new jobs since the grant was initiated as the numerator. Medium grants to existing businesses exhibited a 45% growth in jobs since the grant was initiated (70/154 from the tables above and below). For existing businesses the odds of a micro grant generating job growth was nil, for small grants the odds were 1 in 25 and for medium grants the odds were slightly better than 1 in 10 thus there appeared to be a small chance for job growth to occur. New businesses apparently lost employment and the existing businesses accounted for all of job growth since the grant was initiated. Further, there were no gains in employment for micro grants while medium grants to existing businesses yielded on average 4.3 new jobs since the grant was initiated. Small grants showed a modest success in creating new jobs. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 43 MEPP II CSP BDP Study Total Estimated New Employment since Grant Initiation by Grant Size and New or Existing Business New or existing business Grant Type by Size Total change in full and part time since grant Existing businesses New businesses Total Mean 4.31 -.46 1.85 Median 0 0 0 Sum 70 -8 62 Medium Count 24 17 42 Mean .20 .00 .09 Median 0 0 0 Sum 55 0 55 Small Count 297 330 627 Mean .00 -.16 -.07 Median 0 0 0 Sum 0 -58 -58 Micro Count 493 370 862 Mean .16 -.09 .04 Median 0 0 0 Sum 125 -66 59 Total Count 814 717 1531 Existing businesses in the Industrial/Manufacturing sector, the Service Sector and the Trade sector were each responsible for a portion of the job gains reported. The percentage growth by sector shows that the service sector apparently had the highest growth rate at 13% followed by industrial/manufacturing and then trade. The odds for job creation favored the service sector followed by industrial/manufacturing and then the trade sector (all within the existing business category). International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 44 MEPP II CSP BDP Study Total Estimated New Employment since Grant Initiation by Business Sector and New or Existing Business New or existing business Business Sector of Grant Total change in full and part time since grant Existing businesses New businesses Total Mean .00 .00 .00 Median 0 0 0 Agriculture Sum 0 0 0 Mean .45 -.01 .25 Median 0 0 0 Industrial/Manufacturing Sum 49 -1 48 Mean .07 -.10 -.02 Median 0 0 0 Trade Sum 32 -50 -18 Mean .54 -.22 .19 Median 0 0 0 Service Sum 45 -16 29 Mean .16 -.09 .04 Median 0 0 0 Total Sum 125 -66 59 Looking at the grant size/business sector combinations pinpoints where job growth occurred. Job growth since the grant was initiated can be isolated to three grant size/ business sector categories; all of these are for existing businesses in these grant size and business sectors. Identified were: 1) medium size grants to the Industrial/ Manufacturing sector that exhibited a 53% job growth rate; 2) small grants to the trade sector doubled the number of jobs but this is based on a very small number of firms and is not likely to be statistically significant (more on this below); and 3) small grants in the service sector where a positive job growth rate of 25% was observed. Converted into the odds for job growth, medium size grants awards in the industrial/manufacturing sector had a 1 in 5 chance of achieving growth. A small grant to the trade sector had a 1 in 15 chance, while a small grant to the service sector had a 1 in 8 chance. Small grants to the industrial/manufacturing sector saw a reversal in the size of employment. In the next section we look more closely at whether the relationships indentified in these tables are statistically significant. GLM Model Specification The general linear model was used to assess the null hypothesis regarding employment (including new employment since the end of the grant) and to estimate the effect size in the factorial model proposed in the study design. The General Linear Model (GLM) Univariate procedure in SPSS provides regression analysis and analysis of variance for one dependent variable by one or more factors and/or variables. The factor variables divide the population into groups – these are the business sector categories and the grant size categories. The GLM is International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 45 MEPP II CSP BDP Study used to assess the null hypothesis that business sector and grant size have no effect on employment. Interactions between factors (sector and size) as well as the effects of other factors that may influence employment (gender, business registration, technical training) can be considered. Similarly the effects of covariates and covariate interactions with factors can be included in the analysis. For example, the effect of the education level, number of persons in the household, number of dependents <15, size of grantee contribution and actual size of the grant (provided the categorical grant size variable is excluded) can be included as suitable covariates for exploratory analysis. In general, factor variables are categorical while covariates are scale. We use the GLM when the dependent variable (what we are trying to predict) is a scale variable (in logistic regression it is a dichotomous variable). Here we are attempting to predict average job growth based on the factors we defined in the study design. We are looking to see whether the mean value of job growth is related to membership in one of the factors (grant size/business sector groups). Later we add covariates to this relationship to look for other reasons that may explain job creation. But first we turn to the basic determination of whether job growth is influenced by the factors. Exploring for the significant factors that have an influence on job growth starts first with an overall test of the null hypothesis that grant size and business sector have no influence on job growth. This is tested in an analysis of variance by examining the overall F statistic to see if it is significant. The table from the SPSS GLM procedure is presented below with the F statistic and its significance highlighted. The F statistic is high enough to permit us to reject the null hypothesis. However, the strength of this assertion is low because overall the model only explains 10% or less of the variance. The table also shows that the Business Sector, Grant Size and the interaction of Business Sector with Grant Size and not statistically significant. This means that their contribution to the influence on job growth is not significantly different from zero. However, it bears drilling down to look at the individual contribution made by the interaction effects. This is done in the next table. Tests of Between-Subjects Effects Dependent Variable:Total change in full and part time since grant Source Type III Sum of Squares df Mean Square F Sig. Partial Eta Squared Corrected Model 142.602a 10 14.260 1.981 .038 .107 Intercept 3.685 1 3.685 .512 .475 .003 Business Sector 31.208 3 10.403 1.445 .232 .026 Grant Size 6.553 2 3.277 .455 .635 .005 Business Sector * Grant Size 59.437 5 11.887 1.651 .149 .048 Error 1187.938 165 7.200 Total 1349.000 176 Corrected Total 1330.540 175 a. R Squared = .107 (Adjusted R Squared = .053) International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 46 MEPP II CSP BDP Study In this table of Parameter Estimates we see the estimated influence that each category of grant size and business sector (and their combinations) has on the predicted job growth. Only one of the parameters is statistically significant. This is highlighted below and when translated shows that medium grants to the industrial/manufacturing sector is significant and has a positive influence on job growth (B is positive 3.982). Parameter Estimates Dependent Variable:Total change in full and part time since grant      Parameter B Std. Error t Sig. Intercept 0.474 0.616 0.77 0.443 [Sector_Code=1] ‐0.474 0.803 ‐0.59 0.556 [Sector_Code=2] ‐0.64 0.824 ‐0.777 0.438 [Sector_Code=3] ‐0.297 0.896 ‐0.332 0.74 [Sector_Code=4] 0a . . . [GrantTypeCode=1] ‐0.974 1.476 ‐0.66 0.51 [GrantTypeCode=2] ‐0.724 0.989 ‐0.731 0.466 [GrantTypeCode=3] 0a . . . [Sector_Code=1] * [GrantTypeCode=1] 0.974 1.971 0.494 0.622 [Sector_Code=1] * [GrantTypeCode=3] 0a . . . [Sector_Code=2] * [GrantTypeCode=1] 3.982 1.69 2.356 0.02 [Sector_Code=2] * [GrantTypeCode=2] 1.015 1.315 0.772 0.441 [Sector_Code=2] * [GrantTypeCode=3] 0a . . . [Sector_Code=3] * [GrantTypeCode=1] 0.797 2.49 0.32 0.749 [Sector_Code=3] * [GrantTypeCode=2] 0.483 1.279 0.378 0.706 [Sector_Code=3] * [GrantTypeCode=3] 0a . . . Dependent Variable:Total change in full and part time since grant (continued) [Sector_Code=4] * [GrantTypeCode=1] 0a . . . [Sector_Code=4] * [GrantTypeCode=2] 0a . . . [Sector_Code=4] * [GrantTypeCode=3] 0a . . . a. This parameter is set to zero because it is redundant.   Further analysis (post hoc multiple comparisons) reveals that medium size grants are significantly different than other grant size classes and is related to higher job growth. We can legitimately ask about the influence that other covariates may have on job growth. One of the covariates commonly associated with better prospects for success is education. When education is added as a covariate (it is an ordinal variable captured in Part 1 Question 8) the model’s fit improves and more of the overall variance in job growth is explained. Further analysis, not shown here, shows that the influence of education level of job growth is positive. So we might conclude that education could be more important than business sector or grant size. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 47 MEPP II CSP BDP Study Tests of Between-Subjects Effects Dependent Variable:Total change in full and part time since grant Source Type III Sum of Squares df Mean Square F Sig. Partial Eta Squared Corrected Model 174.099a 11 15.827 2.218 .016 .131 Intercept 15.988 1 15.988 2.240 .136 .014 EducationLevel 31.472 1 31.472 4.410 .037 .026 Sector_Code 36.848 3 12.283 1.721 .165 .031 GrantTypeCode 1.858 2 .929 .130 .878 .002 Sector Code * GrantTypeCode 45.227 5 9.045 1.267 .281 .038 Error 1156.229 162 7.137 Total 1349.000 174 Corrected Total 1330.328 173 a. R Squared = .131 (Adjusted R Squared = .072) Grantee contribution should also relate to positive job growth. The greater the value of grantee contribution the more we might expect to see grantee commitment to job growth. This seems to be the case. When grantee contribution is included as a covariate the model fit improves and the influence of higher levels of contribution is seen in higher job growth (highlighted below). The B value for grantee contribution is slightly positive and significant. When grant value itself (rather than the categorical grant size coded variable) is included as a covariate the influence is significantly positive as we would expect. The introduction of gender as an additional factor did not result in improvement in the model and was not itself significant. A final covariate was introduced. This was the total number of employees at the firm when the grant was initiated. This improved the fit of the model’s ability to explain the variance. The influence of number of employees at the outset was negative; the more employees at the outset the lower the number of new employees that were added. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 48 MEPP II CSP BDP Study Tests of Between-Subjects Effects Dependent Variable:Total change in full and part time since grant Source Type III Sum of Squares df Mean Square F Sig. Partial Eta Squared Corrected Model 208.545a 11 18.959 2.826 .002 .160 Intercept 5.383 1 5.383 .802 .372 .005 FinalGranteeContrib 29.533 1 29.533 4.402 .037 .026 Sector_Code 14.941 3 4.980 .742 .528 .013 GrantTypeCode .992 2 .496 .074 .929 .001 Sector Code * GrantTypeCode 46.004 5 9.201 1.372 .238 .040 Error 1093.489 163 6.709 Total 1324.000 175 Corrected Total 1302.034 174 a. R Squared = .160 (Adjusted R Squared = .103) To focus the above analysis and test for possible confounding influence of higher order parameters with the factors of interest in this study the GLM was done using only the factor dummy variables as was done in the logistic regression. This resulted in a similar outcome to the previous model with the rejection of the null hypothesis stating that none of the factors influenced job growth. As before only medium grants to the industrial/manufacturing sector was statistically significant. It can therefore be said with some confidence that membership in this group predicts the likelihood of job growth (3 jobs more when compared to not being a member of this group according to the B value). What is the cost of generating new or existing employment and job growth? New employment occurs when a start up enterprise is awarded a grant, as opposed to the award of a grant to an existing enterprise. The latter we refer to as existing employment. The cost per new or existing job is calculated by dividing the grant value by the number of new employees. This is done at the grantee level, rather than by dividing total employment by total grant value. The first chart explores grant size by cost per job split between newly created jobs and support to sustainable jobs. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 49 MEPP II CSP BDP Study New or Existing Business    Size of Grant Cost per employee at time of grant Existing businesses New businesses Total Medium Mean 6953 6180 6499    Median 6630 4000 4800    Count 26 21 47 Small Mean 2900 2547 2675    Median 2333 2025 2324    Count 381 560 941 Micro Mean 643 1375 1079    Median 475 1400 833    Count 502 631 1132 Total Mean 1712 1999 1888    Median 950 1500 1500    Count 909 1211 2120 The differences between the cost of new or existing employment are not statistically different. As you would expect there is a statistically valid difference between grant size classes. This difference is somewhat offset by relative differences in grantee contributions. The difference in cost per employee among business sectors is not statistically significant (despite appearing to be so in the following table and chart). This was concluded from a GLM International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 50 MEPP II CSP BDP Study analysis of variance model using cost per employee as the dependent with business sector and new/existing business as the factors. A Tukey multiple comparisons test showed that there was no significant difference in the mean cost of employment across business sectors. New or Existing Businesses      Business Sector of Grant Cost per employee at time of grant Existing businesses New businesses Total Agriculture Mean 2597 3211 2927    Median 1998 2858 2496    Count 126 129 255 Industrial/ Manufacturing Mean 3306 1830 2435    Median 2745 1000 1667    Count 142 168 310 Trade Mean 1044 1945 1597    Median 500 1600 1214    Count 533 713 1246 Service Mean 1884 1559 1657    Median 1975 1225 1392    Count 108 202 309 Total Mean 1712 1999 1888    Median 950 1500 1500    Count 909 1211 2120 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 51 MEPP II CSP BDP Study We looked next at the interaction effects of grant size and business sector. This was again done with the GLM and the results of that model are shown below. Highlighted are the F statistic and significance level of the Sector Code (business sector) and the interaction term Sector_Code * GrantTypeCode (business sector with size of grant). Both of these terms are not significant indicating that in this model they have no statistical effect on the mean value of cost per employee. This is not the end of the story as we next look at the 8 factors of the study design and whether they have a significant effect on the cost per employee. Dependent Variable is the Grant Cost per Employee at Time of Grant Award Source Type III Sum of Squares df Mean Square F Sig. Corrected Model 7.943E8 10 7.943E7 10.426 .000 Intercept 1.394E9 1 1.394E9 183.023 .000 Sector_Code 3.566E7 3 1.189E7 1.560 .200 GrantTypeCode 5.590E8 2 2.795E8 36.687 .000 Sector Code * GrantTypeCode 5.260E7 5 1.052E7 1.381 .232 Error 1.874E9 246 7618033.842 Total 4.228E9 257 Corrected Total 2.668E9 256 R-Squared = .298 (model explains about 30% of the variance) International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 52 MEPP II CSP BDP Study The overall picture of cost per employee by grant size and sector is shown in the table and chart below. We need to determine that the apparent differences are statistically significant. Again we turn to the GLM to assess the eight study design factors, this time presented as independent factors (rather than collectively as the interaction effect). Cost per Employee by Grant Size and Sector Cost per employee at time of grant Factors in the study: grant size and business sector combinations Mean Median Standard Error of Mean medium industrial/manufacturing 6460 4897 1256 small agricultural 2912 2458 310 small industrial 2673 2438 286 small trade 2666 2277 316 small service 2286 2080 229 micro industrial 894 748 82 micro trade 1123 900 166 micro service 952 783 94 Total 2464 1657 201 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 53 MEPP II CSP BDP Study The differences shown are partially statistically different as determined using the multiple comparison tests in the GLM procedure. Once again the medium size grants to the industrial/manufacturing sector stand out. In general the differences within grant size class are not statistically different from one another (e.g., small trade grants are not statistically different from small service grants), but the differences between grant size class are statistically different (as we have seen above). It can be said with confidence that it costs more to create one job with a medium grant than it does with either a small grant or a micro grant. Micro grants cost least of all. The immediate table below confirms that the factors in the model are significant. Of most interest to us is the FactorVariable a categorical variable that has one category for each study design factor. It confirms that we can reject the null hypothesis that there is no difference between factors, and that the factors are statistically significant. Tests of Between-Subjects Effects Source Type III Sum of Squares df Mean Square F Sig. Corrected Model 7.445E8 7 1.064E8 13.767 .000 Intercept 1.592E9 1 1.592E9 206.052 .000 FactorVariable 7.445E8 7 1.064E8 13.767 .000 Error 1.924E9 249 7725907.963 Total 4.228E9 257 Corrected Total 2.668E9 256 a. R Squared = .279 (Adjusted R Squared = .259) In the multiple comparisons table just below, the highlighted figures (also shown with an asterisk) are those that are statistically different from the reference factor shown in the left-most column. The first section of the table shows that the cost per employee for medium size grants to the industrial/manufacturing sector were on average $4174 more expensive than the cost per employee to small size grants to the service sector. Further down the table where micro grants appear in the left hand column you will see that there is no statistical difference between micro grants by sector. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 54 MEPP II CSP BDP Study Multiple Comparisons using the Tamhane Test (does not assume equal variance) (I) Factors in the study (J) Factors in the study Mean Difference (I-J) Std. Error Sig. small agricultural 3548.55 1293.508 .236 small industrial 3787.84 1287.971 .151 small trade 3794.46 1295.112 .154 small service 4173.93* 1276.596 .068 micro industrial 5566.38* 1258.574 .003 micro trade 5337.26* 1266.738 .005 medium industrial/manufacturing micro service 5508.30* 1259.361 .004 medium industrial/manufacturing -3548.55 1293.508 .236 small industrial 239.29 421.452 1.000 small trade 245.91 442.794 1.000 small service 625.39 385.292 .962 micro industrial 2017.83* 320.564 .000 micro trade 1788.72* 351.252 .000 small agricultural micro service 1959.75* 323.638 .000 medium industrial/manufacturing -3787.84 1287.971 .151 small agricultural -239.29 421.452 1.000 small trade 6.62 426.349 1.000 small service 386.10 366.274 1.000 micro industrial 1778.54* 297.436 .000 micro trade 1549.43* 330.280 .001 small industrial micro service 1720.46* 300.746 .000 medium industrial/manufacturing -3794.46 1295.112 .154 small agricultural -245.91 442.794 1.000 small industrial -6.62 426.349 1.000 small trade small service 379.47 390.643 1.000 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 55 MEPP II CSP BDP Study micro industrial 1771.92* 326.975 .000 micro trade 1542.80* 357.112 .002 micro service 1713.84* 329.989 .000 medium industrial/manufacturing -4173.93* 1276.596 .068 small agricultural -625.39 385.292 .962 small industrial -386.10 366.274 1.000 small trade -379.47 390.643 1.000 micro industrial 1392.44* 243.509 .000 micro trade 1163.33* 282.689 .004 small service micro service 1334.36* 247.541 .000 medium industrial/manufacturing -5566.38* 1258.574 .003 small agricultural -2017.83* 320.564 .000 small industrial -1778.54* 297.436 .000 small trade -1771.92* 326.975 .000 small service -1392.44* 243.509 .000 micro trade -229.11 184.998 .999 micro industrial micro service -58.08 124.851 1.000 medium industrial/manufacturing -5337.26* 1266.738 .005 small agricultural -1788.72* 351.252 .000 small industrial -1549.43* 330.280 .001 small trade -1542.80* 357.112 .002 small service -1163.33* 282.689 .004 micro industrial 229.11 184.998 .999 micro trade micro service 171.03 190.274 1.000 medium industrial/manufacturing -5508.30* 1259.361 .004 small agricultural -1959.75* 323.638 .000 small industrial -1720.46* 300.746 .000 micro service small trade -1713.84* 329.989 .000 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 56 MEPP II CSP BDP Study small service -1334.36* 247.541 .000 micro industrial 58.08 124.851 1.000 micro trade -171.03 190.274 1.000 *. The mean difference is significant at the .1 level. Cost of employment analyzed above is based entirely on the grant cost. It could be argued that this should be offset by grantee contribution, especially as we have seen that the micro trade sector contributes less than any of the others. Mean grantee contributions are shown graphically in the bar chart below. Grantee contribution leverages the grant value. To make this a valid hypothesis would require a deeper understanding of what is included in grantee contribution (wages paid can be part of it). However, to test this we can subtract grantee contribution from the grant value and recalculate cost per employee considering grantee contributions. The result showed that the adjusted cost of a job in the small trade sector increases. Small industrial grants become attractive options. Medium grants to the industrial/manufacturing sector remained expensive. Missing from the analysis of employment above are wages paid or earnings made by the created or existing employment. We do not know the value of the employment created except through a possible proxy measure of grantee contribution. Grantee contribution might be used in an attempt to assess the value of employment growth barring another survey done for that purpose. The mean value of grantee contributions hints at this characterization. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 57 MEPP II CSP BDP Study Job growth essentially reduces the cost of employment, or shifts the cost from the grant to the grantee (his contribution). This is an important consideration in the cost per employee. While the cost per employee for medium grants to industrial/manufacturing sector is the highest among the factors tested they are also the only grants that have created job growth over the period since the grant closed. We can now answer the remaining questions in the SOW: 1. At least 12 months after the grant was awarded, has the BDP accomplished the objective of facilitating job growth? Overall job growth in the period at least 12 months after the grant was awarded was limited to less than 2%. A small and disappointing number. However, it was possible to isolate where job growth has occurred. 1.4. Which grant size demonstrated the greatest number of new jobs created (estimate how many grant dollars were needed to create new jobs within the various grant windows)? Employment was first segmented into new jobs created by the grant award, existing jobs sustained by the grant award and job growth created by the grantee stimulated by the grant award. This question relates to new jobs created by the grant award. In total, most new jobs have been created through small grant awards. The highest average number of new jobs created per grant award was 11 and came from medium grants. The average cost per employee was highest for medium grants at an estimated $6180 per new job. Small grants averaged $2550 per new job, and micro grants $1380 per new job. Grant size and business sector combinations show that the cost per employee was highest for medium grants to the industrial/manufacturing sector, and was lowest for micro grants across all business sectors. 1.5. Which business sectors proved to be the most successful in creating new jobs? The trade sector produced the most number of new jobs (most grants were awarded in this sector), but the average number of new jobs created per grant was highest in the agricultural sector. 1.6. Were additional jobs created by the grantee 12 months after the initial grant? If so, how many? Which grant size and type experienced the most post-award job creation? Yes, new jobs were created by the grantees during the 12-month period after the initial grant but these were relatively few. The total weighted number of jobs created amounted (on balance gains minus losses) to less than 60. All new job growth came from business in existence before the grant award. No job growth came from micro grants, some came from small grants, but the majority came from medium grants to existing businesses in the industrial/manufacturing sector. On average, job growth per grantee was highest for medium size grants to existing businesses in the industrial/manufacturing sector. They added an estimated 4.3 jobs per grant award. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 58 MEPP II CSP BDP Study Annex VI. Glossary Alpha: Alpha is the criterion required to establish statistical significance. Assuming that the null is true, alpha is also the proportion of studies expected to result in a Type I error. Confidence interval: Interval that will include the population parameter in a known proportion of all possible studies. “All possible studies means” that if you repeated the study a number of times (drawing a new random sample each time) the outcome of those studies would be within the confidence interval 90% of the time (where 90% is the confidence level). Technically, this is 1-alpha. The confidence interval is the probability of correctly concluding that there is no treatment effect. We are confident that 90% of the time our estimate will be in interval defined by two times the standard error. Confidence level: A level of certainty for the confidence interval - the proportion of studies in which the confidence interval is expected to include the population parameter. In practice this is 1- alpha (see above). Effect Size: The magnitude of the effect - for example the standard difference in means (for a t￾test). Power analysis works with the effect size, which is independent of sample size. Significance tests combined the observed effect with the sample size to yield a combined test statistic. In this study effect size is the expected differences in business success between grant size or grant sector groups. Logistic regression: a type of regression where the dependent variable (the predicted variable) has a binary outcome (a successful business or an unsuccessful one). Normal regression is founded on linear relationships among the variables that include assumptions about the normal distribution of errors. The use of variables with binary outcomes in normal regression violates these expected assumptions. Statisticians have found that when binary variables are transformed using the logit (natural log of the odds) or logistic function the inclusion of binary outcome variables can be accommodated in regression equations. The name “logistic regression” refers to regression where these types of transformations have been used; more generally to regression that predicts a binary outcome. Null hypothesis: Power analysis focuses on the study's potential for rejecting the null hypothesis. In most cases the null hypothesis is the null hypothesis of no effect. For example, this study tests the null hypothesis that there is not difference in the business success of grantees due to grant size or grant sector. The study attempts to disprove (reject) the null hypothesis and thereby show that the alternative proposition that differences in business success is in part due to grant size or grant sector is acceptable with stated confidence. Odds: the likelihood of an occurrence relative to the likelihood of a nonoccurrence. In this study it is the likelihood of business success relative to the likelihood of business failure. Odds refer to a ratio of probabilities. Odds ratios, are referred to in logistic regression and in contingency table analysis, these are the comparison of odds rather than probabilities. Power of a study design: Power is the proportion of studies that will yield a statistically significant effect (assuming the effect size, sample size, and criterion alpha specified in the study design). When designing a study to test the null hypothesis the study design should ensure, to a high degree of certainty, that the study will be able to provide an adequate (i.e. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 59 MEPP II CSP BDP Study powerful) test of the null hypothesis. A rule of thumb is that the power of a study design should be 0.8 or more (assuming the effect size, sample size, and criterion alpha specified in the study design). P-value: In this study we are looking for p-values greater than 0.1 the estimate of the probability for a test of the hypothesis. Usually the p value is compared to the significance level when testing the hypothesis. If the p-value exceeds the designated significance level the alternative hypothesis is accepted; if it does not, the null hypothesis is accepted. Quasi-experimental design: Research designs that have several of the key features of randomized experimental designs, such as pre-post measurement and treatment-control group comparisons, but lack random assignment to a treatment group. The study design used here is quasi-experimental. Factorial design: One model for experimental design. In general a factorial design is used to look at a variety of program variations to see which works best. In factorial design groups are defined by the major variables of interest: is this case the main variables of interest are Business sector and grant size. The purpose of the design is to compare the main effects and the interactions effects of the factors. In order to make statistical inferences about the comparative differences between the factors we need to determine how large the sample should be in each of the groups. Random assignment: Process of assigning the sample into two or more subgroups by chance (businesses studied were not randomly assigned to the different economic sector and grant size groups this was a precondition not under the control of the study). Random selection: Process or procedures ensuring that the businesses selected for the sample in each factorial group were selected by chance. Sample design: Interviewing all the grant awardees to achieve study objectives is both expensive and not necessary. Selecting a random sample of subjects within each group can provide an estimate of the study parameters according to a desired precision and confidence level. The sample design calculates the necessary sample size that will achieve the desired precision and confidence level. Significance level: the notion that a statistical result could be considered significant if it could be shown that the probability of the result being due to chance was 5 % or less (or 1% or 10%). Statistical significance: The statistical significance of a result is the probability that the observed relationship (e.g., between variables) or a difference (e.g., between means) in a sample occurred by pure chance ("luck of the draw"), and that in the population from which the sample was drawn, no such relationship or differences exist. Using less technical terms, one could say that the statistical significance of a result tells us something about the degree to which the result is "true" (in the sense of being "representative of the population"). More technically, the value of the p-value represents a decreasing index of the reliability of a result. The higher the p-value, the less we can believe that the observed relation between variables in the sample is a reliable indicator of the relation between the respective variables in the population. (Source: Statsoft Electronic Textbook). Study domain: a study domain is a segment identified in the overall statistical plan as one for International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 60 MEPP II CSP BDP Study which a certain level of detail and certain data reliability are required. In this special BDP study the factorial group and the study domain are interchangeable. Eight study domains were identified for the BDP study. Type I Error: The error committed when the true effect is null but the study yields a significant p-value and leads the researcher (in error) to reject the null. With alpha set at .05, a type I error would be expected in 5% of trials in which the null is true. A type I error is sometimes referred to as a false positive. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 61 MEPP II CSP BDP Study Annex VII. Bibliography Menard, Scott (2002), Applied Logistic Regression Analysis. (Sage University Papers Series on Quantitative Applications in the Social Sciences, series 07-106). Thousand Oaks, CA: Sage. Mlodinow, Leonard (2008), The Drunkards’s Walk: How Randomness Rules Our Lives, Pantheon Books, New York Hosmer, D.W. and Lemeshow, S. (1989), Applied logistic regression, New York, Wiley. O’Connell, Anna A.(2006), Logistic Regression Models for Ordinal Response Variables. (Sage University Papers Series on Quantitative Applications in the Social Sciences, series 07-146). Thousand Oaks, CA: Sage. Pampel, Fred C.(2000), Logistic Regression: A Primer, (Sage University Papers Series on Quantitative Applications in the Social Sciences, series 07-132). Thousand Oaks, CA: Sage. Sackett, D. L., Straus, S. E., Richardson, W. S., Rosenberg, W., & Haynes, R. B. (2000). Evidence-based medicine. 2nd ed. Edinburgh: Churchill Livingstone. Trochim, William M. K. and Donnelly, James P. (2008), The Research Methods Knowledge Base, 3rd edition, Atomic Dog Publishing. Web-based document. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 62 MEPP II CSP BDP Study Annex VIII. SO Team Comments and IBTCI Responses 1. IBTCI draws the conclusion that the BDP component might have met the need to contain the insurgency, but that this type of programming isn't suited to long-term economic development. That may be true on the face of it, but without better explaining the goals of CSP, the reader may falsely believe that USAID was trying to achieve long-term development goals through BDP. This statement is made up front in the study and again in the conclusions section. We will review the wording in the conclusion sections 2. Although it wasn't a question we asked them to consider, if MSMEs aren't growing to create new employment, why not? This may be our fault for not asking the question in the SOW, but did IBTCI learn anything from this study that can help answer this question (even anecdotal)? Was not possible to determine from anecdotal responses. 3. When IRD accepts a grantee's application, they estimate the number of jobs that an in￾kind grant should generate. In analyzing the data, did IBTCI find that the actual jobs created actually match or exceed the original predictions? The data referred to are available in IRD databases, but it was not provided to us and we did not ask for it. It should be possible to do this analysis if desired. 4. IBTCI noted that some 15% of medium-sized businesses supported by CSP/BDP failed after one year. Several questions: 1) Do we know why they failed? 2) Do we know what happened to the equipment procured? 3) Is there any correlation between these failed businesses and participation in CSP/BDP's training component? We do not have interview information on businesses that failed. If the business had failed it was unusual that an interview was completed (they were not available). 5. All factors considered (cost-per-job; growth potential - or new jobs; and the higher business failure rate of medium-sized grants), which is IBTCI's recommendation? Perhaps IBTCI can provide a descending 'rank' of recommended grant types that CSP should pursue. That might make it too mechanical, e.g., if grant type rank = 3 go to grant size 2. Structure the program differently. Design two components to the BDP. Make one component formal sector business development, and the other informal sector opportunities. The first component is under the ‘surge’ where your objective is to find alternatives to insurgent activities and to offer people alternatives seen by them to have possible positive outcomes. Among these alternatives are the BDP start-up grants that set up unemployed individuals typically in small family businesses. These startups seem to be a successful gambit based on our survey data (where success is measured in the business still being in operation one-year past inception). International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 63 MEPP II CSP BDP Study This initial success enjoyed by the grantee could be underwritten by tracking the start-ups and coupling more successful ventures (a screening mechanism is implied) with business management training that sets them on a path to business growth (business registration, formal book keeping, how to access commercial credit, how to operate company bank accounts). Currently only about one-quarter of micro-grant awardees surveyed said they had received any training that was related to their business. The concept is something like micro-venture capital, or micro-grants to start-ups. The GOI under MOLSA has a grant and business development component (see EG II). MOLSA might have the political will to partner with CSP (useful as an exit strategy for CSP). Under MOLSA’s Social Safety Net (SSN) families that meet the means test for SSN income supplements are required to send their children to school, and the breadwinner to attend vocational education. It would not be out-of￾line to require BDP micro-grant startups to be held responsible for certain achievements as entrepreneurs. Up to now, a case can be made that CSP has bought employment with micro-grants. You don’t get a sense that these businesses are “going anywhere” and overall seem not to have been a source of employment growth. At least some of these might be kick-started as entrepreneurs. The second component focuses on the formal sector [medium and small grants to existing businesses]. Here you are working with established businesses that are looking for expansion capital. It seems logical that larger sized grants would be given to firms that have a documented history (although some may not be currently operating). These are grants to engender economic growth. This is a different objective (SO 8 not SO 7). According to our survey 50-60% of medium and small grantees did receive training that was related to their businesses, they are better educated and their businesses employ more people. These are formal or borderline formal businesses. Here the emphasis could be developing the business model and bringing these businesses to commercial credit and business services. Depending on the size of the business they can be directed to MFI or private banks where Tijara has established grant based loan facilities with private banks. Grants in this component would favor businesses in the industry/manufacturing sector where success in employment growth has been most apparent. Other similar comments: • The study was to measure job growth and determines whether that the business was still in business (i.e. successful). In the Conclusions, Page 4, begins with “…CSP might want to consider shifting the human (technical assistance) and financial resources and focus on medium sized grants.” Granted this study was to focus on job growth, but it was also designed to measure if a business was still successful after one year as a measure of program effectiveness. When you look at only job growth, it could be concluded that medium-sized manufacturing grants were minimally more successful (2% job growth). The 2% job growth is the overall rate. The job growth rate for medium sized grants was 45%; for medium-sized manufacturing grants it was 53%. This is explained in the data analysis annex. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 64 MEPP II CSP BDP Study • However, when you look at 99% success rates of micro still in business versus 85% for medium micro, micro grants win out. The difference is between formal and informal enterprises; informal enterprises are mixed up with family accounts and locations (and in a sense can’t fail) while formal enterprises are separate entities. Medium sized grant holders tend to better fit the image of a business. • Also, from a program management and funding perspective, medium grants are over four times more expensive, $6,180 per job for medium, and $1,380 per job for micro. CSP can employ 4.5 more men and women by funding micro grants, or 2.5 more people with small grants. Yes for accounting purposes that is correct, but it ignores differences between formal and informal business, and possibly the value added created by the different kinds of employment created. This is speculative, and would need more information from the firms, but will an additional job in manufacturing that costs $6,180 (and a high level of contribution) add more to revenues by increasing productivity than a job added as a tradesman? [The Living Conditions Survey of 2004 notes that median hourly wages in 2004 for Agriculture was 375, Construction, manufacturing 571, wholesale, retail 429, and personal services 500. See page 133 of the Analytical Report. Public sector jobs were more than twice as high] • The study did state on page 9 that “In total, most start-up jobs have been created through small grant awards.” Based on cost on a per job basis, micro are more effective. Based on number of jobs generated per grant, small grants produce more jobs. Given both of these are true, how can the study recommend movement to medium grants when they are more expensive and have less number of jobs produced per grant? This only refers to start-up jobs that showed no growth over the year. This may only be a technicality, but we were asked to answer different questions: 1) grantees still in business after one year; 2) number of new jobs created; and 3) Which grant size and type experienced the most post-award job creation? Small grants did best for question 2, while medium grants did best on question 3. • FSO is a little disappointed with one of IBTCI's 'near-term' recommendations on the bottom of pg. 12: They suggest USAID/FSO should "configure a grant strategy for increased job growth and sustainable new enterprises." What exactly does this mean? It's too bland and can be interpreted rather broadly. Per their finding that medium-sized businesses create the most new jobs (despite their higher failure rate), is this they're suggesting we do - and at the expense of our support to those grantees that would fall into the 'livelihoods' category? See the large text box above for a descriptive grant strategy. This section was rewritten in the report. • Page 1, Job Growth, in the study, IBTCI declared that the “sweet spot” was in the medium sized grants in the industrial/manufacturing sector where job grow was International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 65 MEPP II CSP BDP Study the highest. While this sector may have been the one that generated the most “new” jobs, the growth was only 2%. Does this relatively low figure suggest that there wasn’t enough growth to declare any “sweet spot?” The main body of the report glosses over detail found in the annex on data analysis. The 2% you refer to is the overall rate of job growth. Medium sized grants to industrial/manufacturing sector exhibited a 53% job growth rate (confidence interval not calculated, but is likely wide). Medium sized grants overall showed a 45% job growth rate (confidence interval not calculated, but is likely wide). 6. On page 11 under 'Business Sectors and Grant Sizes' - IBTCI states that "on a per grant basis, the industrial/manufacturing sector received the greatest average grant." Can this be simply explained by the sheer cost of manufacturing/industrial equipment simply costs more? Yes, this is likely the case. They also have the highest contribution and probably more assets than other types of business. 7. Also on page 11, under section VI; third bullet under 'Grantee Characteristics' - IBTCI notes that "the level of education is significant and probably has more influence than grant size or business sector in determining job growth." Can CSP/BDP potentially alleviate this challenge by beefing up the BDP (business) training component (i.e. quality and/or length of training)? More information is needed on why and how educational level affects job growth in BDP grants to answer this question. 8. On pg. 12, IBTCI makes a very good point but they are being too kind to FSO - perhaps because they don't want to offend the client. That point is USAID/FSO did not adequately link our BDP grants program with Tijara's micro-finance initiative. In short, we're guilty of stove-piping and we didn't capitalize on a real opportunity to help some of these fledgling businesses grow. This is frankly a valid criticism and they should state it a little more forcefully. This is an oversight that we still have time to fix. Mea culpa. OK. And there are other possibilities with MOLSA (see large text box above). Additional text on potential linkages with Tijara has been added to the report. 9. Top of pg. 13, IBTCI suggests that USAID should conduct a special study to see whether livelihood levels are improving for grantees. Well, maybe. We had actually considered this as part of the BDP study, but for a variety of reasons we decided to keep this analysis more discreet. As I recall, it was going to be rather difficult to measure and at this stage of CSP I don't think we have the time. Yes, livelihood levels are difficult to measure; there was some recent work done by COSIT that we are attempting to obtain that could give us a yardstick. We did ask an earnings question in the Votech questionnaire. It might be possible to use a simple retrospective question about earnings one-year ago and earnings now asking whether it International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 66 MEPP II CSP BDP Study has improved or not. Livelihood is also complicated by the near universal public food distribution to Iraqi families. A small-scale in-depth study could provide needed insight. 10. Also on pg. 13, IBTCI suggests that USAID might consider amending the grant criteria to require grantees to hire at-risk youth. This is an interesting thought, but what about also linking them to our apprenticeship component where appropriate (i.e. skills-specific industrial/manufacturing MSMEs)? OK too, but for now we are having difficulty locating the apprentices and don’t know whether their experience has been positive. IRD has previously reported to IBTCI that they have considered and/or implemented incentives and requirements that BDP grantees hire Votec and Apprenticeship graduates. IBTCI has no data on this part of the program however. 11. On the very last paragraph on pg. 13 under Conclusions, IBTCI suggests that adjusting the cost-per-job to include the grantees' contribution, there could be increased leverage in grant value. IBTCI should expand on this. Are there any interesting findings as a result? We don’t yet know enough about what makes up grantee contribution. It says in the literature that part of what is counted as contribution can be payment of wages. If so then the use of contribution against cost might help to ‘standardize’ cost. You may have noticed from the data analysis that grants in the trade sector attract low levels of contribution. Can contribution be seen as a kind of value added to the grant contribution? Also not all job creation should be thought of as being equal, some jobs created have more value added than others. It may be that contribution is a proxy for this. 12. Pg. 35 under 'grant values, grantee contributions' - It is unclear on the last conclusion point if the grant recipient contribution should be counted as reducing the cost per job to CSP. From the economic point it should be argued that that cost would be included making the small and medium grants cost per job even more expensive. For USAID, what matters is how much it cost the program to create a job. See text box above. 13. FSO would like to know how appropriate it is to throw out the non-response businesses from the analysis as IBTCI has done. Why wouldn’t these businesses be considered a failure? It seems that if these businesses were considered failures it would change the findings some. The non-response rate was 14% of the businesses surveyed. We made this decision before-hand and explained it in one of our methodology meetings with FSO. We attempted to separate those we couldn’t locate into two groups: 1) those who neighbors or contacts said had closed the business and moved on, and 2) those we had no information about. It is the 14% that are under the second classification: these include those where security prevented access; and a few incorrect addresses. 14. FSO would like to hear more about IBTCI’s comment that “there was a wide discrepancy between the survey findings and what is in the database” in reference to expansion International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 67 MEPP II CSP BDP Study grants and grants to start-up enterprises. Why is that? Can IBTCI say whether applicants were dishonest or IRD simply kept bad records? IRD apparently did not place emphasis on making this classification. There would not seem to be a reason for dishonesty. We can make it a recommendation that this distinction is made clear on the database. This is a management correction that IRD can make. 15. Numerous times they point to continue financing of business growth to create more jobs. This could be part of IRD/CSP’s ’09 work plan to tie into Tijara. IBTCI mentions the “manufacturing/industrial” sector, but are they in a position to identify which types of businesses CSP should target? In the IRD database there is a classification for manufacturing/industrial that includes specific businesses within it. These could be listed by IRD. IBTCI does not have enough information to make specific business type recommendations for this sector. 16. Page 10: “The number of existing businesses found in IBTCI’s analysis of the survey data (37%) far exceeded the number classified as expansion or existing businesses according to the CSP BDP database (5%).” Did IBTCI find evidence that these businesses existed BEFORE the “new” or start-up grant? If so (if the jobs existed before the grant), was there evidence that there weren’t any new jobs generated from the grant? Yes, the evidence is based on the data that were collected by asking whether there was employment before the grant. If there was we counted these as existing businesses. Yes, new jobs were not added to existing businesses (by definition). When the grant to an existing business was implemented new jobs were not added. However, there could be job growth between the grant implementation and the time of the survey. These could be compared with job growth expectations from the IRD data.