USAID COMMUNITY STABILIZATION PROGRAM (CSP) COUNTERINSURGENCY (COIN): VOCATIONAL EDUCATION AND APPRENTICESHIP COMPONENT May 31, 2009 This publication was produced for review by the United States Agency for International Development. It was prepared by Harvey Herr and Rich Mason. USAID COMMUNITY STABILIZATION PROGRAM (CSP) COUNTERINSURGENCY (COIN): VOCATIONAL EDUCATION AND APPRENTICESHIP SPECIAL STUDY COMPONENT International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Program, Phase II (MEPP II) 8614 Westwood Center Drive Suite 400 Vienna, VA 22182 Contracted under 267-C-00-05-00508-00 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) i MEPP II CSP Vocational Education and Apprenticeship Study Table of Contents LIST OF ACRONYMS...................................................................................................................................II EXECUTIVE SUMMARY ..............................................................................................................................3 I. INTRODUCTION...................................................................................................................................9 II. USAID’S COMMUNITY STABILIZATION PROGRAM........................................................................10 III. THE PROBLEM STATEMENT ............................................................................................................10 IV. THE DEVELOPMENT HYPOTHESIS.................................................................................................10 V. CSP PROJECT DESIGN.....................................................................................................................11 VI. PURPOSE OF THE VOCATIONAL EDUCATION AND APPRENTICESHIP SPECIAL STUDY.........11 VIII. RESEARCH DESIGN AND STUDY METHODOLOGY.......................................................................12 IX. FINDINGS ...........................................................................................................................................16 X. CONCLUSIONS..................................................................................................................................22 XI. RECOMMENDATIONS.......................................................................................................................23 XII. LESSONS LEARNED..........................................................................................................................24 ANNEX I. USAID/FSO RESPONSE MATRIX TO THE STUDY RECOMMENDATIONS .........................26 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) ii MEPP II CSP Vocational Education and Apprenticeship Study List of Acronyms AoR Area of Responsibility AOTR Agreement Officer Technical Representative BOQ Bill of Quantity CERP Commander’s Emergency Relief Program CFSVA Comprehensive Food Security and Vulnerability Analysis 2008 COIN Counter Insurgency COP Chief of Party COTR Contracting Officer’s Technical Representative CSO Civil Society Organization CSP Community Stabilization Program DAC District Advisory Council EOP End of Project or Program FSO Focused Stabilization Office FY Fiscal Year GOI Government of Iraq HQ Headquarters IBTCI International Business & Technical Consultants, Inc. IHSES Iraq Household Socio-economic Survey IR Intermediate Result LG Local Government LGP Local Governance Program LOP Life of Project or Program M&E Monitoring and Evaluation MEPP II Monitoring and Evaluation Performance Program, Phase II MOLSA Ministry of Labor and Social Affairs MOU Memorandum of Understanding MSME Micro, Small or Medium Enterprise NGO Non-Governmental Organization OJT On-the-job Training PMP Performance Management Plan PC Provincial Council PRT Provincial Reconstruction Team Q Quarter RF Results Framework RFA Request for Application RIG Regional Inspector General SME Small or Medium Enterprise SO Strategic Objective SOW Scope of Work SPSS SPSS predictive analytics software www.spss.com US United States USAID United States Agency for International Development USG United States Government USM United States Military VTC Vocational Training Center WB World Bank International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 3 MEPP II CSP Vocational Education and Apprenticeship Study EXECUTIVE SUMMARY The Community Stabilization Program (CSP) is USAID's largest cooperative agreement worldwide with a current funding level of $644 million during the life of this three-year project. The CSP is implemented by International Relief & Development (IRD) who was awarded the project in May 2006. Strategically, the CSP project is responsible for achieving the mission objective stated in USAID/Iraq’s mission strategy statement of November 2005 (“The USAID/Iraq Transition Strategic Plan 2006-2008”) and found in the Mission’s final results framework (RF) in the August 23, 2006 revision to the Mission Program Management Plan (PMP). CSP supports USAID/Iraq's Strategic Objective 7 (SO 7), Focused Stabilization: reduce the incentives for participation in violent conflict, and Intermediate Results (IR) 7.1 (Unemployment decreased with a focus on young men); 7.2 (Conflict mitigated through increased civil society organization and community activities); and 7.3 (Community infrastructure revitalized and essential services provided by local government). This study focuses on IR 7.1 and quantitatively measures the overall effectiveness of the vocational training program by determining the employment status of the over 14,919 Iraqis that graduated from CSP sponsored programs after December 31, 2007. However, the primary measure of success for CSP has been whether or not it has had a COIN impact through engaging as many participants as possible to get them off the streets in the short-term, thus reducing their vulnerability to be recruited as insurgents. This study looks beyond the short-term outputs of the COIN strategy to the longer term outcomes (results) of CSP’s training for work programs. Outcomes assessed in this study include employment status percentage, whether graduates are employed in their area of study; if they are earning a living wage; and if the apprenticeship program was instrumental in helping graduates obtain long term employment in their field of study. The USAID/Iraq Focused Stabilization Office (FSO) and the Program Office jointly developed a scope of work (SOW) under which IBTCI, the implementing contractor for the Monitoring and Evaluation Performance Program, Phase II (MEPP II), would conduct this special study answering the questions posed by the SOW. Following mission procedures, IBTCI was assigned this task by its Contracting Officer Technical Representative (COTR) in July 2008 after which deliberations began with the CSP and their AOTR. These deliberations sought to clarify any misunderstandings and to coordinate the field work. For reasons explained in the study three pretests of the methodology led to a final March 12, 2009 revision to the SOW. Questions posed in the SOW and answered by this special study are: 1. For graduates completing vocational training after January 1, 2008 and before December 31, 2008, what is the employment rate for these graduates? • What percent of graduates are currently employed in the area of their study? If so, how many months of employment since graduation? What percentage are employed, but outside their area of study? What percentage is under-employed (employed but in unskilled labor)? • If employed in their area of study, does the graduate earn a living wage? • Do you believe the training helped you earn a living? Did the training enhance your life or contribute in a meaningful way to earning a living? International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 4 MEPP II CSP Vocational Education and Apprenticeship Study • Does the graduate realize any income from the skills learned in the vocational training? How relevant was the training to the needed skills for area of study? Did the training contribute in a meaningful way to earning a living? 2. For apprenticeships completed after March 31, 2008 and before December 31, 2008, what percentage of apprenticeships have lead to employment? • Do apprenticeship graduates enjoy a higher employment rate when compared with other VTC graduates? • Did the apprenticeships improve their employability? Did the job after the apprenticeship result in earning a living wage? • Are the apprentices employed outside their area? • Is the apprentice still employed by the business that conducted the apprenticeship? • Did the apprenticeship result in earning a living wage? • Does the former apprentice believe the apprenticeship enhance their overall employability? • Did the apprentice receive the 50% matching salary provided by IRD? Did the apprentice receive a salary from the employer amounting to at least the 50% matching rate? To answer these questions random samples of apprentices and vocational education graduates were drawn from separate datasets extracted from individual databases operated by IRD’s CSP regional offices. The study target population is defined by the dates of interest noted in the questions above. The dataset for vocational education graduates contains only those individuals who completed vocational training after January 1, 2008 that did not go on to do an apprenticeship. The dataset of apprentices contains individuals who entered an apprenticeship after March 31, 2008. These two datasets are mutually exclusive and define the population of interest for this study. Sample size was calculated with consideration for the proposed method of analysis, and estimated non-response. Non-response estimates were informed by three pretests conducted from late September 2008 until February 2009. Very high levels of non-response found during the first two pretests resulted in an investigation into its causes and a re-design of the study. The third pretest using revised retrospective dates obtained satisfactory response rates for apprentices and marginally acceptable response rates for VTC graduates. Based on a third pretest a full study was fielded on March 25, 2009. The study sampled 400 respondents comprised of 150 apprentices and 250 vocational education graduates. The higher number of sampled vocational education graduates was a precaution against anticipated low VTC graduate response rates. IBTCI’s sub-contractor fielded the sample survey concluding the fieldwork on 26 April. Final survey response exceeded expectations: apprentice response was 75% while vocational education graduate response was 82%. concluded data entry on 28 April. Survey data were merged with CSP dataset information prior to the analysis. This allowed survey data to be compared with CSP administrative data. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 5 MEPP II CSP Vocational Education and Apprenticeship Study Analysis of the data was done by IBTCI using SPSS 17.0 and Excel 2007. IBTCI’s data analysis included logistic regression to highlight statistical differences in work experience between apprentices and vocational education graduates. Some follow-up work is continuing to verify the payment of apprentice stipends. Findings VTC graduates. • Forty-six percent of VTC graduates said that they were earning money for work that they did in the past 7 days. • On average employed VTC graduates worked for 8 months since the end of their training. • Eighty-six percent of employed VTC graduates work in their area of study. • Fourteen percent of employed VTC graduates were working outside their area of study. • Underemployment due to work as unskilled labor was not indicated by the survey. Underemployment due to reduced hours of employment did occur. Thirty percent of employed VTC graduates worked 30 hours or less during the seven days that preceded the survey. • There was a strong belief expressed that training helped graduates earn a living wage. This was expressed even by those who were not employed. • Training enhanced the VTC graduates life and contributed in a meaningful way to earning a living. This is demonstrated by the high rate of employment in their area of training. • VTC graduates did realize income from the skills they learned. Caveats are that some of them were already working in the area of their skill before they joined the VTC. • Virtually all respondents (95%) said that the training was worthwhile and provided them with the useful skills. • The study found that VTC graduates did receive the stipend amounts due to them. Apprenticeships • Twenty-nine per cent of apprentices said that they were earning money for work that they did in the past 7 days. • Job duration for apprentices is shorter than for VTC graduates: 6.8 months compared to 8.2 for VTC graduates. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 6 MEPP II CSP Vocational Education and Apprenticeship Study • Apprentices have a lower employment rate than that for VTC graduates. This finding is contrary to expectations. This can partially be due to the longer average period that VTC graduates have been in the job market (10 months) than have Apprentices (7 months on average). • There is no evidence to support the proposition that apprentices are more employable than VTC graduates. • An estimated 70% of employed apprentices earned a “living wage.” Thirty percent fell below the 70,000 ID per capita per month “living wage” threshold. • The earning power of apprentices on average surpassed that of the VTC graduates. • Seventeen per cent of apprentices work outside their vocational specialty. No standard or benchmark has been established to assess if this is acceptable. • Half of the employed apprentices still work for the business that hosted them as an apprentice. • Despite the reality that most apprentices are not employed, 94% believed that the apprenticeship improved their overall prospects for employment. • Did apprentices receive the 50% matching salary provided by IRD as well as the 50% matching amount provided by the host business? Thirty percent of apprentices apparently did not receive their full stipend amount. Conclusions This study has been difficult to implement. Three false starts resulted when sample design pretests failed to locate VTC graduates and apprentices. Clearly this was indicative of an underlying monitoring problem. The problem was recognized by IRD and remedial steps taken, at least in Baghdad. However, the remedy altered how the study was implemented and possibly the study result. Instead of reaching back one year or more to study VTC graduates and apprentices as intended, the study out of necessity looked at more recent examples that had limited employment histories. This may have lead to an underestimate of apprentice employment. The study measured quantitatively the effectiveness of VTC training and apprenticeships in achieving employment. The study found that the overall employment rate was 40% leaving 60% unemployed. A recent nation-wide survey reports unemployment rates, for the same age groups of interest here, to be less than 25%.1 Gauging the success of the program against this benchmark tells us that the results of the program are not encouraging. 1 The “Iraq Household Socio-Economic Survey” IHSES- 2007, Tabulation Report, Central Organization for Statistics and Information Technology (COSIT), Kurdistan Region Statistics Organization, and the World Bank, page 289. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 7 MEPP II CSP Vocational Education and Apprenticeship Study By interviewing only recent program participants; some only a few months after completing training or the apprenticeships a possibility exists that recent apprentices have had insufficient time to find employment (30% completed their apprenticeship 4 months ago or less). Even considering these adjustments the employment rates attributed to VTC training and apprenticeships are not impressive. Eighty-six percent of those who are employed (i.e., 40% of the total respondents) are putting the skills they learned to use; it is unfortunate that more are not employed. We do not have a gauge for assessing whether this level of use of trained skills is acceptable. To expect a higher percentage suggests that the VTC is able to perfectly match training to the demand for skills. A majority of those who are employed say they are self-employed. This includes jobbers, running a small shop or business. From studies of the CSP business development program it is highly probable that these are small family run businesses where accounts are mixed with family accounts (informal businesses). Supporting these small businesses with a grant, or with business skills training was envisioned by the CSP, but was not tested here. Participants said they were satisfied with the courses and their apprenticeships. Since they are paid to attend and participate it is unlikely to find many dissenters. Average earnings made by VTC graduates are slightly below national averages (IHSES), or at least not above them. Calculated per capita incomes linked to the “living wage” benchmark show that many VTC graduates are struggling to keep enough food on the table. For those who are not employed the majority live from family contributions (as dependents of the larger family group). Recommendations 1. CSP’s management experience with the apprenticeship program in Baghdad reinforces the need for early deployment of monitoring and evaluation and Quality Assurance/Quality Control (QAQC) staff in identifying and validating program participants. There can be little doubt that initial beneficiary selection and oversight procedures were insufficient to prevent abuse as the tracking system was not robust enough to locate program participants to validate whether or not they had actually received payment or even existed. IRD’s management remedy closed these loopholes as evidenced by MEPP II’s ability to locate all of the sampled Baghdad participants in the final survey. That final management practice needs to be propagated to other vocational education and apprentice programs. 2. MEPP II experienced difficulty in working with the data provided to them by CSP. This difficultly stems from the independence of the units operating in the field. The M&E units collected information on program participants using different formats. The M&E database structure differs from one location to another: e.g. Kirkuk is different than Mosul which is different than Baghdad. 3. The CSP Rapid Programmatic Assessment Report, November 2007 by IBTCI recommended that CSP introduce an element of sustainability by targeting training-to￾work graduates with BDP grants. The large percentage of employment coming from self-employment show how relevant this is. It was not apparent from this study whether International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 8 MEPP II CSP Vocational Education and Apprenticeship Study grants were made available to VTC graduates and apprentices.2 4. Review the inclusion of women in the VTC program. Women, based on economic activity rates, are over-represented in the training. All but seven of the 65 women interviewed did vocational education in sewing. Completion of the sewing course is awarded with a sewing machine. This is an attractive opportunity for any housewife. Is the program certain that these participants pose a threat of being caught up in the insurgency? 5. Set an employment rate benchmark for the vocational education program by using official government statistics as the guideline. The program should be able to meet or exceed known rates. 6. Investigate why women, especially in Baghdad, are apparently not receiving their full stipend amounts. Lessons Learned 1. CSP oversight of the vocational education graduate and apprentice databases came late in the project cycle. When it did happen and the apprenticeship program was suspended the result was telling. Engage M&E and QAQC early in the program implementation to ensure proper program beneficiary selection. 2. Lack of database uniformity across program implementing units causes problems for overall program management. 3. Necessary revisions to the scope of work significantly reoriented the time frame of the analysis. This may have affected estimates of employment. 4. The VTC program did not set a benchmark for employment rates. 2 IBTCI (2007), Community Stabilization Program Rapid Programmatic Assessment, November 2007, page 11, “The BDP program introduces an element of sustainability into the CSP. The BDP is intended to support grantees that have been identified through other, short-term activities. It was assumed that many of the training-to-work graduates would be likely candidates for BDP grants. Program documentation does not reveal that graduates become BDP grantees. As with job placement this is a logical conclusion for some graduates of the training-for-work program.” International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 9 MEPP II CSP Vocational Education and Apprenticeship Study MEPP II CSP VOCATIONAL EDUCATION AND APPRENTICESHIP SPECIAL STUDY3 I. INTRODUCTION The United States is committed to the future success of Iraq. Within USAID/Iraq’s Transition Strategic Plan 2006-2008, the first of four strategies delineated is “Focused Stabilization: Reduce the incentives for participation in violent conflict.” To help plan and manage the process of assessing and reporting progress towards achieving its strategic objectives, USAID/Iraq (hereinafter the “Mission”) made final its Performance Management Plan (PMP) in August 2006. In the PMP document, consistent with earlier Mission objectives, the strategy to reduce the incentives for participation in violent conflict is identified as Strategic Objective 7 (SO 7). Under SO 7, the NGO International Relief and Development (IRD) was contracted to implement the Mission’s focused stabilization strategy under the Community Stabilization Program (CSP). CSP contributes to the achievement of results under the Mission’s counter-insurgency (COIN) goals through USAID/Iraq’s Strategic Objective 7 of the Mission’s strategic framework. The program has four broad components: 1) short-term employment through the Community Infrastructure and Essential Services; 2) training and employment placement through Vocational Training and Apprenticeship; 3) sustainable job creation through the Business Development Program (BDP); and 4) youth engagement on life skills and conflict mitigation through sports, cultural events and public service campaigns. Through this study, the Focused Stabilization Office (FSO) seeks to measure the results of CSP’s Vocational Training and Apprenticeship component and establish its success in supporting the Mission’s COIN goals through Strategic Objective 7. The agreement officer technical representative (AOTR) for the CSP in USAID/Iraq’s FSO determined that a special study was needed “…to quantitatively measure the overall effectiveness of the vocational training program by determining the employment status of the over 14,919 Iraqis that graduated from CSP sponsored programs after December 31, 2007. However, the primary measure of success for CSP has been whether or not it has had a COIN impact through engaging as many participants as possible to get them off the streets in the short-term, thus reducing their vulnerability to be recruited as insurgents. This study looks beyond the short-term outputs of the COIN strategy to the longer term outcomes (results) of CSP’s training for work programs. Outcomes assessed in this study include employment status percentage, whether graduates are employed in their area of study; if they are earning a living wage; and if the apprenticeship program was instrumental in helping graduates obtain long term employment in their field of study. The CSP The project has begun to wind down as it approaches its September 30, 2009 end date and Mission strategy shifts from COIN programs to more traditional development programs. International Business & Technical Consultants, Inc. (IBTCI) implements the USAID funded Monitoring and Evaluation Performance Program, Phase II (MEPP II). IBTCI receives instructions through its COTR to conduct monitoring and evaluation of partner programs, review partner PMPs or to conduct special studies such as this one. Under MEPP II, IBTCI was asked 3 Names of some organizations have been removed for security reasons. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 10 MEPP II CSP Vocational Education and Apprenticeship Study to undertake this study on behalf of the FSO. IBTCI is committed to providing evidence-based findings that are useful for gauging program performance and management decision making. Under MEPP II, IBTCI is tasked to provide field monitors to assist USAID to monitor projects it cannot otherwise reach. MEPP II has a long-standing relationship with a local partner to provide field monitoring services. The provides field monitors who respond to needs as they arise across Iraq. Field work referred to in this study has been implemented by under the guidance of IBTCI through a purchase order agreement. II. USAID’S COMMUNITY STABILIZATION PROGRAM CSP is USAID's largest cooperative agreement worldwide with overall funding of $644 million during the life of this three-year project. The CSP is implemented by International Relief & Development (IRD) who was awarded the project in May 2006. Since May 2006, USAID has been implementing CSP, a multi-faceted program designed to reduce the incentives for participation in violent conflict by developing and implementing activities that support the social and economic stabilization of Iraqi communities. The program is a non-traditional program for USAID in the sense that rather than focusing on long-term developmental impact, the program focused on short-term results in support of the broader United States Government Strategy in Iraq. The program was targeted at engaging unemployed youths who were the most vulnerable to overtures from violent elements of Iraqi society. Initial funding under the CSP award limited activities to areas of significant insurgent activity in Baghdad. Later, the AoRs for CSP were expanded to cover similar areas in Kirkuk, Mosul/Tel Afar, Ramadi, Falluja, North Babil, Tikrit, Beiji and Diyala. III. THE PROBLEM STATEMENT Unemployed youth are recruited to the insurgency because they do not have other opportunities for earning their livelihood. IV. THE DEVELOPMENT HYPOTHESIS The Iraq Transition Strategy Statement of November 2005 notes that “USG objectives cannot be accomplished through security interventions alone. The USAID strategy provides a focused approach for addressing the non-security issues of governance and market led growth. It is important to increase resources for these two vital areas which, in the short-term, will stabilize areas impacted by the insurgency and mitigate the appeal of insurgent recruitment efforts...” Annex II provides the full development hypothesis for SO 7 as it appears in the Mission Strategy statement. In brief the concept is to focus on communities severely affected by insurgent action by “stabilizing” them. The Transition Strategy Statement goes on to imply that specific activities undertaken will achieve the desired stability leading up to a transition to more traditional development programs. “USAID will focus on employment generation, infrastructure rehabilitation, youth programs, assistance to municipal governments and conflict mitigation” in the cities targeted as being in the strategic USG interest. It is presumed that following the achievement of stability, these “cities will be integrated into USAID’s longer-term development initiatives in health and education, agriculture, micro-credit and building the capacity of International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 11 MEPP II CSP Vocational Education and Apprenticeship Study communities and civil society organizations for advocacy, and the capacity of local government to provide basic services.” V. CSP PROJECT DESIGN The CSP was implemented as a key element to transition Iraq into a stable, democratic and prosperous state. As defined in the RFA the purpose of CSP is to complement military security efforts, and civilian local government development, with economic and social stabilization efforts. The design of the CSP project includes: 1) creation of jobs and development of employable skills with a focus on unemployed youth; 2) revitalization of community infrastructure and essential services; 3) support for established businesses and development of new sustainable businesses; and 4) help to mitigate conflict in selected communities. By carrying out these activities the CSP was expected to achieve measurable progress towards achieving the strategic objective. The CSP PMP provided performance indicators approved by the AOTR that were to provide evidence that strategic objectives were being achieved. The indicators identified and approved for the PMP (# of participants completing vocational skills training and # of apprentices placed) were inadequate to gauge whether training and apprenticeships actually resulted in sought after employment, hence the need for this special study. VI. PURPOSE OF THE VOCATIONAL EDUCATION AND APPRENTICESHIP SPECIAL STUDY This special study is directed at the “the creation of jobs and development of employable skills with a focus on unemployed youth” made available through the vocational technical education program and related apprenticeships. This aspect of the program is abbreviated as Votech. In conjunction with the Ministry of Labor and Social Affairs (MOLSA) the CSP recruited unemployed youth within their area of responsibility (AoR) to join the Votech program. Once Votech graduation was complete some of those trained qualified for an apprenticeship program in the field of their training. The CSP program funded the Votech training and half of the cost of the apprenticeship. This special study seeks to validate whether employment was achieved and whether employment was achieved in the area of study. Annex III is a full description of the Vocational Education and Apprenticeship Program as it appears in the latest agreement modification from September, 2008. Specific questions posed in the SOW (provided in full in Annex I) are listed below: 1. For graduates completing vocational training after January 1, 2008 and before December 31, 2008, what is the employment rate for these graduates? • What percent of graduates are currently employed in the area of their study? If so, how many months of employment since graduation? What percentage are employed, but outside their area of study? What percentage is under-employed (employed but in unskilled labor)? • If employed in their area of study, does the graduate earn a living wage? • Do you believe the training helped you earn a living? Did the training enhance your life or contribute in a meaningful way to earning a living? International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 12 MEPP II CSP Vocational Education and Apprenticeship Study • Does the graduate realize any income from the skills learned in the vocational training? How relevant was the training to the needed skills for area of study? Did the training contribute in a meaningful way to earning a living? 2. For apprenticeships completed after March 31, 2008 and before December 31, 2008, what percentage of apprenticeships have lead to employment? • Do apprenticeship graduates enjoy a higher employment rate when compared with other VTC graduates? • Did the apprenticeships improve their employability? Did the job after the apprenticeship result in earning a living wage? • Are the apprentices employed outside their area? • Is the apprentice still employed by the business that conducted the apprenticeship? • Did the apprenticeship result in earning a living wage? • Does the former apprentice believe the apprenticeship enhance their overall employability? • Did the apprentice receive the 50% matching salary provided by IRD? Did the apprentice receive a salary from the employer amounting to at least the 50% matching rate? VIII. RESEARCH DESIGN AND STUDY METHODOLOGY The special study research design ensures within the limits of time and resources that the questions posed above have a high probability of being answered with a statistically valid response. Providing valid evidence for decision making entails 1) understanding how the questions can be answered (what information are we looking for), 2) finding the best sources of evidence to answer the questions, and 3) appraising the quality and validity of the evidence, while proposing a statistical model that provides the best chance of achieving answers that are statistically significant. Sources of information to answer the questions. The IRD CSP program keeps database(s) of vocational education graduates and apprentices.4 Answering the SOW questions implies either using existing data on program participants, or interviewing participants to obtain what’s needed. The study uses both existing data and survey data. The IRD CSP databases are at the core of the study and are used for their content as well as being the basis for selecting a sample of participants that will be interviewed. How the SOW questions are answered. 4 Each CSP region keeps a slightly different version of these data making it difficult to create a single national database. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 13 MEPP II CSP Vocational Education and Apprenticeship Study Apprentices are to be compared with vocational education graduates who did not go on to do an apprenticeship. Employment rate comparisons between the two are estimated. Within these two groups (apprentices and vocational education graduates) we propose to look at whether they are employed using the skills learned, their sources of earnings, and if they earn a living wage. A simple logistic regression equation was formulated to predict employment (the dependent variable). The independent variable identified the respondent as a VTC graduate or an apprentice. Standard statistical tests (the Wald statistic) demonstrated that employment rates for VTC graduates and Apprentices are statistically different. Overall employment for VTC and Apprentice program beneficiaries can be compared to known national employment rates for youth of a similar age taken from the Iraq Household Socio￾Economic Survey 2007. Ensuring the quality of evidence provided. For statistical precision and accuracy the study design defines a “control group” and a “treatment group” drawn from the homogeneous population of all youth sponsored under CSP who completed a vocational education training program.5 Preliminary screening from the IRD databases selected vocational education graduates and apprenticeships initiated within the reference period of interest (after March 31, 2008 and before December 31 for apprentices, and after January 1, 2008 and before December 31, 2008 for vocational education graduates). The selected apprentices and vocational education graduates were placed in separate data files that became the sample frames from which group members were selected at random.6 Before random samples were drawn from these files additional preparation of the sample frames was needed. The same person was likely to occur in the vocational education graduate file and the apprentice file (vocational education graduates went on to do apprenticeships). To achieve the intended study result vocational education graduates who then went on to become apprentices had to be removed from the vocational education graduate file. The two files were merged and then matched on name/address/telephone with the matching vocational education graduates then filtered out. After the merge and match only vocational education graduates who did not go on to an apprenticeship remained.7 The “treatment group” is drawn at random from the data file of apprentices (the apprenticeship is the “treatment”). The “control group” is drawn at random from those who did not do an apprenticeship. Pretests A questionnaire was designed and pre-tested. Three pretests of the questionnaire and the field monitor procedures were completed before the full sample could be implemented. Initial pretests found extremely low response rates for both vocational education graduates and apprentices. The rates were so low that a second pretest based on a new sample followed, but with a similar result. It was puzzling and discouraging that respondents could not be located. A meeting convened to review our understanding of the program and the database used to select the sample. Discussions unearthed that the Baghdad apprenticeship program had been 5 A very small number of apprentices did apparently not complete vocational training under CSP. 6 These are separate Excel files that were converted to SPSS data files. 7 This is in fact a very elaborate matching and merging exercise simplified here for brevity. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 14 MEPP II CSP Vocational Education and Apprenticeship Study suspended from October to December 2007 (the period of interest in the initial scope of work) and this was thought to be the source of the problem8 . The CSP management decision to suspend had identified weak vetting procedures for apprentice selection and for the businesses that hosted them (Annex VII reports on why the apprentice program was suspended). During the period of program suspension contact information in the database had been updated and corrected. Unverifiable apprentices and businesses were removed from the program and the database. Pre-suspension date data are of undetermined quality and were apparently the source of high non-response in Baghdad.9 The third pretest yielded dramatically better response rates for apprentices.10 Supported by the 3rd round pretest results and discussions with IRD and FSO, the SOW target population was revised to include only post-December 31, 2007 individuals. Based on these findings a full study was fielded in March 2009. Revising the survey instrument. Achieving the correct wording (in Arabic) for some of the questions was challenging. One question asked whether the respondent had been an apprentice. “Apprentice” needed a precise Arabic equivalent that referred clearly to the CSP program. Early attempts missed the mark. Identifying employment is crucial to this study. The pretests showed that employment rates were very low. For this reason, attention was given to ensuring that all forms of “earning a living” were captured, not just wages from formal employment. The concept of employment, while seemingly a basic concept, can be confusing to respondents. For some, employment meant that they had a wage earning job (usually with the government). Employment in this study includes self￾employment and home-based enterprises. To avoid overstating unemployment a series of questions are asked that start with how respondents support themselves and earn their livelihood leading to questions that elaborate respondent’s work history. Annex V shows the final survey instrument. Determining the sample size needed. The study objective is to determine whether or not there is a difference in employment rates between the two groups. In statistical language we are to accept or reject the null hypothesis that there is no difference in employment rates between the two groups. Standard statistical tests (analysis of variance and logistic regression) provide us with the tools to do this. How many participants do we need to interview before we can say conclusively that there is a difference in employment between the two groups? A by-product of the statistical test lets us determine the sample size needed to successfully measure differences between the groups provided we know something ahead of time about the two groups. Pretests provided the estimates of employment rates used to inform the calculation of sample size. Sample size was calculated with consideration for the proposed method of analysis (logistic regression), sample 8 The original SOW for this study set the target population for enquiry as vocational education graduates before September 1, 2007 and apprentices before March 1, 2008. In practice, and based on information from the CSP databases, we could not implement a viable sample of individuals in that time frame. 9 Confirmed by the 3rd pretest results. 10 Response rate for apprentices was 87% while for vocational education graduates it was 57%. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 15 MEPP II CSP Vocational Education and Apprenticeship Study power, the desired precision of the estimates, and estimated non-response. A discussion of this is in Annex IV. The overall sample size for this study is 400. Samples were selected proportional to city size in the sample frame. Two-hundred and fifty vocational education graduates and one-hundred and fifty apprentices were randomly chosen from their separate sample frames. A larger number of vocational education graduates were selected to compensate for their predictably higher non￾response rate (based on pretest information). Table 1 shows distribution of the sample by City and category. Sample size by City and Category of Respondent. City Vocational education graduates Apprentices Total Al Qaim 15 1 16 Baghdad 122 21 143 Basrah 10 15 25 Falluja 5 6 11 Hilla 19 20 39 Mosul 39 13 52 Tal Afar 6 16 22 Kirkuk 34 58 92 Total 250 150 400 Implementation of the Survey field monitors were provided with a list of names of vocational education graduates and apprentices to be interviewed. The list was separated by city. In each city one or more field monitors under the guidance of a supervisor attempted to locate each individual on their list. In many cases the list included a physical address and mobile telephone number of the person to be interviewed. This was not uniformly the case and in some areas (Falluja, Tal Afar, and Hilla) additional research was needed to determine how to make contact with the interviewee (there were lower response rates in these areas). Security was a concern for both those being interviewed and the monitor; precautions were taken. and its monitors gained valuable experience from the three pretests and were well prepared for the full survey. MEPP II worked with to clarify any remaining uncertainties in the questionnaire or field procedures before launching the survey on March 25. MEPP II asked monitors to revisit a respondent when uncertainties arose about certain responses. Collection of valid data calls for a rigorous implementation of the field survey and continuous oversight. The main steps followed in the process of achieving this sample survey are listed in Annex VI. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 16 MEPP II CSP Vocational Education and Apprenticeship Study Response rates for the final survey were consistent or better than what was predicted from the pretests.11 One hundred thirteen of the 150 apprentices responded (75%), while two hundred and six vocational education graduates responded (82%). Field work stopped on 26 April. Data entry was completed on 28 April followed immediately by data validation. Validated data from the survey were merged with database information on the participants. Detailed analytical findings are consigned to Annex VIII with the results of the data analyses reported in the findings section below. IX. FINDINGS Findings are partitioned into four sections: preliminaries, low response rates, vocational education graduates and apprenticeships. Comparisons between apprentices and vocational education graduates appear in the apprenticeship section. Preliminaries: “earning a living wage.” Several of the questions in the scope of work refer to “earning a living wage.” Since this is not defined in the scope of work a definition is provided here. A recent (2007-2008) survey on income and expenditure (IHSES) and another on food security (CFSVA) in Iraq make available data that can be used to form a definition.12 The IHSES calculated per capita income distributions nationally and by governorate. The CFSVA developed a wealth index that it correlated to food shortages. Notably, the CFSVA reported that for households in the lowest wealth index quintile, 56% experienced food shortages during the 30 days preceding their survey (see page 39 of the CFSVA). The wealth index is shown to be strongly correlated to income. Our own survey asks respondents for their earnings last month. These earnings figures are used to place the respondent in an IHSES income distribution group. Those with earnings in the IHSES two lowest income groups (approximately the lowest income quintile) are identified for this study as not earning a living wage (i.e., two lowest income groups estimate the lowest wealth index quintile). Essentially, it is likely that those in the lowest income groups will experience food shortages. The lowest quintile of per capital income has an upper limit of 70,000 ID per capita per month and this was set as the “living wage” benchmark. Respondents with reported per capita income below 70,000 ID were classified as not earning a “living wage.” Annex VIII has a more comprehensive explanation. Low response rates An important finding is the extremely low survey response rates during the first two pretests. In the early rounds of the pretests few of the randomly selected program participants were found. The unlikely chance of finding just 13 out of 40 apprentices during the first pretest, and only 8 11 Sample power was above .90 at this response rate 12 The “Iraq Household Socio-Economic Survey” IHSES- 2007, Tabulation Report, Central Organization for Statistics and Information Technology (COSIT), Kurdistan Region Statistics Organization, and the World Bank; and the “Comprehensive Food Security and Vulnerability Analysis in Iraq”, 2008, Central Organization for Statistics and Information Technology (COSIT), Kurdistan Region Statistics Organization, Ministry of Planning Development Cooperation, Nutrition Research Institute, Ministry of Health, Iraq and the United Nations World Food Program. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 17 MEPP II CSP Vocational Education and Apprenticeship Study out of 40 on the second pretest when these participants were selected from the IRD database portends a more serious problem. monitors worked with both CSP staff and with MOLSA VTC officials to locate participants during these pretests, but this did not improve the result. It should not be difficult to locate apprentices since they are paid by CSP. This is an uncomfortable finding. What does it mean? It means that the CSP program would not have been able to locate participants had they wanted to cross check signatures on timesheets and payment records. Incomplete contact information in the database limited our monitor’s ability to find the respondents; as it would also have done for CSP’s QAQC staff. The problem was acknowledged by Baghdad CSP when they suspended the apprentice program (see Annex VII) and vetted the participants and their employers. A pretest done on apprentices randomly selected from this vetted database located all of the Baghdad participants instead of finding just 20-30% we found 100%. It is more than a data quality issue. Incomplete addresses and telephone numbers on the database were initially ascribed to security concerns, but it can also mask more onerous possibilities. It doesn’t sit well with the reality that apprentices and vocational education graduates were paid by CSP. While this problem was rectified in Baghdad, we did not find that a similar vetting of apprentices was carried out in other regional offices, and response rates in some areas remained low. Characteristics of Apprentices and VTC graduates Gender: One-quarter of the surveyed participants were women. Fewer women than men went on to become apprentices. Age: All survey respondents were between 16 and 35 years old, the target group for the program. The median age was 24 meaning that one-half were less than 24 years old. Persons dependent on the VTC graduate or apprentice: The average household size for VTC graduate and apprentice households was eight persons. The number of household members reportedly dependent on the VTC graduate or apprentice averaged 5 persons. The number is less than the household size because many are single and unmarried while living in a larger household where they are not responsible for others. Marital status: Sixty percent of survey respondents were single having never married. Education level: Survey respondents were not well educated; more than 60% had not completed secondary school. Identifying employment International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 18 MEPP II CSP Vocational Education and Apprenticeship Study Current employment status is the focus of this study. Precautions were taken in the design of the survey instrument to crosscheck employment status. This was done by first asking a generic question about the ways respondents supported themselves and their families since their VTC graduation. Question 1, Part III of the survey instrument asks for categorical responses aimed at covering the range of possible ways of supporting yourself. The response categories divide respondents into those working to make money and those that aren’t. A confirmatory question asked respondents who said they were not doing any kind of work to make money whether they were certain about this. If they did confirm this they were counted as having never been employed. Using this definition 40% of those surveyed were working to make money. Employment rates are affected by the age and gender of the respondent. Economic activity rates for men and women vary significantly in Iraq. Women are much less likely to be in the labor force than are men. When men and women mature labor force participation increases. In the survey older men’s employment rate rose to 70%, for younger men it remained below 35%. Younger men may decide to return to school, or to remain in a dependency status while they decide what to do next. In Iraq society women are less likely to be in the labor force and this shows up in lower survey based employment rates for women. Reported employment came mainly through own-account self-employment (80% - 90%). Few found wage employment in a business or in the public sector. Findings for VTC graduates A. Employment rate for vocational education graduates. The employment rate for VTC graduates who did not go on to become apprentices is reported here. Contrary to expectations based on pretests VTC graduates were more likely to be employed than were apprentices. Forty-six percent of VTC graduates said that they were earning money for work that they did in the past 7 days. B. Months of employment since graduation. Employed VTC graduates worked an average of eight months since the end of their training. For VTC graduates the end of training was after January 1, 2008 through December 31, 2008. The data indicated that some VTC graduates may have already been employed before they entered the training. For example, a self-employed plumber decided to take a vocational education course to improve skills in a trade he already knows something about. His self￾employment could precede his entry into training. This occurred for a few of the respondents. Under Part II, Question 8, it was not uncommon for participants to say that they had improved skills in a field they already knew something about. C. Percent of graduates currently employed in the area of their study. Eighty-six percent of employed VTC graduates work in their area of study. This was determined through questions on employment history. Even though overall employment rates are low, those who did find work tended to find it the field of their vocational training. The unanswered International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 19 MEPP II CSP Vocational Education and Apprenticeship Study question is whether training in other specialties might have given a better result in overall employment. D. Percent employed, but outside their area of study. Fourteen percent of employed VTC graduates were working outside their area of study. E. Percent that are underemployed (employed but in unskilled labor). Underemployment due to work as unskilled labor was not indicated by the survey. Underemployment due to reduced hours of employment did occur. Thirty percent of those employed worked 30 hours or less during the seven days that preceded the survey.13 F. Belief that training helped graduates earn a living wage. Most respondents entered the program to improve skills or increase their chances for a job. These categorical findings are reinforced by open-ended explanations of the categorical response. Overwhelmingly the respondents thought that the skills learned were useful to the workplace. This is reinforced by the high percentage of those who found employment using the skill they learned. G. Did the training enhance the graduates’ life or contribute in a meaningful way to earning a living? For the 85% of those who found employment in their area of study this is certainly true. For those who did not find employment, training has not yet demonstrated a contribution to a better life. Training program participants remained optimistic that the training was a good thing. H. Do graduates realize any income from the skills learned in the vocational training? Yes, graduates realize income from the skills they learned. Caveats are that some of them were already working in the area of their skill before they joined the VTC. Incomes were reported during the survey. I. Relevancy of the training to the needed skills for area of study. Virtually all respondents (95%) said that the training was worthwhile and provided them with useful skills. Five percent said the training did not provide skills useful in the marketplace. Apprenticeships. 13 Working fewer than 6 hours per day is considered part-time work. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 20 MEPP II CSP Vocational Education and Apprenticeship Study A. Employment rate for apprentices. Twenty-nine per cent of apprentices were currently engaged in any type of work that earned money. The same procedure for identifying current employment was followed for apprentices as was done for VTC graduates. Apprentices have a lower employment rate than for VTC graduates. This finding is contrary to expectations. This can partially be due to the longer average period that VTC graduates have been in the job market (10 months) than have Apprentices (7 months on average). Job duration for apprentices is shorter than for VTC graduates: 6.8 months compared to 8.2 for VTC graduates. The original SOW proposed to look at apprenticeships and VTC graduates who completed their studies or apprenticeship one-year or more ago. For reasons explained above the SOW changed the dates of interest to a more recent episode that includes very recent VTC graduates and apprentices. B. Comparing apprentice and VTC graduates employment rates. We can say with confidence that VTC graduate employment rates are higher than apprentice employment rates. Logistic regression was used to reject the null hypothesis that employment rates for VTC graduates and apprentices might be statistically the same. At the 95% confidence level the null hypothesis was rejected. C. Are apprentices more employable than VTC graduates. There is no evidence to support the proposition that apprentices are more employable than VTC graduates. D. Did the job after the apprenticeship result in earning a living wage? An estimated 70% of employed apprentices earned a “living wage.” Thirty percent fell below the 70,000 ID per capita per month “living wage” threshold. The earning power of apprentices on average did surpass that of the VTC graduate. Average per capita monthly income for employed VTC graduates was a little less than 90,000 ID per month. For apprentices the estimated average was 164,000 ID. E. Apprentice employment outside their area of study. Seventeen per cent of apprentices work outside their vocational specialty. No standard or benchmark has been established to assess if this is acceptable. It would be surprising to find that all employment opportunities aligned with the vocational training received. F. Apprentice employment with the business that hosted the apprenticeship. Half of the employed apprentices still work for the business that hosted them as an apprentice. The survey numbers that estimate this finding are small providing limited confidence in this result. The finding expresses confidence in training program as the employer was willing to pick up the 50% wage difference formerly provided by a CSP stipend. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 21 MEPP II CSP Vocational Education and Apprenticeship Study G. Former apprentice’s belief that the apprenticeship enhanced their employability. Despite the reality that most apprentices are not employed 94% believed that the apprenticeship improved their overall prospects for employment. H. Had apprentices received the 50% matching salary provided by IRD? The study includes an assessment of the VTC stipend payments as well as the Apprentice stipend payments.14 To compare the VTC stipend, Part II, Question 9 asked respondents how much they were paid while they were in the vocational education program. This amount was compared to VTC stipend amounts provided by CSP. Stipend amounts were in US dollars converted to Iraqi Dinar (ID) using the exchange rate of 1000 ID = $1. Stipends to vocational education students are fixed at either $75 (75,000 ID) or $100 (100,000 ID) per month. A larger amount was paid to trainees in the construction trades. The stipend amount was subtracted from the reported VTC payment amount (collected from the respondent in Part II, Question 9). A positive difference indicated that payments received exceeded the stipend. Negative differences flagged discrepancies that needed to be looked at individually. Few negative differences were found and these ranged from the equivalent of $5 to $10 an amount easily attributable to exchange rate differences. It was concluded that vocational education trainees were paid the correct stipend amounts. Stipends paid to apprentices are more complex. There is not a uniform stipend amount, but a variable amount depending on local market conditions. CSP provided a list of apprentice stipends by city and construction or non-construction trades. Fifty percent of that amount is reimbursed by CSP to the business owner. Payments to the business owner are done on a reimbursement basis only, supported by payment receipts and timesheets. The apprentice should receive the full amount unaware that 50% is paid by CSP to the business owner. We were informed to expect considerable variance within Baghdad in the amount of the stipend paid with differences from one area to another. A computer program allocated these referenced stipend amounts to the survey data. As before, the stipulated stipend amounts were subtracted from the amounts respondents said they received as an apprentice. The analysis found that 30% (25 cases) of the apprentices reported receiving less than the stipulated apprentice stipend amount. The majority of these cases are in Baghdad. Further, if you are a woman apprentice it seems you are 5 to 6 times more likely to be paid less than your stipulated stipend amount. In our data this appears to be a systematic problem, and therefore troubling. To verify this finding we went back to CSP to validate the documented stipend amounts actually paid to the specific individuals indentified as receiving less than the stipulated amount. Concurrently, the field monitors went back to the same respondents to confirm the amounts they said they were paid during the apprenticeship (Part IV, Question 15). There is still room for error and misunderstanding in how the numbers were reported, but these findings need an explanation. 14 Annex VIII provides details of the stipends paid. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 22 MEPP II CSP Vocational Education and Apprenticeship Study I. Did the employer match the 50% salary paid by IRD? This is the same assessment used for the CSP 50% payment amount above. Business owners hosting apprentices pay them directly the full amount due. Fifty percent of that amount is reimbursed by CSP to the business owner. Payments to the business owner are done on a reimbursement basis only, supported by payment receipts and timesheets. The apprentice should receive the full amount not necessarily aware that 50% is paid by CSP to the business owner. X. CONCLUSIONS The study has been difficult to implement. Three false starts resulted when sample design pretests failed to locate VTC graduates and apprentices. This too is a finding, but outside the intended focus of the SOW. In MEPP II’s four-year experience in Iraq it has never encountered such low survey response rates (including locating IDP recipients of food and non-food distributions). Clearly this was indicative of an underlying management problem. The problem was recognized by IRD and remedial steps taken, at least in Baghdad. However, the remedy altered how the study was implemented and possibly the study result. Instead of reaching back one year or more to study VTC graduates and apprentices as intended, the study, out of necessity, looked at more recent examples that had limited employment histories. This may have lead to an underestimate of apprentice employment. The study measured quantitatively the effectiveness of VTC training and apprenticeships in achieving employment. The study found that the overall employment rate was 40% leaving 60% unemployed. A recent nation-wide survey reports unemployment rates, for the same age groups of interest here, to be less than 25%.15 Gauging the success of the program against this benchmark tells us that the results of the program are not encouraging. By interviewing only recent program participants; some only a few months after completing training or the apprenticeships, a possibility exists that recent apprentices have had insufficient time to find employment (30% completed their apprenticeship four months ago or less). Even considering these adjustments, the employment rates attributed to VTC training and apprenticeships are not impressive. The study could not address the issue of whether long-term employment had been generated because the sample was drawn from only the more recent VTC graduates and apprentices. Official employment statistics measure the employment rate of persons who say they are in the labor force (persons who are actually looking for paid work, and not a student, retired person or a housewife). We can assume that everyone in the vocational education and apprentice program is in the labor force and looking for work, but this may not be the case. For example, the percent of women in the program (25%) exceeds the percent of women from the general population in the labor force (<20%). It is possible to conclude that some of the women join the program (vocational courses in sewing) to take advantage of a paid interlude that earns 75,000 ID per month and provides a sewing machine at the end, but have no intention of seeking 15 The “Iraq Household Socio-Economic Survey” IHSES- 2007, Tabulation Report, Central Organization for Statistics and Information Technology (COSIT), Kurdistan Region Statistics Organization, and the World Bank, page 289. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 23 MEPP II CSP Vocational Education and Apprenticeship Study employment. A similar argument can be made for some of the younger participants who may complete a vocational education course and then return to a student’s status. Eighty-six percent of those who are employed (i.e., 86% of the 40% of the total respondents who are employed) are putting the skills they learned to use; it is unfortunate that more are not employed. We do not have a gauge for assessing whether this level of use of trained skills is acceptable. To expect a higher percentage suggests that the VTC is able to perfectly match training to the demand for skills. A majority of those who are employed say they are self￾employed. This includes jobbers, running a small shop or business. From studies of the CSP business development program it is highly probable that these are small family run businesses where accounts are mixed with family accounts (informal businesses). Supporting these small businesses with a grant, or with business skills training was envisioned by the CSP program, but was not tested here. Participants said they were satisfied with the courses and their apprenticeships. Since they are paid to attend and participate it is unlikely to find many dissenters. Average earnings made by VTC graduates are slightly below national averages (IHSES), or at least not above them. Calculated per capita incomes linked to the “living wage” benchmark show that many VTC graduates are struggling to keep enough food on the table. For those not employed the majority of them live from family contributions (as dependents of the larger family group). The study validated that the stipends paid to the VTC participants had been received. There were few anomalies, and these were attributed to random errors rather than systematic fraud. Validating that stipends paid to apprentices were actually received in full has been more complicated. The study found that 30% of apprentices were receiving less than the stipulated stipend amount. Women apprentices were 5 to 6 times more likely to be among those not receiving their full stipend amounts. This needs further investigation. (see Annex VIII for greater detail). XI. RECOMMENDATIONS 1. CSP’s management experience with the apprenticeship program in Baghdad reinforces the need for early deployment of Monitoring and Evaluation and QAQC staff in identifying and validating program participants. There can be little doubt that initial beneficiary selection and oversight procedures were insufficient to prevent abuse as the tracking system was not robust enough to locate program participants to validate whether or not they had actually received payment or even existed. IRD’s management remedy closed these loopholes as evidenced by MEPP II’s ability to locate all of the sampled Baghdad participants in the final survey. That final management practice needs to be propagated to other vocational education and apprentice programs. 2. MEPP II experienced difficulty in working with the data provided to them by CSP. This difficultly stems from the independence of the units operating in the field. The M&E units collected information on program participants using different formats. The M&E database structure differs from one location to another: e.g. Kirkuk is different than Mosul which is different than Baghdad. It was necessary to reconcile the different formats to achieve a program-wide data set of apprentices and vocational education graduates that we could use for this study. This same lack of uniformity would hinder program-wide management International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 24 MEPP II CSP Vocational Education and Apprenticeship Study studies similar to this one. It is recommended that each of the operating units maintain the same database structure and collect the same information. This extends to using the same forms and operating procedures. This need not affect the independence of management. 3. The CSP Rapid Programmatic Assessment Report, November 2007 by IBTCI recommended that CSP introduce an element of sustainability by targeting training-to￾work graduates with BDP grants. The large percentage of employment coming from self￾employment show how relevant this is. It was not apparent from this study whether grants were made available to VTC graduates and apprentices. 4. Review the inclusion of women in the VTC program. Women, based on economic activity rates, are over-represented in the training. All but 7 of the 67 women interviewed did vocational education in sewing. Completion of the sewing course is awarded with a sewing machine. This is an attractive opportunity for any housewife. Is the program certain that these participants pose a threat of being caught up in the insurgency? 5. Set an employment rate benchmark for the vocational education program by using official government statistics as the guideline. The program should be able to meet or exceed known rates. 6. Investigate why women, especially in Baghdad, are apparently not receiving their full stipend amounts. XII. LESSONS LEARNED 1. CSP oversight of the vocational education graduate and apprentice databases came late in the project cycle. When it did happen and the apprenticeship program was suspended the result was telling. Ensure that M&E and QAQC are engaged early in the program implementation to ensure proper program beneficiary selection. 2. Lack of database uniformity across program implementing units across provinces causes problems for overall program management. 3. Necessary revisions to the scope of work significantly reoriented the time frame of the analysis. This may have affected estimates of employment. 4. The VTC program did not set a benchmark for employment rate. Internatio Monitoring and Evaluation Per nal Business & Technical Consultants, Inc. formance Project, Phase II (MEPP II) MEPP II CSP Vocational Education and Apprenticeship Study 25 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 26 Annex I. USAID/FSO Response Matrix to the Study Recommendations IBTCI Paragraph IRD/CSP Response Planned Actions/ Timeline for Follow up USAID/FSO Comments Recommendation #1: CSP’s management experience with the apprenticeship program in Baghdad reinforces the need for early deployment of Monitoring and Evaluation and QAQC staff in identifying and validating program participants. Apprenticeship program participants must be VoTech graduates---all VoTech graduates are potential apprentices. In all CSP cities (including Baghdad), apprenticeship program participants are tracked and validated by CSP apprenticeship placement officers. CSP Iraq-wide apprentice documents and reporting forms for pre-assessment, follow-up and monitoring, evaluation, and post￾assessment are utilized uniformly by all CSP cities, and included in the CSP FOM. The validating of apprentices and payments to apprentices is included in our response to Recommendation #6. Timeframe: IRD to document the procedure by May 31, 2009; IBTCI to upload the final report and annexes to the DEC upon final report completion in June 2009. USAID suggests that IRD document the procedure; that IBTCI incorporates this matrix and IRD’s response as an annex to this report; and that IBTCI loads the final report on the DEC per MEPP II contractual requirements for deliverables. Recommendation #2: MEPP II experienced difficulty in working with the data provided to them by CSP. This difficultly stems from the independence of the units operating in the field. The M&E units collected information on program participants using different MEPP II clearly misunderstands the role of M&E on implementing participant databases and accordingly misuses the term “M&E database”. CSP M&E uses two data structures for information gathering and reporting: the tracking sheets and the M&E database. Both data structures are fixed and uniform across all IRD to report to USAID on the uniformity of M&E databases for other CSP program components by June 15, 2009. If they are not uniform, develop an action plan for The VoTech and apprenticeship programs are now largely closed, therefore, it would be a make work exercise to retroactively revise the databases. However, USAID/FSO asks IRD to report to USAID on the MEPP II CSP Vocational Education and Apprenticeship Study International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 27 MEPP II CSP Vocational Education and Apprenticeship Study IBTCI Paragraph IRD/CSP Response Planned Actions/ Timeline for Follow up USAID/FSO Comments formats. The M&E database structure differs from one location to another: e.g. Kirkuk is different than Mosul which is different than Baghdad. program components. M&E tools for collecting information are unified in CSP. CSP M&E neither collects nor maintains a data base of participant addresses, phone numbers and email addresses. Each CSP cities’ EGY unit developed this database for VoTech and apprentices that was appropriate for their city. CSP implementing partners for the VoTech programs across CSP cities were different and often required different data. This information was made available to MEPP II; however information like mobile phone numbers is not collected in cities that have no mobile phone service. correcting. uniformity of the databases for the other ongoing CSP components. Recommendation #3: The program already envisions supporting VTC graduates with grants. The large percentage of employment coming from self￾employment show how relevant this is. It was not apparent from this study whether grants were made available to VTC graduates and apprentices. Although, CSP supports making VoTech trainees aware of the MSME grants, CSP does not envision this connection as a strategic result. Trainees, while in class, are made aware of CSP MSME grants program through multi presentations by CSP MSME officers. The CSP VoTech graduates who enroll in the apprenticeship programs carry this knowledge of CSP MSME grants with them and may share it with other non-CSP sponsored colleagues in the apprenticeship hosting businesses. Closed. While a useful finding, CSP never envisioned this connection between VTC and BDP as a strategic result of the program. CSP has largely phased out of vocational education (support to MoLSA ended Jan 2009). If there are lingering activities in Baqubah The report references that the CSP program already envisions supporting VTC graduates with grants. However, while this is an interesting finding and should be incorporated into future USAID programs, CSP regrettably did not envision this potential linkage between the VoTech and BDP components from the start. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 28 MEPP II CSP Vocational Education and Apprenticeship Study IBTCI Paragraph IRD/CSP Response Planned Actions/ Timeline for Follow up USAID/FSO Comments However, CSP’s most successful grants were awarded to grantees aged more than 35 years due to their past working experience, capital and knowledge of starting new businesses. Many of the graduates were unlikely to meet the match requirement for the grants. Moreover, the business skills training was mainly oriented, but not limited to, approved grantees who are expected to have had prior experience and knowledge. In general, though it would have been ideal to provide the graduates with grants, there is not enough confidence that the graduates or apprentices possessed enough business skills and capital to meet the BDP grant program requirements. with industrial schools, IRD should make graduates aware of BDP grants. Thus, there is nothing of substance to report on awareness raising amongst VTC graduates. Recommendation #4: Review the inclusion of women in the VTC program. Women, based on economic activity rates, are over-represented in the training. All but seven of the 58 women interviewed did vocational education in sewing. Completion of the sewing course is awarded with a sewing machine. This is an attractive opportunity for any housewife. Is the program certain that these participants pose a threat of being caught up in the insurgency? Yes, there have been multiple attacks carried by women in 2008. Women can earn income and are caretakers of the younger generation who are vulnerable for recruitment in violent acts. Women are part of CSP’s targeted age group. Currently, sewing is seen as a job that a woman can do from her home. However, it is not CSP’s specialty to conduct security analysis in relation to women’s participation in violent acts. Closed. USAID/FSO has reviewed the participation of women under a COIN strategy and finds it relevant for female suicide bombers. This is also a priority of MNF-I. USAID will target resources in Baqubah in particular for this purpose under Phase IV of the CSP extension. While CSP largely focuses on unemployed young men, USAID/FSO is responsive to the upswing in female suicide bombers in Iraq, particularly in Baqubah where there have been 17 instances in recent months. Thus, the CSP program is addressing these needs through engaging at risk women in targeted areas. USAID/FSO believes that it is a valid COIN argument International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 29 MEPP II CSP Vocational Education and Apprenticeship Study IBTCI Paragraph IRD/CSP Response Planned Actions/ Timeline for Follow up USAID/FSO Comments Timeframe: October 2009 to February 2010. to reduce vulnerabilities for potential female suicide bombers. Recommendation #5: Set an employment rate benchmark for the vocational education program by using official government statistics as the guideline. The program should be able to meet or exceed known rates. National employment rates can not be compared to CSP employment rates or be used as a guideline as the targeted populations are different. National statistics are based on tremendously larger samples, while CSP statistics are based on extremely smaller samples. Overall, CSP is expected to reduce unemployment by 1.25%; however, this can not be, primarily, associated with VoTech or apprenticeship programs as CSP generates employment though its grants and CIES activites. Closed. CSP already measures national unemployment rates as part of the M&E plan. IRD already measures the unemployment rate using Ministry of Planning statistics for youth between the ages of 17-25 and 25+ as part of its M&E plan and as a requirement of CSP Modification 12. However, it is difficult to attribute CSP alone to nation-wide changes in employment outcomes as CSP. Recommendation #6: Investigate why women, especially in Baghdad, are apparently not receiving their full stipend amounts. CSP pays stipends based on actual days of training attended by the trainees. For example, if a trainee in a construction course misses four days in a month (average 22 training days). The trainee will receive $82 out of the $100 monthly stipend. The payment of stipends to the trainees in Baghdad and other CSP cities is established, controlled and monitored. This process involves VTC, EGY and finance personnel, in addition to the trainee who must bring photo identification. At the time of payment the trainee signs a receipt. Timeframe: the Baghdad office is closing June 30, 2009. However, M&E will continue to August 31, 2009. IRD will complete these site visits within this time frame and report back to USAID. USAID/FSO recommends that IRD QAQC or EGY officers randomly spot check and conduct visits to a sample of women apprentices in Baghdad. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 30 MEPP II CSP Vocational Education and Apprenticeship Study IBTCI Paragraph IRD/CSP Response Planned Actions/ Timeline for Follow up USAID/FSO Comments In addition, trainees know when payments are due to them and any delays, or reduced payments, will lead to near riots as CSP has experienced.