June 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by James Statman and Richard Rousseau with International Development Group LLC under the Learning Evaluation and Analysis Project-II (LEAP-II) task order. Evaluation Final Performance Evaluation of “Tackling Youth Employment in Tunisia” Photo credit: Mashrou3i EVALUATION REPORT FINAL PERFORMANCE EVALUATION OF “TACKLING YOUTH EMPLOYMENT IN TUNISIA” JUNE 27, 2016 Learning Evaluation and Analysis Project-II (LEAP-II) Activity Order # 17 Contract Number: AID-OAA-I-12-00042/AID-OAA-TO-14-00046 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i TABLE OF CONTENTS Executive Summary .................................................................................................................................... 1 Evaluation Purpose & Evaluation Questions ......................................................................................... 11 Project Background .................................................................................................................................. 13 Methodology & Limitations ..................................................................................................................... 16 Evaluation Methodology ......................................................................................................................... 16 Limitations .............................................................................................................................................. 18 Findings and Conclusions ......................................................................................................................... 19 EQ1. To what extent has the HP-UNIDO implementation approach been effective in creating sustainable employment? ........................................................................................................................ 19 EQ 2. Relative to a traditional contract model, were there any benefits to using the GDA approach? Were there any externalities of using a GDA approach that facilitates external resource partners and a PIO approach that attracts the resources of additional donors? .............................................................. 30 EQ3. To what extent did UNIDO’s cooperation with existing SBSs contribute to employment outcomes? (Consider UNIDO’s business model, years of experience working with indigenous parastatals and cost structure.) ................................................................................................................ 31 EQ4. To what extent can the employment information collected by the HP-UNIDO activity be independently validated and deemed credible? ....................................................................................... 35 Recommendations ..................................................................................................................................... 39 Annexes ...................................................................................................................................................... 41 Annex I: Evaluation Statement of Work ................................................................................................. 41 Annex II: Key Informant Interview (KII) Guide .................................................................................... 54 Annex III: Focus Group Discussion (FGD) protocol - Beneficiaries ..................................................... 62 Annex IV: Individual Beneficiary Survey Questionnaire ....................................................................... 63 Annex V: Institutional Beneficiary Survey Questionnaire ..................................................................... 73 Annex VI: Evaluation Design Matrix ..................................................................................................... 76 Annex VII: List of Key Informant Interviews (KIIs) & Focus Group Discussions (FGDs) ................... 81 Annex VIII: List of Job Creation Programs in Tunisia ........................................................................... 88 Annex IX: Complexities of Measuring Jobs Created and their Costs ..................................................... 90 Annex X: Disclosure of Any Conflicts of Interest .................................................................................. 91 ii LIST OF TABLES Table 1.1 Jobs Created and Projected as Reported by UNIDO 3/2016 (4) .................................................. 3 Table 4.1 Group Meetings Held In Sousse ................................................................................................. 17 Table 4.2 Summary of Survey Populations, Samples, and Response Rates .............................................. 17 Table 5.1 Jobs Created and Projected as Reported by UNIDO 3/2016 (4) ................................................ 19 Table 5.2 Participatns in HP-LIFE Training and Coaching by Age (1) ..................................................... 26 Table 5.3 Estimated Jobs Created per Enterprise/Entrepreneur Assisted .................................................. 32 Table 5.4 Costs per Job Created ................................................................................................................. 37 Table 5.5 Payback Period ........................................................................................................................... 38 LIST OF FIGURES Figure 3.1 Tackling Youth Employment in Tunisia Mashrou3i ................................................................ 15 Figure 5.1 Impact of Mashrou3i on Personal Income ................................................................................ 21 Figure 5.2 How did Mashrou3i affect your own employment status? ....................................................... 24 Figure 5.3 How many persons have you hired as a result of Mashrou3i? ................................................. 24 Figure 5.4 Mashrou3i Participants by Gender and Region ........................................................................ 25 Figure 5.5 Mashrou3i Impact on Employment Status by Gender .............................................................. 26 Figure 5.6 Highest Level of Study by Mashrou3i Participants .................................................................. 27 Figure 5.7 Period of Unemployment after Receiving Degree .................................................................... 27 Figure 5.8 Employment Status ................................................................................................................... 28 Figure 5.9 Ratings of Small Business Support (SBS) Organizations in Kairouan by Mashrou3i Participants .................................................................................................................................................. 33 Figure 5.10 Ratings of Small Business Support (SBS) Organizations in in Kasserine by Mashrou3i Participants .................................................................................................................................................. 34 Figure 5.11 Ratings of Small Business Support (SBS) Organizations in Le Kef by Mashrou3i Participants .................................................................................................................................................................... 34 Figure 5.12 Ratings of Small Business Support (SBS) Organizations in Sidi Bouzid by Mashrou3i Participants .................................................................................................................................................. 35 iii ACRONYMS AOR Assistance Officer Representative ANETI National Agency for Employment and Independent Work APIA Agency for the Promotion of Agricultural Innovation APII Agency for the Promotion of Industry and Innovation BFPME Bank for Financing Small and Medium Enterprises BTS Tunisian Bank for Solidarity FGDs Focus Group Discussions FTF Face-to-Face GDA Global Development Alliance GOI Government of Italy GOT Government of Tunisia HP Hewlett Packard HP-LIFE Hewlett Packard’s Learning Initiative for Entrepreneurs IDG International Development Group LLC ISET Higher Institute of Technological Studies KIIs Key informant interviews Mashrou3i “Tackling Youth Employment in Tunisia” project ODCO Office for the Development of the Center West PIO Public International Organization SBS Small Business Support SME Small and Medium Enterprise SWOT Strengths, Weaknesses, Opportunities, and Threat ToR Terms of Reference UNIDO United Nations Industrial Development Organization USAID U.S. Agency for International Development USG U.S. Government UTICA Union of Industry, Trade and Handicrafts 1 EXECUTIVE SUMMARY Evaluation Purpose Between 2012 and 2016, the “Tackling Youth Employment in Tunisia” project (Mashrou3i) aimed to create employment for young Tunisians in four interior governorates: Le Kef, Kairouan, Kasserine, and Sidi Bouzid.1 USAID support was provided through a Global Development Alliance (GDA) with Hewlett Packard (HP), the United Nations Industrial Development Organization (UNIDO), and the Government of Italy (GOI). The activity is implemented by UNIDO in partnership with the HP Foundation and local Tunisian partner organizations. USAID funding of $1.5 million was provided through a Program Contribution Agreement with UNIDO that was signed on September 28, 2012 and which, following two no-cost extensions, is scheduled to end on September 30, 2016. The purpose of the final performance evaluation is to document and assess the ways and extent to which Mashrou3i succeeded in creating sustainable employment/livelihoods for its target populations. The evaluation was conducted in March-May 2016, with fieldwork taking place March 23 to April 8, 2016 in Tunis, Sousse and Kairouan, Tunisia. Evaluation Questions The performance evaluation was conducted to specifically address four overarching evaluation questions: 1) To what extent has the HP-UNIDO implementation approach been effective in creating sustainable employment in the interior regions in which the activity is implemented? 2) Relative to a traditional contract model, were there any benefits (e.g., lower costs, greater leverage) to using the GDA approach for the implementation of the HP-UNIDO activity? Were there any externalities (positive or negative) of using a GDA approach that facilitates external resource partners and a PIO approach that attracts the resources of additional donors? 3) To what extent did UNIDO's cooperation with existing Tunisian Small Business Support (SBS) organizations contribute to employment outcomes? Consider UNIDO's business model, years of experience in working with indigenous entities that are government funded (parastatals), and cost structure. 4) To what extent can the employment information collected by the HP-UNIDO activity be independently validated and be deemed credible? Program Background Mashrou3i is organized into two complementary but distinct technical components: 1. Component One: Support for Small Business Service Organizations (SBSs) 2. Component Two: Support for Entrepreneurs and Enterprises, consisting of: 2.1 HP-LIFE Entrepreneurship Training and Coaching 2.2 Support to Start-Ups and Existing Businesses 1 In late 2015 Mashrou3i commenced program activities in four additional governorates: Gafsa, Kebili, Medenine and Tataouine. However, these are not included within the ambit of this evaluation. 2 Methodology and Limitations The evaluation drew upon qualitative and quantitative methods to obtain evidence sufficient to support conclusions and recommendations addressing the evaluation questions. These methods included: document review; 41 key informant interviews (KIIs); one focus group discussion (seven participants); four group meetings (17 participants total); one HP-LIFE training observation; and five business site￾visits. The evaluation team also conducted two online surveys of project participants: (1) an individual and enterprise beneficiary survey (which covered entrepreneurs trained and/or coached by HP-LIFE, start-ups, and existing enterprises that received project services), and (2) an institutional beneficiary survey (which covered SBSs that received capacity building services from Mashrou3i). The individual and enterprise beneficiary survey generated a total of 272 valid responses out of a total of 1,463 participants to whom the survey was sent, constituting a response rate of 18.6 percent. The institutional survey was sent to 23 regional SBS organizations, of which 13 responded, constituting a response rate of 57 percent.2 Limitations included security constraints, which prevented the evaluation team from conducting data￾collection site visits to Le Kef, Kasserine and Sidi Bouzid. The team was able to conduct two site visits to Kairouan. Due to time and budget constraints, the evaluation team was not able to conduct a statistically significant, random sample survey of individuals and enterprises that received support from Mashrou3i. The evaluation team addressed these limitations by conducting additional KIIs and group meetings in Tunis and Sousse with some participants and beneficiaries from all four target regions, and by utilizing differing methods and data sources to address each evaluation question. Findings and Conclusions EQ1: To what extent has the HP-UNIDO implementation approach been effective in creating sustainable employment? Conclusion: Considered in the context of political, social and economic instability during 2014-2015 and the challenges posed by the current security situation, Mashrou3i has made notable progress in generating sustainable employment for youth in the four interior governorates in which it has been implemented, slightly surpassing its target for direct jobs creation and thereby demonstrating the efficacy of its intervention approach. Findings/Evidence: From October 1, 2012 through March 31, 2016, UNIDO reports having generated 1,035 direct jobs, or 103.5 percent of the life of activity target of 1,000 direct jobs.3 Further, Mashrou3i reports having placed 44 HP-LIFE trainees in jobs and it projects that within the following 12 months an additional 575 jobs will be created from current program activities, constituting a grand total of 1,654 direct jobs created, placed or in-process. This represents 165.4 percent of the life of activity jobs target. 2 Survey respondents did not respond to every question on the relevant questionnaire and so the number of responses (Ns) may differ from item to item. Throughout this report the number of responses (N) noted in the text, figures, and tables pertains to the number of respondents who answered that specific survey question. 3 The Program Contribution Agreement, evaluation Statement of Work and Mashrou3i “Results Chain and Main Expected Results” chart all refer to a target of 2,000 jobs including indirect and seasonal jobs. However, subsequent to fieldwork the evaluation team was informed by USAID/Washington that shortly after the start of Mashrou3i implementation, USAID and UNIDO agreed to revise the target to 1,000 direct jobs, dropping the inclusion of seasonal or indirect employment. This report therefore uses this as the target. 3 The table below presents the jobs created and projected as reported by UNIDO by activity or type of enterprise. The evaluation team conducted a detailed analysis of the job data provided by UNIDO to produce Table 1.1 below and confirmed that supporting documentation exists for the jobs numbers included in the table. Table 1.1 Jobs Created and Projected as Reported by UNIDO 3/2016 (4) # Activity/Type of Enterprise Jobs Created through March 2016 Note HP-LIFE Trainees Placed through March 2016 Note Additional Jobs to be Created as Reported by UNIDO Note Totals 1 HP-LIFE Support to Entrepreneurs: 471 44 307 822 Start-ups enabled/facilitated by HP-LIFE program 121 (1) 121 New positions hired by start-ups enabled by HP￾LIFE Program 350 (1) 307 (1) 657 Found employment elsewhere 14 (1) 14 Hired by existing enterprises 30 (1) 30 2 Deep-dive business coaching support for start￾ups in cooperation with local organizations (5): 135 268 403 Start-ups established as a result of business coaching 16 (2) (2) 16 New positions hired by start-ups established from business coaching 119 (2) 268 (2) 387 3 Support to Existing Enterprises 429 (3) Additional jobs expected, but not quantified 429 Totals 1,035 44 575 1,654 Notes (1) 'Copy of Survey_HP-LIFEentrepreneurs_16_March_2016_2.xlsx', UNIDO, March 2016 (2) 'Startups_job creation_data_Jan_2016', UNIDO, January 2016 (3) 'Tableau_accompagnement EE Mars 2016', UNIDO, March 2016 (4) 'Copy of Mashrou3i project - JOBS Data -mars 2016', UNIDO, March 2016 (5) According to UNIDO, these start-ups initially received training, but not coaching under HP-LIFE. According to the online survey conducted by the evaluation team, 53 percent (N=99) of the 188 HP-LIFE beneficiaries responding indicated that Mashrou3i helped them to start a business; 11 percent (N=20) that the project helped them expand an existing business; 10 percent (N=19) that it helped them to find a job; and 27 percent (N=50) that it had no impact upon their employment situation. For the online survey of individual entrepreneurs and enterprises, slightly more than half of the respondents (85 out of 163) noted an increase in jobs within their businesses, some greater than ten new 4 positions: 36 percent (N=58) reported an increase of 1-5 positions following participation in Mashrou3i; nine percent (N=15) reported an increase in 6-10 employees; seven percent (N=12) reported more than 10 new employees; and 48 percent (N=78) reported no change. 1a) Which aspects of the implementation approach have been the most effective in creating sustainable employment and why? Conclusion: The support to young entrepreneurs and support to existing enterprises have been the most effective programmatic components of Mashrou3i in creating jobs. This may be related to the more intensive training and coaching associated with these activities. Findings/Evidence:  Of jobs created to date, UNIDO reports that: 471 jobs (46 percent) were created by the HP-LIFE support to young entrepreneurs; 429 (41 percent) through support to existing enterprises; and 135 jobs (13 percent) created were attributed to the Mashrou3i “Deep Dive” business coaching activity in conjunction with local business support organizations. More than 12,000 Tunisians have taken at least one online HP-LIFE e-learning module. In the focus group discussion (Kairouan) and in group meetings with young entrepreneurs from Le Kef, Kasserine, and Sidi Bouzid, participants consistently stated that the online e-learning modules and group training were helpful to them in enabling them to create a business idea, and to enhance their business and finance skills.  In group meetings with young entrepreneurs from all four governorates (N=17), many stated that they faced serious challenges in access to finance and several described major bureaucratic hurdles in obtaining necessary permits or registration. Many seemed intimidated by a business environment they perceive as somewhat discouraging to young entrepreneurs, and by what they described as a general lack of entrepreneurial culture in their regions. As one participant stated, “we have no role models or success stories” from which to draw inspiration or guidance. 1b) Which populations have benefited most from the activity and why? How have women and youth in the four regions differentially been reached and benefited? Conclusion: Although women have participated in the program at slightly higher rates than men4 , the evaluation data do not indicate differential program outcomes by region or gender. Approximately 75 percent of men and women reported that Mashrou3i had helped them find a job, or start or expand a business. Findings/Evidence:  Mashrou3i does not disaggregate jobs created by gender or age.  Approximately 87 percent (N=1,354) of the HP-LIFE training and coaching participants for whom UNIDO provided birth dates were between the ages of 20 and 34 years (Table 5.2). The vast majority of those who responded to the survey of individuals and enterprises (94 percent, N=247) reported having post-secondary education in a variety of fields of study, including IT, engineering and other technical or scientific fields, manufacturing, and “other services.” 4 UNIDO reports (Project Progress Report 1 January 2013 – 15 March 2016) that of 1,136 HP-LIFE entrepreneurs, 588 (52%) are women. 5  Mashrou3i participants in group meetings conducted by the evaluation team studied a broad range of subjects including engineering and IT, and also optometry, physical therapy, agronomy, communications, journalism, and humanities.  A majority (59 percent, N=150) of total respondents (N=253) (to a question about the period of unemployment after graduation) reported having been unemployed for more than one year after obtaining of their degree; 11 percent (N=27) reported having been unemployed for six months to one year; 14 percent (N=36) one to six months; and 16 percent (N=40) less than one month (Figure 5.7, page 29). Of those who responded to a question about current employment status (N=263), forty-eight percent (N=127) of survey respondents described themselves as currently unemployed; 37 percent (N=96) as self-employed; and 15 percent (N=40) as employed (Figure 5.8). 1c) How were participants identified? Can anything be concluded about the general characteristics (e.g. educational or professional background of participants vs. non-participants? Conclusion: The most striking characteristic of HP-LIFE FTF participants, based on the survey data and UNIDO KIIs, is the high level of education of participants, with 94 percent of survey respondents (N=247) indicating the completion of post-secondary education5 . Furthermore, a very high percentage of program participants have studied in scientific, technical, and/or engineering fields. Findings/Evidence:  HP-LIFE online e-learning curriculum. Access to the HP-LIFE online e-learning curriculum (25 modules) is free and open to anyone in Tunisia with computer access to the internet.  Four-day business coaching workshops. Trainers from the local partners interview all HP￾LIFE applicants for the face-to-face training, and nominate candidates for the four-day business coaching workshops whom they judge to have the most viable business ideas.  “Deep-dive” coaching. The candidates for the “deep-dive” coaching for start-up development and support to existing enterprises replied to a request for proposals asking for business ideas from prospective young entrepreneurs in the four target governorates. Based on this call, 44 proposals were selected based on technical merit and perceived likelihood of success. Similarly, 88 enterprises in the four regions were profiled and selected for technical support based on their assessed job growth potential. 1d) How did the HP-UNIDO approach of working directly with local communities influence individual motivations or job training or placement outcomes? Conclusion: By engaging local partners for service delivery at the community level, the Mashrou3i model encourages access and makes use of local knowledge for job creation. The incorporation of the HP-LIFE e-learning curriculum into Higher Institute of Technological Studies (ISET) courses is a particularly striking example of this process. HP-UNIDO efforts to strengthen these institutions, while relatively modest in scope, can be plausibly linked to improved or expanded service delivery and likely placement outcomes. The approach of working through local institutions increased awareness of the availability of HP-LIFE online training and the subsequent enrollment of about 12,000 Tunisians. 5 The evaluation team was unable to obtain data on the level of education of the owners or managers of the 52 companies referred to by UNIDO as ‘existing enterprises’, or of the owners of an estimated 60-70% of the 42 e start-ups (i.e. those who participated in so-called “deep-dive” coaching) who did not also receive HP-LIFE FTF training. 6 Findings/Evidence:  Online survey responses received from 13 SBS partners suggest success in building their service delivery capacity. Out of a total of 11 organizations that responded to a question about the impact of Mashrou3i on their services to entrepreneurs, 55 percent (N=6) reported that Mashrou3i helped them to add new services and improve existing services; 36 percent (N=4) reported improvement to existing services; and nine percent (N=1) indicated no impact on their service provision.  The decision of the Ministry of Higher Education to formally incorporate the HP-LIFE e-learning modules into required entrepreneurship courses at 25 ISETs throughout the country – and to have its instructors trained in this approach, methodology, and content – strengthens these institutions by incorporating a standardized IT-based business skills curriculum and instructional platform, including exercises across campuses and instructors, and by expanding the HP-LIFE beneficiary population. EQ2: Relative to a traditional contract model, were there any benefits to using the GDA approach? Were there any externalities of using a GDA approach that facilitates external resource partners and a PIO approach that attracts the resources of additional donors? Conclusion: The GDA approach, including the Italian Development Cooperation and USAID, and the engagement of an experienced Public International Organization (PIO) as the major implementing partner, leveraged financial resources, human capital, and institutional experience in support of program results; enlisted the extraordinary technical expertise of a major private sector corporation; and demonstrated the value and potential of such a unique multi-donor public-private partnership (PPP) model. Findings/Evidence:  The GDA mechanism resulted in a significant leveraging of USAID program resources, more than doubling the USG investment for the HP-LIFE program. The GDA approach also leveraged the expertise and in-country experience of UNIDO, a PIO with which USAID has not frequently collaborated, incorporating its long-term partnerships with HP and the GOI.  FGD and KIIs suggest that this unique collaboration, involving the multilateral engagement of the Governments of Tunisia, Italy, and the United States; a U.N. agency; and a private-sector corporation – thereby also creating a PPP within its structure – is itself a noteworthy model in accordance with the cooperative thrust of international development policy and the principles of local ownership and priority-setting.  The GOI believes that Mashrou3i has been highly effective and greatly values its working partnership with the GOT, USAID, HP, and UNIDO. In late 2015, the GOI significantly increased its investment, providing resources to extend the program to four additional governorates: Gafsa, Kebili, Medenine, and Tataouine (these additional four governorates are not covered in this evaluation). EQ3: To what extent did UNIDO’s cooperation with existing SBSs contribute to employment outcomes? (Consider UNIDO’s business model, years of experience working with indigenous parastatals and cost structure.) Conclusion: UNIDO’s cooperation with SBSs has contributed both directly and indirectly to employment creation. However, program participants do not consistently rate the quality and impact of the services delivered by some local partners as acceptable. 7 Findings/Evidence:  The evaluation online survey responses from regional small business support organizations suggest a significant positive impact on local service delivery: out of 11 organizations that responded to a question about program impacts six reported that Mashrou3i helped them to add new services and improve existing services; four said that the program helped improve existing services; and one reported no impact.  In group meetings and KIIs, project beneficiaries, including HP-LIFE aspiring entrepreneurs and owners of existing small businesses, expressed a range of opinions concerning the value of the local implementing partners in assisting their entrepreneurial project. While some found the local SBSs to be helpful in providing technical advice, assistance, and support, many others expressed the opinion that the SBS personnel with whom they consulted are not sufficiently skilled, experienced, or interested in providing the necessary support the entrepreneurs needed.  While acknowledging the potential cost and sustainability benefits of utilizing local SBS structures to deliver entrepreneurial training, advising, and coaching services, several Tunisian experts in enterprise development suggested that parastatal organizations generally lack the actual hands-on expertise, business experience, and motivation that are found in the private sector.  While acknowledging financial, material and technical support from UNIDO, some participating Business Center representatives indicated in KIIs that they regarded this level of support insufficient to produce a significant and sustainable increase in institutional services. EQ4: To what extent can the employment information collected by the HP-UNIDO activity be independently validated and deemed credible? Conclusion: The evaluation team was able to independently validate a sample of Mashrou3i employment information, and found no evidence of error or misrepresentation of results. HP-UNIDO makes diligent efforts to monitor and track its program activities and to collect, analyze, and report credible employment results and has modified its program design accordingly. Best practice, however, would suggest a more systematic and rigorous performance monitoring and results-tracking approach that more clearly separates the data collection and analysis process from service delivery functions and personnel. Findings/Evidence:  Comparison of UNIDO-reported jobs from a small sample of companies with the self-reported jobs creation figures by the same companies in the online survey proved to be almost identical, suggesting that the information reported by UNIDO can be considered credible. 4a) How does HP-UNIDO define and measure employment creation? Conclusion: HP-UNIDO does not have a formal definition for ‘employment creation’ that is linked to a specific action or set of actions undertaken or supported by the activity. HP-UNIDO measures employment creation through enterprise surveys that compare enterprise employment levels at different points of time. Findings/Evidence:  Mashrou3i relies on periodic surveys to determine whether new jobs have been created in the assisted enterprises. UNIDO considers sustainable employment as a job that has existed for one year or more. Through its most recent (March 2016) survey, UNIDO has verified with the entrepreneurs all start-up numbers reported in the April 2016 Progress Report. UNIDO does not 8 count those enterprises (or their employment figures) that were established in 2014 or 2015 and subsequently closed or could otherwise not be validated through the survey process.6 4b) Given the methodology and procedures used to collect employment information, to what extent are the results from the routine activity monitoring likely to be over or under-estimated? Conclusion: Given the current methodology and procedures used by UNIDO, there may be discrepancies between actual and reported results for individual participant enterprises. However, based on the analysis of the supporting documentation from UNIDO and the results of the online survey, the evaluation team believes that the employment information is reasonably accurate. Findings/Evidence: Monitoring surveys administered by Mashrou3i and local partner staff may tend to over-estimate program activity and employment results due to social desirability effect in which respondents are reluctant to provide more negative information to people with whom they cooperate in project implementation. Conversely, the telephonic surveys may fail to reach employed individuals and functioning businesses and start-ups, which are then excluded, thereby underestimating results. However, the evaluation team did not find evidence of systematic over-estimation or under-estimation of employment created. 4c) What methodologies, processes, systems or structures have facilitated or impeded the collection of high quality and credible employment data? Conclusion: The absence of a separate, full-time monitoring and reporting function within the project can threaten the collection of high quality and credible employment data. Findings/Evidence:  While UNIDO is committed to quality service delivery and data collection, analysis, and reporting systems, Mashrou3i’s monitoring and evaluation function may not receive sufficient attention and expertise. Like much of the Mashrou3i implementation model, the employment data collection process is built around personal/professional relationships rather than a more systematic, rigorous, and independent performance-monitoring plan. The HP-UNIDO data collection approach can be laborious, and the reported data can be open to possible bias and misinterpretation, which limits its credibility and utility for program decision-making. 4d) Based on the data collected by HP-UNIDO, can cost per job created – including management costs and in-kind contribution – be accurately estimated? Provide simple benefit/cost calculation to determine whether the benefits of the jobs created exceeded costs (e.g. compare cost per job with wage data). Conclusion: The cost per job created can be estimated, but the accuracy of such a calculation is subject to a high degree of uncertainty due to questions about which costs to use, the reliability of employment data, and the attribution of jobs to program. A benefit/cost calculation can also be made, but includes a similar level of uncertainty. 6 UNIDO measurement of employment supported through SBSs is restricted to those start-ups that received ‘deep-dive’ coaching through the SBSs. The employment data reported by UNIDO only reflect jobs that resulted from HP-LIFE training/coaching, or support to start-ups and existing enterprises. They do not include any other employment that might have been encouraged by SBSs. 9 Findings/Evidence:  Based on the figure of 1,035 jobs created and the USAID contribution of $1.5 million, the cost per job is $1,449. Based on the higher employment figure of 1,654, which includes 44 job placements and 575 anticipated jobs, the cost per job drops to $907.  If the full activity cost including the HP and Italian Government contribution ($1,847,748) is included in the calculation, the estimated cost/job reported increases to $3,235 based on 1,035 jobs created. If one adds anticipated jobs and employment placements, the figure drops to $2,024. A more accurate measure would look at total cost per job created and would include not only Mashrou3i Program costs, but also all costs incurred by entrepreneurs to create new jobs.  Assuming a median monthly salary of 600 TDN (or approximately $300 USD), the investment in this job creation program would achieve full payback from three to 10.8 months, depending on the cost/job calculation used. Of course, the payback period using the full costs per job created (i.e. the cost/job including costs incurred by entrepreneurs to create new jobs) would be higher. Recommendations 1. Continue support for job creation in Tunisia’s Interior Regions: Given the high levels of unemployment and poverty in Kairouan, Kasserine, Le Kef, and Sidi Bouzid, and the potential negative repercussions on security and stability in Tunisia and the region, USAID should continue to provide assistance for job creation activities in these governorates. As further explained in other recommendations, USAID should consider mechanisms for providing business support and advisory services that rely more heavily on private service providers and well￾qualified business coaches who have actual private sector business experience. 2. Continue to work with HP-LIFE: It is recommended that USAID continue to work with HP￾LIFE on two levels. First, in expanding and improving the HP-LIFE online courses and face-to￾face training and coaching for young entrepreneurs. Second, in light of the very positive working relationship Mashrou3i has established with the Ministry of Higher Education in general, and the Higher Institutes of Technological Studies (ISET) in particular, USAID should continue to work with the Ministry and with the ISETs to integrate the HP-LIFE curriculum into existing entrepreneurship courses at the ISETs, and possibly other higher education institutions, and to actively promote use of the e-learning program across the interior governorates. Such collaboration with the Ministry will complement the ongoing support by the Business Reform and Competitiveness Project (BRCP) for the establishment and strengthening of career centers at institutions of higher learning. 3. Work through private business support service providers to provide enterprise development, entrepreneurship, and employment services: In light of a) the significant network of private business services providers in Tunisia, including those that have been nurtured by USAID-funded programs in the past, as well as b) the need for private enterprises to be supported by people and organizations who understand business, USAID should work through indigenous, private service providers and organizations to expand technical assistance and training for enterprise development, entrepreneurship, and employability in any future programs. 4. Support entrepreneurship development programs that provide coaching by experienced business experts: Given that business coaching by qualified business professionals has been identified in Tunisia and world-wide as a critical element in the success of many new ventures, and that new businesses in Tunisia face a plethora of administrative and financial obstacles, 10 USAID should support indigenous systems and programs that provide on-going business coaching by qualified, experienced small-business experts over a three- to four-year period. 5. Reorient Regional Business Centers and API Regional Offices: As Tunisian parastatal regional business support organizations do not have the background or qualifications to advise entrepreneurs and enterprise owners on how to run a business, USAID should encourage the GOT to consider the policy, operational, and service benefits of transforming Regional Business Centers and API offices into business development advocates. Rather than business service providers, such organizations could refocus their efforts on helping entrepreneurs and enterprises obtain necessary government approvals and gain access to privately supplied business advisory services and finance. Such an advocacy role may help to overcome the administrative bottlenecks identified in KIIs with young entrepreneurs. 6. Improve the business enabling environment: In conjunction with its ongoing, critical support for tax and customs reforms in Tunisia, USAID should continue to encourage the GOT to improve the overall business climate by reducing the administrative burdens on entrepreneurs and enterprises, including the reform of existing labor and bankruptcy codes. These are two areas identified in KIIs as serious obstacles to job creation in Tunisia. 7. GOT Employment Programs: In conjunction with Recommendation 6, USAID should coordinate its efforts with other donors to encourage the GOT to review and revise employment programs that inadvertently discourage workers from seeking full time employment in the private sector. 8. Job Creation Data Collection and Analysis: In order to strengthen and ensure the reliability, validity, and utility of the Mashrou3i job creation data, and in light of the expansion of services to four additional governorates, it is recommended that UNIDO employ a qualified local Monitoring and Evaluation/data-base professional to establish and operate the Mashrou3i data collection and analysis system, and should also utilize an independent Tunisian research organization to implement qualitative and quantitative data collection and analysis. Reforms, such as those suggested in Recommendations 6 and 7, will complement and reinforce whatever efforts USAID undertakes to fund new entrepreneurship, employability, and enterprise development programs. 11 EVALUATION PURPOSE & EVALUATION QUESTIONS USAID commissioned an independent and external final performance evaluation of the “Tackling Youth Employment in Tunisia” (Mashrou3i) activity, which is implemented through a Global Development Alliance (GDA) managed by USAID’s Middle East Bureau with Hewlett Packard (HP), the United Nations Industrial Development Organization (UNIDO), and the Government of Italy (GOI). Launched in late 2012, the GDA aims to create job opportunities for young Tunisian women and men in four vulnerable regions in the Center West of Tunisia: Le Kef, Kairouan, Kasserine, and Sidi Bouzid. It is the only USAID-sponsored job creation activity in Tunisia that exclusively targets areas outside of the coastal regions. Over the life of the activity, Mashrou3i aims to reach approximately 10,000 aspiring and existing entrepreneurs, with a focus on youth, and create at least 1,000 direct jobs in the targeted governorates. The purpose of the evaluation is to understand the extent to which the activity has been successful in creating sustainable employment/livelihoods in the vulnerable interior regions of Tunisia where the activity is implemented. Findings will assist USAID/Washington, USAID’s Tunisia Office, implementing partners, and other relevant stakeholders to understand the effectiveness of the HP-UNIDO approach, and will help partners better understand how to support the Government of Tunisia’s (GOT) vision to promote private sector development through small and medium enterprise (SME) creation and growth. The evaluation was conducted in March-May 2016 with fieldwork taking place from March 23 to April 8, 2016 in Tunis, Sousse, and Kairouan, Tunisia. The evaluation team consisted of four consultants:  Dr. James M. Statman – Team Leader  Mr. Richard Rousseau – Economic Growth Technical Expert  Dr. Medhi Ben Braham – Monitoring and Evaluation Specialist  Ms. Samia Boussaid – Administration and Logistics Manager Evaluation Questions This report presents responses supported by evidence to the following evaluation questions, in order of priority, and sub-questions: 1) To what extent has the HP-UNIDO implementation approach been effective in creating sustainable employment in the interior regions in which the activity is implemented? a) Which aspects of the implementation approach have been the most effective in creating sustainable employment and why? b) Which populations have benefited most from the activity and why? In particular, the evaluation team should consider how youth in each of the four regions and women have differentially benefited from and been reached by the activity. c) How were participants identified? Can anything be concluded about the general characteristics (e.g., educational or professional background) of participants vs. non￾participants in the HP-UNIDO program? 12 d) How did the HP-UNIDO approach of working directly with local communities influence individual motivations or job training and placement outcomes? 2) Relative to a traditional contract model, were there any benefits (e.g., lower costs, greater leverage) to using the GDA approach for the implementation of the HP-UNIDO activity? Were there any externalities (positive or negative) of using: a) a GDA approach that facilitates external resource partners? b) a Public International Organization (PIO) approach like UNIDO’s that attracts the resources of additional donors? 3) To what extent did UNIDO's cooperation with existing Tunisian Small Business Support (SBS) organizations contribute to employment outcomes? Consider UNIDO's business model, years of experience in working with indigenous entities that are government funded (parastatals), and cost structure. 4) To what extent can the employment information collected by the HP-UNIDO activity be independently validated and be deemed credible? a) How does HP-UNIDO define and measure employment creation? b) Given the methodology and procedures used to collect employment information, to what extent are the results reported from the routine activity monitoring likely to be over-estimated or under-estimated? c) What methodologies, processes, systems or structures have facilitated or impeded the collection of high quality and credible employment data? d) Based on the data collected by HP-UNIDO, can costs per job created (including program management and in-kind) be accurately estimated? To the extent feasible, provide a simple benefit-cost calculation as to whether the benefits of jobs created exceeded costs (e.g., such as by comparing estimated cost per job with wage data). 13 PROJECT BACKGROUND Mashrou3i is implemented through a GDA managed by USAID’s Middle East Bureau with HP, UNIDO, and GOI. UNIDO is responsible for implementing the project in Tunisia. USAID provided $1,500,000, and HP and the GOI provided an additional $1,847,748 in cost sharing, bringing the total budget to $3,347,748. According to the Program Contribution Agreement between USAID and UNIDO, the purpose of Mashrou3i is to “create opportunities for young Tunisian women and men in four vulnerable regions, Le Kef, Kairouan, Kasserine, and Sidi Bouzid.” The program was envisioned to directly and indirectly promote the creation of new, and the growth of existing, companies and entrepreneurial endeavors. As per the Agreement, the project seeks to leverage UNIDO’s expertise in the field of private sector development and history of work with local partners to stimulate enterprise creation and development, with a focus on the agro-food and environment sectors. The project was developed in line with the GOT’s strategy to promote private sector development through SMEs creation and growth by boosting employment, particularly among youth. The project included a capacity building component for national and regional service providers to strengthen the support provided to SMEs. The goals of the project are to: (1) reach roughly 10,000 aspiring and existing entrepreneurs, including many youth, and (2) create at least 2,000 jobs, including direct, indirect and temporary jobs.7 The Mashrou3i award commenced on October 1, 2012 with a planned completion date of March 31, 2015. The project received a no-cost extension in February 2015 extending the period of performance to December 31, 2015, and it received a second no-cost extension in December 2015 that extended the completion date to September 30, 2016. The Mashrou3i’s program is organized into two complementary but distinct technical components: 1. Component One: Support for Small Business Support Organizations The HP-UNIDO program strategy utilizes local parastatal Small Business Support organizations (SBSs) to deliver Mashrou3i services within the four target governorates. This approach structurally embeds the program within these existing institutions – and builds upon their knowledge and working relationships with local businesses, banks and government institutions - as a means of promoting sustainability and cost-efficient service delivery. SBS capacity building under Mashrou3i was implemented through the delivery of a limited amount of information technology (IT) equipment and through training and technical support for participating SBS organizations and staff. Training and technical assistance support consisted of an initial Strengths, Weaknesses, Opportunities, and Threat (SWOT) analysis workshop that enabled the local implementing partners to assess their institutional strengths and limitations; and training workshops in fostering “green businesses” and on models of support for local small businesses. 7 The Program Contribution Agreement, evaluation Statement of Work and Mashrou3i “Results Chain and Main Expected Results” chart all refer to a target of 2,000 jobs including indirect and seasonal jobs. However, subsequent to fieldwork the evaluation team was informed by USAID/Washington that shortly after the start of Mashrou3i implementation, USAID and UNIDO agreed to revise the target to 1,000 direct jobs, dropping the inclusion of seasonal or indirect employment. This report therefore uses this as the target. 14 Related to component Two (below) training was also delivered on the HP-LIFE program and curriculum to SBS personnel and to trainers and coaches recruited from local and national education and service institutions, thereby strengthening the capacity of these institutions and building the cadre of HP-LIFE program providers. 2. Component Two: Support to Start-ups and Existing Enterprises 2.1 HP-LIFE Entrepreneurship Training and Coaching: is built around the HP e-learning platform comprised of 25 interactive modules addressing core business, finance and entrepreneurial knowledge and skills. Individuals can enroll without cost in one or more modules and receive certification for those completed. Developed by HP initially in Spanish, Mashrou3i translated the HP-LIFE curriculum into French and Arabic for use in Tunisia. Individuals typically learn of the HP-LIFE program through the Mashrou3i Facebook page and other social media, from friends and associates, through local implementing partners or other youth services organizations, through outreach informational sessions conducted by UNIDO at colleges, job fairs or other events and, in some instances, as part of the curriculum in required courses in entrepreneurship at the local Higher Institutes of Technological Studies (ISET) branches. Young people who have successfully completed at least three modules can then apply to participate in a three-day, face-to-face group training experience conducted by UNIDO-trained personnel at a local participating business center or SBS organization. This process, which is aimed at helping the aspiring entrepreneur to create a viable business idea, also links the participants with trainers and others at the SBS, with local businesses, and with their peers to provide access to on-going support, coaching and advice as they move to implement their idea. Among participants who complete the three-day training, the trainer identifies those participants whose business ideas seem to be of particular merit and have a good likelihood of success. Participants may then be nominated by the trainer to participate in an additional four-day intensive workshop aimed at assisting them to create a comprehensive business plan to direct and guide the development of their enterprise. The trainees then continue to work with their local implementing partner organization to implement the business plan, register their start-up, access finance as required, and commence operations. 2.2 Support to Start-Ups and Existing Businesses in the Regions: Following a formal competitive process in which young people from the four regions submitted ideas for new businesses, Mashrou3i, through its local partners, initiated an intensive on-going process of coaching, training and support to facilitate the development of viable start-ups. In addition, the partner entities helped identify local businesses in each locality with potential for expansion and job growth. These businesses then receive intensive one-on-one technical assistance and support from UNIDO experts in SME finance, marketing and communication and other services from the local SBSs, addressing their specific business needs and opportunities. Component 1 and Component 2.2 are managed by UNIDO in Tunis, Tunisia. Component 2.1 is managed from UNIDO’s headquarters in Vienna, Austria, which is also responsible for overall project management and reporting. The chart on the following page (Figure 3.1) shows the flow of participants through Mashrou3i’s components. 15 Figure 3.1: Tackling Youth Employment in Tunisia Mashrou3i8 8 UNIDO initially approved 44 start-ups for ‘Deep-dive’ Business Coaching; 42 of these are included on detailed UNIDO lists provided to the evaluation team. UNIDO indicates that 36 start-ups are currently active, and estimates that 30%-40% of these attended HP-LIFE FTF training and HP-LIFE business coaching. HP‐LIFEOn‐line Courses 12,000 individualparticipants HP‐LIFEFace to Face Training 1,369 participants TACKLINGYOUTH EMPLOYMENT IN TUNISIA MASHROU3I HP‐LIFE Business Coaching 154 participants 2.2. Support to Start‐ups and Existing Enterprises 2.2.2 Deep‐diveBusiness Coaching for Start‐upsin collaboration with SBSs 42 participants of which: ‐7 HP‐LIFE Training only ‐2 HP‐LIFECoachingonly ‐16 HP‐LFETraining and coaching ‐17 Other 1. Support for Small Business Service (SBS) Organizations 23 institutionalparticipants Other, i.e. Non participantsin HP‐LIFETraining and/orCoaching 2.1. HP‐LIFE 2. Support for Entrepreneurs and Enterprises 2.2.1 Supportto Existing Enterprises 52 enterprises 16 METHODOLOGY & LIMITATIONS Evaluation Methodology Data Collection Methods The evaluation drew upon qualitative and quantitative methods to obtain data and evidence sufficient to support conclusions and recommendations addressing the evaluation questions (see Annex VI for the Evaluation Design Matrix). Each data collection method is described and discussed in detail below. As noted in the Limitations section (below), security restrictions necessitated a reconfiguration of data collection methods, preventing all planned field-based KIIs, observational visits and FGDs in Le Kef, Kasserine and Sidi Bouzid. With assistance from UNIDO, the evaluation team compensated for these limitations through training observation, KIIs and group meetings in Sousse with participants from the three regions that were inaccessible. The team also conducted additional KIIs in Tunis with representatives of partner organizations from these regions who were able at short notice to travel to Tunis. The evaluation team wishes to acknowledge the extraordinary level of cooperation, openness and professional support provided by UNIDO that enabled the evaluation to proceed despite these significant challenges. Document Review The evaluation team conducted a comprehensive desk review to obtain background information on substantive issues, contextualize research questions, focus and prioritize primary research by highlighting informational gaps, and identify key stakeholders. Documents reviewed included the agreement documents, progress reports, presentations, results data, work plans, participant lists, survey data and publicity materials. In addition to documents received immediately prior to initiation of fieldwork, HP and UNIDO in Tunis and Vienna were immediately responsive and forthcoming in furnishing and clarifying documents. Key Informant Interviews (KIIs) The evaluation team conducted semi-structured individual and, at times, small group interviews with key stakeholders, using KII guides (see Annex II). The interviews were semi-structured to allow for an open framework, conversational communication and more detailed follow up questions where necessary. KIIs lasted approximately one hour. In cases where respondents did not have this time available, the interviews were abbreviated and only central themes were explored. In total, the evaluation team conducted 41 KIIs (39 in-person and two via telephone). Annex VII presents the list of KIIs. Focus Group Discussions (FGDs) and Group Meetings The evaluation team conducted one FDG at the Cyber Park in Kairouan (one of 18 such facilities nationally established by the Ministry of Technology and Communications) with seven beneficiaries in attendance (two males and five females). Three evaluation team members were present at FGD. One guided the discussion in French and Arabic and the others took notes and prepared a summary for review (the FGD was not recorded or transcribed). See Annex III for the FDG instrument. The evaluation team conducted four group meetings in Sousse with HP-LIFE participants from the four targeted regions who had already completed the four-day business plan training and were receiving further advanced coaching from HP-LIFE trainers. See Table 4.1 below for details. Training Observation The evaluation team was able to observe HP-LIFE training sessions with aspiring entrepreneurs from all four target regions and discuss the training with the trainer and participants. 17 Table 4.1: Group Meetings held in Sousse Participant’s Location Males Females Total Le Kef 3 0 3 Kairouan 4 2 6 Kasserine 2 1 3 Sidi Bouzid 3 2 5 Total 12 5 17 Business Site Visits The evaluation team conducted five site-visits to businesses in and around Kairouan whose owner or manager had participated in one or more Mashrou3i program components or events. These businesses included a plastic bottle recycling business that exports to China; an ice cream manufacturing plant; a spice and pickling business; a center for autistic and other mentally and physically challenged children; and a computer/IT services company. Online Survey The Evaluation Team conducted two separate surveys using Survey Monkey software in order to obtain data to answer the four evaluation questions. The first survey covered the regional business support organizations that were part of Program Component 1 and that assisted in delivery of services to SBS organizations (see Annex V for the survey instrument). The second survey targeted individual entrepreneurs and enterprises that had received support from one or more of the activities under Program Component 2 (see Annex IV for the survey instrument). Given the severe restrictions on travel within three of the four governorates assisted by the activity, these surveys became a very important method of data collection. The surveys were distributed by e-mail to addresses provided by UNIDO. E-mail reminders were sent several times between the initial distribution date of March 25, 2016 and the closing date of April 16, 2016; telephone follow-up was also conducted with a number of existing enterprises. The first survey, (i.e. the one of regional business support organizations), generated 13 valid responses from the 23 organizations surveyed; there were no invalid responses. The second survey of individual entrepreneurs and enterprises generated a total of 272 valid responses out of a total of 1,463 program participants (individuals and enterprises) to whom the survey questionnaire was sent; 30 responses were deemed invalid (27 were duplicates and 3 were not found on UNIDO lists of participants). Table 4.2 below provides detailed information concerning survey populations, samples, and response rates. Table 4.2 Summary of Survey Populations, Samples, and Response Rates Comp. # Program Activity Type of Recipients # Supported by Program # Surveyed # of Valid Responses Response Rate 1 Institutional Capacity Building Public Institutions 23 23 13 57% Subtotal Component 1 23 23 13 57% 2.11 Online plus face-to-face training Individuals 1,215 1,215 175 14% 2.12 Online plus face-to-face training and coaching Individuals 154 154 57 37% 2.2 Start-ups Enterprises 42 42 17 40% 2.3 Existing Enterprises Enterprises 52 52 28 54% Subtotal Component 2 1,463 1,463 272 19% Total (Comp. 1 and 2) 1,486 1,486 285 19% Notes: Sum of individual and enterprise valid responses (277) for Component 2 exceeds the 'Subtotal Component 2' for '# of Valid Responses' (272) due to the participation of several entrepreneurs in both HP-LIFE coaching and the support to start-ups activity. 18 The evaluation team also requested contact information from UNIDO and HP for the individuals who only took the online courses, but who did not enroll in either the Face to Face Training or Coaching offered by HP-LIFE. HP explained that such information was not available due to systems changes that took place after the 2015 division of HP into two separate operating enterprises. HP estimates that about 12,000 individuals enrolled in the HP-LIFE online courses in Tunisia. Limitations There were several limitations that should be noted. Security Constraints: Prevented the evaluation team from conducting data-collection site visits to Le Kef, Kasserine, and Sidi Bouzid. The team was able to conduct two site visits to Kairouan but had to return to Sousse before nightfall. These security restrictions significantly reduced the number of FGDs with aspiring entrepreneurs, observational visits to business and local SBSs; and KIIs with entrepreneurs, SBS and college personnel and other program beneficiaries. With assistance from UNIDO the evaluation team was, however, able to observe an HP-LIFE training course in Sousse and conduct group meetings with some participants from each governorate. Further, representatives from local implementing partners were able to travel to Tunis for individual and group KIIs. This enabled the evaluation team to obtain sufficient data to adequately address each of the evaluation questions and sub-questions but certainly decreased the richness, variability and reliability of the information and constrained inter-regional comparisons. Survey size and KIIs: Due to time and budget constraints, the evaluation team was not able to conduct a statistically significant, random sample survey of individuals and enterprises that received support from Mashrou3i. However, survey questionnaires were sent to the total universe of those beneficiaries for whom the evaluation team had valid e-mail contact details. Out of a total survey population of 1,463 program participants, 272 valid responses were received. In addition, a separate survey was conducted of 23 local SBS organizations supported by Mashrou3i, and 13 valid responses were received. While the survey was not scientifically and statistically valid, it did cover a sizable share of the beneficiaries and yield useful information. Similarly, individual and group KIIs and meetings with program beneficiaries were opportunistic; organized through UNIDO based on availability and – for those from the target regions where the evaluation team could not visit – willingness and ability to travel to Tunis on short notice to meet with the team. To mitigate these constraints, the evaluation team triangulated and verified data from various sources whenever practicable and paid particular attention to any out-lying or unusual perspectives or opinions. 19 FINDINGS AND CONCLUSIONS This section presents the evaluation findings and conclusions organized by each of the evaluation questions (EQ) and sub-questions. For each, we first present the conclusion addressing the question and then the specific findings or evidence supporting that conclusion. EQ1. To what extent has the HP-UNIDO implementation approach been effective in creating sustainable employment? Conclusion: Considered in the context of political, social, and economic instability during 2012-2016 and the challenges posed by the current security situation, the Mashrou3i project has made considerable progress in generating sustainable employment for youth in the four interior governorates in which it has been implemented. Findings/Evidence: 1.1 From October 1, 2012 through March 31, 2016 UNIDO reports having produced 1,035 jobs, or 103.5 percent of the life of activity target of 1,000 direct jobs.9 UNIDO further reports having placed 44 HP-LIFE trainees in jobs, and projects that within the following 12 months an additional 575 jobs will be created from current program activities, constituting a grand total of 1,654 jobs created, placed, or in-process. This represents 165.4 percent of the life of activity jobs target. The evaluation team conducted a detailed analysis of the job data provided by UNIDO to produce Table 5.1 below and confirmed that supporting documentation exists for the jobs numbers included in the table. Table 5.1 Jobs Created and Projected as Reported by UNIDO 3/2016 (4) # Activity/Type of Enterprise Jobs Created through March 2016 Note HP￾LIFE Trainees Placed through March 2016 Note Additional Jobs to be Created as Reported by UNIDO Note Totals 1 HP-LIFE Support to Entrepreneurs: 471 44 307 822 Start-ups enabled/facilitated by HP-LIFE program 121 (1) 121 New positions hired by start-ups enabled by HP-LIFE Program 350 (1) 307 (1) 657 9 The Program Contributing Agreement, evaluation Statement of Work and Mashrou3i “Results Chain and Main Expected Results” chart all refer to a target of 2,000 jobs including indirect and seasonal jobs. However, subsequent to fieldwork the evaluation team was informed by USAID/Washington that shortly after the start of Mashrou3i implementation, USAID and UNIDO agreed to revise the target to 1,000 direct jobs, dropping the inclusion of seasonal or indirect employment. This report therefore uses this as the target. 20 Table 5.1 Jobs Created and Projected as Reported by UNIDO 3/2016 (4) # Activity/Type of Enterprise Jobs Created through March 2016 Note HP￾LIFE Trainees Placed through March 2016 Note Additional Jobs to be Created as Reported by UNIDO Note Totals Found employment elsewhere 14 (1) 14 Hired by existing enterprises 30 (1) 30 2 Deep-dive business coaching support for start-ups in cooperation with local organizations (5): 135 268 403 Start-ups established as a result of business coaching 16 (2) (2) 16 New positions hired by start-ups established from business coaching 119 (2) 268 (2) 387 3 Support to Existing Enterprises 429 (3) Additional jobs expected, but not quantified 429 Totals 1,035 44 575 1,654 Notes: (1) 'Copy of Survey_HP-LIFEentrepreneurs_16_March_2016_2.xlsx', UNIDO, March 2016 (2) 'Startups_job creation_data_Jan_2016', UNIDO, January 2016 (3) 'Tableau_accompagnement EE Mars 2016', UNIDO, March 2016 (4) 'Copy of Mashrou3i project - JOBS Data -mars 2016', UNIDO, March 2016 (5) According to UNIDO, these start-ups initially received training, but not coaching under HP-LIFE. Additional support has been provided by experts through local partners. 1.2 The process of creating jobs through the operational establishment of new enterprises cannot be expected to immediately generate employment. The Mashrou3i March 2016 Progress Report states that of 1,136 entrepreneurs participating in HP-LIFE training, the majority (57 percent) were in the project (idea) phase; 18 percent in the business plan preparation phase; 11 percent were seeking funding; one percent recently received funding; five percent were in the business start-up phase; and eight percent had established operational enterprises. This suggests an active pipeline of entrepreneurial endeavors moving towards the establishment of start-ups, a moment defined by company registration. 1.3 Feedback from Mashrou3i participants (N=188) supports the assertion of program success. Fifty￾three percent (N=99) of HP-LIFE beneficiaries responding to the online evaluation survey indicate that Mashrou3i helped them to start a business; 11 percent (N=20) that the project helped them expand an existing business; 10 percent (N=19) that it helped them to find a job; and 27 percent (N=50) that it had no impact upon their employment situation (Figure 5.2, page 26). Although survey respondents cannot be considered to constitute a statistically significant sample of HP-LIFE 21 participants, the fact that almost 74 percent of respondents assert that the program was linked to an improvement in their employment situation is evidence for the efficacy of the model. While, as noted above, the majority of respondents assert that their employment situation has improved, Figure 5.1 below shows that 58 percent (N=104) of online survey respondents state that their income level had not changed; 40 percent (N=71) indicate that the program resulted in an increase in income; and two percent (N=4) report a decrease (Total N=179). Based on the evaluation team’s meetings with a number of entrepreneurs, it is likely that some portion of those respondents who reported that Mashrou3i had aided them in starting a project have not yet seen any impact on their incomes. 1.4 The online survey also asked program participants “How many persons have you hired as a result of Mashrou3i”? Slightly more than half of the respondents (85 out of 163) noted an increase in jobs within their businesses, some greater than ten new positions: 36 percent (N=58) reported an increase of 1-5 positions following participation in Mashrou3i; nine percent (N=15) reported an increase in 6- 10 employees; seven percent (N=12) reported more than 10 new employees; and 48 percent (N=78) reported no change (Figure 5.3, page 26). As noted, about half of respondents reported no jobs growth and several stated that the increases in employment should be attributed to their personal efforts, to other factors or to a combination of these. Some noted specific jobs numbers: “50” and “20 new jobs”; “six permanent and 100 seasonal”; “16 jobs (of which five are professional)”. Several offered general endorsements of the project or noted that it generated hope and proof that despite the often bleak economic context “job creation is possible.” 1.5 Several young people who were in the process of actually launching their business stressed that they “did not know how to manage a company” prior to their involvement with HP-LIFE and that the training and coaching had been very helpful. But still they commented that there is “a big difference between theory and actually operating a business” and despite being in the program they felt that they were “facing our problems alone.” As one participant stated: “anyone can start a company, but the challenge is survival.” In the face of what they see as a lack of business culture and rampant corruption “even in the Business Center” they stress the importance of finding experienced people 22 who can guide them through the challenge of business survival. (Source: KIIs, FDG and group meetings) 1a) Which aspects of the implementation approach have been the most effective in creating sustainable employment and why? Conclusion: Based on UNIDO data for all participants, the evaluation survey and KIIs, HP-LIFE support to young entrepreneurs and Mashrou3i support to existing enterprises have been most effective in creating jobs. This may be related to the more intensive training and coaching associated with these activities. Findings/Evidence: 1a.1 Mashrou3i generates jobs though several distinct and somewhat complementary activities. Of jobs created to date, UNIDO reports that: 471 jobs (46 percent) were created by the HP-LIFE support to young entrepreneurs; 135 jobs (13 percent) created were attributed to the Mashrou3i “Deep Dive” business coaching activity in conjunction with local business support organizations; and 429 (41 percent) through support to existing enterprises (see Table 5.1, page 20). 1a.2 Of the 575 jobs UNIDO projects to be created in the future, as a result of its activities through March 2016, 307 (53 percent) are attributed to HP-LIFE support to young entrepreneurs and 268 (47 percent) to “Deep Dive” coaching with SBOs. No estimate was given to the evaluation team for additional jobs in existing enterprises, although UNIDO does anticipate some further growth. 1a.3 More than 12,000 Tunisians have taken at least one online HP-LIFE e-learning module. HP-UNIDO reports that participants take an average of 13-14.5 of the 25 course modules, and complete 90 percent of the courses initiated. The significant uptake of HP-LIFE e-learning modules suggests a continued market for entrepreneurship and business management training. For the vast majority of Tunisian participants, the online HP-LIFE e-learning process is their only formal engagement with Mashrou3i. Results data from HP-LIFE worldwide suggest that the experience of completing the interactive modules provides concretely useful business knowledge and skills10. However, because of changes to the HP corporate structure and concurrent changes to the HP-LIFE online platform, HP is unable to readily disaggregate information by country, nor can the current system report by user location or governorate. The evaluation was also unable to find a means to identify and obtain information on project outcomes from individuals whose only engagement with Mashrou3i was online, nor to identify those who participated in the modules as part of a college entrepreneurship course curriculum. As course adoption continues to increase and online use continues, the ability to collect information from these users becomes increasingly important. 1a.4 Additional support for the utility of HP-LIFE derives from a 2015 survey UNIDO conducted of Tunisian participants who completed the three-day face-to-face training on the HP-LIFE curriculum. This training experience establishes a professional relationship between the young aspiring entrepreneurs, the trainers/coaches from the local partner institutions and their peers and serves as an entry point for further coaching support and possible participation in the subsequent Mashrou3i four￾day business planning workshop. Participants reported that the online e-learning modules and group 10 The FY 2015 “Rolling Survey” of HP-LIFE e-learners (from all countries) taken six months subsequent to HP-LIFE registration indicates that 78% stated that they had applied the information or skills acquired through the e-modules and 84% agreed that the modules were helping them achieve their professional goals. 23 training were helpful in enabling them to create a business idea, and to enhance their business and finance skills. According to a survey implemented by UNIDO, of 376 aspiring Tunisian entrepreneurs (HP-LIFE participants who had taken the three-day face-to face (FTF) training but were not current business owners) surveyed, 305 (81 percent) stated that they actively utilize business concepts and techniques they acquired from the training. This group reported establishing 63 new enterprises generating 339 direct jobs. Of the 376 respondents, 91 (24 percent) indicated that the training improved the feasibility of their project. An additional 120 respondents (32 percent) stated that the training confirmed their business idea, and 31 (8 percent) reported changing their business idea as a result of the training; five of these then went on to find employment. Similarly, of 66 established entrepreneurs (who had also completed the FTF training) surveyed by UNIDO, 59 (89 percent) asserted that they actively use the knowledge and skills acquired through the program in their work which has resulted in the development or improvement of their businesses. These entrepreneurs credited HP-LIFE with increasing business efficiency (32 percent), enabling them to acquire additional clients (26 percent), increasing revenue (20 percent), decreasing costs (16 percent), hiring new employees (six percent) and with the creation of 21 direct jobs. 1a.5 Group meetings with young entrepreneurs from all four governorates (N=17) who had completed the four-day business plan coaching showed a broad range of technical/professional backgrounds and progress. Many stated that they faced serious challenges in access to capital, particularly for operating expenses, and several described major bureaucratic hurdles in obtaining necessary permits or registration (for others this was not a problem). About one-third had business cards and business brochures or other marketing materials and almost all were able to enthusiastically and quickly describe their venture and “unique value proposition” in what the trainer called their “elevator talk.” Of the 17 beneficiaries, all but one seemed clear on her/his business plan, marketing strategy, market segment and immediate next steps and all expressed the belief that HP-LIFE had enabled them to learn critical business skills and concepts. Still many seemed somewhat stalled by a business environment that is not particularly encouraging to young entrepreneurs and what they described as a generalized lack of entrepreneurial culture in their regions. As one group put it, “we have no role models or success stories” upon which to draw inspiration or guidance. 1a.6 As Figure 5.2 shows below, evaluation survey data indicate that the participants identified by UNIDO as ‘start-ups’ were more likely to say that the program had no impact on their personal employment situation than participants identified by UNIDO as either HP-LIFE training or HP-LIFE coaching participants. Thirty-six percent of those participants identified as ‘start-ups’ by UNIDO said there was no impact on their employment situation as a result of Mashrou3i, whereas 26 percent of HP-LIFE training participants and 25 percent of HP-LIFE coaching participants said there was no impact. 24 7% 10% 2% 10% 50% 53% 65% 53% 7% 11% 8% 11% 36% 26% 25% 27% Start-ups (N=14) HP-LIFE F2F Training (N=176) HP-LIFE F2F Business Coaching (N=51) Total (N=188) Figure 5.2 How did Mashrou3i affect your own employment status? No impact Helped me expand an existing business Helped me start a business Helped me find a job 1a.7 However, a somewhat different picture emerges when looking at employment creation. Below Figure 5.3 indicates, as would be expected, start-ups outperform those who have only participated in HP-LIFE training. 33% 11% 50% 35% 48% 19% 67% 38% 48% 36% 19% 0% 8% 10% 9% 29% 22% 4% 8% 7% Existing Enterprises (N=21) Start-ups (N=9) HP-LIFE F2F Training (N=136) HP-LIFE F2F Business Coaching (N=40) Total (N=163) Figure 5.3 How many persons have you hired as a result of Mashrou3i? >10 6-10 1-5 0 1b) Which populations have benefited most from the activity and why? How have women and youth in the four regions differentially been reached and benefited? Conclusion: The evaluation found no data suggesting differential program outcomes by region or gender. UNIDO data on all participants indicate that women have participated in the program at slightly higher rates than men, although this varies somewhat from region to region, as shown in Figure 5.4 below. Eighty-seven percent of all program participants are age 34 or less. UNIDO, however, does not disaggregate jobs created by gender or age. The evaluation survey found that 74 percent of women 25 reported a positive impact from participation in Mashrou3i (i.e. found a job, helped start a business, or helped expand an existing business), while 73 percent of men reported a positive impact. Findings/Evidence: 1b1. Data provided by UNIDO indicate that women constitute slightly more than half of Mashrou3i participants with small variation across governorates (see Figure 5.4 below). The evaluation survey data show that Mashrou3i has benefited males and females in roughly equal proportions with just over 70 percent of both groups of respondents reporting a positive impact as a result of program assistance as shown in Figure 5.5 below. 26 7% 14% 9% 46% 51% 48% 20% 8% 16% 27% 26% 27% Male (N=137) Female (N=72) Total (N=209) Figure 5.5 Mashrou3i Impact on Employment Status by Gender No impact Helped me expand an existing business Helped me start a business Helped me find a job 1b.2 According to UNIDO data on program participants compiled by the evaluation team, approximately 87 percent of HP-LIFE training and coaching participants are between the ages of 20 and 34 years (highlighted in green), as shown in Table 5.2 below. Table 5.2 Participants in HP LIFE Training and Coaching Activities by Age (1) Age Group (years) # in age group % of total cumulative # up to and including this age group cumulative % of total 15-19 3 0.2% 3 0.2% 20-24 146 8.3% 149 8.5% 25-29 645 52.0% 794 60.5% 30-34 376 26.6% 1170 87.1% 35-39 130 8.7% 1300 95.8% 40-44 35 2.8% 1335 98.6% 45-49 12 0.8% 1347 99.4% 50-54 5 0.4% 1352 99.8% 55-59 1 0.1% 1353 99.8% 60-64 1 0.2% 1354 100.0% 65-69 0 0.0% 1354 100.0% Totals 1354 100.0% 1354 100.0% (1) Includes participants in HP-LIFE training and coaching conducted by staff, three regional business centers in Kairouan, Kef, and Sidi Bouzid, and ODCO in Kasserine and other local institutions. 27 1b.3 Descriptive data from the evaluation online survey provides characteristics of HP-LIFE program beneficiaries. The vast majority (94 percent) of respondents report having post-secondary education in a variety of fields of study, as shown in Figure 5.6 below. A large number of participants studied IT, engineering and other technical or scientific fields, manufacturing, and “other services.” As illustrated below in Figure 5.7, a majority of respondents (59 percent) report having been unemployed for more than one year after obtaining of their degree; 10 percent report having been unemployed for 6 months to one year; 15 percent report having been unemployed for 1-6 months; and 16 percent report having been unemployed for less than one month. 16% 14% 11% 59% Figure 5.7 Period of Unemployment after Receiving Degree (Total N=253) <1 month (N=40) 1-6 months (N=36) 6 months -1 year (N=27) >1 year (N=150) 28 Further, as the following Figure 5.8 shows, 48 percent of survey respondents described themselves as currently unemployed;11 37 percent as self-employed; and 15 percent as employed. 15% 48% 37% Figure 5.8 Employment Status (Total N=263) Employed (N=40) Unemployed (N=127) Self-employed (N=96) Mashrou3i participants with whom the evaluation team met in group meetings studied a broad range of subjects including engineering and IT, but also optometry, physical therapy, agronomy, communications, journalism, and humanities. 1b.4 In KIIs, UNIDO personnel stated that the “ideal” HP-LIFE beneficiary (in terms of benefiting most and likelihood of success) would be a young person in the “interior regions” of Tunisia with a post￾secondary degree in engineering, IT or other technical subjects, but noted that successful participants have studied a variety of subjects including communications and journalism. 1c) How were participants identified? Can anything be concluded about the general characteristics (e.g. educational or professional background of participants vs non￾participants? Conclusion: The processes for identifying candidates for HP-LIFE e-learning curriculum, HP-LIFE, and “deep dive” coaching are described in detail below (the source of these descriptions are KIIs with UNIDO staff). The most striking conclusion to be drawn both from the Mashrou3i Program population data and the survey data is the high level of education of participants, 94 percent of whom have completed post￾secondary education. Furthermore, a very high percentage of program participants have studied in scientific, technical, and/or engineering fields. Findings/Evidence: 1c.1 Access to the HP-LIFE online e-learning curriculum (25 modules) is free and open to anyone in Tunisia with computer access to the internet. Awareness of HP-LIFE is promoted through the Mashrou3i Facebook page, through friends, colleagues, college lectures and advisors, local business support institutions, electronic social networks, and active outreach and awareness events from HP- 11 While this high percentage of unemployed respondents could be an artifact of the survey sampling procedure, it may also reflect the amount of time required from the inception of a business idea to the actual creation of a business plan, financing and actually launching a new business. Many of these unemployed may be in the “pipeline” for new businesses. 29 UNIDO at colleges, job fairs, and other events. As such, initial entry into Mashrou3i and potential access to training, business coaching and other program resources is limited only by knowledge of the program, interest, online access, and basic computer skills. 1.c.2 UNIDO describes the HP-LIFE selection process as focusing on inclusion rather than exclusion. Trainers from the local partners personally interview all applicants for the face-to-face training to ensure that they are motivated, understand the challenges they will face and that their basic interest and ideas seem plausible and realistic. No one with a reasonable idea and desire to participate is excluded. There are no written selection criteria (nor statistics kept on those rejected). In this process Mashrou3i expects trainers to rely on their professional judgment, similarly to trainers nominating candidates for the four-day business coaching workshops whom they judge to have the most viable business ideas. 1c.3 A more targeted and competitive process was used for the selection of candidates for the “deep-dive” coached start-up development and for support to existing enterprises. Selection started with a request for proposals announced on the Mashrou3i Facebook page asking for business ideas from prospective young entrepreneurs in the four target regions. Based on this call, 78 proposals from aspiring entrepreneurs were received and vetted by a committee consisting of HP-UNIDO and local partner institution staff. Of these, 44 (56 percent) were selected and approved by the National Steering Committee based on technical merit and perceived likelihood of success. Partner institutions then initiated the formal coaching process aimed at creating successful business start￾ups. Eight participants subsequently dropped out, losing interest or motivation or securing employment or other opportunities, leaving 36 active participants. Similarly, 88 existing enterprises in the four regions were profiled and selected for technical support based on their assessed job growth potential. About 52 were selected for further support. 1d) How did the HP-UNIDO approach of working directly with local communities influence individual motivations or job training or placement outcomes? Conclusion: By engaging local partners for service delivery at the community level, the Mashrou3i model encourages access and makes use of local knowledge for job creation. This local accessibility may also support individual motivation for training and business support. The incorporation of the HP-LIFE e￾learning curriculum into ISET courses is a particularly striking example of this process. HP-UNIDO efforts to strengthen SBSs, while relatively modest in scope, can be plausibly linked to improved or expanded accessible local service delivery, including business counseling and advising, and possibly to placement outcomes. Findings/Evidence: 1d.1 The Mashrou3i program model utilizes well-established local business support and education institutions in the four regions – including the Business Centers in Le Kef, Kairouan, and Sidi Bouzid, the Office for the Development of the Center West (ODCO) in Kasserine, and ISET branches – as partners in service delivery, support, building capacity, and structurally embedding the program within these ongoing structures. In this way, HP-UNIDO’s work with communities is mediated through these public entities and is enhanced or constrained by their capacities, level of functionality, service, motivation, and their reputation within their respective communities. 1d.2 UNIDO reports capacity-building support activities including: initial SWOT analysis workshops with local implementing partner organizations to establish areas for capacity-building support; “Green Business” training in all regions; workshops for ISET and UTICA (Union of Industry, Trade and Handicrafts) personnel; training for ISET and APERE educators on HP-LIFE e-learning 30 methodology; training of trainers in the four regions and provision of IT equipment to regional partners. 1d.3 The evaluation of online survey responses received from the regional SBS partner organizations suggests success in building their service delivery capacity. Out of a total of 11respondents, 54 percent (N=6) reported that Mashrou3i helped them to add new services and improve existing services; 36 percent (N=4) reported improvement to existing services; and nine percent (N=1) indicated no impact on their service provision. 1d.4 The ISETs’ decision to formally incorporate the HP-LIFE e-learning modules into its required entrepreneurship courses and to have its instructors trained in this approach, methodology and content strengthens these institutions by incorporating a standardized IT-based business skills curriculum and instructional platform, including exercises across campuses and instructors, and by expanding the HP-LIFE beneficiary population. EQ 2. Relative to a traditional contract model, were there any benefits to using the GDA approach? Were there any externalities of using a GDA approach that facilitates external resource partners and a PIO approach that attracts the resources of additional donors? Conclusion: The GDA approach, including the Agency for Italian Development Cooperation (GOI) and USAID and the engagement of an experienced Public International Organization (PIO) as the major implementing partner, clearly leveraged financial resources, human capital and institutional experience in support of program results, enlisted the extraordinary technical expertise of a major private sector corporation, and demonstrates the value and potential of such a unique multi-donor PPP model. (Annex VIII includes a list of donor-funded programs in Tunisia.) Findings/Evidence: 2.1 The GDA mechanism resulted in a significant leveraging of USAID program resources, more than doubling the USG investment for the HP-LIFE program. 2.2 The GDA approach also leveraged the expertise and in-country experience of UNIDO, a PIO with which USAID has not frequently collaborated, incorporating UNIDO’s long-term partnerships with HP and the GOI. The FGD and KIIs suggest that this unique collaboration, involving the multilateral engagement of the Governments of Tunisia, Italy and the United States; a U.N. agency; and a private￾sector corporation – thereby also creating a PPP within its structure – is itself a noteworthy model in accordance with the cooperative thrust of international development policy and the principles of local ownership and priority-setting. 2.3 The GOI asserted that the GOT and bilateral and multilateral donors have concurred that addressing the challenges of youth employment in the marginalized regions is a key development focus and that within this environment of constrained job availability entrepreneurship represents the primary programmatic approach to plausible, sustainable jobs growth. Yet, the GOI believes that there continues to be inadequate communication, sharing of program details and coordination among donors resulting in lost-efficiencies and possible duplication (from KII with the GOI). The GOI believes that Mashrou3i has been highly effective and greatly values its working partnership with the GOT, USAID, HP and UNIDO. In late 2015 the GOI significantly increased its investment, providing resources to extend the program to four additional governorates: Gafsa, Kebili, Medenine and Tataouine (these additional four governorates are not included in this evaluation). 31 2.4 The HP Foundation – the newly created corporate structure under which the HP-LIFE program is managed – also believes that Mashrou3i has been a valuable program for building business and entrepreneurial skills and knowledge among young Tunisians, and values its partnerships with UNIDO and USAID (telephonic KII with HP Foundation). HP remains committed to supporting and expanding HP-LIFE in Tunisia, is in the process of updating and improving its platform (which will enable greater and more discrete data collection and analysis on results), and is working to include two additional modules in the HP-LIFE curriculum. 2.5 The GDA mechanism and participation of UNIDO (a PIO), the GOI, and HP – all of which had a working relationship prior to the creation of the GDA collaboration for implementation of Mashrou3i – generated administrative procedures and monitoring and reporting requirements more flexible and seemingly less rigorous (or perhaps rigid) than those employed in the administration of typical USAID contract activities. The SOW for the agreement between USAID and UNIDO was a one￾page general statement of overall program goals and objectives. An overall assessment of the benefits and possible costs of a GDA mechanism as compared to more traditional contracting approaches should be derived from an examination of several such activities. The lessons learned from this single example suggest that: (1) the GDA mechanism successfully facilitated the cooperative engagement and contribution of resources and expertise from public and private institutions with differing organizational cultures, bureaucratic traditions and administrative requirements, without engendering apparent friction or conflict; and (2) the GDA mechanism permitted the degree of programmatic flexibility and adaptability necessary to rapidly refine interventions to increase participation and improve outcomes. EQ3. To what extent did UNIDO’s cooperation with existing SBSs contribute to employment outcomes? (Consider UNIDO’s business model, years of experience working with indigenous parastatals and cost structure.) Conclusion: UNIDO cooperation with SBSs has contributed both directly and indirectly to employment creation. However, program participants do not consistently rate the quality and impact of the services delivered by some local partners as acceptable. Findings/Evidence: 3.1 In its design for Mashrou3i, UNIDO builds upon its forty-year experience of working in Tunisia (its Tunis office was established in 1995 following the period of economic reforms) and has a substantial history of working with the local parastatal business support and educational institutions that provide cost-efficient services in the four target governorates. 3.2 An examination of UNIDO data on employment creation (see summary table under Question 1) suggests that the collaboration with local institutions on ‘deep dive coaching’ for start-ups was less successful in absolute terms in creating jobs than the two other support programs that worked more independently from local support institutions, (i.e. HP-LIFE support to start-ups and existing enterprises). According to UNIDO data, of the 1,035 total jobs created to date by the program, only 135 came from the “deep dive” coaching done in collaboration with SBSs, whereas 471 jobs are attributed to HP-LIFE support for start-ups and another 429 to program support for existing enterprises, as shown below in Table 5.3. To some extent this can be attributed to the fact that the number of enterprises participating in the “deep dive” start-ups coaching (44 selected; 36 remain) was 32 fewer than the number participating in the program for existing enterprises (52) and the number of HP-LIFE entrepreneurs who completed online training, face-to-face training, and face to face coaching (154). However, a relative comparison of jobs per enterprise created still shows that support to existing enterprises was more successful than ‘deep dive coaching’ to start-ups, and that the latter was comparable to the HP-LIFE support to entrepreneurs when counting only those 154 HP supported entrepreneurs who completed online training and face-to-face training and coaching. Table 5.3 Estimated Jobs Created per Enterprise/Entrepreneur Assisted # Activity/Type of Enterprise Jobs Created through March 2016 Additional Jobs to be Created as Projected by UNIDO Total Jobs # of Entrepreneurs and/or Enterprises Assisted Jobs per Enterprise based on Jobs Created to Date Jobs per Enterprise based on Projected Jobs 1 HP-LIFE Support to Entrepreneurs 471 307 778 154 3 5 2 Deep-dive business coaching support for start-ups 135 268 403 42 3 10 3 Support to Existing Enterprises 429 429 52 8 8 3.3 The evaluation survey responses from regional SBS organizations suggest a significant positive impact on local service delivery. As noted above, out of 11 responding organizations six reported that Mashrou3i helped them to add new services and improve existing services; four said that the program helped improve existing services; and one reported no impact. Although due to security and time constraints the evaluation team was unable to independently validate or confirm these results, they are encouraging and indicative of the utility of this programmatic model. 3.4 In group meetings and KIIs, project beneficiaries, including HP-LIFE aspiring entrepreneurs and owners of existing small businesses, expressed a range of opinions concerning the value of the local implementing partners in assisting their entrepreneurial projects. While some found the local SBSs to be helpful in providing technical advice, assistance and support, many others expressed the opinion that the SBS personnel with whom they consulted were not sufficiently skilled, experienced or interested in providing the necessary support the entrepreneurs needed. 3.5 While acknowledging the potential cost and sustainability benefits of utilizing local SBS structures to deliver entrepreneurial training, advising, and coaching services, several Tunisian experts in enterprise development and expansion, including university faculty, staff from development agencies and a business consultant, suggested that parastatal organization generally lack the actual hands-on expertise, business experience and motivation that are found in the private sector. 3.6 While acknowledging financial, material, and technical support from UNIDO, some participating Business Center representatives indicated in KIIs that they regarded this level of support as insufficient to produce a significant, sustainable increase in institutional service building capacity and results. 33 3.7 The survey of individuals, entrepreneurs, and enterprises, asked participants within each region to rate the services provided by Mashrou3i’s partner organizations located within their respective regions. Mashrou3i worked with seven public sector support agencies that are found in each region: the National Agency for Employment and Independent Work (ANETI), the Agency for the Promotion of Industry and Innovation (APII), the APII Incubator, the Agency for the Promotion of Agricultural Innovation (APIA), Business Center, ISET, and the ODCO. ANETI reports to the Ministry of Labor; API and the Business Center report to the Ministry of Industry, APIA to the Ministry of Agriculture; ISET to the Ministry of Higher Education; and ODCO to the Ministry of Regional Development. In addition, participants in Kairouan were asked their opinions of the Cyber Park in Kairouan which is managed by the Ministry of Information Technology. 3.8 The following graphs (Figures 5.9, 5.10, 5.11, and 5.12) show the participant ratings by region and organization in order of the combined score ‘Excellent’, ‘Very Good’, and ‘Good’. The organization on the left side of each graph received the highest scores, while the organization on the right side of each graph received the lowest. Ratings of these agencies varied by region, with some organizations scoring well in some regions but not as well in others. The only exception is ODCO, which received consistently lower ratings. 9% 10% 9% 18% 13% 16% 14% 42% 17% 23% 25% 21% 30% 31% 34% 43% 13% 45% 44% 38% 32% 41% 32% 42% 20% 13% 16% 13% 15% 9% 11% 11% 4% 10% 6% 10% 11% 3% 9% 0% Cyber Parc (N=35) APII Incubator (N=40) ISET (N=32) Business Center (N=39) APII Incubator (N=47) APIA (N=32) ANETI (N=44) ODCO (N=24) Figure 5.9 Ratings of Small Business Support (SBS) Organizations in Kairouan by Mashrou3i Participants Excellent Very Good Good Fair Poor 34 25% 19% 18% 33% 21% 29% 32% 8% 19% 29% 14% 29% 29% 29% 32% 33% 35% 24% 36% 36% 17% 16% 14% 6% 14% 7% 7% 16% 21% 14% 12% 14% 7% 0% 5% APII (N=24) ANETI (N=21) ISET (N=17) APII Incubator (N=21) APIA (N=14) ODCO (N=14) Business Center (N=19) Figure 5.10 Ratings of Small Business Support (SBS) Organizations in Kasserine by Mashrou3i Participants Excellent Very Good Good Fair Poor 25% 17% 17% 20% 30% 38% 33% 13% 21% 28% 25% 20% 33% 31% 41% 21% 21% 35% 25% 19% 22% 13% 24% 17% 10% 20% 6% 17% 17% 17% 10% 0% 5% 6% 4% API Incubator (N=24) API (N=29) Business Center (N=29) ISET (N=20) APIA (N=20) ODCO (N=27) ANETI (N=27) Figure 5.11 Ratings of Small Business Support (SBS) Organizations in Le Kef by Mashrou3i Participants Excellent Very Good Good Fair Poor 35 EQ4. To what extent can the employment information collected by the HP-UNIDO activity be independently validated and deemed credible? Conclusion: The evaluation team was able to independently validate a small sample of Mashrou3i employment information and found no evidence of error or misrepresentation of results. HP-UNIDO makes diligent efforts to monitor and track its program activities and to collect, analyze and report credible employment results, and has modified their program design accordingly. Best practice, however, would suggest a more systematic and rigorous performance monitoring and results-tracking approach that more clearly separates the data collection and analysis process from service delivery functions and personnel. Findings/Evidence: 4.1 In order to answer this question, the evaluation team compared the numbers of jobs created at 20 existing companies reported by UNIDO to the responses provided by these same 20 companies in the evaluation online survey. Twenty of the 52 companies who responded to the survey answered the question: “How many persons have you hired as a result of Mashrou3i?” UNIDO results were lower than survey results in nine cases, higher in five cases, and the same in six cases. Overall, UNIDO estimated that 142 jobs had been created in these 20 companies, while the survey results showed 139, a difference of only three jobs in total. Based on this limited assessment, the evaluation team finds that the results being reported by UNIDO can be considered credible. Differences in specific numbers may be due to several factors, including the timing of the UNIDO survey (February 2016) versus that of the Evaluation Team survey (March 2016), and the level to which respondents understood the nature of the questions being posed in the UNIDO survey and the Evaluation Team survey. 36 4a) How does HP-UNIDO define and measure employment creation? Conclusion: HP-UNIDO does not have a formal definition for ‘employment creation’ that is linked to a specific action or set of actions undertaken or supported by the activity. HP-UNIDO measures employment creation through enterprise surveys that compare enterprise employment levels at different points of time. The surveys are conducted by UNIDO and local partner staff. Findings/Evidence: 4a1. According to KIIs with UNIDO staff, Mashrou3i relies on periodic surveys to determine whether new jobs have been created in the assisted enterprises. UNIDO considers sustainable employment as a job that has existed for one year or more. Through its most recent (March 2016) survey, UNIDO has verified with the entrepreneurs all start-up numbers reported in its April 2016 Progress Report. UNIDO does not count those enterprises (or their employment figures) that were established in 2014 or 2015 that subsequently closed or could otherwise not be validated through the survey process. 4b) Given the methodology and procedures used to collect employment information, to what extent are the results from the routine activity monitoring likely to be over or under￾estimated? Conclusion: Given the current methodology and procedures used by UNIDO, there may be discrepancies between actual and reported results for individual participant enterprises. However, based on the analysis of the supporting documentation from UNIDO and the results of the online survey, the evaluation team believes that the employment information is reasonably accurate. There is no evidence of systematic over-estimation or under-estimation of employment created. Findings/Evidence: 4b.1 Monitoring surveys administered by Mashrou3i and local partner staff may over-estimate program activity and employment results due to social desirability effect in which respondents are reluctant to provide more negative information to people with whom they cooperate in project implementation. Conversely, the telephonic surveys may fail to reach employed individuals and functioning businesses and start-ups which are then excluded, thereby underestimating results. As noted above the evaluation found no evidence of systematic over or under estimation of level of activity or results. 4c) What methodologies, processes, systems or structures have facilitated or impeded the collection of high quality and credible employment data? Conclusion: The absence of a separate, full-time monitoring and reporting function within the program impedes the collection of high quality and credible employment data. Findings/Evidence: 4c1. UNIDO is clearly committed to quality service delivery to its target beneficiaries. Data collection, analysis and reporting systems, while addressed, may not receive sufficient attention and expertise. Like much of the Mashrou3i implementation model, the employment data collection process is built around personal/professional relationships rather than a more systematic, rigorous and independent project monitoring plan and data system. As noted above, the HP-UNIDO approach can be laborious and open to possible bias, and the reported data vulnerable to confusion and misinterpretation, which limits its credibility and utility for program decision-making. 37 4d) Based on the data collected by HP-UNIDO, can cost per job created – including management costs and in-kind contribution - be accurately estimated? Provide simple benefit/cost calculation to determine whether the benefits of the jobs created exceeded costs (e.g. compare cost per job with wage data). Conclusion: The cost per job created can be estimated, but the accuracy of such a calculation is subject to a high degree of uncertainty due to questions about which costs to use, the reliability of employment data, and the attribution of jobs to program interventions (see Annex IX regarding the ‘Complexities of Measuring Jobs Created and their Costs’). A benefit/cost calculation can also be made, but includes a similar level of uncertainty. Findings/Evidence: 4d1. Various costs per job created are shown in Table 5.4 below: Table 5.4 Costs per Job Created Source of Contribution Type of Contribution Total Contribution Costs/job based on jobs created Costs/job based on jobs created and projected USAID Cash $ 1,500,000 $ 1,449 $ 907 Government of Italy Cash $ 1,137,748 $ 1,099 $ 688 Hewlett Packard Cash $ 410,000 $ 396 $ 248 USAID, GOI, HP Cash $ 3,047,748 $ 2,945 $ 1,843 Hewlett Packard In-kind $ 300,000 $ 290 $ 181 USAID, GOI, HP Cash and in-kind $ 3,347,748 $ 3,235 $ 2,024 Participating Enterprises and Entrepreneurs Cash and in-kind (information not available) Totals $ 3,347,748 $ 3,235 $ 2,024 Note: As detailed in Table 1.1, Jobs created to date = 1,035; Jobs created to date and jobs projected = 1,654. The estimated cost per job in the Mashrou3i project ranges from a low of $907 to a high of $3,235 depending on which estimates of costs and jobs are used. However, these estimates are not the total cost per job, which would also include the costs that are incurred by the organization or individual that created the job. Such costs may include such items as training, tools, equipment, supplies, space, and social insurance. USAID costs/job: Based on the figure of 1,035 jobs created (see Table under Question 1), as reported by UNIDO, and the USAID contribution of $1,500,000, the cost per job is $1,449. Based on the higher employment figure of 1,654, which includes 44 job placements and 575 anticipated jobs12, the cost per job drops to $907. Total Cash Contributions/job: If HP in-kind contributions are excluded, then the cost/job is $2,945 based on jobs created to date, and $1,843 based on jobs created to date and projected. 12 Mashrou3i identifies “anticipated” or “projected” jobs through period discussion with a sample of participating businesses and revises these figures on a regular basis. 38 Total Program costs/job: If the full activity cost, including cash and in-kind contributions from HP and the Government of Italy ($1,847,748), is used, then the estimated cost/job reported increases to $3,235 based on the 1,035 jobs created to date. If one adds anticipated jobs and employment placements, the figure drops to $2,024. Total Costs/job: A more accurate measure would look at total cost per job created and would include not only Mashrou3i Program costs, but also all costs incurred by entrepreneurs and enterprises to create new jobs. 4d.2 Cost benefit analysis: Assuming a median monthly salary of 600 TDN (or approximately $300 USD), the investment in this job creation program would achieve full payback in three to 10.8 months depending on the cost/job calculation used, as shown in Table 5.5 below. The payback period using full costs per job created would be higher. Table 5.5 Payback Period Costs per Job based on: Payback in months based on Jobs Created to Date Payback in months based on Jobs Created to Date and Projected USAID Costs/Job 4.8 3.0 Total Cash Contributions/Job 9.8 6.1 Total Cash and In-kind Contributions (USAID, GOI, HP)/Job 10.8 6.7 Total Cash and In-kind Contributions (USAID, GOI, HP, Others)/Job NA NA 39 RECOMMENDATIONS Drawing upon the findings and conclusions of the Final Performance Evaluation of the “Tackling Youth Employment in Tunisia” activity, the evaluation team recommends: 1. Continue support for job creation in Tunisia’s Interior Regions: Given the high levels of unemployment and poverty in Kairouan, Kasserine, Le Kef, and Sidi Bouzid, and the potential negative repercussions on security and stability in Tunisia and the region, USAID should continue to provide assistance for job creation activities in these governorates. As further explained in other recommendations, USAID should consider mechanisms for providing business support and advisory services that rely more heavily on private service providers and well￾qualified business coaches who have actual private sector business experience. 2. Continue to work with HP-LIFE: It is recommended that USAID continue to work with HP￾LIFE on two levels. First, in expanding and improving the HP-LIFE online courses and face-to￾face training and coaching for young entrepreneurs. Second, in light of the very positive working relationship Mashrou3i has established with the Ministry of Higher Education in general, and the Higher Institutes of Technological Studies (ISET) in particular, USAID should continue to work with the Ministry and with the ISETs to integrate the HP-LIFE curriculum into existing entrepreneurship courses at the ISETs, and possibly other higher education institutions, and to actively promote use of the e-learning program across the interior governorates. Such collaboration with the Ministry will complement the ongoing support by the Business Reform and Competitiveness Project (BRCP) for the establishment and strengthening of career centers at institutions of higher learning. 3. Work through private business support service providers to provide enterprise development, entrepreneurship, and employment services: In light of a) the significant network of private business services providers in Tunisia, including those that have been nurtured by USAID-funded programs in the past, as well as b) the need for private enterprises to be supported by people and organizations who understand business, USAID should work through indigenous, private service providers and organizations to expand technical assistance and training for enterprise development, entrepreneurship, and employability in any future programs. 4. Support entrepreneurship development programs that provide coaching by experienced business experts: Given that business coaching by qualified business professionals has been identified in Tunisia and world-wide as a critical element in the success of many new ventures, and that new businesses in Tunisia face a plethora of administrative and financial obstacles, USAID should support indigenous systems and programs that provide on-going business coaching by qualified, experienced small-business experts over a three- to four-year period. 5. Reorient Regional Business Centers and API Regional Offices: As Tunisian parastatal regional business support organizations do not have the background or qualifications to advise entrepreneurs and enterprise owners on how to run a business, USAID should encourage the GOT to consider the policy, operational, and service benefits of transforming Regional Business Centers and API offices into business development advocates. Rather than business service providers, such organizations could refocus their efforts on helping entrepreneurs and enterprises obtain necessary government approvals and gain access to privately supplied business advisory services and finance. Such an advocacy role may help to overcome the administrative bottlenecks identified in KIIs with young entrepreneurs. 40 6. Improve the business enabling environment: In conjunction with its ongoing, critical support for tax and customs reforms in Tunisia, USAID should continue to encourage the GOT to improve the overall business climate by reducing the administrative burdens on entrepreneurs and enterprises, including the reform of existing labor and bankruptcy codes. These are two areas identified in KIIs as serious obstacles to job creation in Tunisia. 7. GOT Employment Programs: In conjunction with Recommendation 6, USAID should coordinate its efforts with other donors to encourage the GOT to review and revise employment programs that inadvertently discourage workers from seeking full time employment in the private sector. 8. Job Creation Data Collection and Analysis: In order to strengthen and ensure the reliability, validity, and utility of the Mashrou3i job creation data, and in light of the expansion of services to four additional governorates, it is recommended that UNIDO employ a qualified local Monitoring and Evaluation/data-base professional to establish and operate the Mashrou3i data collection and analysis system, and should also utilize an independent Tunisian research organization to implement qualitative and quantitative data collection and analysis. Reforms such as those suggested in Recommendations 6 and 7 will complement and reinforce whatever efforts USAID undertakes to fund new entrepreneurship, employability, and enterprise development programs. 41 ANNEXES Annex I: Evaluation Statement of Work Statement of Work Final Performance Evaluation of “Tackling Youth Employment in Tunisia” I. Purpose and use of evaluation The purpose of this statement of work is to conduct a final performance evaluation of the “Tackling Youth Employment in Tunisia” activity, which is implemented through a Global Development Alliance (GDA) managed by USAID’s Middle East Bureau with Hewlett Packard (HP), the United Nations Industrial Development Organization (UNIDO), and the Government of Italy. The objective of the activity is to create job opportunities for young Tunisians. The purpose of this final evaluation is to understand to what extent the activity has been successful in creating sustainable employment/livelihoods in the vulnerable interior regions of Tunisia in which the activity is implemented. The evaluation should assist USAID/Washington, USAID’s Tunisia Office, the implementing partners, and other relevant stakeholders in understanding the effectiveness of the HP-UNIDO approach in creating employment opportunities in the interior of Tunisia. The evaluation will help multiple partners better understand how to support the Government of Tunisia’s (GOT) vision to promote private sector development through small and medium enterprise (SME) creation and growth that will lead to increased employment, particularly among youth. Lessons learned will be shared with relevant stakeholders and used to inform the design of follow-up job creation programming. II. Identification of Award Award Title: Tackling Youth Employment in Tunisia Implementing Partner: United Nations Industrial Development Organization, Hewlett Packard, and the Government of Italy Mechanism: Public International Organization (PIO) Grant Agreement Activity Number: Award Duration: Award Budget: AID-OAA-IO-12-00002 October 1, 2012 - March 31, 2015 (No Cost Extension until December 31, 2015) $1,500,000.00 USD Cost Sharing Amount: $1,847,784.00 (through Hewlett Packard and the Government of Italy) Country of implementation: Tunisia III. Background The Tunisian revolution brought into focus the need for governance and employment creation. However, in the four years following the revolution, the economic and social situation has only marginally improved and general unemployment remains high, reaching over 30% among young university graduates. Unemployment rates are even higher in the South, 42 Central and North West governorates, especially among women and youth. The GOT has identified boosting employment, particularly among youth, as a national priority, and a number of donors support entrepreneurship and job creation-related programming. To date, there have been no evaluations of USAID jobs-related activities in Tunisia. Aside from World Bank analyses of the effectiveness of GOT programs, USAID is not aware of any evaluations of other donor or GOT programs. The 2012 “HP-LIFE Program Process and Outcome Evaluation” (http://pdf.usaid.gov/pdf_docs/pdact908.pdf) covered five countries but did not include Tunisia because at the time of the evaluation Tunisia was deemed to have an insufficient number of HP-LIFE trainees to be included. IV. Description of Activity Launched in early 2013, USAID's Global Development Alliance (GDA) with UNIDO, Hewlett Packard, and the Government of Italy has a central goal of creating job opportunities for young Tunisian women and men in four vulnerable regions in the Center West regions of Tunisia: Le Kef, Kairouan, Kasserine, and Sidi Bouzid. The activity is in line with the GOT’s strategy to promote private sector development through small and medium enterprise (SME) creation and growth that will lead to increased employment, particularly among youth. Currently, this is the only USAID-sponsored job creation activity in Tunisia that exclusively targets areas outside of the coastal regions. Leveraging UNIDO's expertise in the field of private sector development, the activity works directly and indirectly to promote the creation of new, and the growth of existing, enterprises, with a special focus on the agro-food and environment (e.g., waste management and recycling) sectors. Additionally, the activity has supported institutional capacity building by providing business institutions with information technology (IT) and supporting university educators through trainings on integrating Hewlett Packard’s “Learning Initiative for Entrepreneurs” (HP-LIFE) online courses into their university curricula. HP-LIFE is a global program that trains students, entrepreneurs, and small business owners on applying IT and business skills, with the aim of establishing and growing businesses, building successful companies, and creating jobs. The HP-LIFE curriculum is intended to provide budding entrepreneurs with both hard and soft skills to structure their thinking, management and advocacy to obtain financing and build and grow their businesses. Much of the project centers around HP's Learning Initiative for Entrepreneurs (HP-LIFE) curriculum, which is a 25 chapter e-Learning tool available for existing and future entrepreneurs. Thus far, more than 9,000 Tunisians have taken HP-LIFE online courses, of which over 1,100 have continued with face-to-face training allowing them to receive business mentoring and coaching as they launch their own business projects. An additional 30 educators and 2,000 students will be trained through HP-LIFE workshops offered at the Higher Institute of Technological Studies (ISETs) and in universities in the four regions. ISETs are intermediaries that help deliver the training. Existing Tunisian Small Business Support (SBS) organizations help to identify entrepreneurs and companies that need support. Annex One provides the Results Framework for the activity. V. Relationship to other USAID/USG, Donor, and GOT Activities/Initiatives There are a large number entrepreneurship, business start-up, and small business growth activities managed by the USG and other donors in Tunisia. The resources spent by donors, in combination with Tunisian national initiatives, represent a diverse set of approaches and visions for addressing Tunisia’s employment needs. A few donor activities, HP-UNIDO among them, include a focus on jobs in Tunisia’s interior, while many activities focus on the coastal zones and greater Tunis. Some projects focus on early stage businesses or pre-startup entrepreneurship while others focus on later stage businesses. Many have a dimension of strengthening or working through Tunisian training and education systems. This rich diversity in programmatic approaches provides an opportunity to learn about the types of programs that work most effectively in markets such as Tunisia. 43 USG-supported entrepreneurship and job creation focused programming in Tunisia includes: ● USAID supports the Center for Entrepreneurial and Executive Development (CEED) to create a center for management and entrepreneur training in Tunis; the Business Reform and Competitiveness Project (BRCP) to strengthen existing businesses and business service providers; and the HP-UNIDO GDA. Annex Two includes a brief summary of these activities. ● The State Department’s Middle East Partnership Initiative (MEPI) implements the Regional Chambers of Commerce activity, which has led to a University Entrepreneurship and Leadership Program at four regional universities; and Women’s Enterprise for Sustainability (WES), which provides women with entrepreneurship and management training both in Tunis and interior regions; ● The Commerce Department’s Regional Investments to Support Entrepreneurship (RISE) partnership is a new platform between the Commerce Department and a number of top U.S. companies and non-profits to increase training, mentoring, access to finance for entrepreneurs, as well as to support the development of incubators and accelerators for Tunisian entrepreneurs, and improve the commercial environment. ● The Overseas Private Investment Corporation (OPIC) supports the Tunisia Credit Guaranty Facility in partnership with leading Tunisian financial institutions to increase the credit available to Tunisian SMEs and stimulate growth in Tunisia’s private sector. The interrelationship and interdependence among various USG and other donor projects is not well documented, and, as such, it is not known to what extent various projects augment or duplicate what other programs are doing. Anecdotally, among USAID-supported projects, there appears to be strong program collaboration; for instance, USG-funded project staff may refer beneficiary businesses/clients to other USG-funded projects that enable them to obtain the available services most closely tailored to their needs. For example, upon receiving their HP-LIFE certification, beneficiaries from the HP-UNIDO activity are sometimes referred to USAID’s BRCP activity, which helps entrepreneurs to fine tune their business proposals using a value chain analysis and learn to obtain financing. Once entrepreneurs get to the start￾up phase, BRCP may refer them to the USAID-funded CEED project, which provides mentoring and pairs entrepreneurs with a more experienced business manager to coach them with real local expertise. CEED holds regular meetings with different donors and angel investors who provide knowledgeable assistance to these young startups to improve their success rate. V. Results to Date Over the life of project, HP-UNIDO aims to reach roughly 10,000 aspiring and existing entrepreneurs, with a focus on youth, and create at least 2,000 jobs (including direct, indirect, and temporary employment) in the targeted governorates. As of May 2015, based on survey data, the activity has created 548 total direct jobs (including 79 in start-ups and 469 enterprise development hires). With regard to direct assistance to entrepreneurs, results as of May 2015 include: ● More than 9,000 Tunisians took HP-LIFE online courses, of which 1,136 entrepreneurs attended face-to-face trainings (52% women) ● 36 start-ups and 80 high growth SMEs are being supported with technical assistance and coaching ● More than 1,400 students and aspiring entrepreneurs took part in HP-LIFE e￾Learning workshops and thematic sessions 44 ● 134 HP-LIFE entrepreneurs coached to develop their business plan ● Manual for young entrepreneurs developed and distributed With regard to capacity building of business support institutions, results as of May 2015 include: ● 16 trainers from public and private sector trained and certified on HP-LIFE e￾learning ● 55 educators of the Higher Institute of Technological Studies (ISET) and of other universities trained on HP-LIFE e-Learning to enrich their business courses ● IT equipment provided to support HP-LIFE e-Learning workshops in the four governorates ● 76 representatives of business support institutions trained on “greening your business” ● Assessment of local business support institutions VI. Award Modifications On March 20, 2015, USAID’s Office of Assistance and Acquisition granted a no cost extension for the activity, which extended the period of award from March 15, 2015 to December 31, 2015. The rationale for the no cost extension was to extend the implementation period in response to early implementation challenges. Despite the best efforts of UNIDO, many activities could not be implemented in the initially anticipated timeframe, mainly due to the political challenges faced by Tunisia at the early stages of the project. These challenges resulted in difficulties in the mobilization of the national partners "Centres d' affaires" and the Office de Developpement du Centre Ouest (ODCO) - for Kasserine - to conduct business coaching, as well as administrative difficulties in signing agreements with national partners and a low response level of enterprises. In addition, recurrent periods of inaccessibility of the targeted regions due to security issues, mainly in Kasserine, Kef and Sidi Bouzid, delayed implementation of the project. VII. Evaluation Questions The evaluation should answer the following questions, in order of priority: 1) To what extent has the HP-UNIDO implementation approach been effective in creating sustainable employment in the interior regions in which the activity is implemented? a) Which aspects of the implementation approach have been the most effective in creating sustainable employment and why? b) Which populations have benefited most from the activity and why? In particular the evaluation team should consider how youth in each of the four regions and women have differentially benefited from and been reached by the activity. c) How were participants identified? Can anything be concluded about the general characteristics (e.g., educational or professional background) of participants vs. non-participants in the HP-UNIDO program? d) How did the HP-UNIDO approach of working directly with local communities influence individual motivations or job training and placement outcomes? Note: For the purposes of this evaluation question, “effective” is defined as actual employment/livelihoods created versus expectations at the beginning of activity implementation. “Sustainable employment” is defined as positions created by employers or self-employment lasting (or expected to last) at least one year post-intervention. If in conducting the evaluation the evaluation team identifies concerns with the validity of this definition of “sustainable employment,” these concerns should 45 be documented in the final evaluation report. The evaluation team should propose an evaluation methodology that tries to identify the role of the HP-UNIDO implementation approach in creating employment versus other factors that could be responsible for employment creation over the same timeframe. 2) Relative to a traditional contract model, were there any benefits (e.g., lower costs, greater leverage) to using the GDA approach for the implementation of the HP-UNIDO activity? Were there any externalities (positive or negative) of using a) a GDA approach that facilitates external resource partners and b) a PIO approach like UNIDO’s that attracts the resources of additional donors? 3) To what extent did UNIDO's cooperation with existing Tunisian Small Business Support (SBS) organizations contribute to employment outcomes? Consider UNIDO's business model, years of experience in working with indigenous entities that are government funded (parastatals), and cost structure. 4) To what extent can the employment information collected by the HP-UNIDO activity be independently validated and be deemed credible? a) How does HP-UNIDO define and measure employment creation? b) Given the methodology and procedures used to collect employment information, to what extent are the results reported from the routine activity monitoring likely to be over-estimated or under-estimated? c) What methodologies, processes, systems or structures have facilitated or impeded the collection of high quality and credible employment data? d) Based on the data collected by HP-UNDIO, can costs per job created (including program management and in-kind) be accurately estimated? To the extent feasible, provide a simple benefit-cost calculation as to whether the benefits of jobs created exceeded costs (e.g., such as by comparing estimated cost per job with wage data). Based on evaluation findings, the evaluation team should identify actionable recommendations that will help USAID to better design and implement effective youth employment programming going forward, particularly for youth in vulnerable regions of Tunisia. To the extent feasible, recommendations also should speak to considerations for replicating or scaling up the program in other settings. VIII. Existing Activity Data/Information USAID will provide the evaluation team with the following activity documents: ○ Activity concept paper ○ Activity agreement and no cost extension ○ Activity quarterly and annual reports ○ Activity Work Plan ○ Budget information ○ Data from HP-UNIDO survey of beneficiaries ○ One-page results summaries ○ Recently completed State Department (Bureau of Near East Affairs) assessment of economic growth programming in the MENA region ○ Other documents, as available and requested by the evaluation team 46 The evaluation team should complete the document review prior to arriving in Tunisia. The evaluation team may also request and review additional resources to the extent necessary to perform its work, and USAID will do its best to provide these resources to the extent they are available. IX. Evaluation Design and Methodology 1) Overall Evaluation Design Evaluators will use a mix of quantitative and qualitative data collection and analysis methods to generate answers to the evaluation questions listed above. The evaluation must follow the principles and guidelines for high quality evaluations outlined in the USAID Evaluation Policy (January 2011) and ADS 203. 2) Data Collection Methods The evaluation team must develop data collection tools that will provide high quality data to answer the evaluation questions. Data collection methods may include, but are not limited to, a combination of the following: ● Review of relevant documentation (e.g., activity reports, survey data, etc.); ● Key informant interviews (KIIs) (which should include, but are not limited to, KIIs with USAID/Washington Economic Growth Team, including the AOR; HP-UNIDO staff; and activity beneficiaries); ● Focus group discussions or group interviews with beneficiaries (e.g.,, HP-LIFE trainees; aspiring entrepreneurs; enterprises; university beneficiaries) and other counterparts and stakeholders; ● On-line surveys of key informants and training participants; ● Process-tracing, contribution analysis, or other “case” based causal analysis Fieldwork is expected to take place in Tunisia. The evaluation team is expected to conduct fieldwork in each of the four governorates in which the activity is implemented. The evaluation team must propose a methodology that identifies as representative of a cross-section of activity beneficiaries as possible. All beneficiary data should disaggregated by gender and geographic location. USAID will facilitate the introduction of the evaluation team to UNIDO’s Tunis office and the project managers of the HP-UNIDO activity, which will in turn facilitate the evaluation team’s fieldwork. UNIDO will be able to provide data on the beneficiaries of the HP-LIFE trainings (almost 1,200 entrepreneurs so far) that includes: (i) name, DOB, education, degree, years of professional experience, status of project (project idea/business plan/start-up, existing business etc.), sector, activity, investment, number of jobs foreseen and already created, number of HP-LIFE certificates etc.; and (ii) data from the HP-LIFE impact surveys conducted, including the entrepreneurs’ replies. In designing the evaluation, the evaluation team should propose an approach to validate the employment data collected under the HP-UNIDO activity. USAID will provide the evaluation team with the tool HP-UNIDO uses to collect employment data, as well as facilitate an introduction with the HP-UNIDO data manager. Prior to beginning field work, the team is required to share data collection tools with USAID for review and feedback before they are used in the field (e.g., key informant interview questionnaires, focus group discussion protocols, online survey questionnaires, etc.). During this discussion, the evaluation team should also provide USAID with the proposed list of beneficiaries and stakeholders that the team intends to interview during the evaluation and the schedule of meetings. 3) Evaluation Design Matrix and Analysis Plan 47 Prior to beginning field work, the evaluation team must submit for USAID review and approval an evaluation design matrix that details the proposed methodology for data collection and analysis for each of the evaluation questions. The analysis plan should detail what procedures will be used to analyze qualitative and quantitative data; how data from focus group discussions and key informant interviews will be transcribed and analyzed; and how the evaluation will weigh and integrate qualitative data with quantitative survey and performance monitoring data to reach conclusions. 4) Data Quality and Data Limitations The evaluation team must ensure that the data collected clearly and adequately answer the evaluation questions, is sufficiently precise to present a fair picture of performance, and is at an appropriate level of detail to inform conclusions and actionable recommendations. USAID requires that any issues potentially affecting the quality of evaluation data (including the data validity, integrity, timeliness, precision, and reliability) be discussed and documented in the evaluation planning stage and assessed on an ongoing basis during evaluation implementation, including during data collection and analysis. All limitations and measures to address or overcome limitations should be discussed by the evaluation team and USAID in the implementation phase and detailed in the final report. The final evaluation report should be clear and transparent about any notable limitations and if and how they may affect the evaluation’s findings, conclusions, and recommendations. X. Evaluation Deliverables 1. Evaluation Design Matrix, Analysis Plan, and Evaluation Work Plan: The Contractor will submit a final evaluation design matrix, including analysis plan, and evaluation workplan to the Task Order Contracting Officer’s Representative (TOCOR) covering (a) the overall design strategy for the evaluation, (b) the detailed evaluation methodology, including data collection and analysis plan for the evaluation, proposed interviewees, proposed questionnaires, and other data collection tools, and (c) a specific evaluation implementation timeline and fieldwork schedule (using Gantt chart). The evaluation design matrix, analysis plan, data collection tools, evaluation work plan, and budget must receive USAID approval prior to the evaluation team commencing field work. 2. Evaluation Team Planning Meeting. Prior to the onset of the evaluation, the evaluation team should expect to have at least two in-person or phone meetings with USAID, including, but not limited to, the evaluation TOCOR and the activity’s AOR to: a. Review the evaluation questions; b. Review the evaluation work plan; c. Finalize team members’ roles and responsibilities; d. Discuss the draft evaluation matrix; analysis plan; data collection methods, and instruments and tools; e. Review and clarify any logistical and administrative procedures for the assignment; f. Discuss other issues, as needed. The evaluation team should review the evaluation questions with USAID for relevance and feasibility prior to the design matrix being finalized. If any adjustments are made to the evaluation questions, these changes, along with detailed rationale for the changes, should be clearly documented in an Annex to the evaluation. Any changes must receive the approval of the evaluation TOCOR. 3. Draft Evaluation Report and Briefing: The Contractor will submit a draft report and conduct a detailed briefing of preliminary findings and recommendations of the evaluation to USAID, including but not limited to the evaluation TOCOR and activity AOR, within three weeks of field 48 work completion. Recommendations for future job creation programming in Tunisia will be addressed. USAID will review the report and provide written comments on the draft within 7 days of receipt. 4. Final Evaluation Report: The final evaluation report must be submitted within seven working days from of receiving USAID’s comments. The evaluation report should not exceed 30 pages, excluding the title page, annexes, and evaluation data. The report must be submitted initially in English, electronically, and within 60 days a French translation must be submitted. At the time of submission of the final English language report, the survey instruments and data sets, per ADS 203 and ADS 579, must be submitted on a flash drive to the evaluation TOCOR. The final evaluation report should include: 1. Executive Summary (including activity background, activity purpose, evaluation questions, methodology, findings, conclusions, recommendations, and lessons learned, not to exceed four pages). 2. Introduction and Background (including evaluation purpose, background of the activity, award description). 3. Body of Report organized as follows: a. Evaluation Questions; b. Methodology; c. Limitations; d. Key Findings (Evidence); e. Conclusions; and f. Recommendations. 4. Annexes to include the following: statement of work (including any modifications), reference list of documents, list of persons contacted and affiliation (the evaluator should notify USAID if there are any concerns from a privacy or confidentially perspective), detailed methodology (including a discussion of the limitations of the methodology employed), data collection tools and protocols, evaluation team disclosures of real or potential conflicts of interest, references, and any statements of difference submitted by activity implementers, team members, or USAID. 5. Evaluation Data: per guidelines in ADS 203 and ADS 579, any raw quantitative data (e.g., survey responses, etc.) in electronic form collected by the evaluation team, along with relevant codebooks for the data. The data should be organized for use by those not fully familiar with the project or the evaluation. The Contractor will submit four (4) hard copies of the Final Report and one (1) electronic copy in Adobe format to the evaluation TOCOR. The contractor will submit the final English language version of the evaluation report to USAID’s Development Experience Clearinghouse (DEC) within 30 days of USAID’s approval of the report. The contractor should post the French version of the evaluation to the DEC within 7 days of USAID approval. The final evaluation report must conform to the Criteria to Ensure the Quality of the Evaluation Report found in Appendix I of the USAID Evaluation Policy (http://www.usaid.gov/evaluation). This evaluation will not be considered “final” until the evaluation TOCOR has confirmed, in writing, that the report has met all of the quality criteria. General evaluation report guidelines include: ● The evaluation report must represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not, and why; 49 ● The report must include the evaluation Scope of Work as an annex. All modifications, whether in technical requirements, evaluation questions, evaluation team composition, methodology, budget, or timeline must be agreed upon in writing by the AOR; ● Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides must be included in an Annex in the final report; ● Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparison groups, etc.) and what is being done to mitigate the threats to validity; ● Evaluation findings must be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and supported by strong quantitative or qualitative evidence; ● Sources of information must be properly identified and listed in an annex; ● Recommendations must be supported by a specific set of findings; and ● Recommendations must be action-oriented – organized according to whether recommendations are short-term or long-term, practical, and specific, with defined responsibility for the action. XI. Illustrative Timeline and LOE: Stage Activities included Timeframe Planning Review of relevant award info/documents 1 week Evaluation Work Plan submitted and approved; initial evaluation design matrix submitted Evaluation Team meeting with USAID Preparations Finalize, with USAID approval, evaluation design matrix and detailed data analysis plan 1.5 weeks USAID meeting with the Evaluation Team to discuss fieldwork schedule, list of interviewees, and data collection tools. Finalize list of interviewees and site visit schedule 50 Field implementation Conduct Evaluation Fieldwork in Tunisia 2.5 weeks Data Analysis Analysis of data and findings 2 weeks Final Evaluation Report and Dissemination USAID feedback incorporated and final evaluation report submitted 3 weeks Dissemination of final evaluation report findings, conclusions, and recommendations via in-person meeting or teleconference Total Duration 10 weeks *A week could be 6 working days. XII. Evaluation Team Composition The following three positions are considered key personnel. - Evaluation Team Leader: The Team Leader will be a senior-level evaluation expert, with at least 10 years of experience in leading and conducting evaluations of development activities, with experience in social science evaluation methods, especially performance evaluations, qualitative data collection methods, and other relevant data collection/analysis techniques. Minimum Master degree required. Experience working with USAID evaluations is required. The Team Leader will ideally have technical economic growth expertise or experience managing economic growth activities, preferably in the MENA region. Technical experience related to job creation and/or workforce development programming is strongly preferred. - Economic Growth Technical Expert The Economic Growth Technical expert should be an expert in economic growth programming, with a particular focus on entrepreneurship and job creation programming in the Middle East and North Africa. Previous experience with USAID and economic growth-related evaluations strongly preferred. - Local M&E Specialist The local M&E specialist should have strong experience in social science methods and at least 5 years of relevant experience, including experience conducting performance evaluations. Relevant economic growth technical experience strongly preferred. All members of the evaluation team should have superior writing, English language, interpersonal, and analytical skills. The names of the evaluation team leader and economic growth technical expert should be included in the Contractor’s proposal, along with key skills relevant to this evaluation. Current Curriculum Vitae for the evaluation team leader and economic growth technical 51 expert should be included as an annex to the Technical Proposal. A curriculum vitae for the local M&E specialist should be provided to USAID during the initial evaluation team planning meeting. Please note that the evaluation team may be accompanied by an independent evaluator from UNIDO who would serve as an additional member of the evaluation team. All salary, travel, and other costs related to the independent evaluator will be covered by UNIDO. XIII. Evaluation Management 1. Logistics The evaluation team is solely responsible for logistics but should confer with USAID’s Middle East Bureau and the USAID Tunisia Office regarding security or other concerns. USAID will provide key documents to the evaluation team and provide introductions to the implementing partners prior to the initiation of field work. The evaluation team is responsible for arranging in-country flights, vehicle rental, and/or drivers as needed for their field work and site visits. They will also need to arrange their own hotel arrangements and procure their own work/office space, computers, internet access, printing and photocopying in line with relevant USAID regulations. Evaluation team members will be required to make their own payments. 2. Period of Performance The evaluation is expected to commence in mid-to-late February 2016. The expected duration of the evaluation is approximately 10 weeks from commencement to the delivery of the final report. 52 Annex One. HP-UNIDO Results Framework 53 Annex Two. USAID Economic Growth Programming in Tunisia Current Activities  Tax and Customs Reform Project: USAID provides technical assistance in support of the Government of Tunisia’s economic reform agenda necessary to grow the economy and create new economic opportunities for Tunisians.  Business Reform and Competitiveness Project (BRCP): BRCP works with high potential firms in Tunisia to strengthen and enhance entrepreneurship in critical areas and leverage sustainable debt and equity financing for SMEs. The project expands employment opportunities with partner firms by working through key partner organizations in the public and private sector. The program added 3963 jobs to private Tunisian firms during its first year of operation.  Tunisian-American Enterprise Fund (TAEF): USAID has provided $60 million to the TAEF to promote the development of the Tunisian privately owned small and medium-sized enterprises through direct investments into those Tunisian enterprises. USAID expects to provide an additional $40 million to the TAEF in FY 16 and FY 17 to capitalize the fund at $100 million. The TAEF made its first investment in 2014 and plans to make additional investments over the next year.  Center for Entrepreneurial and Executive Development: CEED drives economic growth by developing, connecting and mentoring entrepreneurs to strengthen their businesses so they can create jobs and in turn accelerate economic prosperity.  HP/UNIDO: USAID has partnered with the Government of Tunisia, Government of Italy, HP, and UNIDO to foster job creation and entrepreneurship with a focus on youth in interior regions by promoting youth-led enterprise creation and development. The project is on track to create nearly 1,000 jobs by the end of 2015. Past Activities  Tunisia SME Development: From Oct 2012-September 2014, this $1.2 million program worked to strengthen Tunisia’s economic competitiveness and its prospects for sustained growth by promoting greater involvement of small and medium enterprises (SME) in trade- and investment-related activities. The project supported small business development centers (SBDCs) in Tunisia and technical assistance to export-ready or near export-ready SMEs to boost Tunisian exports and promote broad-based economic growth.  Tunisia ICT Workforce Development: From June 2012-September 2013, USAID completed an $8 million project to support competitiveness, trade and export in Tunisia’s internet communication and technology (ICT) sector. The project featured three components: - Component 1: Expanded ICT entrepreneurship and SME development. - Component 2: Increased workforce development and employment in Tunisia’s ICT and ICT￾enabled sectors. - Component 3: Removal of business and regulatory constraints to SME growth in Tunisia. 54 Annex II: Key Informant Interview (KII) Guide Individual and group semi-structured instrument Targets:  USAID: AOR and others  HP‐UNIDO personnel  Participating business support organizations: ISETs, SBSs  Trainers  Government of Tunisia: ministries and in governorates  Donors  Local youth and workforce development experts  Participating entrepreneurs  Other beneficiaries Note: Skip items not relevant to the particular target interview. 1. Identifying information: 1.1 Name(s)    1.2 Affiliation(s) 1.3 Title/position(s) 1.4 Gender 1.5 Location of KII 2. Did you play a role in the design, organization or funding of the HP UNIDO Mashrou3i project? If so please describe: 2a. Do you think that there were benefits of USAID awarding this contract under the Global Development Alliance (GDA) approach with a Public International Organization: 55 3. The objective of the HP‐UNIDO Mashrou3i Project is to “create job opportunities for young Tunisians.”   Briefly, what are the challenges Tunisian young people face in securing sustainable employment and entrepreneurship? 4. For representatives of implementing partners and cooperating organizations only: What role does your organization play in supporting and implementing the HP‐UNIDO Mashrou3i Project? Please describe: 4.a How did your partnership in the HP-UNIDO Mashrou3i Project affect your organization’s capacity to deliver services for young people? 56 5. What are the particular employment challenges facing young people in the interior governorates, young women and marginalized groups such as handicapped populations? (if not mentioned in Q3 Above): 6. In what ways has the HP‐UNIDO Mashrou3i Project succeeded in creating sustainable employment for Tunisian young people (for you)?: Comments: 7. In what ways if any, has SBS participation in the HP‐UNIDO Mashrou3i Project contributed to the achievement of employment outcomes? Please explain: 8. The HP UNIDO Mashrou3i Project consists of several components: Of which specific component(s) are you knowledgeable (or participated)? Briefly describe your involvement in the component(s):   HP on‐line Coaching/mentoring:   Workshops: Other:   None: 57 training: 9. Which components of the HP‐UNIDO Mashrou3i Project do you think have been most and least effective in creating sustainable employment and why is this so? Are there other programs and projects working on youth employment and entrepreneurship that have been particularly effective? What has made them successful? 10. In what specific ways can the HP‐UNIDO Mashrou3i Project improve in its goal of helping young people gain sustainable employment? 11. Which populations do you think have benefited most from the HP‐UNIDO Mashrou3i Project? Why is this so?: 58 12. Are there any populations of Tunisian vulnerable young people for whom the HP‐UNIDO Mashrou3i Project would not be an appropriate or useful intervention? What would need to done to succeed with these specific populations? Please explain:     13. In what ways does UNIDOs practice of working with local community groups, organizations and parastatals contributed to the effectiveness of the HP‐UNIDO Mashrou3i Project? Please explain: 14. From your perspective, have there been any particular challenges affecting HP UNIDO’s work with these local organizations? How have they been addressed? 59 15. What can you tell us about the process through which the HP‐UNIDO Mashrou3i Project recruits and selects participants? (Alternate: please describe the process through which you were recruited and selected to participate in the HP‐UNIDO Mashrou3i Project?): 15a. Do you have any suggestions as to how to improve the recruitment process? 16. Do you know of other projects (or, have you participated in other projects) addressing youth employment issues in Tunisia that you regard as particularly successful or interesting?  Please describe: 60 17. What do you see as the trade‐offs in implementing youth employment and training projects in the interior governorates rather than in the population centers as  a means of addressing youth unemployment in the interior? 18. For respondents with knowledge of the HP‐UNIDO employment information system only: How would you describe the strengths and weaknesses of the employment information system? 18a. What specific procedures are employed to test the data’s validity and reliability? 18b. Do you have any suggestions to improve the validity, reliability or efficiency of the employment information collection, storage and analysis system?: 61 19. Do you have any further thoughts, comments or suggestions about the HP‐UNIDO Mashrou3i Project that you would like to share?   We greatly value and appreciate your participation in the evaluation. Thank you. 62 Annex III: Focus Group Discussion (FGD) protocol - Beneficiaries Evaluation Team Note: the FGDs seek to elicit qualitative nuance and context that will help us better understand, interpret and situate the data from the large survey instrument. Illustrative quotes and differing viewpoints may be particularly valuable. Time may not permit addressing all prompts at each FGD. PROTOCOL: Introduction: Since 2013, Mashrou3i has been working with young people in communities in the Center West regions of Tunisia acquire information technology (IT) and business skills that will help then find employment or create or expand their businesses. Each of you here today participated in one or more Mashrou3i sponsored activities or events. The aim of this focus group discussion is to enable us to learn as much as we can about your experiences and your views of the project. Ground Rules: First, here are a few “ground rules” to help us enjoy a productive discussion: 1. Only one person should speak at a time; 2. Please no side conversations with those sitting near you; 3. Let’s avoid having one or two people dominate the conversation; and 4. Be sure to hear from everyone; we want to hear as many different voices, stories and perspectives as possible. Opening Prompt (optional, as a way of encouraging discussion): To get started, we will go around the room asking everyone to briefly respond to the following question: What one key fact should we know about your local community that is important in understanding the challenges facing young people in finding employment or starting or expanding small businesses? Follow-On Prompts: 1. How did you first learn of Mashrou3i? Why did you want to participate in the project? 2. In what ways has your life changed as a result of participating in Mashrou3i? 3. How do the challenges of employment and small business development in your community affect you as young women or men? 4. Has Mashrou3i provided additional support beyond the HP-LIFE program? How useful this been to help you achieve your goals? Are there areas in which you need additional advice or support? 5. Overall, what is your assessment of Mashrou3i? What are your suggestions for making it more effective? Concluding Statement: Thank you so much for participating in this focus group discussion. Your contributions have been quite helpful to our evaluation work. Should you find that you have other inputs to share or other comments or suggestions please contact us at: jstatman@internationaldevelopmentgroup. com. 63 Annex IV: Individual Beneficiary Survey Questionnaire 64 65 66 67 68 69 70 71 72 73 Annex V: Institutional Beneficiary Survey Questionnaire # Results Framework Reference Category Question Response 1 Response 2 Response 3 Response 4 Response 5 1 NA Basic Info Name of organization x x x x 2 NA Basic Info Address x x x x 3 NA Basic Info Region Tunis etc. etc. etc. etc. 4 NA Basic Info Last Name of person interviewed x x x x 5 NA Basic Info First Name of person interviewed x x x x 6 NA Basic Info Title x x x x 7 NA Basic Info E-mail address x x x x 8 NA Basic Info Telephone x x x 9 NA Basic Info Ownership of organization? Governm ent Private x x x 10 NA Basic Info If private, what is the legal status? Non-profit Limited Liability company Corporati on x x 11 NA Basic Info What phrase best describes your organization? Business support organizati on Training provider Both 12 NA Services Received Have you received any services from Mashrou3i? (If yes, then rate the following services that you received on a scale of 1 to 5, where 1 is poor and 5 is excellent. Do not rank services that you did not receive.) Yes No x x x 13 1.2.3 Services Received Assistance in providing training for graduates and young entrepreneurs. 1. Poor 2. Fair 3. Good 4. Very Good 5. Excellent 14 1.2.4 Services Received Assistance in the implementation of a quality control on the training you provide. 1. Poor 2. Fair 3. Good 4. Very Good 5. Excellent 74 # Results Framework Reference Category Question Response 1 Response 2 Response 3 Response 4 Response 5 15 1.2.5 Services Received Assistance in the development of a training and assistance module related to access to finance for start-ups. 1. Poor 2. Fair 3. Good 4. Very Good 5. Excellent 16 1.2.6 Services Received Assistance in the development of a training and assistance module related to environmental training (Green business plan and TEST methodologies) 1. Poor 2. Fair 3. Good 4. Very Good 5. Excellent 17 1.2.7 Services Received Training on TEST methodology 1. Poor 2. Fair 3. Good 4. Very Good 5. Excellent 18 1.3.2 Services Received IT equipment to facilitate HP-LIFE trainings 1. Poor 2. Fair 3. Good 4. Very Good 5. Excellent 19 1.3.3 and 1.3.4 Services Received HP-LIFE e-learning training of trainers 1. Poor 2. Fair 3. Good 4. Very Good 5. Excellent 20 NA Impact Rate the value or utility of the HP-LIFE program as a tool for training and empowering entrepreneurs and employees. 1. Poor 2. Fair 3. Good 4. Very Good 5. Excellent 21 NA Impact How has Mashhou3i affected your organization's services to entrepreneurs? (mark one only) Helped us to offer new services. Helped us to improve existing services. Helped us to both offer new services and improve existing services. No impact. x 22 NA Impact How has Mashhour3i affected the number of your clients? (mark one only) Increased it. Decrease d it. No impact. x x 23 NA Impact Overall, how has Mashhour's affected your ability to serve your clients? (mark one only) Improved it. Weakene d it. No impact. x x 75 # Results Framework Reference Category Question Response 1 Response 2 Response 3 Response 4 Response 5 24 NA Impact Do you have additional comments that you would like to make about Mashrou3i? 25 NA Basic Info May we contact you for additional information? (yes, no) Yes No x x x 76 Annex VI: Evaluation Design Matrix Evaluation Question Sub-questions Evidence Source(s) Analysis 1. To what extent has the HP-UNIDO implementation approach been effective in creating sustainable employment? How do beneficiaries rate HP-UNIDO effectiveness in helping them achieve sustainable employment? FGD e-survey KII Narrative Mean rating scores; Qualitative analysis; Case analysis Are there regional/gender differences in Beneficiaries’ assessments? FGD e- survey KII Narrative Above data disaggregated by gender and location; 4x2 tables How do sustainable employment results compare to activity targets? PMP Quarterly reports Yearly comparisons How do ISET and university personnel beneficiaries rate HP￾UNIDO effectiveness in helping beneficiaries achieve sustainable employment? KIIs Mean rating scores; Qualitative analysis How do GOT personnel assess HP-UNIDO effectiveness in helping them achieve sustainable employment? KIIs Qualitative assessments How do those who received HP LIFE training at ISETs or universities rate the effectiveness and usefulness of their training? FDGs e-survey Mean rating scores; qualitative assessments Does case study analysis support causal relationship between HP UNIDO intervention and sustained employment? Narrative Process tracing 1a) Which aspects of the implementation approach have been the most effective in creating sustainable employment and why. Do beneficiaries who have taken HP LIFE on-line courses differ in their assessment of effectiveness from those who also received mentoring and coaching, or ISET or university-based training? e-survey Compare mean scores What component(s) of the intervention model do participants rate as most important and useful? e-survey FDG Ratings; qualitative comments What component(s) of the intervention model do HP UNIDO, ISET and university personnel believe to be most efficacious? KIIs Ratings; qualitative comments Do employment outcomes differ for those participating in on￾line learning; on-line learning plus mentoring; ISET or university workshops? HP UNIDO Employment data base reports Compare mean outcomes by months employed 1b) Which populations have benefited most from the activity and why? How have women and youth in Are there regional/gender differences in the employment outcomes of participants? HP UNIDO employment data base reports Employment outcomes disaggregated by gender and location Did HP UNIDO take any steps during implementation to KIIs KII qualitative input; 77 Evaluation Question Sub-questions Evidence Source(s) Analysis the four regions differentially been reached and benefited? facilitate inclusion of particular hard-to-serve populations? Quarterly reports document review Which sub-populations, including youth, do HP UNIDO personnel believe benefitted most from the program? KIIs KII qualitative analysis Are there some sub-populations that HP UNIDO personnel found less-likely to benefit from their intervention? KIIs KII qualitative analysis Are there regional/gender differences in participants’ assessments of the usefulness of the program? FGDs e-survey Qualitative analysis mean ratings 1c) How were participants identified? Can anything be concluded about the general characteristics (e.g. educational or professional background of participants vs non￾participants? What procedures did HP UNIDO utilize to inform potential participants of their program? KIIs Quarterly report Qualitative analysis; document review How did participants learn of the program? What factors led them to apply? KIIs Quarterly report Qualitative analysis; document review What was HP UNIDOs strategy in selecting participants (e.g. most qualified; most vulnerable; gender balance; other)? KIIs HP-UNIDO proposal and quarterly reports Qualitative analysis; document review What procedures and criteria were used to select/exclude participants? KIIs Qualitative analysis What percentage of interested applicants were selected? HP-UNIDO data base reports Document review How do participants assess the selection process? e-survey Mean scores 1d) How did the HP UNIDO approach of working directly with local communities influence individual motivations or job training or placement outcomes? How do participants view the value of accessing the program through local institutions? FGDs Qualitative analysis To what extent do local institutions participating in the project or receiving technical assistance and capacity-building support believe that these interventions have improved job training and client placement? KIIs Qualitative analysis How did HP UNIDO identify and select local implementing partners? KIIs Qualitative analysis What factors affected local partners’ success in recruiting participants? KIIs Qualitative analysis How does HP-UNIDO assess the value and importance of working through community institutions? KIIs Qualitative analysis Are there any lessons learned from other employment/livelihoods projects with similar populations and locations to support a community-focused design? KIIs Qualitative analysis 78 Evaluation Question Sub-questions Evidence Source(s) Analysis 2. Relative to a traditional contract model, were there any benefits to using the GDA approach? Were there any externalities of using a GDA approach that facilitates external resource partners and a PIO approach that attracts the resources of additional donors? How do USAID, HP, UNIDO and GOI managers assess the value and utility of the GDA mechanism as compared to more traditional contractual approaches? KIIs Qualitative analysis Are records available documenting USAID’s decision to adopt the GDA approach for this activity? Contract documents Document review Is there evidence to support the assertion that the activity agreement was awarded in less time than other USAID activities in Tunisia? USAID award data Document review 3. To what extent did UNIDO’s cooperation with existing SBSs contribute to employment outcomes? (Consider UNIDO’s business model, years of experience working with indigenous parastatals and cost structure.) How do HP UNIDO personnel rate the importance of SBSs in the project? KIIs Compute mean scores; qualitative analysis How do HP UNIDO personnel rate the effectiveness of cooperation with SBSs? KIIs Compute mean scores; qualitative analysis How have the SBSs worked with UNIDO prior to this project? KIIs Qualitative analysis How do SBS managers present their role in the project? KIIs Qualitative analysis How do SBS personnel rate the effectiveness of their cooperation with UNIDO on this project? KIIs Qualitative analysis What suggestions do SBS and UNIDO personnel have to improve cooperation and outcomes? KIIs Qualitative analysis 4. To what extent can the employment information collected by the HP￾UNIDO activity be independently validated and deemed credible? Based on the summative findings of item 4a-4c (below) does the ET find sufficient evidence to support the validity and credibility of the project employment data system? 4a) How does HP-UNIDO define and measure employment creation? What is the operational definition of employment creation used in the project? HP-UNIDO documents Document review Does the project definition consistent with that used by relevant GOT structures? GOT documents Document review What methodology is used by HP UNIDO to collect employment data? HP-UNIDO reports and procedures; KIIs Document review; qualitative analysis 79 Evaluation Question Sub-questions Evidence Source(s) Analysis How are the employment project data managed, analyzed and stored? KIIs; HP-UNIDO procedures; on-site review Document review; qualitative analysis; direct observations Are the procedures for data collection standard across the four governorates? KIIs; document review Document review; qualitative analysis Were there any changes in the process or timing of data collection during the course of the project? KIIs; document review Document review; qualitative analysis 4b) Given the methodology and procedures used to collect employment information, to what extent are the results from the routine activity monitoring likely to be over or under-estimated? What procedures were used to assess the validity and reliability of the employment collection and analysis system? HP-UNIDO procedures; KIIs Document review; qualitative analysis What were the primary challenges the project faced in implementing the data collection system? KIIs; project procedures Document review; qualitative analysis Were any changes made during the course of the project to strengthen the data system? KIIs; project procedures Document review; qualitative analysis 4c) What methodologies, processes, systems or structures have facilitated or impeded the collection of high quality and credible employment data? What gaps or limitations, if any, do HP UNIDO personnel identify in their data collection system? KIIs Qualitative analysis Does a rapid review of the HP UNIDO data collection system indicate any gaps or limitations that could plausibly affect the validity and reliability of the data? Systems review (KIIs and design review) Document review; qualitative analysis What exogenous factors if any, may have affected the validity or reliability of the data? KIIs Qualitative analysis 4d) Based on the data collected by HP-UNIDO, can cost per job created – including management costs and in-kind contribution - be accurately estimated? Provide simple benefit/cost calculation to determine whether the benefits of the jobs created exceeded costs (e.g. compare cost per job Does HP UNIDO have full-cost data available? Data base output Document review Are data readily available to reasonably estimate wages of program participants? Data base output Document review Can wage data be disaggregated by type of program participation (i.e. on-line training; on-line plus coaching; workshop)? Data base output Document review Can wage data be reasonably disaggregated by gender and location? Data base output Mean wages compared, 4x2 tables 80 Evaluation Question Sub-questions Evidence Source(s) Analysis with wage data). Does HP UNIDO data document change in value or profits of participating entrepreneurs? Data base output Comparison of mean value by year 81 Annex VII: List of Key Informant Interviews (KIIs) & Focus Group Discussions (FGDs) Mtg # Person # Organization Acronym Gov (G) Priv (P) Int'l (I) Office or Unit if applicable First Name Last Name Title Region Product Day and Date Time Meeting in 1 1 U.S. Agency for International Development USAID I Glenn Rodgers Tunis Thursday, March 24, 2016 9:00 Tunis 2 U.S. Agency for International Development USAID I Rym Ben Aisa Tunis Thursday, March 24, 2016 9:00 Tunis 3 U.S. Agency for International Development USAID I Slim Bostangi Tunis Thursday, March 24, 2016 9:00 Tunis 2 4 United Nations Industrial Development Organization UNIDO I Maurizio Bonavia Tunis Thursday, March 24, 2016 11:00 Tunis 3 5 United Nations Industrial Development Organization UNIDO I HP-LIFE Mahmoud Chouchene Director Tunis Friday, March 25, 2016 9:00 Tunis 4 6 Agence de Promotion de l'Industrie et d'Innovation APII G Centre de soutien a la Creation d'entreprises (CSCE) Mohamed Arfa Director Tunis Friday, March 25, 2016 10:00 Tunis 5 7 Agence Nationale pour l'Emploi et le Travail Independent ANETI G Jalel Amri Cadre (HP LIFE Facilitator) Kairouan Monday, March 28, 2016 10:00 Sousse 6 8 Cyber Parc Kairouan G Maryiem Zrigue Directrice (HP LIFE Facilitator) Kairouan Monday, March 28, 2016 10:00 Sousse 7 9 Focus Group of HP-LIFE Trainees P Kairouan Monday, March 28, 2016 10:00 Sousse 82 Mtg # Person # Organization Acronym Gov (G) Priv (P) Int'l (I) Office or Unit if applicable First Name Last Name Title Region Product Day and Date Time Meeting in 8 10 Institut Superieur d'Etudes Technologiques ISET ISET G Anis Hamrouni Vice Director, Department Head (HP LIFE Educator) Kairouan Monday, March 28, 2016 11:00 Sousse 9 11 Speech Pathologist P Owner Kairouan Monday, March 28, 2016 13:00 Sousse 10 12 Karwisoft P Sofien Khelif CEO Kairouan Monday, March 28, 2016 14:00 Sousse 11 13 Career Center Kairouan G Director Kairouan Monday, March 28, 2016 15:00 Sousse 12 14 Agence de Promotion de l'Industrie et d'Innovation APII G Incubator Director Kairouan Monday, March 28, 2016 16:00 Kairouan 15 Centre d'affaires G Mehdi Oueslati Director Kairouan Monday, March 28, 2016 16:00 Kairouan 13 16 Institut Cyrta de Formation P Yemen Yakoubi Directeur Le Kef Training Tuesday, March 29, 2016 10:00 Sousse 17 Sirhan P Sirhan Founder, CEO Le Kef Paper Tuesday, March 29, 2016 10:00 Sousse 18 Tunisian Cloud P Bassem Lamouchi Founder, CEO Le Kef IT Tuesday, March 29, 2016 10:00 Sousse 14 19 United Nations Industrial Development Organization UNIDO I HP-LIFE All Tuesday, March 29, 2016 12:00 Sousse 15 20 Shawki P Shawki Kasserine Interior Decoration Tuesday, March 29, 2016 14:00 Sousse 83 Mtg # Person # Organization Acronym Gov (G) Priv (P) Int'l (I) Office or Unit if applicable First Name Last Name Title Region Product Day and Date Time Meeting in 21 Tewfik P Tewfik Kasserine Apparel Tuesday, March 29, 2016 14:00 Sousse 22 Wafa P Wafa Kasserine On-line Magazine Tuesday, March 29, 2016 14:00 Sousse 16 23 Ahmed P Ahmed Sidi Bouzid Architecture Wednesday, March 30, 2016 10:00 Sousse 24 Shahed P Shahed Sidi Bouzid Optician Wednesday, March 30, 2016 10:00 Sousse 25 Somim P Amin Ouni Sidi Bouzid Nails Wednesday, March 30, 2016 10:00 Sousse 26 Wiam P Wiam Sidi Bouzid Plant nursery Wednesday, March 30, 2016 10:00 Sousse 27 Younes P Younes Sidi Bouzid Metal Furniture Wednesday, March 30, 2016 10:00 Sousse 17 28 Compass IT P Hodami Kairouan Wesite and software developer Wednesday, March 30, 2016 14:00 Sousse 29 Geomatics Engineering P Mohamed Anis Ben Romdhane CEO and Founder Kairouan GIS Wednesday, March 30, 2016 14:00 Sousse 30 Infirmary P Hoda Kairouan Infirmary Wednesday, March 30, 2016 14:00 Sousse 31 MACDEV P Amin Shared Kairouan ICT Developer Wednesday, March 30, 2016 14:00 Sousse 32 NT Nouvelle Technologie P Walid Hlioui Manager Kairouan 3D Printing Wednesday, March 30, 2016 14:00 Sousse 33 Romeo Call P Salem Hammami Kairouan Call Center Wednesday, 14:00 Sousse 84 Mtg # Person # Organization Acronym Gov (G) Priv (P) Int'l (I) Office or Unit if applicable First Name Last Name Title Region Product Day and Date Time Meeting in Center March 30, 2016 18 34 Rawen P Tewfik Segni Manager Kairouan Pickles Wednesday, March 30, 2016 12:00 Sousse 19 35 GIPA P Hakim Manager Kairouan Ice cream Thursday, March 31, 2016 10:00 Tunis 20 36 Sabri Hammami Recyclage SHR P Sabri Hammami Owner Kairouan Recycled plastic Thursday, March 31, 2016 12:00 Tunis 21 37 Rawen P Tewfik Segni Manager Kairouan Pickles Thursday, March 31, 2016 14:00 Tunis 22 38 Ministry of Higher Education MHE G Direction Generale des Parternariat Ismail Haddad All Friday, April 01, 2016 Tunis 39 Ministry of Higher Education MHE G Direction Generale des Parternariat Friday, April 01, 2016 Tunis 23 40 Centre d'affaires G Sidi Bouzid Friday, April 01, 2016 Tunis 24 41 Le Marbrerie Briu P Ouannes Missaoui Director General Kasserine Friday, April 01, 2016 Tunis 25 42 Rachid Nafti P Rachid Nafti Rachid Nafti Independent Consultant Tunis Sunday, April 03, 2016 Tunis 26 43 Agence de Promotion de l'Industrie et d'Innovation APII G Sidi Bouzid Regional Office Mohsen Akrimi Director Sidi Bouzid Monday, April 04, 2016 10:00 Tunis 44 Agence de Promotion de APII G Le Kef Incubator Abbes Bouloubi Director Le Kef Monday, April 04, 11:00 Tunis 85 Mtg # Person # Organization Acronym Gov (G) Priv (P) Int'l (I) Office or Unit if applicable First Name Last Name Title Region Product Day and Date Time Meeting in l'Industrie et d'Innovation 2016 27 45 Centre des Jeunes Dirigeants d'Entreprise CJD P Amira Ellouze Member of the Executive Bureau All Monday, April 04, 2016 11:00 Tunis 46 Centre des Jeunes Dirigeants d'Entreprise CJD P Wafa Laamiri Présidente Nationale Monday, April 04, 2016 11:00 Tunis 28 47 Confederation des Entreprises Citoyens de Tunisie CONECT P Douja Ben Mahmoud Gharbi First Vice President All Monday, April 04, 2016 12:00 Tunis 29 48 Office de Development Centre Ouest ODCO G Rafiq Giesme Kasserine Monday, April 04, 2016 15:00 Tunis 30 49 Esprit University P Alaya Bettaieb All Tuesday, April 05, 2016 10:00 Tunis 31 50 Silatech I Ali Munif All Tuesday, April 05, 2016 11:00 Tunis 32 51 L'Agence Italienne pour la Cooperation au developpement I Cristina Napoli Representative All Wednesday, April 06, 2016 10:00 Tunis 33 52 U.S. Agency for International Development USAID I Kay Freeman Acting Representative All Wednesday, April 06, 2016 11:00 Tunis 53 U.S. Agency for International Development USAID I Rym Ben Aisa Wednesday, April 06, 2016 11:00 Tunis 54 U.S. Agency for International Development USAID I Slim Bostangi Wednesday, April 06, 2016 11:00 Tunis 86 Mtg # Person # Organization Acronym Gov (G) Priv (P) Int'l (I) Office or Unit if applicable First Name Last Name Title Region Product Day and Date Time Meeting in 55 U.S. Agency for International Development USAID I Wednesday, April 06, 2016 11:00 Tunis 34 56 Banque Africaine du Developpement BAD I Souk Ittanmia Wassila Dridi Expert in Follow-up and Evaluation All Wednesday, April 06, 2016 12:00 Tunis 57 Banque Africaine du Developpement BAD I Souk Ittanmia Mohamed Wednesday, April 06, 2016 12:00 Tunis 58 Banque Africaine du Developpement BAD I Souk Ittanmia Baha Director Wednesday, April 06, 2016 12:00 Tunis 35 59 Business Reform and Competitiveness Project BRCP I Amel Mankai Chief of Party Thursday, April 07, 2016 9:00 Tunis 36 60 Center for Entrepreneurship and Executive Development CEED I Wafa Makhlouf Executive Director Thursday, April 07, 2016 13:00 Tunis 61 Center for Entrepreneurship and Executive Development CEED I Sonia Ben Hajji Project Manger Thursday, April 07, 2016 13:00 Tunis 37 62 Reseau Entreprendre RE P Allagui Anis External Consultant, Ernst & Young Thursday, April 07, 2016 15:00 Tunis 63 Reseau Entreprendre RE I Zeinab Messaouid Director Thursday, April 07, 2016 15:00 Tunis 38 64 United Nations Industrial Development Organization UNIDO I Maurizio Bonavia Friday, April 08, 2016 9:30 Tunis 87 Mtg # Person # Organization Acronym Gov (G) Priv (P) Int'l (I) Office or Unit if applicable First Name Last Name Title Region Product Day and Date Time Meeting in 65 United Nations Industrial Development Organization UNIDO I Najah Cheniti Friday, April 08, 2016 9:30 Tunis 39 66 Centre d'Orientation et de Reconversion Professionelle CORP I Youssef Fennira Director Friday, April 08, 2016 11:00 Tunis 88 Annex VIII: List of Job Creation Programs in Tunisia # Program Name or Type Acronym Implementer Acronym Funding Agency Type of assistance Region 1 Business Reform and Competitiveness Project BRCP Pragma USAID Enterprise Development Coastal 2 Mashrou3i UNIDO USAID, Cooperation Italien Enterprise Development Interior 3 Center for Entrepreneurship and Executive Devlopment CEED Center for Entrepreneurship and Executive Devlopment CEED USAID Entrepreneurship Development Coastal 4 Creation d'entreprise et formation des entrepreneurs CEFE National Agency for Employment and Independent Work ANETI Government of Tunisia Entrepreneurship Development All 5 Entrepreneurship Development Esprit University Tunisian Businesses Entrepreneurship Development Coastal 6 Institut Superieur d'Etudes Technologiques ISET Institut Superieur d'Etudes Technologiques ISET ISET Tunisian Government Entrepreneurship Development All 7 Mashrou3i UNIDO USAID, Cooperation Italien Entrepreneurship Development Interior 8 Reseau Entreprendre RE Qatari Friendship Fund Qatari Friendship Fund Entrepreneurship Development Coastal 9 Souk Ittanmia African Development Bank AfDB African Development Bank Entrepreneurship Development All 10 Thniti (My road) Confederation des Entreprises Citoyens de Tunisie CONECT Qatari Friendship Fund Entrepreneurship Development Coastal 11 Business Reform and Competitiveness Project BRCP Pragma USAID Career Guidance, Training, Placement Coastal 12 Centre d'Orientation et de Reconversion Professionelle CORP German Tunisian Chamber of Commerce and Industry Germany Career Guidance, Training, Placement Coastal 13 Education for Employment EFE Education for Employment EFE Career Guidance, Training, Placement All 14 Fursati Ministry of Higher Education Career Guidance, Training, Placement All 15 Nagehni Silatech Career Guidance, Training, Placement 16 Tounes Ta3mil Ministry of Higher Education MHE Silatech, Qatari Friendship Career Guidance, Training, Placement All 89 # Program Name or Type Acronym Implementer Acronym Funding Agency Type of assistance Region Fund (QFF) 17 Microfinance Banque Tunisienne de Solidarite BTS GOT Finance All 18 Microfinance Enda Finance 19 Microfinance Taysir Finance Kef 20 SME Finance Banque pour le financement des petits et moyennes entreprises BFPME GOT Finance All 21 Tunisian American Enterprise Fund TAEF Tunisian American Enterprise Fund USAID Finance Coastal 22 Business Reform and Competitiveness Project BRCP Pragma USAID Policy All 23 Centre des Jeunes Dirigeants d'Entreprise Centre des Jeunes Dirigeants d'Entreprise CJD Tunisian Businesses Networking Coastal * Programs mentioned more than once implements more than one job supporting activity. 90 Annex IX: Complexities of Measuring Jobs Created and their Costs While job creation and the cost of job created can be important indicators of the success of program, there are limitations to both measures that should be noted. 1) Conflicts between Private and Public Objectives: While job creation is a central public objective, it is squarely at odds with the objectives of those who actually create jobs, including private enterprises, non-profits, and government. Good managers in any organization, public or private, increase employment only to help meet the demands of their clients. To the extent they can, managers seek to control costs by minimizing job creation to provide the highest value per unit cost. 2) Attribution: Attribution describes a direct causal link between activities and outcomes, based on an estimation of the counterfactual situation. In the complex world of private sector development, attributing outcomes to activities can be extremely challenging. There are many changes that may occur without any influence from a development project. It nonetheless seems reasonable to tentatively attribute job creation to a donor when there is evidence jobs were created and somehow linked to donor activities in the following illustrative areas:  development and marketing of a new product;  establishment of new markets for existing products;  raising finance for business expansion; and  reducing costs that result in increased competitiveness and expanded sales. 3) Job Destruction: New jobs in one company may result in a loss of jobs at other companies (either in the country assisted or elsewhere), thus reducing net social gains. In other words, development programs may just be slicing the pie differently, rather than making the pie bigger. Even if the job creator is producing a better and/or less costly product, one has to ask whether the social benefits of such support outweigh the social costs. 4) Calculating Costs: Calculating the costs of a new job is difficult and can be thought of from different perspectives. There is the cost to any one donor agency that can be measured. All donor support can be measured. The costs to a firm can be measured. Or all costs in total, the overall social cost, can be measured. Furthermore, it should not be assumed that a cheaper job is a better job. A job in a capital intensive industry costs more than one in a labor intensive industry. A job in a start-up is probably more expensive than a job in an existing enterprise. However, there are times when such ‘expensive’ jobs make sense from an economic point of view. 91 Annex X: Disclosure of Any Conflicts of Interest 92 X Middle East Bureau/Technical Services U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523