education and employment alliance: An Evaluation of Partnerships in Support of Youth Employability Global Report April, 2010 2 ACKNOWLEDGMENTS The Education and Employment Alliance (EEA) Evaluation Report represents the tireless dedication, remarkable teamwork, and significant contributions of many individuals and organizations. First and foremost, the EEA study owes a deep debt of gratitude to the young people, their parents, trainers, and community members who participated in this global evaluation. We also wish to express our significant gratitude to our main counterpart at the United States Agency for International Development (USAID), Chris Capacci-Carneal, for her strong support and guidance during the program and in the conduct of this evaluation. We are similarly indebted to previous counterparts for the program at USAID/Washington including Roberta Cavitt, Joe Williams and our original partner in this effort, Seema Agarwal-Harding. Significant support was also provided by the following individuals who served as our most recent, main USAID counterparts in each country of operation and contributed to the design and implementation of each EEA country program: Robert Burch of USAID/Egypt; Sourav Banerjee of USAID/India; Chimi Thonden of USAID/Indonesia; Jane Casewit of USAID/Morocco; Zachary Orend of USAID/Pakistan; and Tom Crehan of USAID/Philippines. They join many others at USAID in Washington and in each country who we have had the pleasure of working with over the last five years. We are also indebted to the following organizations and individuals for their leadership and invaluable contributions in making this program a success and for the wide number of partnerships and friendships forged through EEA: • The National Secretariats, namely Nahdet El Mahrousa in Egypt; the QUEST Alliance in India; Indonesia Business Links in Indonesia; the Rural Support Programmes Network in Pakistan; the Consuelo Foundation in the Philippines; and our country advisory team in Morocco; • EEA alliance members in each country, including Steering Committee members, Country Advisors, lead implementing partners, business leaders, leverage partners and government officials. Finally, we are extremely appreciative of the efforts of the global evaluation team members who helped design, write and edit the global and country evaluations contained in this final report. Special thanks and recognition is due to Lin Lin Aung, whose tireless energy and dedication was always on display in managing a complex evaluation process involving multitudes of stakeholders across five countries. She was supported by a wonderful team including Mariam Swift (Morocco), Marcia Miranda (Philippines), and Shereen Nasef (Egypt). These evaluation team leaders worked closely with numerous EEA team members at IYF overseeing global and country operations and providing support in editing and designing the final versions of this report. We are grateful to all of them for their critical efforts. This evaluation exemplifies collaboration at its best, and such collaboration was only fitting for a program focused on the power of partnership. We are grateful to all who contributed, and trust their efforts will be of important utility to others seeking to use alliance-based approaches to support youth needs. Awais Sufi Vice President, International Youth Foundation 3 CONTENTS Section I: Introduction ................................................................................................................... 3 Section II: Summary of Global Results.......................................................................................... 10 Section III: Findings and Analytical Review of Program Achievements ......................................... 13 Intermediate Result I: Alliance Building and Leveraging Resources ..................................................... 13 Intermediate Result II: Improved Access to and Relevance of Education and Employability Training ... 25 Intermediate Result III: Improved Prospects for Employment and Successful Entrepreneurship .......... 32 Intermediate Result IV: Positive, Indirect Effect on Families and Communities of Youth ..................... 39 Section IV: Lessons Learned and Recommendations .................................................................... 41 A. Building and Sustaining Alliances .................................................................................................. 41 Part 1: Identifying Effective Secretariats ......................................................................................... 41 Part 2: Establishing Country Alliances and Steering Committees.................................................... 42 B. Assessing EEA Employability Outcomes......................................................................................... 44 Part 1: Utility of the Alliance Approach in Providing Effective Skills Training .................................. 44 Part 2: Internships/Apprenticeships .............................................................................................. 45 Part 3: Job Placement Services ...................................................................................................... 46 Part 4: Sustainability and Scalability of Employability Projects ....................................................... 47 Conclusion ................................................................................................................................... 48 FIGURES Figure 1: EEA Results Framework ............................................................................................................ 7 Figure 2: EEA’s Accomplishments and Challenges ................................................................................. 10 Figure 3: IR I Indicators ......................................................................................................................... 13 Figure 4: Country-level Sustainability Status ......................................................................................... 14 Figure 5: Leverage Contributions by Country ......................................................................................... 15 Figure 6: Number and Types of Alliance Partners by Country ................................................................. 15 Figure 7: Project-level Costs by Country ................................................................................................. 22 Figure 8: Alliance Building and Program Support Costs by Country ....................................................... 23 Figure 9: Unit Cost per Beneficiary ........................................................................................................ 23 Figure 10: IR II Indicators ...................................................................................................................... 25 Figure 11: Direct Youth Beneficiaries: Participation, Completion, and Dropout Rates ............................. 26 Figure 12: Classification of Gender Participation in EEA Programs by Country ....................................... 27 Figure 13: Number of Trainers Trained by Country ................................................................................ 27 Figure 14: Youth Satisfaction with Trainers ........................................................................................... 28 Figure 15: Youth Satisfaction with EEA Training .................................................................................... 29 Figure 16: IR III Indicators ..................................................................................................................... 32 Figure 17: Overall Job Placement and Entrepreneurship Success Rates by Country ............................... 34 Figure 18: Job Placement Rates by Country ........................................................................................... 34 Figure 19: Entrepreneurship Success Rates by Country ......................................................................... 36 Figure 20: Youth Satisfaction with Internships or Apprenticeships ......................................................... 37 Figure 21: Job Satisfaction of Youth ...................................................................................................... 38 Figure 22: IR IV Indicators ..................................................................................................................... 39 ANNEXES Annex A: Project-level Sustainability Status Annex B: Typology of EEA Partners and Leverage Contributions by Country Annex C: EEA Egypt Evaluation Report Annex D: EEA Indonesia Evaluation Report Annex E: EEA Morocco Evaluation Report Annex F: EEA Pakistan Evaluation Report Annex G: EEA Philippines Evaluation Report Annex H: EEA India (QUEST) Evaluation Reports Annex I: EEA Global Evaluation Framework and Surveys 4 Section I: Introduction A. Background Youth between the ages of 15 and 25 account for 1.1 billion of the world’s population, out of which some 400 million are either unemployed, underemployed or live on less than US$2 a day.1 Youth in general are three times – five times in Southeast Asia - more likely than adults to be unemployed. The World Bank estimates that in 2010 the worldwide number of youth in this age group will reach 1.8 billion, and 1.5 billion of them (83%) will live in developing nations. The vast majority lack basic education, marketable skills, decent employment, and are not positively engaged in civil society. Out-of-school, unemployed or underemployed youth are at higher risk of becoming victims or perpetrators of violence and crime, teen pregnancy, receipt and transmission of HIV/AIDS and other diseases. Such negative outcomes not only impose costs on young people and their families, but also on the economy and society at large.2 In response to the alarming rise in global youth unemployment and the resulting disengagement and vulnerability of this age group, the International Youth Foundation (IYF), with support from the United States Agency for International Development (USAID), implemented the Education and Employment Alliance (EEA) Program in six countries with high youth unemployment rates, particularly among young Muslim populations. Target countries included Egypt, India, Indonesia, Morocco, Pakistan and the Philippines. Youth employment stands at 25% and above in most of these target countries - with one out of every three or four youth likely to be unemployed. For example, in the Philippines, the International Labor Organization estimates that over 2.5 million Filipino youth are unemployed, accounting for more than half of the unemployed in the country. In these countries, youth struggle to have access to educational opportunities, and are likely to become unemployed or underemployed under exploitative terms, such as low pay and long hours of work. A high percentage of youth drop out of high school due to poverty and other constraints, and subsequently, are unable to make a full and successful transition to adulthood as productive, healthy and engaged citizens. B. What is the EEA Program? The Education and Employment Alliance was initiated in 2005 as cooperative agreement between IYF and the Asia and Near East Bureau of USAID. At its core, EEA’s mandate was to create and expand education and employment opportunities for disadvantaged youth through innovative pilot projects conducted in strong partnership with stakeholders across the public and private sectors, as well as civil society. Such multi-country, multi-stakeholder, multi-sector partnerships were anticipated to provide a dynamic and effective platform to better leverage financial and human resources in support of youth development. At the same time, they were intended to support the collaborative design, testing and implementation of alliance-based interventions in support of youth education and employability needs. Throughout the alliance building process, the program hoped to demonstrate that such partnerships could help youth better acquire marketable employability skills, gain decent employment or self-employment, and make a full transition into adulthood as successful, engaged and productive citizens. Simultaneously, EEA sought to promote the long term sustainability and scalability of activities involving partners across all sectors. Alliances were forged and reinforced at the global, country, and community or project levels. Global Level: With overall management by IYF, global level alliances were developed by establishing a common framework for alliance development across all six countries; sharing lessons learned across interventions; providing crosscutting technical assistance; creating linkages to multinational corporate partners; and supporting the leveraging of their resources. Country Level: Building on this global platform, Steering Committees within each country were established to help formulate and guide specific alliance-based projects in support of the needs of youth. In this respect, country level alliances looked to establish an insightful and credible steering committee of multinational and local businesses, local governments and community associations that could come together to: 1 International Labour Organization, “New ILO Study Says Youth Unemployment Rising,” Press Release. 2 EQUIP3 guide to conducting USAID cross-sectoral youth assessments. 5 • Discuss their current activities that supported young people in securing better livelihoods, as well as where they perceived critical gaps that needed to be addressed in the educational and employment domains; and • Work collaboratively in ways that would bring the unique skills, expertise and resources of partners from across all sectors to address such needs. To ensure that such dialogues were action-oriented and purposeful, in each country a local organization or team was designated to serve an Alliance Secretariat. This Secretariat was first and foremost tasked with working closely with IYF and the Steering Committee to design multi-sector pilot programs that would be approved by the Steering Committee for receipt of approximately $3 million in Alliance “seed” funding to be leveraged by other partners. As programs moved forward (between five and eleven projects depending on the country), the Secretariat and IYF worked collaboratively to help ensure project and overall alliance program success through the monitoring and evaluation of project activities; the building of partner capacity to implement programs; the sharing of effective practices across pilot programs; and the leveraging of resources to expand and sustain program activities. Community Level: These global and country level alliance structures were in direct support of alliance building at the project or community level, where more than 310 EEA partners across sectors helped implement and support pilot projects designed to address education and employability challenges faced by young people in each targeted country. In sum, EEA and partners supported 43 such pilot programs across its six countries (inclusive of EEA/India programs), which collectively benefited over 1.2 million young people through a variety of activities intended to improve learning outcomes, develop new employment skills and help young people start their own businesses. Taken together, including India, EEA programs leveraged approximately $12.7 million in partner contributions in cash and in-kind resources to build upon USAID support. Of particular interest and importance for the purpose of this evaluation, a number of such projects went through more than one stage of implementation, with partners across sectors building on their initial cooperation to build upon what they viewed to be successful initiatives and expand and better sustain these initial partnership activities. EEA pilot activities covered a wide array of programmatic focus areas,3 including: • Job preparedness programs that looked to combine technical/vocational and life skills, equivalency education, internship/apprenticeship training, mentorship and job placement services - specifically designed to address the unique needs of underserved youth such as out-of￾school and other at-risk populations (Morocco, Indonesia, Philippines and Pakistan) • Career counseling, mentorship, and job-matching programs directed at students and graduates of higher educational institutions (Egypt) • Entrepreneurship support and leadership development programs (Indonesia, Egypt, Philippines and Morocco, Pakistan on a smaller scale) • Programs that supported the creation of new digital tools and learning approaches that increase student learning at the basic education level and also better prepare youth for work (India) • Programs that directly linked schools and training programs with businesses to ensure educational programs meet the needs of industry (All EEA countries in general) • Programs that were specifically targeted to improve the education and employability of girls and young women (all EEA countries) It should be noted that given the unique nature of the EEA/India program, it was concluded that program activities in this country should be evaluated separately from the other five countries which are collectively assessed in this global EEA report. While EEA programs in Egypt, Indonesia, Morocco, Pakistan and the Philippines focused almost entirely on school-to-work transitions and employability programs, the India program centered largely on innovative and effective uses of educational technology, with large numbers of beneficiaries receiving basic educational support through innovative uses of technology. Moreover, in India, in consultation with the USAID Mission, IYF established a field office structure with the aim of the field office “spinning off” as an independent entity to promote alliance-based structures, while in other countries Secretariats were existing organizations that integrated EEA activities into their programming portfolio. Evaluations of several EEA/India programs under its QUEST Alliance are provided in Annex H of this report. 3 A table listing each of the EEA programs, core implementing partners, a general description of activities and targets, and sustainability status is included as Annex A of this report. 6 C. Project Evaluation Objectives, Scope and Methodology This evaluation was overseen by a four-person evaluation team formed in early 2009 as the EEA program entered into its final year of operation. The report is structured in four main sections: Introduction; Summary of program results by country; Findings and analytical review of program achievements by intermediate result; and Lessons learned and recommendations. As it developed the scope and methodology to be undertaken in this final evaluation, the evaluation team had numerous consultations with USAID in Washington and in each country, as well as IYF management and key alliance stakeholders within each country program. Such consultations demonstrated the desire across all stakeholders that the evaluation team center its attention on two key questions: 1. Did the EEA alliance/multi-stakeholder approach bring forward unique benefits in terms of education and employability outcomes for young people in comparison to more traditional development models that placed less emphasis on multi-sector engagement, planning and design? 2. Did the EEA alliance-building approach and the use of multi-stakeholder partnerships to design, implement and oversee programs help promote the sustainability and scalability of interventions? This evaluation seeks to address such questions in a way that builds upon the overall approach of monitoring and evaluation set forward by the program from its outset. In this respect, the evaluation team centered its work on the following intermediate results (IRs) for the EEA program, using these overall guideposts to help best frame survey questions, data collection and analysis: 1. Alliance building and leveraging of resources (i.e. to what extent did EEA succeed in building partnerships and in promoting sustainable and scalable youth employability programs?) 2. Improved access to and relevance of educational and employability training 3. Improved prospects for employment and successful entrepreneurship 4. Positive, indirect effect on families and communities of youth (IR IV was added as part of the evaluation to determine whether there is any indirect effect on the families and communities, although it was not included in the original monitoring and evaluation plan utilized by the program.) Within each of these IR’s, a number of relevant indicators set forth are listed in the EEA Results Framework below: 7 Figure 1: EEA Results Framework Education and Employment Alliance Goal: To identify and give young men and women education and employment opportunities to build a better future Intermediate Result I: Alliance building and leveraging resources Indicator 1.1: In-kind and cash resources leveraged for the project through private sector, government, civil society and other partners Indicator 1.2: Number of private sector, government and civil society partnerships formed through the program to jointly offer employment and business development training activities in targeted areas Indicator 1.3: The effectiveness of the EEA Approach in building partnerships, and promoting sustainable and scalable youth employability programs Indicator 1.4: Non-target institutions requested to adopt the EEA program, models and tools as part of their activities Intermediate Result II: Improved access to and relevance of education and employability training Indicator 2.1: Number of young men and women participating in EEA programs Indicator 2.2: Number and percentage of young men and women completing EEA-supported programs Indicator 2.3: Number of teachers, trainers, facilitators and counselors trained under EEA supported programs Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in EEA-supported programs Indicator 2.5: Level of satisfaction of EEA trainees with the quality and relevance of the training, counseling and services received through EEA programs Indicator 2.6: Level of employers’ satisfaction with the soft skills and/or technical skills of EEA-trained youth Intermediate Result III: Improved prospects for employment and successful entrepreneurship Indicator 3.1: Number and percentage of trainees who get a job within six months of completing the program Indicator 3.2: Number and percentage of EEA trainees starting their own businesses within six months of completing the program Indicator 3.3: Number and percentage of youth with satisfactory internships or apprenticeships Indicator 3.4: Number and percentage of placed youth who declare having a satisfying and quality job Intermediate Result IV: Positive indirect effect on communities and families of youth Indicator 4.1: Number and percentage of employed youth financially helping their families with household, health, and education expenses Indicator 4.2: Number and percentage of youth who have reported improved financial and social conditions of their families as a result of their financial support 8 Findings with respect to each of these indicators are discussed in detail in the following sections. Reflective of the multiple levels of activity undertaken by the global and country level EEA programs, the evaluation process was divided into two stages: 1. A country-level evaluation framework, which included questionnaires,4 focus group discussion protocols and reporting templates that were developed and finalized in late-2008. Surveys were then made available, translated into several local languages, and tested and adapted by each country in early-2009. Country report drafts were then made available for review and feedback, and were subsequently revised based on feedback received from USAID and other stakeholders. 2. A global evaluation framework with a set of common indicators was finalized in early-2009, and data aggregation from country level reports and analysis of overall findings began in June-2009. Findings from this global evaluation are provided within this report; however, they build heavily on aggregated findings from the country level reports referenced above. The evaluation used a mix of qualitative and quantitative methods including focus group discussions with youth, the administration of surveys, and key informant interviews. The survey tools were developed to measure how EEA performed against indicators set forth under each IR and to generate discussions leading to a better understanding of the motivations, behavior and perspectives of EEA partners, employers and the youth. As a part of the final evaluation, 711 youth from across all five countries of focus were surveyed through questionnaires, one-on-one interviews, group interviews, and focus group meetings. In addition, 70 employers and 62 alliance partners of whom 30 were implementing partners and 32 were country-level alliance members were interviewed. Finally, given the inherently qualitative nature of some aspects of alliance building that are not capable of precise measurement, this report includes anecdotal facts and testimonials that illustrate the utility of the alliance for youth, implementing partners, employers and their families. D. Evaluation Challenges and Limitations While the evaluation was conducted with significant rigor and extensive support from numerous program partners in data collection, several constraints are noteworthy. Such constraints often were centered around the inherent difficulty of aggregating numerous programs of different types, timeframes, partnership structures and partner capacity under one uniform umbrella for the purpose of data aggregation and analysis. More specifically, the team faced the following challenges: Time Constraints: Most projects were completed between September and December 2008, with the exception of projects in Egypt, which continued until April 2009, and some Moroccan projects that continued until June 2009. Due to time and budget limitations, the process of collecting the data for the final project reports started in February 2009, with an understanding that some available data, particularly for the programs still ongoing, would not be based on fully completed projects. Because of this difficulty, the evaluation team relied on the triangulation method to measure particular change such as learning acquisition to ensure findings provided a realistic view of program achievements. Where appropriate, the team also attempted to divide certain projects into training cohorts that had completed training and placement activities, and those cohorts that were still within training programs as the evaluation data collection was being finalized. Such division of training cohorts is footnoted as relevant. Data Uniformity: EEA pilot projects significantly varied in terms of the target beneficiaries, age, type and duration of training offered and placement strategies. For example, Egypt provided two to five-day job readiness courses to very large numbers of university students and basic job matching support, while Indonesia and the Philippines offered an average six-month (for in-class and on-the-job) vocational skills training to more limited numbers of out-of-school youth, together with direct job placement services with employers who frequently participating in training. Morocco on the other hand provided a much longer skills training and placed youth in one-year long internships. Such diversity made an overall evaluation, particularly data aggregation and analysis, extremely challenging. Therefore, while the evaluation team made every effort to compare and present the results of interventions across all countries, some exceptions were important to consider and have been put forward despite the team’s original hope for full 4 Five surveys were developed for different stakeholder groups: youth, employers, alliance members, implementing partners and secretariats. See Annex I. 9 uniformity in presentation. Other country-specific challenges have also been elaborated in country reports and where such challenges have been found, the evaluation team has sought to fully disclose such issues. Nonetheless, while such limitations are inherent in any multi-country evaluation of this nature, the evaluation team has tried to provide an objective analysis of program outcomes across five countries and overall believes that the evaluation produces significantly relevant program learnings with a high degree of quality for further analysis. Tracking Graduates: Another challenge was difficulty tracking and accessing program graduates from EEA-supported job training programs, particularly those who were working far away from original training sites. In this respect, as EEA concluded, it was challenging to get appointments with graduates who had already begun working to attend survey administration and focus group discussions. While random sampling was not an option, surveys and focus group discussions were administered after work in the evening or on weekends to maximize youth involvement. Where possible, surveys were mailed out to some graduates and follow-up telephone interviews took place. Therefore, purposive sampling is considered another limitation of the study. Additionally, in limited cases where data were available on program participants but not on job placement, particularly six months after program completion as required under the EEA global indicator, the team has not included the number of beneficiaries under such programs when performing an overall assessment of employment outcomes of the program. Evaluator Independence: Finally, this study is inherently limited by the lack of a fully external evaluation of program outcomes, as each of the evaluators has over time provided evaluation and other support to EEA country programs and partners during the course of program implementation. Given both resource constraints and the desire to leverage the broad experience of program participants to fully compare and contrast approaches used in the different countries for governance, program design, leveraging of resources and other elements, a strategic decision was made to utilize participants in the overall program to support this evaluation. In the interest of promoting learnings and improvement, the EEA teams have done their best to fully express challenges and problems within programs and the alliance structure, grounded in the reality of their experience in the program. However, the study does have this inherent limitation, which is important to disclose fully. 10 Section II: Summary of Global Results Overall, this evaluation finds that the EEA program has shown important progress in core activity areas of the program, most specifically in: • Developing a broad platform for building public-private alliances that increased resources and expertise available for supporting youth employability aims • Increasing the avenues for long term sustainability and scaling of successful pilots, particularly by providing a means for local partners with strong community connections to “plug into” alliance-based structures and integrate best practices into their own networks • Promoting greater relevance and support for training programs by closely involving employers in program design, training and oversight of activities Specific outcomes of the program within these overall findings are presented in the following sections, using the specific indicators set forth by the EEA program’s Intermediate Results as a guide. In summary, the most relevant findings under each of these Intermediate Results include: Figure 2: EEA’s Accomplishments and Challenges Accomplishments Challenges IR I: Alliance Building and Leveraging Resources • Thirty-five pilot projects were designed and implemented across five countries. Approximately 78% of 37 partner organizations had not previously worked with USAID, demonstrating the broad engagement of new stakeholders in youth employability programs. • $9.3 million was raised in leverage contributions of which 31% was in cash and the rest were in-kind contributions. • Partnerships were forged with 319 multi-sector organizations (25% represented governments, 45% local and multinational corporations and 30% local NGOs). • Utility of the alliance approach demonstrated through improved program quality, relevance, scalability and sustainability. • Seventy-seven percent of thirty responding implementers rated the EEA alliance experience as better than other project experiences using non-alliance approaches. • Approximately forty percent of the 35 projects supported by EEA continued to be active following the conclusion of USAID support. • Alliance building process was more time-consuming and resource￾intensive than anticipated. Challenges included lack of sufficient resources allocated for alliance building and coordination efforts, and complexities involved in working with multiple partners. • Some partners felt alliance building focus took away their time and focus from the actual delivery of programmatic activities. IR II: Improved Access to and Relevance of Education and Employability Training • An aggregated 29,873 (vs. 24,210 target) youth participated in various technical/vocational, entrepreneurship and life skills training - 57% male and 43% female. • A total of 26,006 completed the training achieving 87% completion rate. Dropout rate was relatively low at 13%. • A total of 432 trainers were trained (vs. 388 targeted). • Youth satisfaction with the training was high. Over ninety percent of 558 respondents reported gaining new technical and life skills. Of nearly 300 youth respondents, 73% rated their internships as good or excellent. • Employer satisfaction was high. Of employers surveyed, 79% rated the overall performance of EEA graduates as either good or excellent. • Sixty percent of employers interviewed rated EEA graduates as better than other employees of similar age • Oversubscription and inability to serve all interested youth posed challenges • Monitoring and ensuring the quality of training as durations and curricula varied by project. • There was difficulty recruiting female participants for certain types of technical training in a few countries. • Not all EEA graduates were ready to work as many were enrolled in high school or universities. As such, comparing country data proved to be challenging. 11 groups. Additionally, 96% of responding employers stated that EEA program was closing the gap between employers’ needs and the employees’ qualifications. IR III: Improved Prospects for Employment and Successful Entrepreneurship • Of 8,580 graduates ready to work, 4,801 graduates (56%) secured jobs or set up small businesses. Of these 4,801 youth, 3,843 were placed in jobs and 958 set up small businesses. Against an initial target of 7,046 youth to become employed or self-employed, EEA attained the overall employment rate of 68% against this target. • Approximately 84% of over 200 respondents (working youth) were satisfied with the overall work environment while 60% were happy with their salaries. • Youth-led enterprises created over 500 additional jobs. • Most of the new businesses created are sustainable and viable. Eighty percent of the 127 respondents (entrepreneurs) reported making profits. • Half of the surveyed entrepreneurs who had previous jobs reported to be earning more income. • Finding jobs for out-of-school youth with no job experience in fluctuating job markets and in the midst of global economic crisis. • Some trainees were still undergoing one to two-year internships at the time of evaluation, and placements could not be tracked. • Negotiating fair compensation because of reluctance of employers to hire out￾of-school youth and offer fair compensation packages was difficult. There was greater need for post￾training support than anticipated given transition of marginalized youth into unfamiliar job environments. IR IV: Positive Indirect Effect on Families and Communities • Of 238 youth responded, 75% were found to be financially helping their families. • Of 295 youth responded, 82% reported varying degrees of improvements in social and economic situations of families. • Some youth were still earning minimum wage and trying to make ends meet. EEA’s global accomplishments are built upon activities and achievements within each EEA country of operation. Full evaluations from Egypt, Indonesia, Morocco, Pakistan and the Philippines are included as separate annexes (see Annexes C-G) of this report, which tracks program achievements against the intermediate results and indicators utilized in this global framework. In summary, significant findings with respect to each country include: Egypt: EEA Egypt piloted innovative, new approaches to provide employability skills training for 17,644 young Egyptians at universities and youth centers. This large number of youth served by EEA was a direct result of the program’s success in penetrating Egypt’s challenging public sector to directly reach youth with needed services. In fact, twelve MOUs have been signed with additional faculties at Cairo University and four more public universities to replicate the career development model pending available resources. Additionally, through its alliance building model, the program has been able to build an extensive network of private sector and NGO partners that leveraged nearly $3 million in cash and in kind resources to increase service offerings and reach a larger number of participants. As detailed further in the EEA Egypt report, while extremely successful in accessing youth for the provision of needed training and career counseling, the program focused more heavily on short courses and career counseling, with a less robust job and internship matching service provided as compared to other EEA programs with more lengthy interventions for smaller cohorts. As such, of those youth tracked, a total of 484 trainees obtained a job and 402 set up small businesses out of 5,875 students who were ready to enter the employment market or start small businesses. As the program looks to expand based on the available opportunities, such findings and approaches are being re-examined and refined to both maximize the program’s entry into the formal sector while providing more robust job training and placement services. Indonesia: EEA Indonesia reached a total of 13,202 people of which 12,686 are direct youth beneficiaries and 516 are indirect who received additional jobs created through enterprise development.5 Of 1,286 graduates who were ready to work, 1,074 youth became employed or self-employed. EEA Indonesia contributed to establishing a culture of working in alliances and worked with over 65 partners that provided over $1.27 million in leverage contributions. A number of subgrant projects have demonstrated to be sustainable with continued support from alliance partners. Among important challenges faced by the program was a higher level of management support 5 This aggregated number included 7,655 Nike factory operators who received a life skills training, 3,232 vocational high school students who received entrepreneurship training, and nearly 1,000 youth who received a two-day short course on managing T-shirt franchises in Bandung. 12 and coordination efforts than anticipated to implement eleven projects, which in turn brought forward a perception by some partners that the alliance building process was too challenging and time-consuming. In another area, EEA Indonesia learned that graduates who started working or running businesses after six months of completion still needed substantial support from the program, which faced time and resource constraints in attempting to provide such support before the overall EEA program ended. Morocco: MEEA has been devoted to developing and expanding job training, placement and entrepreneurship programs for disadvantaged youth in Morocco. MEEA implemented seven training and placement projects in such sectors as computer refurbishment and repair, textile, tourism, preschool teaching, and basic vocational trades, and raised over $2.4 million in leverage. A total of 2,238 youth participated in MEEA-supported technical and life skills training of whom 1,688 completed the training. Three-hundred graduates have been placed in jobs, 15 graduates have set up small businesses, and over 700 are currently undergoing one to two-year internships. The MEEA approach fostered a new culture in Morocco through the six NGOs that implemented the MEEA projects. MEEA appears to have shifted its partners’ perspective from working in isolation using their limited resources and capacities to working in partnership with other organizations and leveraging resources, sharing knowledge, exchanging services and expertise. In this respect, the Alliance grew to include over forty active partners ranging from local businesses and associations to government ministries, international foundations and multinational corporations. Using such partnerships, one of MEEA’s key successes is the replication or scale up of six pilot projects. These new partnerships have helped and will continue helping with project continuation, sustainability, and most importantly, with placement in internships and jobs. Pakistan: EEA-Pakistan supported six workforce development projects which trained a total of 1,406 youth and placed 551 youth in jobs in customer service, IT, and hospitality industries. Additionally, more than 10,000 trainees across 100 vocational training institutes throughout Punjab benefitted from basic IT training improved through EEA. EEA-Pakistan provided capacity building support to sub-grantees who in turn built project￾focused public-private alliances that added value in terms of technical inputs around curriculum development, community mobilization, and job placement. The Alliance also raised leverage resources totaling $862,742 in support of expanding these programs. Among significant challenges faced was the inability of some project partners to track graduates who got jobs through quasi-government training institutes or set up small businesses in remote areas. Additionally, the program faced significant difficulty in recruiting female participants in some technical training programs because of social norms, although it was able to create opportunities for those who were willing and able to participate. For example, a hospitality training program revealed employment opportunities for women to cook in restaurants and hotels, a trade previously considered to be exclusively reserved for Pakistani men. One highlight of the training programs initiated under EEA was the ASK Development project, which has continued to train young people and place them in jobs with follow-on funding from the National Vocational and Technical Education Commission (NAVTEC). Philippines: EEA Philippines supported six integrated employability skills training projects and reached an aggregated 3,036 youth. Of 2,669 youth who completed the training, 1,794 have been placed in jobs and 167 have set up small businesses. Overall program success is attributed to the ability of partners to help prepare graduates for the job market by providing both technical and soft skills and acquire government educational equivalency certificates in place of high school diplomas. Lack of willingness from industries to hire out-of￾school youth with minimal formal educational experience and work experience was a major challenge, but active liaison and networking with industries and follow-up meetings with internship providers proved to be helpful. As in Indonesia, youth in the Philippines expressed a desire to receive more post-training, counseling support particularly after their short-term contracts expired – which most partners were not able to provide because of time and resource constraints. As part of its sustainability strategy, EEA supported local governments in developing their own community supported employability programs for youth which built upon successful approaches used during the EEA program. This new initiative has been launched as the Youth Productivity Services in Misamis Oriental which is reaching 450 additional youth and testing a platform for long term support by local governments of community-based training initiatives. 13 Section III: Findings and Analytical Review of Program Achievements Intermediate Result I: Alliance Building and Leveraging Resources Intermediate Result I for the EEA program focused on EEA’s ability to build and develop effective alliances in support of youth education and employability needs. In this respect, the design of the EEA program anticipated that the program could set forward measurements for the effectiveness of alliance building processes, as well as for the ability of created alliances to promote more scalable, sustainable and cost effective interventions in support of youth needs. As it looked to measure EEA’s performance in this respect, the program focused on several indicators that could help assess progress toward this intermediate result. They included: Figure 3: IR I Indicators LEVERAGE • Indicator 1.1: In-kind and cash resources leveraged for the EEA program through private sector, government, civil society and other partners ALLIANCE BUILDING • Indicator 1.2: Number of private sector, government and civil society partnerships formed through EEA to offer employment and entrepreneurship training SUSTAINABILITY AND SCALABILITY • Indicator 1.3: Effectiveness of the EEA alliance approach in building partnerships and promoting sustainable and scalable youth employability programs • Indicator 1.4: Non-target institutions that requested to adopt the EEA program, models and tools as part of their activities Summary of Intermediate Result I Findings Several findings discussed in more detail in the following subsections are of note. In particular, with respect to leveraging of resources and building of partnerships, EEA made significant strides. In this respect, across five countries, over $9.3 million in cash and in-kind resources were leveraged from a large array of over 300 partners across the public and private sectors as well as civil society. This amount exceeded original targets set forth at 3:1 against USAID seed funding of approximately $2.9 million, and also exceeded a 1:1 ratio against overall USAID funding of $9 million toward both country and global activities. Similarly, with respect to partnerships, a significant additional finding relates to the number of new partnerships formed with groups who traditionally have not worked with USAID. With respect to the 37 primary implementing partners in the program who have received direct USAID support, 78% of 37 partner organizations (including Secretariats) had not received support from USAID before the EEA program. While direct surveys on this issue were not undertaken with non-USAID funded recipients under the program (e.g., local corporate leverage partners), anecdotally, many of these partners, likely at a similar percentage, have also not worked in partnership with USAID previously. Additionally, EEA earned high marks for its collaborative strategy. Seventy-seven percent of thirty implementing partners rated the EEA model and experience as better or much better compared to previous programs they had conducted in the youth employment field but without a strong focus on forming partnerships and gathering leverage from other stakeholders to support activities. This expansive network of partnerships formed was found to be directly relevant with respect to indicators attempting to measure sustainability, scalability and cost effectiveness of the overall program. In general terms, the five country evaluations find that USAID-seed funds were able to attract the attention of a significant number of new partners and fuel additional contributions from these partners in support of programs. The interest of these partners, grounded in direct contributions they made to project activities, helped set the stage for further, second stage activities once initial projects matured and showed success. The second stage of these partnerships in turn helped expand the number of beneficiaries reached, and brought forward new avenues for sustainability beyond USAID support through revenue generation, 14 continuation of activities with implementing partner resources or contributions of other local, national or international stakeholders. In sum, approximately forty percent of the 35 projects supported by EEA continued to be active following the conclusion of USAID support. The figure below summarizes the sustainability status of each country, and section 1.3 below provides further examples of programs sustained following EEA interventions. Figure 4: Country-level Sustainability Status Country Country-level Sustainability Status Egypt • Alliance activities largely continue with the new grant from IYF/Samsung benefitting approximately 2,500 youth. • MOUs have been signed with 12 university faculties to replicate the career development model. • Revenue-generating models will be introduced at universities for self￾sustainability. Indonesia • National alliance – Steering Board – expressed interest in continuing its role as an organized group advocating for youth employability issues. It may be sustained with support from the National Secretariat. • Two projects are being replicated and are providing life skills and entrepreneurship training for an estimated 30,000 Nike factory operators and 3,000 vocational high school students. Morocco • Ministry of Education’s Casablanca Academy’s vocational training centers adopted the EEA skills training model. • A number of NGOs and donors (such as Silatech, Finnish Children & Youth Foundation, Near East Foundation, Fondation Auteuil) requested to fund the EEA project model and methodologies for expansion of phase 1 projects into new government or NGO venues. • Emploi Habilité training program is being replicated in Tétouan-Tangiers. • National alliance ceased activities as EEA funding ceased. Pakistan • Approximately 10,000 trainees at Vocational Training Institutes anticipated benefiting annually from the curriculum improved through EEA. • The National Vocational and Technical Education Commission provided $100,000 to an EEA partner, ASK Development, to continue a second phase of EEA skills training. • Additional investments are being made by outside partners to scale and sustain EEA pilot activities in the Service and Compressed Natural Gas industries. • National alliance did not continue post Secretariat operations ending in early￾2009. Philippines • Provincial government launched an alliance project, based on the EEA model, to train 450 youth in Misamis Oriental. • Similar alliance-driven skills training activities continue benefitting 600 youth under the new IYF/Wrigley grant for the Rizal Province. • Working with local networks including trade associations in growth areas helped ensure long term sustainability and effective outcomes. • National alliance ceased operations as Secretariat operations ended in mid-2009. While such findings are compelling, the evaluation also found that developing and nurturing an alliance￾based framework to support programs presented unique challenges that are important to address in future efforts. In this respect, EEA partner NGOs found that building relationships with non-traditional development partners, such as corporations, governments and local corporations, was a challenge that required more time and effort since trust and positive relationships needed to be developed. Similarly, NGOs found that new knowledge and skills were needed, particularly in linking youth training with employment, strategizing how to build alliances and identifying other resources to sustain projects. As such, it was perceived across all countries that allocating time for alliance coordination and management work cut into project implementing schedules and the ability to reach anticipated outcomes under IR II. Detailed evaluation findings concerning alliance building and leveraging resources are presented under each of the following indicators under the program’s global evaluative framework. 15 Indicator 1.1: In-kind and cash resources leveraged for the project through private sector, government, civil society and other partners The EEA alliance structures helped to leverage a high ratio of cash and in-kind resources for youth, most of whom were at-risk, out of school and poor. The original targets of 3:1 against seed funds have been exceeded, with USAID country seed grants of $2,890,133 leveraged by a total of $9,338,218 in cash and in-kind resources, or a ratio of 3.2 to 1. Cash contributions totaled $2,930,333 or 31% of the total leverage. In-kind contributions equaled $6,407,885 or 69% of the total leverage. In-kind contributions were donated in forms of land, equipment, curriculum materials, learning software, trainers and stipends for interns, among other items. See Annex B for the typology of alliance partners and leverage contributions by country. Individual country leverage ratios ranged from $1.5 to $1 in Pakistan to as high as $5.3 to $1 in Egypt, as described in Figure 5 below. It should be noted also that such ratios likely do not fully capture all resources accessed by the program in support of overall objectives, particularly time donated by Steering Committee members and the full resource costs donated through drawing linkages to governments. Similarly, to the extent new phases of programs have been initiated in the last quarter of the program to support the scalability and sustainability of EEA interventions, these have not been counted given the reduced management time at the country level available to track and calculate such contributions. Figure 5 presents leverage contributions by country: Figure 5: Leverage Contributions by Country Country Number of Projects USAID Grant Leverage Contributions Leverage Total Cash In-kind Ratio Egypt 5 $565,500 $2,985,245 $1,043,104 $1,942,141 5.3 :1 Indonesia 11 $581,592 $1,272,310 $386,958 $885,353 2.2:1 Morocco 7 $538,825 $2,415,240 $543,719 $1,871,521 4.5:1 Pakistan 6 $586,248 $862,742 $177,829 $684,913 1.5:1 Philippines 6 $617,968 $1,802,681 $778,723 $1,023,958 2.9:1 Total/Average 35 $2,890,133 $9,338,218 $2,930,333 $6,407,885 3.2:1 Indicator 1.2: Number of private sector, government and civil society partnerships formed through EEA to offer employment and business development training activities While the program did not set overall goals for the number of partnerships formed in support of program goals, the number of such partnerships formed significantly exceeded expectations. In this respect, the evaluation study finds that EEA has been effective in forging vibrant public-private alliances that were critical in generating new leverage resources to support and expand programs, and to drive a partnership model that can engender greater innovation and broader support amongst relevant stakeholders. Through 35 pilot projects, EEA established relationships with 319 public-private sector organizations, including local governments, local and multinational corporations, community-based organizations, implementing partner NGOs and training institutions. Figure 6 presents the number of EEA partners by sector in each country. Figure 6: Number and Types of Alliance Partners by Country Country Government Private Sector NGO International Organization Total Number of Partners Egypt 9 47 21 3 80 Indonesia 7 36 20 2 65 Morocco 9 21 14 1 45 Pakistan 5 3 11 0 19 Philippines 51 35 21 3 110 Total 81 142 87 9 319 Partnerships were forged with both local and multinational corporations which were potential employers to: 1) leverage more cash and in-kind resources; 2) serve on steering or advisory committees; 3) design or improve curricula; 4) motivate and mentor youth during the training; and 5) offer apprenticeships and subsequently jobs. On the field level, a number of EEA projects involved communities, parents and 16 teachers who provided training venues and other support to make sure youth completed their training and found jobs. Additionally, EEA worked closely with the national and local government agencies in various areas including: • Recruiting youth from hard-to-reach areas and outreach efforts. For example, in the Philippines, over sixty percent of 86 youth surveyed said they learned about the program through local governments. • Providing financial and other support to the implementation of youth employability training activities. For instance, EEA Egypt worked closely with public universities and the National Youth Council and its Centers to provide employability training to youth. In West Java, the local government donated land to young entrepreneurs to set up catfish farms. • Offering services such as trade competency testing and necessary licensing, often to help out-of￾school youth who lack high school diplomas to get jobs in countries such as the Philippines and Indonesia. • Replicating the EEA model and approaches as locally-owned initiatives or serving as part of local committees established to continue providing support to youth-led businesses in countries such as Indonesia, Morocco, and the Philippines. The study also involved 32 national alliance members who shared the reasons they participated in the alliance.6 Responses are presented in an order of importance to these organizations: • to help youth acquire relevant employability skills and become employed • Youth Development is part of our institutional concerns • to support community development • to engage in good corporate social responsibility (CSR) practice • because of an invitation from a reputable organization The partnership-building experience, however, was not without challenges. While at a macro level, alliances seem to yield valuable benefits, the alliance development process can appear unwieldy and challenging especially in countries where working in alliances was a relatively new idea such as Indonesia and Morocco. Significant amount of resources were required for developing and strengthening alliances. Some sub-grantees in Egypt interviewed shared that the time involved in creating alliances and meeting leverage requirements at times took focus away from the delivery of core programmatic activities. Some sub-grantees in Indonesia also expressed their reservations about implementing similar alliance projects in the future because of the complexities involved, such as the need to: take a participatory approach in design and implementation; invest more time and resources in engaging a broad segment of partners with a broad diversity of experience in such efforts; and negotiate with various parties in the case of conflicts. Nonetheless, despite such challenges as articulated by partners, it appears that such alliance building efforts did yield tangible benefits as noted in the overall findings of this IR. Indicator 1.3: Effectiveness of the EEA Alliance approach in building partnerships and promoting sustainable and scalable youth employability programs This section focuses on the effectiveness of the EEA program model through a number of relevant subtopics, including: • The value and effectiveness of the Secretariat • The role of Steering Committees • The value of an overall alliance approach in promoting sustainable and scalable projects • The sustainability of alliance-based interventions • The scalability of alliance-based interventions • The cost-effectiveness of the EEA model In each of these areas, the study gathered perspectives from an array of alliance partners, including implementing partners, Steering Committee members and private sector partners, through surveys and interviews. 6 National alliance members include Steering Committee members, corporate partners and other alliance partners who were not lead implementing partners. A separate survey was developed solely for lead implementing partners. See Annex I. 17 Effectiveness of Secretariats: During the first two years, IYF invested significant resources in building the capacity of National Secretariats based on their identified needs, particularly in areas of building and reinforcing alliances, setting up a sub-award monitoring system, leverage raising, conducting due diligence, financial management and overall program management. The Secretariats subsequently shared their acquired knowledge and skills in these areas with all implementing partners through capacity building workshops and regular technical support visits. The Secretariats - with substantial support from Steering Boards in countries such as Morocco and Egypt - also played a critical role in establishing linkages with the private sector and governments for some projects. The ability to attract local businesses to join the alliance and play a substantive role has been noted as one of the major achievements of the EEA program. All 32 national alliance members interviewed stated that the Secretariats were effective in supporting the work of the alliance. The following are the three areas frequently quoted as most effective across the five countries: • Designing innovative youth education and employability projects • Implementing a transparent grant-making process • Providing technical assistance to partners during the start-up and throughout implementation The study also asked thirty lead implementing partners how effective the Secretariats were in providing technical assistance and meeting their organizational needs. Across five countries, all implementers reported receiving assistance from the Secretariat in the following areas: designing and implementing youth employability projects using the alliance approach, technical training, partnership linkages, institutional development, monitoring and evaluation, financial management, and trouble shooting. Of 28 implementing partners (out of thirty) responded to this specific question, 25 or 89% rated the quality of assistance received as either good or excellent – which demonstrated the importance of establishing local management structures on the ground to provide direct assistance. Of these 28 respondents (partners), one partner from Indonesia rated the assistance as fair, expressing a desire to receive more support with entrepreneurship interventions particularly with marketing and product development. Two partners from Egypt and Morocco rated it as poor, citing their desire for more monitoring and technical support visits. There was a sense among some alliance members that it would have been useful for alliance members to participate more actively and collectively in outreach events and advocate for a broader policy framework to promote EEA. Effectiveness of Steering Committees: EEA established Steering Committees and the study assessed the role and contributions of the Committee members across five countries. The Committees were composed of multi-sector alliance members, including development practitioners, government officials, corporate partners and civil society groups. Depending on the country and individual, Steering Committee members may have represented their organizations or themselves. The study finds that Steering Committees were most active in the alliance building, program design and early implementation stages across five countries. A robust process put in place to encourage Board participation resulted in collaboration among Board members and alliance partners with Secretariats playing a facilitation role. At the program outset, the Board met quarterly or more to review proposals, provide suggestions to strengthen program designs, and approve subgrant activities that would best support EEA program goals. In Egypt and Morocco, Steering Board members made major contributions to the program by opening up relationships with public institutions, gaining formal agreements with the National Youth Council, and forming linkages with the private sector. These contacts proved critical to the integration of the EEA model into both public and private sector environments. It was also noted however that most EEA countries struggled to sustain active participation of some Board members after seed funds were allocated. With less focused activity where they could direct their insight and advice, Board participation naturally waned in most countries. In Indonesia, however, some Steering Committee members were also corporate partners who provided financial support to projects; and therefore, several members continued to be closely involved in program implementation and helped brainstorm ideas to strengthen select activities throughout the program. Effectiveness of the Alliance Approach: Evaluation findings suggest that the use of an alliance approach had an effect on the following elements of the EEA program: program quality, sustainability and scale, and resource efficiency. 18 1) Program Quality: The alliance approach encouraged EEA projects to engage corporate partners or employers meaningfully from the program outset. EEA’s strategy and ability to include its multi-sector alliance partners in various aspects of program implementation beyond leverage contributions appears to have positive effects on the quality and relevance of training provided and types of internships and jobs made available. Strategies, such as having corporate partners review and approve training modules and serve as mentors for participants, helped EEA training activities to be demand-driven, high quality, and relevant. Projects that provided employers with access to a pool of qualified graduates and helped them see business benefits beyond social benefits, achieved strong placement rates. Of particular note, as discussed in more detail under indicators 2.4-2.6 below, youth were highly satisfied with training programs and 79% of employers rated beneficiary youth as either good or excellent. 2) Sustainability and Scalability: EEA encouraged its partners to consider sustainability and scalability at the program outset, and particular attention was given to successful pilots. As projects showed tangible results, project-level alliances explored different opportunities for scale￾up and sustainability. An estimated forty percent of projects were successful in sustaining or scaling-up EEA project activities with support from EEA or outside partners (detailed in the sustainability and scaling up sections below). At the project level, the study finds strong examples across countries of tested partnerships starting a second stage to expand EEA initiated efforts to both scale and sustain efforts perceived as successful. Examples include the following: a. In Morocco, the Ministry of Education’s Casablanca Academy’s vocational training centers adopted EEA’s IT and life skills training. b. Similarly, in the Philippines, the local government in Misamis Oriental is running a locally-owned vocational training program, modeled on EEA, benefitting an initial group of 450 students while Wrigley Company is supporting alliance activities in Antipolo for 600 additional youth. c. In Egypt, a government youth center adopted EEA’ s “One Stop Shop” projects with a view towards outreach to 4,500 other centers across the country. Samsung Electronics is supporting EEA project activities at both universities and youth centers reaching 2,500 additional youth. d. In Pakistan, a government agency is working with an EEA partner, ASK Development, implementing a second phase of EEA skills training. e. In Indonesia, Junior Achievement is continuing to work with EEA corporate partners to implement a second phase of its entrepreneurship training benefitting an additional 3,000 vocational high school students. 3) Resource Efficiency: Initial USAID investments made in implementing 35 projects were supplemented by leverage contributions from partners at an average 3:1 leverage ratio. The leverage requirements made certain that partners were not overly dependent on EEA funding, and as a result of contributions, interventions appeared reasonably cost efficient at a unit cost of approximately $600 per trainee (see the cost-effectiveness section below for details). More importantly, alliance partners occupied roles as equal stakeholders, in that they contributed their own resources while the EEA approach, methodologies and partnerships were tested in the most resource-intensive pilot phase. 4) Social Empowerment Effects: Anecdotal evidence suggested that the use of the alliance approach improved the self-confidence and social standing of graduates. Interviews with field program managers suggested that the approach also provided them with a sense of community, particularly those who are either employed or self-employed and contributing to their families. Alliance Participants’ Perspectives and Attitudes Surveys with partners also explored how national alliance and implementing partners generally benefitted from working with alliances. In this area, survey results indicated that the partners felt greater capabilities after EEA in terms of developing new partnerships in support of their work and in designing and implementing demand-driven, alliance-based interventions. For most partners in Indonesia and Morocco, for example, working in public-private alliances was a novel idea, and their EEA experience helped them learn how to build and maintain alliances. The following are the areas commonly identified by partners as added benefits in their survey responses: • Ability to become part of the extensive network of public-private sector organizations that work on youth employability issues, and access to financial and technical support from the alliance 19 • Enhanced credibility of individual organizations by being associated with the country-alliance • Learning management accountability and transparency • Systematic exchange of ideas, expertise and good practices around designing, implementing and evaluating employability projects, as well as motivating companies and communities to maximizing empowerment programs • Strengthened organizational capacity in terms of program management, communications and business development skills • Improved training modules and methodologies • Ability to access a large pool of graduates and recruit employees Furthermore, implementing partners were asked to what extent EEA influenced them to improve their capacities to work with youth or become advocates, and 93% of thirty partners reported varying degrees of influence or change. Specifically, 19 or 63% of thirty partners reported a great deal of influence whereas 33% or ten partners reported some influence. Of thirty implementing partners surveyed, 29 partners have had experience implementing youth projects using non-alliance approaches. When they were asked to compare their EEA experience with other project experience, 23 or 77% of thirty partners who responded rated the EEA alliance experience as better or much better because of the following reasons: • Receipt of significant support from government and businesses • More interaction and linkages with partners and stakeholders together with responsiveness of EEA • Opportunity to improve organizational capacity in managing youth programs by interfacing with others from whom they could learn • Likelihood of alliance programs such as EEA to be more transparent and manageable because of the greater number of partners taking an interest in the initiative Five or 17% of thirty implementing partners responded felt the EEA approach was similar to other program models, whereas two partners recounted the EEA experience as worse expressing their reservations about implementing similar alliance projects in the future because the complexities involved, such as the need to take a participatory approach in design and implementation, to invest more time and resources in engaging a broad segment of partners with a broad diversity of experience in such efforts, to focus consistently on finding new leverage contributions, and to negotiate with various parties in the case of conflicts. Of seven partners who rated the experience as similar or worse, six were in Indonesia and one represented Morocco. This finding appears closely linked with the fact that the alliance building was a relatively new idea. Additionally, the Indonesia Alliance, due to USAID rules in this country, had to conduct due diligence on over thirty companies that were program supporters which proved to be time￾and resource-intensive and caused delays. Additionally, survey findings suggest that a majority of partners felt that the use of alliance approach has had a positive influence on program outcomes. When asked to determine the level of effect of the EEA alliance approach on the program using a scale of one to five, the highest ratings of four to five were given by 79% of 28 implementing partners responded, with the rating of three by the remaining 21% of the partners. Scaling up: A review of project outcomes and activities across all countries, together with data gathered as a part of evaluation surveys and focus groups with key stakeholders, suggests that the EEA program has provided a number of important avenues for scaling up project activities through the participation and support of project partners. No precise model can be identified for such scale-up occurring in different countries, but in a number of circumstances project partners (initially at the suggestion of and with the participation of IYF, national secretariats and/or USAID), gathered together based on a common understanding of challenges in the development of youth and came to general agreement about the need to support such activities. USAID seed funds were then used as an incentive and driver throughout the partnership development process, ensuring that partnership agreements and commitments would be rewarded with concrete resources and program implementation would take place. As programs were designed and implemented, substantial technical assistance was provided by IYF, national steering committees, national secretariats, USAID and other advisors to the program to help ensure soundness of the technical intervention and the integration of key partners that could support the program in the long term. One critical area of focus for most of these programs was attention provided to 20 the scalability prospects for the program over time, a focus that helped to encourage partners and implementers to provide strong support for successful implementation during the pilot phase. As pilot phases matured, partnerships were tested and such avenues for scalability were more fully explored. In certain cases, assuming both time and resource availability, USAID seed funds were provided to support such scaling up. For example, in Morocco, the Emploi Habilité training program providing life skills, technical skills and internships in partnership with local government, is presently being replicated in the region of Tétouan-Tangier, as a first step towards scaling up the program more broadly across other cities. Building on the success of its first phase, the second phase took the model of IT and life skills training and integrated this package into the Ministry of Education’s Casablanca Academy’s vocational training centers – Centres de Formation par Apprentissage (CFA). In other cases, scaling was through new resources, provided by either non-USAID resource partners in phase one of a pilot or other funders that were not present during phase one as partners or donors. For instance, in Egypt, public sector involvement is a key component of the program design in order to promote scale and sustainability, and the program’s strategy to partner with formal, public sector institutions to build the foundation for this expansion is the key ingredient to scalability. In this respect, a government Youth Center adopted EEAs “One Stop Shop” package of services, which EEA encouraged with a view towards outreach to 4,500 other centers across the country that could potentially learn from this model. Additionally, twelve MOUs have been signed with additional faculties at Cairo University and four more public universities to replicate the career development model pending available resources. One finding of particular note was that scaling through public institutions was promoted substantially through leveraging certain members of steering committees or other advisors who had a unique understanding and familiarity with the workings of government. Such understanding and familiarity was often vital to moving complex partnership models through bureaucratic systems and finalizing the commitment of public institutions to reform. Additionally, while no comprehensive findings can be made in this respect, the alliance platform also appears to have served to benefit reform-minded steering committee members, who were able to use this platform to promote concrete, resource supported reforms and build a constituency in support of more systemic change. Scaling up, however, also presented unique challenges that should be addressed in future efforts. In particular, insufficient time was available to fully monitor and observe and to provide needed technical assistance to programs during their scale up phases. This was directly relevant to initiatives in the public sector, where focusing on the quality of government trainers and providing constant feedback and mentoring to the system is essential. Similarly, resource constraints in specific areas related to government compensation and incentives can also pose real obstacles to the scaling of initiatives. In this respect, a consistent challenge appears to be the provision of incentives or additional pay to government employees who, after participating in an EEA supported effort, are expected to integrate new teaching approaches and activities into their daily responsibilities, with no or minimal additional compensation. Alliance Sustainability: Alliance sustainability was viewed through two prisms – the sustainability of project-level activities, and the sustainability of broader, alliance-based systems and approaches among the broader set of stakeholders participating in EEA country alliances. Project Level Sustainability: In assessing the sustainability of EEA program’s specifically, the evaluation team focused primarily on the ability of EEA program activities to sustain themselves without USAID support upon the conclusion of the EEA program. In this area, the evaluation team found that by program end, in all countries, approximately forty percent of EEA projects directly effecting youth were being continued, either adopted by government, sustained with new donor resources, or continued on by EEA partner NGOs as part of their programs and strategic directions. The training curricula developed or improved for many EEA programs in Indonesia, Pakistan and the Philippines also continued to be utilized, which is another mark of sustainability for activities funded under EEA. Sustainability in this area appears closely linked with the same partnership factors that are relevant to discussions of scalability and cost effectiveness. In this respect, at the outset, the sustainability of the projects appears to be aided by the ability of the project design teams to integrate large numbers of resource and technical partners into the implementation of project activities. Such partners create a diverse array of supporters that provide different inputs into both planning and implementation, and such inputs heightened their interest in monitoring project outcomes given their own contribution to the efforts. 21 As the project unfolded, initial assessments were then made by each of these partners, often informally, about the value of the intervention and its prospects for long term success in training youth and helping them access jobs or be prepared for employment. These informal assessments were followed up through direct consultations between such partners, IYF and national secretariats with a focus on exploring new pathways for both improvements to projects and long term sustainability. Such consultations, coupled with the broad array of supporting partners that had been developed, allowed for an opportunistic approach to exploration of sustainability options, grounded in the experience and relationships already formed. More broadly, survey results show that 75% of alliance partners surveyed were using their own networking capacity to develop and sustain partnerships. For example, in Indonesia, Prestasi Junior Indonesia has obtained funding from EEA partners to implement phase two of its vocational training program. Similarly, in Pakistan, NAVTEC is providing additional funding to the EEA partner - ASK Development to implement the phase two of this EEA-initiated program. In addition to the creation of country platforms and a group of strong supporters, EEA appears to be particularly successful in institutionalizing the model on a broader national or provincial level in Egypt, Morocco and the Philippines where the national or local governments adopted the EEA alliance approach and methodologies continuing to work with EEA partners. Examples included activities in the Philippines, where building on EEA’s partnership with the Metal Works Industry Association of the Philippines (MIAP) and the success of phase one, the provincial government has adopted the EEA model and mobilized local resources to train 450 additional youth in collaboration with MIAP. Country-Level Alliance Sustainability: The evaluation team also assessed the sustainability of country￾level alliance structures as apart from project-based activities. Such alliances were those managed on a daily basis by country secretariats, with strategic guidance from a prominent group participating in steering or advisory committees. In assessing the long term sustainability of such country-level activities, it is important to first identify in detail country-level responsibilities of such structures. Across the EEA program, common expectations were set for country alliances. Specifically, such country level alliances were responsible for: • Assessing gaps in education and employment programs for youth with the support of a respected group of stakeholders from across the public and private sectors and civil society • Identifying new partners from across sectors to design new pilot interventions to address such gaps • Approving the allocation of seed funds to partners for implementation of such interventions, and then monitoring implementation, providing needed technical support in ensuring quality outcomes, and supporting effort to scale and sustain such initiatives • Providing capacity building for partners across the program in areas of common need, including in monitoring and evaluation, life skills programming, leveraging of resources, and job placement • Sharing and distributing lessons learned and advocating for adoption of best practices by other stakeholders within the country of operation Such activities were largely focused on project activities and would appear to enhance prospects for the sustainability of specific projects as detailed in the previous section. Of course, project level activities were anticipated to bring forward areas of common interest and learnings, particularly with respect to capacity building for partners implementing EEA projects and beyond, and the spreading of such learnings. The evaluation has found that overall such broader efforts by country alliances toward spreading lessons learned and building capacity of partners in areas of mutual need were very well received. However, despite the effectiveness of such activities, the evaluation found that sustainability of alliances at a country level was elusive and largely unrealistic. This finding is largely based on the predominant focus of national alliances on project level activities rather than broader advocacy and reform in areas of education and employment. Similarly, experience suggests that advocacy and reform efforts of this nature would likely require a concerted focus in just these areas, often undertaken by large trade associations or other coalitions with this sole mandate, and not a project specific focus like EEA. At another level, while country level support to individual projects was considered extremely important for the success of programs, finding direct support for secretariat-led activities from donors outside USAID was largely impractical. It is worthy to note that EEA Secretariats and partners were able to find numerous partners willing to dedicate cash and in-kind contributions to specific projects, as such donors could better appreciate the concrete benefits their support would provide to young people. Fundamentally important areas of support such as capacity building, technical assistance, project design, monitoring 22 and evaluation and leveraging of resources are more difficult to secure from such donors, and are instead better understood by development assistance agencies such as USAID. Nonetheless, despite this overall finding, some areas of national level alliance building have longer term prospects of success, and systems in place for country level activities could be re-initiated should additional support or interest from a variety of quarters become present. Cost Effectiveness: The overall cost effectiveness of the EEA program was assessed at two levels. The first level is related to country program subgrants, which include both seed grants administered by USAID and cash and in-kind leverage contributions provided by other partners to support EEA country programs. The second level more broadly includes costs classified as alliance building and program support costs. This second level includes costs outside of those directly incurred by project implementers. They accordingly include costs related to the functioning of IYF as the Secretariat, local consultants and advisors in five countries, as well as IYF direct and indirect expenditures attributable to the EEA program. Each of these cost categories are discussed in more detail below. Subgrant Project-level Costs: As noted above, subgrant project-level costs include seed funds and leverage contributions invested toward 35 individual projects which typically covered curriculum design or improvement, training, job placement costs, and project specific costs of NGO subgrantees. The amount of seed funding support from USAID toward individual projects is a useful metric to assess cost effectiveness given that such funds are directly focused on implementation of projects, are directly leveraged by investments from other parties, and are best positioned for increased efficiency when additional beneficiaries are added to successful pilot programs. Across five countries, the USAID support of $2.89 million in seed funds that were coupled with leverage investments of $8.75 million reached a total of 29,871 beneficiaries. If all project-level costs are taken in account, this brings the cost to $390 per beneficiary. Because of significant leverage raised by EEA, only $97 of this total was borne directly by USAID and $293 was shouldered by the alliance partners. Figure 7: Project-level Costs by Country Country Number of Beneficiaries USAID Funds Leverage Total Project Budget Cost Per Beneficiary Egypt 17,644 $ 565,500 $ 2,415,739 $ 2,981,239 $ 169 Indonesia 5,547 $ 581,590 $ 1,272,311 $ 1,853,901 $ 334 Morocco 2,238 $ 538,825 $ 2,399,889 $ 2,938,714 $ 1,313 Pakistan 1,406 $ 586,248 $ 859,526 $ 1,445,774 $ 1,028 Philippines 3,036 $ 617,968 $ 1,802,681 $ 2,420,649 $ 797 Total/average 29,871 $ 2,890,131 $ 8,750,146 $ 11,640,277 $ 390 Alliance Building and Program Support Costs: A more expansive examination of costs by necessity also includes alliance building and program support costs of the EEA program. These costs supported significant, collective efforts by IYF, National Secretariats in each country and local program consultants working alongside project implementers. Such global and national assistance was provided in numerous areas related to program activities, including: • managing the design and implementation of alliance based interventions • providing international technical expertise on successful employability approaches • assisting in leveraging of resources • providing capacity building support to implementing partners in skills training, internship and job placement, and post training support to entrepreneurs • grant administration services • financial and managerial oversight • USAID regulatory compliance • direct monitoring and evaluation support Funding at this level also enabled IYF to convene three global summits in Egypt, India and Indonesia with project partners attending from each EEA country, and to publish case studies and other learning materials, with an overall aim to share program learnings with a broader group of EEA alliance members and the development community. Upon examination of IYF accounting records, these costs taken together 23 with indirect costs attributable to the EEA program, totaled approximately $6 million across all five countries. This USAID-borne cost adds approximately $203 on a per beneficiary basis - see Figure 8 below. Figure 8: Alliance Building and Program Support Costs by Country Country Number of Beneficiaries Alliance Building and Program Support Costs Cost Per Beneficiary Egypt 17,644 $ 846,755 $ 48 Indonesia 5,547 $ 1,084,899 $ 196 Morocco 2,238 $ 1,227,932 $ 549 Pakistan 1,406 $ 686,162 $ 488 Philippines 3,036 $ 640,658 $ 211 Global 0 $ 1,588,981 $ - Total/average 29,871 $ 6,075,387 $ 203 Taken together, these overall costs of both seed funds and Alliance building program support costs total approximately $593 per beneficiary, and appear equivalent - if not cheaper - compared to other good practice country projects of this duration. The World Bank estimates that the unit cost for youth employment programs in Latin America vastly varies between $500 and $2,000.7 Similarly, the International Labor Organization estimates that non-formal education programs normally cost between $700 and $1,500.8 Of the EEA unit cost of $593 per beneficiary, $300 (51%) was born by USAID and $293 (49%) was contributed by alliance partners. Figure 9: Unit Cost per Beneficiary Cost Category USAID Alliance Contributions Overall Cost Per Beneficiary Project-level $ 97 $ 293 $ 390 Alliance building and program support $ 203 $ - $ 203 Total/average $ 300 $ 293 $ 593 Finally, and perhaps most significantly, a full assessment of cost efficiency would also require a more thorough assessment of the effect of alliance building activities on long term sustainability and scalability of proven practice programs. For example, the EEA model of programming demonstrates the significant increase in beneficiaries and benefits possible through up-scaling under an alliance building approach, particularly with government or quasi-government agencies as partners. Should the new efforts to substantially expand the EEA model be successful, the cost effectiveness of initial EEA investments could be dramatically expanded. While the longitudinal study required for this type of evaluation goes beyond the scope and resources available for this study, such factors should be addressed more directly in similar employability projects in the future. Indicator 1.4: Non-target institutions requested to adopt the EEA program, models and tools as part of their activities The evaluation also focused on how many institutions have adopted or requested to adopt the alliance approach, model, and methodologies in each EEA country which serves as a proxy indicator of the EEA program sustainability. In this respect, the evaluation team felt this indicator would be helpful in determining the level of local buy-in and ownership of different models and methodologies that EEA tested. In addition to a majority of target organizations continuing to use the EEA model and 7 Tesliuc, Cornelia. 2008. “What works in LAC to address youth unemployment?” World Bank power point presentation available at: http://info.worldbank.org/etools/docs/library/243472/day8CorneliaTesliucLACYouthApr9Session2.pdf 8 Hans, Christiaan. May 2002. “International Labor Organization’s Mekong Project: Non-formal education and rural skills training.” http://www.ilo.org/public/english/region/asro/bangkok/child/trafficking/downloads/tia-2-publication.pdf 24 methodologies to implement similar projects in the future, nearly thirty non-target organizations across five countries – ranging from various government agencies to NGOs - requested to adopt the EEA program, models and tools as part of their activities. A few country examples worth highlighting include the following: In Morocco, all six implementing partners are continuing to use IYF’s Passport to Success life skills curriculum to complement technical training components. They have also developed new partnerships with other local NGOs, international organizations and private sector entities, which contributed to ensuring the continuation, replication or expansion of individual projects. A new set of donors and partners include: the Silatech Foundation of Qatar, Finnish Children and Youth Foundation, Near East Foundation, Fondation D’Auteuil and the Moroccan Ministry of Labor. Additionally, some implementers introduced the alliance approach to their new partners who in turn have adopted for their own operations. For instance, EEA partner PlaNet Finance has incorporated the alliance model into their operations and developed new partnerships with other local associations (such as Darna Association) to implement similar interventions. In the Philippines, nine out of 22 municipalities in Misamis Oriental have requested to adopt and implement the EEA model committing their own funds. The pilot, known as the Youth Productivity Services, is reaching 450 additional youth with the allocated budget of nearly $90,000. If this pilot is successful, EEA/YPS will be integrated as part of the Unlad Kabataan Program, demonstrating strong potential for institutionalizing the EEA model and approaches. In Egypt, career facilitation and counseling did not exist in public universities before EEEA’s career centers introduced it among its activities. There is a tremendous need for career counseling services because most university students do not have a broad understanding of the various career tracks available to them. Beyond the actual replication of the university career development and youth center models, EEA developed an important strategic partnership with the US-based National Career Development Association (NCDA) which has the potential to create a ripple effect if this partnership can be expanded to reach more students at more public universities in Egypt with needed services. 25 Intermediate Result II: Improved Access to and Relevance of Education and Employability Training This section of the evaluation focuses on Intermediate Result II of the EEA overall evaluation framework, and more directly looks at whether the EEA alliance-based programming approach has helped improve target youth’s access to quality education and/or employability training and job placement services. The analysis is set forth through the lens of the following indicators. Figure 10: IR II Indicators ACCESS • Indicator 2.1: Number of young men and women participating in EEA programs • Indicator 2.2: Number and percentage of young men and women completing EEA programs QUALITY AND RELEVANCE • Indicator 2.3: Number of teachers, trainers, facilitators and counselors trained under EEA supported programs • Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in EEA supported programs • Indicator 2.5: Level of satisfaction of young trainees with the quality and relevance of the training, counseling and services received through EEA programs • Indicator 2.6: Level of employer satisfaction with EEA-trained youth Summary of Intermediate Result II Findings A total of 29,873 youth (vs. 24,210 targeted) participated in various technical/vocational, entrepreneurship and life skills training between 2006 and 2009. An aggregated 26,006 completed the training (vs. 17,813 targeted) and 3,867 dropped out, thereby achieving an 87% completion rate. In terms of gender balance, 57% of participants were male and 43% were female. Indicators 2.1 and 2.2: Number of young men and women participating in and completing EEA training projects In sum, EEA exceeded anticipated totals for number of youth participating in and completing EEA training programs. Across the five countries, a total of 29,873 youth ages 15 to 24 participated in various EEA training programs, versus 24,210 targeted.9 Of these participants, an aggregated 26,006 youth or 87% of total participants completed the training, with 3,867 youth (13% dropout rate) not finishing training. Taken together, these figures indicate EEA reached 5,663 more participants than originally anticipated in the development of projects and 8,193 more individuals completed programs than originally anticipated. In assessing access and overall participation of youth, the team has erred on the side of conservatism. In this respect, it is important to note that this 29,873 figure does not include over 7,655 people who received life skills training at Nike factories in Indonesia because the training provided was short and provided to young operators who are already working as part of their professional development training. Approximately 10,000 youth in Pakistan who received Microsoft IT training were also not counted for purposes of employment statistics under IR III as tracking of graduates in these programs was insufficient to assess their employability outcomes. Training activities were designed based on local interests and needs, and therefore, were different in nature, scope and duration. However, generally speaking, EEA training programs had three fundamental components: technical/vocational skills training, life skills training, and on-the-job training followed by counseling and job placement services. These components were present in all countries except for Egypt which in its two largest programs provided career development training and counseling courses that 26 complemented academic work in universities and was coupled with less rigorous job matching services. Figure 11 presents a comparative analysis of participation, completion and dropout rates across five countries. Figure 11: Direct Youth Beneficiaries: Participation, Completion, and Dropout Rates Country Number of participants Number of youth who completed EEA Training Number of youth who dropped out Target Actual Target Actual Egypt 11,750 17,644 8,252 14,759 2,885 Indonesia 5,079 5,549 4,063 5,504 45 Pakistan 1,430 1,406 1,287 1,386 20 Philippines 2,958 3,036 2,702 2,669 367 Morocco 2,993 2,238 1,509 1,688 550 Total 24,210 29,873 17,813 26,006 3,867 As illustrated in the figure above, three EEA countries – Egypt, Indonesia and Philippines - exceeded their targets in terms of the number of participants. Pakistan and Morocco achieved 98% and 80% of their targets respectively. In terms of the number of youth who completed the training, Egypt, Indonesia and Pakistan met or exceeded their targets. Philippines and Morocco reached 97% of original targets. The scope, type and duration of training differed in each country based on specific interventions designed based on local needs of target groups. EEA-Egypt in large part concentrated on providing complementary job readiness training (short courses) and job matching services through public universities which allowed them to reach larger numbers of youth, many of them with higher degrees of educational attainment. In contrast, other countries offered longer, more comprehensive employability skills training relevant for underserved and more traditionally vulnerable target youth populations. In these cases, training generally lasted longer – from a minimum six months to a maximum two years – with a focus on providing integrated life, technical/vocational, equivalency education, job placement support and follow￾up mentoring support based on local needs and frequently in direct partnership with industry. Recruitment Process: The recruitment process was also examined to determine whether EEA has improved access to education and employability training for target youth and what difference the alliance approach has made. In this respect, evaluation surveys and analysis suggest that use of the alliance approach encouraged EEA to work with a large array of local governments, youth groups and NGOs to not only mobilize local resources but also to publicize the program more widely to better reach vulnerable, hard-to-reach youth groups. Survey results show that nearly thirty percent of 447 survey respondents learned about the EEA program through local governments and youth groups. In Indonesia and Philippines, higher percentages of youth learned about the EEA program through these avenues, seventy percent and forty percent of respondents respectively. Survey data also indicate that as pilot activities completed, communities and graduates became advocates for the program and contributed to the recruitment process. For instance, over fifty percent of the youth surveyed heard about the program from their friends or family members who are EEA alumni, and the rest learned about EEA from their former schools and the media (newspaper, radio, TV and brochures). Selection of Trainees: Upon receipt of applications, across all programs, implementing partners interviewed prospective candidates to assess their interests, motivation, education and other background information. In this process, all EEA implementing partners used the following selection criteria to screen and select young participants: 1) between 15 and 24 years of age; 2) socio-economic situation considerations – youth from economically disadvantaged families; and 3) demonstration of aptitude, commitment and potential. This adherence to selection criteria appears to have helped avoid confusion and maintain transparency and accountability as EEA generated high levels of interest among young people and local government officials who frequently advocated for selection of their own constituents. EEA’s outreach efforts also appeared to be effective in reaching youth from different backgrounds and geographic areas. EEA served youth with various backgrounds including university students (Egypt), youth with only elementary or middle school-level education (Indonesia), as well as out-of-school and rural youth (Indonesia and the Philippines). Each project had its own criteria used in recruitment and selection of beneficiaries, and these criteria included household income, educational attainment and 27 employment status of youth as set forward in recruitment forms. While baseline data were not available for the overall EEA project, partner surveys and interviews as a part of this evaluation confirmed that partners adhered to such criteria in selecting trainees, particularly from economically disadvantaged families. Gender: There was no overall gender strategy integrated into the project, although an equal number of male and female beneficiaries were targeted and individual projects sought to recruit equal numbers of both male and female participants. In terms of gender balance, two out of five EEA countries – Indonesia and Morocco - were able to attain nearly equal ratios even though some technical training projects such as engine repair and broiler chicken farm operator training had difficulty recruiting female participants. Pakistan received a slightly higher ratio of male participants. The country experienced some difficulty recruiting female participants since social norms made mobilization of young women into technical programs more difficult, particularly for its compressed natural gas station operator training project. In Egypt, the total number of males considerably exceeded the total number of females. This appears to have been due to the fact that EEEA worked largely through the faculties of engineering at Cairo and Ain Shams Universities which traditionally enroll many more male students than female students. Nevertheless, it should be noted that women accessed EEEA youth center projects at higher rates than EEEA university programs. Figure 12 presents the classification of gender participation by country. Figure 12: Classification of Gender Participation in EEA Programs by Country Country Male Female Egypt 67% 33% Indonesia 48% 52% Morocco 54% 46% Pakistan 59% 41% Philippines 70% 30% Challenges with respect to youth participation and completion were largely around oversubscription and inability to meet needs. Some countries struggled to limit the number of trainees based on agreed activities and budget. This was particularly the case in Egypt and Indonesia where EEA garnered high interest and received more applications than anticipated from unemployed youth. Pakistan and Philippines faced difficulty recruiting female participants because of the nature of some technical training provided, such as chicken farm operator training and engine repair. Indicator 2.3: Number of teachers, trainers, facilitators and counselors trained under EEA supported programs EEA provided training to 432 trainers in life skills, IT, and technical skills, exceeding its initial target by 44 as indicated in Figure 13. Such trainers were provided a combination of orientation, mentoring, counseling, placement and training services, and several training of trainers sessions were organized between 2005 and 2008 in all five countries. Figure 13: Number of Trainers Trained by Country Target Number of Trainers Actual Number of Trainers Egypt 102 111 Indonesia 24 26 Morocco 97 141 Pakistan 54 51 Philippines 111 103 Total 388 432 The training of trainers varied according to country needs. In Morocco, Philippines, and Pakistan, country programs adapted IYF’s Passport to Success life skills curriculum and trained trainers specifically to deliver life skills lessons during and beyond EEA. For technical/vocational training activities, EEA relied largely on locally accredited training institutions for training curricula and trainers minimizing the need to recruit and train new trainers. However, while this strategy allowed creativity and flexibility and 28 promoted sustainability, it proved to be more challenging because it required a greater level of monitoring and quality control to ensure quality and effectiveness of a series of training offered. More generally and consistent with best practices, country programs that conducted pilot or mid-term evaluations performed better because they were able to garner feedback from both youth and employers and improve as necessary. In this process, youth were also given an opportunity to provide feedback and suggestions – which in itself is a worthwhile and useful process because it was the first time for many students to critique their trainers who were encouraged to leave the rooms to give privacy to students. Since EEA interventions were designed to provide a combination of services in various countries within different cultural contexts, program success became largely dependent upon the quality of teachers, trainers and counselors. Accordingly, EEA graduates were asked to rate their trainers/teachers in the following four areas during the evaluation using a five point rating scale:10 overall performance, technical knowledge, ability to provide mentorship and counseling, and capacity to keep youth engaged and motivated. An average of 557 students responded to each question and over ninety percent of students rated their trainers as either good (55%) or excellent (36%). Students were most satisfied with trainers’ overall performance and capacity to keep them engaged and motivated throughout the program as indicated in Figure 14. Figure 14: Youth Satisfaction with Trainers Very poor Poor Fair Good Excellent N/A Number of respondents Overall Performance of Trainers 1% 1% 5% 56% 37% 0% 526 Technical Knowledge 0% 4% 4% 58% 32% 2% 583 Ability to Provide Mentorship 1% 3% 3% 54% 34% 5% 575 Capacity to Keep Youth Engaged and Motivated 1% 4% 2% 50% 40% 3% 544 From a sustainability point-of-view, the use of alliance approach and leverage requirements has encouraged EEA partners to contribute their human resources. Universities, regional education and professional training institutions, ministries, local associations, national NGOs, and private sector entities have shared knowledge and expertise of their own trainers, teachers and facilitators with EEA. Such in￾kind contributions have reduced the overall project cost, ensured full adoption and buy-in of the programs by the partners, and promoted program sustainability. Additionally, EEA seed funds enabled partners to refine their curricula and upgrade the skills of trainers. Capacity building efforts have meant that trainers are positioned to provide the follow-up support and mentoring to EEA graduates to the extent their own institutional resources allow. They will also be prepared to provide similar in-country training, counseling and mentoring services to other youth beyond EEA with tested curricula and teaching methodology. Also of interest, through interviews conducted as part of this evaluation, trainers in both Morocco and Pakistan reported that they themselves had undergone transformation after participating in training of trainers for life skills programs. In interviews conducted in Morocco, for example, trainers reported that EEA experience changed the way they look at teaching and interacting with youth, and had a positive influence on teacher-student relationships. They also reported improvements in their teaching methodology, coaching styles now catered to meet the needs of youth, and improved team spirit among trainers. Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in EEA supported programs As noted in this report’s introductory section on evaluation limitations, as EEA used different curricula and methodologies across its 35 pilot projects in the countries of focus, this evaluation was not able to standardize and systematically measure learning gains among youth across the program. To address this limitation, during country evaluations, country teams used a combination of quantitative and qualitative measurement tools to assess the progress of youth in acquiring technical, life and other skills. These 10 Five-point scale using indices such as very poor, poor, fair, good and excellent. 29 different instruments have allowed the evaluation team to triangulate the results to ensure data accuracy and validity. Some of the instruments used include, but not limited to: 1. Classroom-based tests administered to youth throughout the training and at the end of each training program 2. Government-administered trade exams 3. Youth self assessment questionnaire: which asks youth to assess their own capacities before and after the program 4. Employers’ assessment of the performance of EEA trainees as interns and employees: this is through an interview of employers and direct supervisors of youth that can give first hand observations and assessments of youth job performance (See indicator 2.6)11 While instruments used by each program varied under this indicator, the global evaluation performed a survey of 503 youth trainees who provided self assessments of their learning gains and perceptions of the training’s value. This self-reporting questionnaire results show that over 469 (93%) of the 503 respondents reported that they have gained technical knowledge and skills needed for employment. Surveyed youth in all five countries confirmed through self assessment that the EEA supported programs provided them not only with the technical skills they needed for employment, but most importantly, with the soft skills that they felt were needed to be accepted in the job market. Nearly 85% consider that life skills they have acquired through the EEA programs were good to excellent, and 75% consider that technical skills they have learned from the EEA programs were good to excellent. In terms of life skills acquisitions, the most significant changes were reported in areas of communication skills, listening skills and self-confidence. For example, 51% of 334 respondents felt that they experienced significant improvements in their communication skills; 50% of 327 respondents for listening skills; 48% of 370 respondents for self-confidence and 43% of 307 respondents for conflict-resolution. Similarly, 77% of 365 youth surveyed reported that they felt more positive about their future as a result of their participation in EEA. Indicator 2.5: Level of satisfaction of EEA trainees with the quality and relevance of the training, counseling and services received through EEA programs Survey results indicate that program graduates were generally pleased with the quality and relevance of training provided. It appears that graduates appreciated most a number of activities/approaches used: dynamic mentoring relationships with trainers; practical applications of job skills or the internship component of programs; life skills teaching methodologies; and follow-up technical and social support as they began internships and jobs. Youth were asked to rate their level of satisfaction with five aspects of the training, and Figure 15 presents the results in these areas. On average, 82% of youth rated the training as either good or excellent, and the ratings were most positive for the content, quality of training materials and life skills taught. As presented in Figure 15, 74% of youth respondents felt that the training received was highly relevant to their employment needs. Figure 15: Youth Satisfaction with EEA Training Very poor Poor Fair Good Excellent Number of respondents Overall Content of the Training 0% 2% 5% 59% 33% 558 Quality of Materials 0% 3% 7% 52% 34% 560 Relevance of Training to Employment Needs 1% 4% 13% 52% 22% 504 Life Skills Acquired 1% 2% 6% 50% 34% 545 Technical Skills Acquired 1% 4% 9% 48% 27% 542 11 Examples of the instruments used under this indicator differed in each country, although all attempted to maintain a standard that would help ensure project outcomes and youth learning gains would be appropriately measured. For example, projects in the Philippines tracked the number of youth who passed the government trade exams and received certificates to determine whether graduates met the competency standards established in specific industries. Some projects in Indonesia such as engine repair upheld similar standards and arranged for students to take local and national competency exams. In Pakistan and Morocco where IYF’s Passport to Success life skills training was conducted, implementing partners used specific tools - such as mastery exams and surveys - to measure the learning gains of students. 30 Of 711 youth surveyed, 184 were entrepreneurs who received entrepreneurship training, financial and business support from EEA. Of 184 entrepreneurs, 87% reported that they gained the necessary knowledge and skills to establish small businesses. Ninety-five percent felt prepared as entrepreneurs. On average, 72% of entrepreneurs rated the quality of coaching services received as good (43%) or excellent (29%). Youth participated in the evaluation also identified areas for improvement for similar interventions in the future and provided suggestions as follows: • Technical training: In some EEA countries such as Morocco and Indonesia, youth considered the EEA life skills training to be more valuable and higher quality than the technical training component, citing a desire for technical training materials to be reviewed and improved, and strengthening the skills of some trainers. • Training Facilities: In certain countries, the facilities used for life skills and technical training by the projects did not provide for the most conducive learning environment. For example, in Morocco, participants and trainers complained about broken and unsanitary lavatories, inadequate lunch, and rest areas or absence thereof, and inadequate and unequipped classrooms. For two projects in Indonesia, participants of engine repair and carpet making training expressed a desire to have larger quantities of and more advanced equipment and tools to practice with and use. • Training Duration: All countries except Morocco and Egypt provided an average of six-month training programs. In Morocco trainings generally exceeded this timeframe, with one to two years of training inclusive of an internship performed. In Egypt, many courses through the University Career Centers were only for a few days. In general, training durations shorter than six months were considered by youth to be too short and not deep enough to provoke significant and lasting technical gains or attitudinal shifts. From the perspective of the partners, some projects faced time and resource constraints and therefore attempts were made to condense training activities. Some partners were also worried that youth might lose interest if the training was too long as many had to work to earn income. In most cases, the duration of training was not tested before interventions were designed to reach an optimal level for some EEA projects which in turn likely affected the level of satisfaction and outcomes. Indicator 2.6: Level of employers’ satisfaction with the soft skills and/or technical skills of EEA trained youth This evaluation involved seventy employers who provided job opportunities to 684 youth and internship opportunities to 1,037 youth across five countries. The process of surveying and interviewing employers helped to capture the employers’ opinions, satisfaction levels with EEA training programs, as well as suggestions for similar alliance-based interventions in the future. Employers represented a wide range of industries including information technology and hardware, telecommunications, trading, apparel, hotel and hospitality, banking and financial services, consulting, academic institutions and research units, and agriculture and farming companies. Entrepreneurial and technical sectors included automotive repair, mobile maintenance unit, ship construction, house construction, auto repair, mechanical works and fabrication, processed food production, medical services, animal husbandry, handicrafts, and others. Over seventy percent of the employers surveyed learned about the program through implementing partners. Following market assessments, EEA partners in the different countries used different strategies to approach and recruit employers. In countries where employers were involved more thoroughly throughout the program, programs seem to have performed better in terms of teaching relevant skills and placing youth beneficiaries in internships and jobs. Through a review of program reports and interviews with EEA managers, implementing partners and employers, the following strategies were indicated by stakeholders as most effective in improving internship/job placement rates and achieving high levels of employer satisfaction: • Involving potential employers in curriculum design and improvement, which helped ensure that training materials were relevant and demand-driven. • Identifying and making prior arrangements with businesses to offer internships, preferably with stipends, meals or other forms of compensation for young interns. • Conducting regular consultative and feedback meetings with internship providers to monitor and improve the performance of trainees, which increased their chances to be offered jobs as seen in Morocco, Indonesia and the Philippines. This was seen a critical step towards placing youth in 31 good jobs because internship is a probationary period for many youth before they are offered jobs and close monitoring has made a huge difference. • Providing opportunities to potential employers or corporate partners to give competency exams to graduates and recruit them at the training site. This enabled employers to choose from a pool of qualified graduates and corporate partners were able to place youth in jobs quickly and effectively resulting in high job placement rates in Indonesia. The study also sought to determine how employers learned about the program: Seventy-two percent of the seventy respondents were approached directly by partners, 19% said the program was introduced and referred to them by their contacts, four percent learned about the program from program brochures, and two percent learned about the program from newspapers and advertisements. The rest mentioned other ways such as referrals by the trainers, companies’ public relations efforts to reach out to NGOs, personal contacts and youth approaching companies themselves through internship and job applications. In terms of employers’ level of satisfaction with the performance of EEA graduates, 58% of the seventy employers surveyed rated the overall performance of EEA graduates as good, and 21% as excellent. Employers commented the EEA graduates had the required basic knowledge and skills, were serious, eager to learn, and ambitious. Employers also commented that the CVs that were provided to them through the different EEA programs were of high quality, and that they (the employers) were willing to offer jobs and internships to EEA youth despite the fact that most EEA graduates had no prior work experience. Sixteen percent of seventy employers surveyed rated the overall technical skills of EEA graduates as excellent, 54% as good, and 27% as fair. The study also asked whether employers would be willing to offer jobs to EEA youth again and 98% of employers said they would consider offering jobs, internships or apprenticeships to EEA graduates in the future. Philippines received some negative responses from larger companies where automation and more complex skills were needed for the work. While largely positive about student capabilities, employers also presented some recommendations to mend the weaknesses of some EEA programs. These recommendations included: • Further improve technical skills of EEA youth by increasing duration and scope of training • Further enhance life skills by adding more topics, particularly work ethics and communication skills of EEA youth to work effectively in professional settings • Ensure continuity of the training program and hiring processes • Conduct regular assessments of employers’ needs and adapt the curriculum to respond to the latest needs and requirements • Hold consultative and feedback meetings with internship, apprenticeship and job providers In addition, employers were asked to compare EEA graduates with other employees from similar age groups. In Indonesia and the Philippines, EEA graduates had to compete with and were often compared to graduates of similar or longer two to four-year vocational/technical training programs. Thirty-two percent of the seventy respondents thought the EEA youth were similar to their non-EEA peers in terms of general performance and professionalism, while 49% of respondents reported that EEA graduates were much better than their non-EEA peers, and 11% said EEA youth employees were the best. The evaluation results also reveal that 83% of employers thought that EEA has been able to provide qualified graduates for the business sector that met their needs and expectations, and 96% of responding employers stated that EEA program was closing the gap between employers’ needs and the employees’ qualifications. Statements by surveyed participants also suggest that working with the Alliance has had any changes on human resource management practices of partner companies. For example, forty percent of surveyed employers said that they were more open to the idea of providing internships and mentorship support to youth in general, and 29% of surveyed employers said that their companies plan to become more involved in EEA-like alliance-based programs focusing on youth employability. Fifteen percent stated that they were more likely to recruit youth through EEA and other similar programs as a result of participation in the EEA program. In sum, EEA appears to have served as a catalyst to encourage corporations through their CSR efforts to be part of programs that helped youth and responded to community needs. This in turn helped employers gain a positive image within their respective communities driven by their involvement in youth employment programs. 32 Intermediate Result III: Improved Prospects for Employment and Successful Entrepreneurship This section seeks to address the issue of whether EEA was able to improve prospects for employment and entrepreneurship for target youth and measure the effectiveness of the EEA alliance approach in placing youth in jobs within six months of graduation. The following indicators were used to help guide these assessments: Figure 16: IR III Indicators CHANGE IN EMPLOYMENT STATUS • Indicator 3.1: Number and percentage of trainees who got a job within six months of completing the program • Indicator 3.2: Number and percentage of EEA trainees starting their own businesses within six months of completing program YOUTH SATISFACTION WITH AND QUALITY OF INTERNSHIPS AND JOBS • Indicator 3.3: Number and percentage of EEA trainees with satisfactory internships or apprenticeships • Indicator 3.4: Number and percentage of placed youth who declare having a satisfying and quality job As noted previously, concerning limitations of this evaluation study, a few factors relating to timing of the evaluation and the comparability of different projects supported in EEA countries posed specific challenges in presenting findings under this Intermediate Result. Three issues are of particular note: • Timing of Project Completion versus Timing of the Evaluation: Deadlines for the completion of the overall EEA Global program necessitated that country evaluations would be required to begin in December 2008 and end by August 2009. Unfortunately, particularly with respect to the largest program in Morocco, Emploi Habilite, the larger cohorts of trainees were still in training￾related internships as this overall global evaluation was being finalized. Additionally, under the evaluation framework set forward at the outset of EEA activities, it was anticipated that a period of six months would be appropriate to track placement given the desire to ensure placement efforts could bear fruit and to measure the sustainability of employment generated through the program. This time challenge was exacerbated under this indicator, as a number of projects also lacked this six month time frame to fully assess outcomes under this intermediate result. As such, in certain instances, it became premature to fully measure employment rates, trends and sustainability in a consistent way across all EEA programs and countries. To address this challenge and present the most useful data for analysis, the team only included cohort classes that completed the training in early-2009 to allow sufficient time for tracking and assessment of employment outcomes under this IR. Where the division of cohorts has taken place in this respect, we have made an appropriate notation. • Comparability of Country Data: An additional challenge was in the full comparability of job training and placement programs. As noted above under IR II, four countries – Indonesia, Morocco, Pakistan and the Philippines - largely utilized comprehensive training programs that integrated life skills, technical skills, and internship/job placement activities that lasted from six months to a year. While some differences were present in terms of timing and training and placement approaches, such programs were reasonably comparable and data has been included in both global and country-specific tables. With respect to programs in Egypt, however, the substantial bulk of beneficiaries were participants in a more abbreviated range of career development training courses for university students lasting from approximately two days to one week in length. Additionally, as typical with such career development centers in other countries, job placement was more focused on career counseling services and job matching for a subset of participants in Center activities. As such, the comparability of such programs both in terms of the rigor and length of training interventions and job placement was less than optimal and in some respects caused significant variations among country outcomes and overall average outcomes under this intermediate result. 33 • Eligibility for Jobs: EEA provided a range of training interventions through 35 pilots. Some projects were very focused on direct job training and placement for those ready to enter the job market. However, a number of other projects targeted young people were not eligible or ready for jobs right away. These projects included programs for vocational high school students in Indonesia that were not graduating; first, second and third year university students in Egypt; and students who are currently undergoing one to two-year internships in Morocco. As such, for the purpose of assessing employment outcomes, only data and analysis relevant to employable cohorts are presented under IR III. As such, a distinction must be made between the total number of graduates who are counted under IR II and the total number of graduates eligible for jobs or self-employment who are counted under IR III. Summary of Intermediate Result III Findings Taking into account these limitations, an aggregated 8,580 youth were determined to be eligible and ready for employment or self-employment in the time frame allotted. Of these, 4,801 have been placed in jobs or set up small businesses within the six months of completion. This figure represents 56% of the total number of youth completing the training. This overall figure of 4,801 youth can be broken into 3,843 youth (80%) who have been able to get decent work upon graduation and 958 youth (20%) that have been able to set up small businesses with EEA support. Across five countries, EEA set an initial target of 7,046 youth to become employed or self-employed. Therefore, against original targets, EEA attained the overall employment rate of 68% against this original target. With respect to employment training programs specifically, 7,129 individuals completed programs, and 3,843 found jobs, representing an overall employment rate of 54%. EEA graduates are placed in variously sized-firms representing a wide range of industries such as engineering, marketing and sales, office administration, textile, hotel and restaurant management, chicken farming, auto-repair, welding and seaweed production. Similarly, out of the 1,451 youth that underwent entrepreneurship training, 958 were able to set up small businesses with EEA support, representing a success rate of 66%. The study also seeks to determine whether youth who had prior jobs are better off working as entrepreneurs, and assess the viability of new businesses established. Survey results show that 44% of 293 young entrepreneurs responded had jobs prior to joining EEA, and nearly half of them reported to be earning more income as a result of joining EEA. In terms of vaiability of these businesses, survey results point to positve results and trends. For instance, 84% of 94 survey respondents have been able to cover expenses with earnings and 80% of 127 respondents reported making profits. Based on this data, it is concluded that a majority of businesses will be able to sustain themselves but whether the remaining ten to twenty percent of businesses will be able to survive in this financial climate without funded oversight is still in question. Individual country outcomes in both employment and entrepreneurship are provided in Figure 17 below: 34 Figure 17: Overall Job Placement and Entrepreneurship Success Rates by Country Countries Number of youth completing employment entrepreneurship training Number of youth getting jobs or setting up small businesses Job placement and entrepreneurship success rate against completion Egypt 2,750 886 32%12 Indonesia 1,286 1,074 84% Morocco 912 315 35%13 Pakistan 963 565 59% Philippines 2,669 1,961 73% Total/Overall Rate 8,580 4,801 56% Indicator 3.1: Number and percentage of EEA trainees who got a job within six months of completing the program EEA placed a total of 3,843 youth in jobs across five countries, resulting in an overall placement rate of 54%. As illustrated in Figure 18 below, success in reaching employment targets varied from one country to another, with a high of 84% in Indonesia and a low of 21% in Egypt. Figure 18: Job Placement Rates by Country Countries Number of youth completing and ready to work Number of youth placed in jobs Job placement rate against completion Indonesia 850 714 84% Morocco 568 300 53% Pakistan 963 551 57% Philippines 2,440 1,794 74% Egypt 2,308 484 21% Total/Overall Rate 7,129 3,843 54% Based on the alliance partner survey results and interviews, job placement success appears attributable to the following factors in most countries: • Industrial knowledge of EEA’s implementing partners • Ability of partners to respond quickly to local market conditions and work in close partnership with local employers who are in high growth industries • Ability of local partners to establish good partnerships with local businesses, and engage them in meaningful ways during training and placements • Development and execution of strategies to monitor the performance of interns and follow up with consultative, feedback meetings with employers to help address trouble areas; and with internships providers – to ensure EEA was meeting the needs of employers 12 As noted above in the introductory section to IR3, the comparability of some programs in Egypt as related to programs in other countries is somewhat limited. This was due to substantially shorter training courses and job matching services supported by Career Development Centers in Egypt, rather than more rigorous job placement conducted in concert with employer-supported training and internships. In this respect, it is interesting to note that the more limited youth center training cohorts in Egypt, where training programs were roughly comparable to the training programs in the other four countries, had a job placement and entrepreneurship success rate of approximately 58%. If only this cohort is factored into the global averages for job placement and entrepreneurship success, the average rises from 56% in Figure 17 to 66% overall. 13 The Morocco EEA program’s low overall rates of placement and entrepreneurship are largely due to the poor performance of one entrepreneurship project out of six total projects that experienced substantial management challenges during the project. As such, 344 young people were considered trained by this project, but only 15 were able to start new enterprises given poor follow-up by the implementing partner, resulting in a four percent success rate for this project specifically, As demonstrated by Figure 18, the five remaining projects, all related to job training, had an overall placement rate of 53%. 35 • Success in nurturing close relationships with local business and partners using appropriate cultural norms around charity and supporting the disadvantaged Conversely, the reasons for lower employment rates for some EEA countries appear to have included the following factors: • Original placement targets were set unrealistically high and projects were overly optimistic about their ability to place trainees. • Some sub-grantees did not conduct sufficiently rigorous labor market assessments at the beginning of the programs to help ensure the availability of jobs. • Labor markets and demands were constantly changing from the time the training was designed vs. the time for job placements. • The onset of the global economic crisis during the pendency of training programs caused significant reluctance on behalf of employers to commit to full way employment for trainees. • Some projects could not facilitate or track job placement for women in some remote and traditional areas where job opportunities were rare, which was the case for Pakistan and part of Indonesia. Egypt in particular fell short in reaching its original target of 2,497 youth placed in jobs, and had a lower employment rate of 21%, having placed 484 youth in jobs. Such lower placement is likely due to unrealistic assumptions about the possibilities of placement through short term training and job matching services, as well as a very challenging employment market in Egypt during the last year of the program during the global economic crisis. Additional information regarding the comparability of Egyptian job training and placement programs to other countries is provided above. In order to have a sense of the effectiveness of job placement services, the team asked youth how long it took to find jobs in the youth follow-up survey. Of the 166 youth who responded to this particular survey question, 88% of them reported finding jobs within six months of completing the training, and ten percent reported taking longer than six months to find jobs. Higher percentages of youth were able to find jobs quickly within one to three months in Egypt and Pakistan [92% and 84% respectively]. In terms of the types of jobs offered to EEA youth, each country had its own experience worth highlighting: Egypt: Employers included multinational companies, national companies (where multinational and national companies were mostly for engineering job and/or internship opportunities), small enterprises such as processed food companies and vocational centers and self-employment. Jobs and internships offered were for the following positions: engineers, administrative, marketing and sales, technical wand factory workers, trainers, and quality controllers. Indonesia: EEA’s requirement of placing youth in jobs upon graduation was new to some partners, and therefore, Indonesia Business Links (Secretariat) and IYF took a hands-on approach and invested significant resources in outreach and post-training support, working closely with implementing partners and aiming for a high job placement rate. Corporate partners and potential employers were given access to recruit from a pool of qualified graduates after competency exams. Graduates were placed in jobs with small to medium-sized enterprises in the following sectors: automotive and motorcycle repair, chicken farming, computer assembly, financing and sales, T-shirt design and making, and fish filet production – across West Java (Bogor, Karawang, Bandung), Central Java (Pati), East Java (Surabaya, Pasuruan and Jombang) and the Northern part of Greater Jakarta. Morocco: Sectors covered included tourism, textile, sales, and services (including plumbing, electricity repair, welding, and mechanics), IT, and office administration. The tourism sector hired more youth. There were three kinds of employers that offered internships and jobs to EEA graduates: • Large international companies and Franchises such as Bull, Munisys (IT), New Rest, Fruit of the Loom (Textiles), Accord (Hotels), and McDonalds (Restaurants). These large companies hired large numbers of young EEA interns at a time. • Medium-sized enterprises such as member enterprises of the Association of Women Entrepreneurs, small textile enterprises, hotels and restaurants, steel and machine shops, auto repair, and construction companies. Each of these medium-sized companies hired one to five interns. 36 • Small and micro-enterprises generally ran by one or two people, such as auto repair workshops, or small service companies. These enterprises would typically hire one or two interns at a time. Pakistan: Pakistan focused on the following industries: hotel and restaurant management, customer relations and services, compressed natural gas (CNG) station management, embroidery and knitting for women. The program faced challenges facilitating and tracking job placements in some areas for women in some remote areas; however, the job placement rate for women in the hospitality sector was higher because of their willingness to work in kitchens. Philippines: For formal employment, youth were placed in the following industries: steel and machine shops, auto repair, ship building and construction. Employers in two projects were partners of EEA from the beginning, specifically Habitat for Humanity and Sitangkai Seaweeds. They had significant interest in the outcome of the youth training programs since the quality of training outcomes had a direct effect on their core businesses of building homes or growing seaweed. Additionally, many graduates were also employed through industry or business associations that EEA had partnered with. These were employer￾members of the Metal Industry Association of the Philippines and the Davao City Chamber of Commerce. These employers offered internships to graduates and hired those who excelled. Indicator 3.2: Number and percentage of EEA trainees starting their own businesses within six months of completing the program Across five countries, EEA enabled 958 youth or 66% of those who completed the entrepreneurship training to receive the necessary skills and establish small businesses as presented in the table below: a total of 402 in Egypt, 360 in Indonesia, 15 in Moroco, 14 in Pakistan, and 167 in the Philippines. Against its original target of helping 808 youth become entrepreneurs, EEA has been able to help 150 more entrepreneurs than targeted. In addition to the entrepreneurship training, EEA provided financial assistance, equipment and mentoring support to youth to start and subsequently expand small, sustainable businesses. Additionally, youth-led businesses in Indonesia created 516 jobs for the communities in West Java. Additional and significant numbers of jobs may have also been created in other country entrepreneurship programs, but such job creation numbers were not tracked regularly. Jobs created within Indonesia were in sorting, harvesting, and marketing of catfish, production of fish feed, and processing and packaging of fish filets. These jobs have become primary or secondary sources of income for many coastal families and youth. In Pakistan, entrepreneurship was not part of the program design; however, 14 graduates decided to start small businesses with their newly acquired confidence and skills. Gaining access to capital, however, became a major hindrance, and graduates expressed a desire to receive some type of financial support from EEA. As indicated in Figure 19, in Morocco, the Entrepreneurship Spirit Project trained 344 youth but failed to provide substantive support to a majority of graduates given very low capacity and challenges with the implementing partner. Only 15 students from this project set up small businesses after receiving assistance from microfinance institutions. Figure 19: Entrepreneurship Success Rates by Country Countries Number of youth completing entrepreneurship training Number of youth setting up small businesses Entrepreneurship success rate against completion Egypt 442 402 91% Indonesia 436 360 83% Morocco 344 15 4% Pakistan 0 14 NA Philippines 229 167 73% Total/Overall Rate 1,451 958 66% Youth satisfaction rate was generally high in this area – with over seventy percent of 184 young entrepreneurs surveyed expressing satisfaction with the training and support received. Additionally, 87% 37 of these 184 youth felt they gained the necessary knowledge and skills to start up a small business from EEA as a result of participation in the program. The evaluation also attempts to determine whether youth are better off working as entrepreneurs and assess the viability of new businesses established. Survey results are positive. Eighty-four percent of 94 entrepreneurs surveyed said they were able to cover business expenses with earnings. Eighty percent of 127 respondents said they were already making profits. Based on these data, it is concluded that a majority of businesses will be able to sustain themselves. Whether the remaining ten to twenty percent of businesses will be able to survive in this financial climate without funded oversight is still in question. Despite ultimate successes in this area, entrepreneurship projects were challenging to start up, according to surveys conducted in the evaluation. Interviews with program managers indicate that EEA countries struggled to implement entrepreneurship projects at the outset of activities due to five reasons: 1. Cultural norms that discourage youth to become entrepreneurs because of a different level of prestige compared to formal jobs or due to uncertainties to earn stable income 2. Youth not having enough confidence or experience 3. Financial climate and external shocks such as natural disasters and sharp increase in consumer prices that threaten the existence of enterprises 4. Inadequate financial resources allocated for entrepreneurs within countries 5. Lack of sufficient experience of partners to implement entrepreneurship components of their overall employability interventions To address such issues, as projects were underway, EEA learned the need to provide more hands-on business and social support from the alliance. The alliance helped youth stay focused and motivated, and assisted them in mobilizing additional resources to counterbalance external shocks and reinstate project activities if crises occurred. EEA also hired outside experts to strengthen and sustain businesses in precarious situations where in-house expertise was not sufficient. For example, Indonesia provided more comprehensive financial and business support to youth to start businesses with significant contributions from the alliance partners. In two Indonesian projects focusing on chicken farming and auto-repair shops, youth had their parents contribute resources and made them family-run businesses which then increased the likelihood of such enterprises being successful and sustainable. Indicator 3.3: Number and percentage of EEA trainees with satisfactory internships or apprenticeships Internships were an integral part of EEA programs. The duration of internships generally varied between two to four months, with some programs in Morocco offering one to two- year internships during which trainees received a complement to their technical training and put into practice their newly learned life skills. In certain cases, trainees received small stipends for their work as interns or apprentices, and in some of these circumstances, especially for trainees in the tourism sector, the best performing interns received employment contracts at the end of their internship period. Youth were asked to rate their internship experience using a five-point scale and results are presented below: seventy-five percent of youth surveyed rated the internship experience as good or excellent while eight percent rated as very poor or poor. Overall internship experience was considered the most worthwhile by EEA graduates – followed by technical skills learned and coaching. Figure 20: Youth Satisfaction with Internships or Apprenticeships Very poor Poor Fair Good Excellent Number of respondents Overall Internship Experience 2% 6% 13% 43% 32% 293 Relevance of Internship Assignments 2% 4% 19% 45% 24% 270 Technical Skills Learned 1% 7% 15% 43% 31% 275 Coaching Received During Internships 3% 5% 11% 41% 32% 275 38 Some negative responses – poor or very poor - were driven by the following factors: • Internships did not fit with the training because some youth participated in basic technical training and were not prepared to use sophisticated tools and equipments at factories they interned. Indonesia, Morocco and Philippines had a few of these cases, for example, for training such as engine repair. In some cases, supervisors refused to allow females to take on substantive work. This was especially the case for girls completing their internships in the field of computer hardware repair in Morocco. In some rare cases in Morocco, female interns were subject to subtle yet real harassment from their direct supervisors. This was the case for young women who were completing their internship in the hotel industry. When such incidents were reported, remedial action was immediately taken. • In other cases, interns were asked to perform tasks that they felt were not directly related to their internships. This was the case for the young Moroccan pre-school teachers who worked in small private pre-schools, and for some interns in Indonesia and the Philippines to some extent. Indicator 3.4: Number and percentage of placed youth who declare having a satisfying and quality job EEA graduates were asked to rate their level of satisfaction with the following aspects of their jobs: overall work environment, relevance to the field of study, support from supervisors and peers, professional growth opportunities, salary and other compensation. Survey results reveal the following: • While 84% of 205 respondents were satisfied with the overall work environment, they were not as satisifed (40% unhappy) with salary and other compensation provided to them at current workplace. For example, around fifty percent of the Egyptian trainees considered that they were not paid enough in comparison with the tasks they were implementing. It was also the case for 25% of the employed EEA youth in Pakistan. • Professional growth opportunities appear to have been made available for EEA graduates with 75% of youth reporting as happy or very happy. • In terms of training relevance, 75% of youth were happy and the rest were not, which suggests that there are still some cases of skills mismatches. • Eighty-three percent of youth were satisfied with the support receiving from their supervisors and peers. In Egypt, Morocco and the Philippines, about twenty percent of youth expressed a desire to receive more support from supervisors and peers, and the figure goes up to 38% in Indonesia and down to ten percent in Pakistan. Some of the issues were that youth were viewed suspiciously by older workers or found employment far away from home and needed more technical and social support. Figure 21: Job Satisfaction of Youth Very unhappy Unhappy Happy Very happy Number of respondents Overall Work Environment 2% 12% 61% 23% 205 Relevance to the Field of Study 6% 10% 49% 26% 195 Support from Supervisors and Peers 5% 12% 45% 38% 193 Professional Growth 8% 13% 48% 27% 190 Salary 7% 26% 47% 16% 205 Other Compensation 14% 19% 43% 17% 181 It is important to note that trainees’ level of satisfaction was greatly influenced by the field they ended up in. For example, those working in tourism and IT are more satisfied with their jobs than those performing more physical tasks. Entrepreneurs who were running successful businesses were found to be happier as they have their own businesses, make profits and can hire additional employees. Indonesia and Morocco were able to ask whether graduates who had jobs previously were now earning more income as a result of participating in EEA programs. Eighty-five percent of 65 respondents said yes in Indonesia while only 29% answered positively for Morocco. 39 Intermediate Result IV: Positive, Indirect Effect on Families and Communities of Youth While not included in the original monitoring and evaluation framework for the program, Intermediate Result IV was added to the final evaluation to assess the indirect effect of EEA on the families and communities. As such, two proxy indicators were used: Figure 22: IR IV Indicators INDIRECT EFFECT ON FAMILIES AND COMMUNITIES OF YOUTH • Indicator 4.1. Number and percentage of employed youth financially helping their families with household, health and education expenses • Indicator 4.2. Number and percentage of youth who have reported improved financial and social conditions of their families as a result of their financial support Indicator 4.1: Number and percentage of employed youth financially helping their families with household, health and education expenses It is not uncommon in developing countries, especially in Africa and Asia, for children to share their incomes with their families and help pay for daily household expenditures or education for their siblings once they start working – as a way to repay their parents’ support and as part of becoming a responsible adult. Survey results show that most of the youth share their income with their parents and siblings now that they are able to work and earn regular income through jobs or enterprises that they have set up. Most of the youth in all five countries reported that they can begin saving for their future, invest in their children’s education and cover health expenses as they begin working. However, youth involvement in their community as a whole was not automatic. While most youth showed willingness to help their immediate family and friends, in some countries, such as Morocco, their commitment to help a wider community has not happened yet. This was not part of the original EEA program design, and would need more support and supervision from alliances if youth were to play leadership roles in their communities. Life skills, counseling and mentoring offered through the EEA programs contributed to change this attitude and to promote a sense of responsibility among EEA youth. The study however shows that EEA has demonstrated to have a multiplier effect, and had a positive if indirect effect on families. With improved livelihood, youth are able to contribute to promoting social and economic development of their families. Survey results are presented below: • Sixty-three percent of 238 respondents are now sharing their income with parents and siblings for daily household, health and education expenses. • Twelve percent of youth have their own families and are using the income to support their spouses and children. • Forty-three percent of youth use their earnings mainly for personal consumption such as rent, food and clothing. • Forty-two percent of youth are beginning to save part of their earnings for future use and investment. Indicator 4.2: Number and percentage of youth who have reported improved financial and social conditions of their families as a result of their financial support The study seeks to determine not only how many youth are helping their families but also how many feel that they now have the ability to improve the financial and social conditions of their families. Survey results show that 82% of 295 graduates responding stated that that their financial support (with the income they shared) helped improve their family’s financial and social situation. • Philippines reported the highest percentage [92%] of 93 youth reporting improved financial and social situation. • In Egypt, 83% of 12 respondents reported improved financial and social situation. • In Pakistan, 82% of 60 respondents reported improved financial and social situation. • In Indonesia, 81% of 85 respondents reported improved financial and social situation. 40 • Morocco reported the lowest percentage of 42% of 45 youth surveyed reporting some improvements – because youth were primarily using their earnings for personal consumption and felt that their income was not enough to share with families. One significant issue in Morocco that potentially resulted in such perception was the existence of government incentives which subsidized employers to hire youth at only the minimum wage. The unfortunate consequence of such subsidy was all employers providing wages at this level, rather than any higher amount to compete for qualified employees. Philippines and Indonesia’s higher numbers of youth reporting improved situations appear closely linked with the highest percentages of Indonesian and Filipino youth reporting to be earning more income as a result of participation in the program. It should also be noted that some families and communities have benefitted directly from new jobs created by EEA-supported enterprises, particularly in Indonesia. Overall, 18% of 295 graduates surveyed felt that there was “no change” because earnings from entry-level jobs were not enough to support their families after spending it for personal needs. Morocco reported the highest percentage of youth [58%] reporting “no change” whereas other countries reported much lower percentages [ten to twenty percent] who stated that their income did not help improve the social and financial situation of their families. 41 Section IV: Lessons Learned and Recommendation This evaluation has reviewed the progress, achievements and challenges of EEA projects dispersed over five countries, presenting what we hope is a useful framework for reflection on lessons learned and recommendations for future efforts of this nature. These lessons learned are presented in light of the two overarching aims of this study: • The building and sustaining of country- and community-based alliances through the EEA operational framework and • An assessment of how employment-related outcomes were affected by EEA alliance building approaches. Each of these overarching aims is broken down into areas of learning in the sections below, followed by specific recommendations provided in light of such lessons learned. A. Building and Sustaining Alliances EEA’s approach appears quite unique in its use of country and community/project level alliances of stakeholders from the public, private and civil society sectors to support youth employability development aims. Within each country, as noted in discussions of Intermediate Result I, alliance building was carried out at the global, country and community levels, first through the work of IYF and the establishment of Secretariats and Steering Committees to guide overall program activities. This was then followed by the creation of project-level alliances to achieve specific aims, and these alliances were provided with tailored technical assistance in needs assessments, program design, life skills and technical training interventions, the leveraging of resources, and monitoring and evaluation, among other areas. Part 1: Identifying Effective Secretariats EEA has been generally successful in identifying NGOs best-suited to serve as Secretariats and providing them with the technical support necessary to undertake the unique implementation approach supported by this program. This includes the formation of Steering Committees and project-based local alliances in support of programs. The study finds that capacity building was provided in the following areas critical for the development of community-based interventions with multiple stakeholders: designing a subgrant award and monitoring system, identifying and recruiting alliance partners, structuring partner relationships, conducting due diligence on partners, developing or enhancing curricula and other learning materials, training of trainers, and improving program and financial management practices. Recommendations: • Understand that alliance building is a unique skill that requires an emphasis on building relationships, finding areas of common interest and providing creative approaches to structuring workable partnerships among diverse stakeholders. • The development of partnerships across sectors is particularly challenging at times, as alliance conveners must be able to speak multiple “languages” of different sectors and translate those languages between partners as deals are contemplated, negotiated and finalized. Such skills are distinct from the focus of many NGOs aimed primarily towards implementation of projects, necessitating that significant effort be taken to find the appropriate organization for undertaking this type of “secretariat” or brokering role. In fact, in some circumstances, initial focus was placed on more implementation focused organizations for this role, and ultimately structural changes were required to better utilize the strengths of different organizations in support of overall alliance building and implementation aims. • From EEA’s experience, secretariat organizations might either be more associational in nature to begin with, or may have used alliance based approaches in their implementation of projects, enabling them to see both the value and also the challenges of partnership. As alliance-based programs are developed, it is critical to assess and understand the areas where secretariats need most support and guidance and provide them with the appropriate capacity building and guidance as they begin to learn this role. These steps have proven to be essential in ensuring that Secretariats were prepared and had sufficient capacity to undertake tasks associated with 42 alliance building, which also helped promote sustainability of the alliance approach and good management practices. Part 2: Establishing Country Alliances and Steering Committees In terms of setting up steering committees or national alliances intended to provide overall guidance, EEA used two approaches. Some countries such as Indonesia recruited mid-level experts and practitioners, some of whom represented an institutional interest in the success of EEA as corporate donors or NGO partners. As a general matter, these alliance members invested significant time and resources during program design and implementation stages to help projects get off the ground. Other countries such as Morocco and Egypt recruited high-level figures who were not able to dedicate as much time and efforts on EEA but had wider networks that proved to be useful at program outset to establish alliances, generate leverage and support long term sustainability and scalability efforts. Both approaches appear to have their benefits and disadvantages. Programs appeared more likely to get full attention of steering board members and useful guidance under the first approach, where more technical advisors could give fairly direct advice, building on specific experience and filtering best practices into the programmatic discussions. To the extent possible, they would also provide direct connections to prospective leverage and other partners, helping to expand the benefits of programs. Such advisors appeared more likely to continue to maintain interest and focus on the project progress as the overall program matured, taking a keen interest in project outcomes. In the end, however, such technical focused teams did not always have the direct ability to garner broader interest in initiatives essential for a more dramatic expansion of activities and the integration of lessons learned into larger initiatives being undertaken by national stakeholders. Conversely, those alliances with higher level representation seemed to be successful in engaging members early on in the process to set strategic direction and help distribute grant funds. However, as these key activities concluded, alliance members were more difficult to engage, with other high priority national initiatives taking their attention. This proved more challenging as alliances attempted to present lessons learned and gain further programmatic guidance from alliance members as programs matured. Meetings were often difficult to schedule and frequently alliance members may have designated a less senior person to represent them in meetings. Nonetheless, in key areas, the presence of a higher level advisory committee had important benefits. As alliances began to engage leverage partners and other key stakeholders, particularly in government, the marketability and credibility of program interventions increased significantly by virtue of its association with a high-level board. Similarly, while difficult to engage at times, certain advisory members became extremely important champions for several efforts, particularly when challenges arose that required high￾level brokering and mediation of disputes among sectors and as programs went from a first to a second or third stage with more partners, particularly in the public sector. Recommendations: • Consider recruiting a mix of both high-level and mid-level experts and practitioners focusing on both local and international issues - combined with some policy makers – to maximize contributions and program benefits. One possible approach is to create two bodies – one more strategic and visible, and another more technical and focused on working level issues. However, it is important in this area to ensure that such structures do not increase bureaucracy and management burdens on the alliance. • Clearly formulate strategies, incentives and mechanisms to ensure Secretariat-like structures are effectively coordinating Committee activities, providing regular information and updates, actively seeking the support and guidance of Committee members, and maintaining the momentum for participation throughout the program. In this respect, Board and Alliance members should be given regular opportunities for providing their feedback at all stages of the implementation, particularly during program design and post-midterm evaluations. In case Board or alliance members have time constraints to attend regular meetings, efforts should be made to have a telephone conversation or individual meeting to solicit their feedback and guidance and gather their support for particularly critical areas of need. • Train local staff on how to generate public interest and forge public-private alliances; reinforce alliances in appropriate cultural contexts; and negotiate and resolve conflicts if crises arise. 43 Knowledge transfer to local staff and to implementation partner staff helps promote the scalability and sustainability of alliance-based projects. • Dedicate sufficient program management support staff to fill the various roles that alliance programs demand. • Develop alliances based on mutual respect and understanding. If possible, go beyond legalistic, formal partnerships and build vibrant alliances with a shared sense of purpose and responsibility, knowledge sharing and open communication. To the greatest extent possible, leverage local cultural approaches that encourage caring for and partnering with local communities to achieve social good. • Clearly define roles and responsibilities of each alliance member and enable members to keep each other accountable and responsible throughout the project. Part 3: Forging Project-level or Local Alliances Forging local alliances was regarded as the most critical and challenging stage in alliance building in order to show real results in overall efforts. At this phase, IYF worked closely with National Secretariats to build their capacity in developing local alliances and working with selected implementing partners to develop and implement alliance based projects. The study finds that capacity building was done strategically and successfully, particularly in light of two indicators tracked globally - the number of partnerships and amount of leverage resources obtained. In this respect, capacity building efforts enabled EEA to forge 319 partnerships and mobilize over $9 million in leverage in support of the 35 pilot activities by program end. The EEA program demonstrates the significant increase in beneficiaries and cost efficiency possible under the multi-stakeholder approach – with the cost of less than $600 per beneficiary of which roughly only fifty percent was borne by USAID. Multiple and different strategies were used to recruit corporate, NGO and government partners: countries used IYF and Secretariats’ institutional contacts, recruited partners through wider networks of Committee and alliance partners, and identified and visited potential partners door-to-door. This latter effort was time consuming but proved to be highly effective in Indonesia and Morocco in recruiting local businesses that were potential employers. For certain projects, initial alliances had been already established before implementing partners submitted proposals to EEA, which were then built upon and expanded in support of program requirements. In several cases, the IYF global and national secretariats provided substantial support in recruiting corporate partners for innovative project areas before providing formal approvals. Overall, EEA’s ability to engage the private sector for program design and employment support has been looked upon as crucial to the program success. Having potential employers review and approve the curriculum materials helped to confirm that skills training provided was demand-driven and relevant. Giving local business partners access to a pool of qualified graduates and the ability to test and recruit them through competency exams or internships also helped secure high job placement rates. Regular feedback meetings with employers have also enabled the program to improve the quality of graduates produced. More generally, the study found that alliance development efforts are more effective when public and private service actors and private businesses stop looking at their work with the alliance as mere charitable contributions, but rather as a part of their broader corporate responsibility and a critical community and business issue. In this respect, as corporate partners – particularly at the local level - become more engaged, they begin to working collaboratively with implementing partners in designing program content and on the job training modules, monitoring, and sharing knowledge and expertise. All had the end goal to better prepare youth for employment and help their bottom lines. Recommendations: • Assess and meet the needs of Secretariats before they are tasked with building local alliances, especially in areas of finding and negotiating with right partners and raising leverage resources in support of programs. Their ability to forge vibrant alliances will ultimately go far in determining program outcomes. • Review proposals with a structured but consultative approach, enabling stakeholders from all sectors to identify flaws in program design, avoid duplication and maximize synergy, and ultimately find ways to contribute themselves to help promptly establish local alliances. • Clearly define and agree upon roles and responsibilities of each partner to achieve optimal results. Moreover, it is important to ensure a reasonably even distribution of roles and 44 responsibilities, both in terms of fairness and also broadening the platform for expansion of initial program successes through multiple channels. Project-based memoranda of understanding (MOUs) are one avenue to ensuring clarity around roles. However, negotiation of documentation around alliances should not become the focal point of efforts. Rather, the formality of alliance must give way to the practical realities of implementation, and the MOU should be viewed as a step in building trust around a partnership framework that can be expanded once tested and proven successful after its first phase. • Find innovative ways to mobilize and recruit local businesses to support the program – such as outreach visits and calls presenting the program and benefits of joining the alliance – to assure long term effectiveness and sustainability. Global corporate partners are also vital for alliance prominence, recognition, resources and rigor – but optimally partnerships should include a broad array of smaller, local companies to help generate increased appeal in local communities and direct effect in terms of job placement. In this respect, local businesses should be engaged meaningfully by tapping into their knowledge and expertise and involving them in curriculum design and pilot activities. Similarly, they should be given access to program graduates for internships and job opportunities, and regular feedback regarding the performance of graduates should be requested to improve the quality and relevance of the training. B. Assessing EEA Employability Outcomes Part 1: Utility of the Alliance Approach in Providing Effective Skills Training The use of the alliance approach influenced the overall EEA program design and implementation, particularly as it supported strong connections to employers and the private sector. The study shows that engaging potential employers and other stakeholders through this approach has made the training programs more comprehensive, demand-driven and relevant to industry needs. It also mobilized community support to make sure that participants completed their courses and continued on to find productive work. In particular circumstances, CSR community empowerment projects were focused on hard-to-reach areas and, in these areas, conducting community research together and then co-designing the project with corporate partners and local community development experts has proven to be helpful in ensuring that economic activities are locally driven, relevant and viable. Recommendations: • Engage all key stakeholders during the design stage – particularly training providers, prospective employers and local experts – to increase the quality, relevance and effectiveness of the training. It is equally important to keep in mind, however, that while such engagement of stakeholders had a significant value, engaging new partners in program design and implementation presented certain challenges. In particular, while new implementation partners often had very strong community connections, they had less experience in implementation of quality employability programs. As such, while EEA provided design support for interventions, it often provided less mentorship during implementation and relied significantly on implementing partners to deliver the education/vocational training. This resulted in less desirable employment outcomes in certain projects, highlighting the need to perhaps conduct more thorough review of training materials and testing of modified training durations, as well as more deliberate coaching and capacity building during the implementation phase of activities. • Put in place a robust process for Board and Alliance participation that can encourage meaningful collaboration at various stages of implementation and improve the program design, quality and effectiveness. However, the establishment of such processes requires considerable human resources in terms of time for coordination of all efforts. While in the end alliance building can encourage strong local ownership and better long term sustainability prospects, those designing such projects must be conscious of the level of effort required to fully ensure long term success. It should also be noted that such alliance building poses unique challenges in that time spent in this pursuit should not be substituted for quality mentorship and capacity building during the implementation phase of projects. • Increase program effectiveness by integrating post-training follow-up support as a key component of program designs. This evaluation found that the transition for disadvantaged youth into a formal working environment poses both new and exciting opportunities but also frequent challenges in terms of adjusting to new norms, expectations and often critical feedback. 45 As such, mentorship for youth should be provided for at least the first six months as youth enter the labor market. This can also help correct cases of skills mismatches and encourage responsiveness to employer needs. • Engage youth in meaningful ways as equal stakeholders of the alliance. Solicit their feedback and consult with them on a regular basis to ensure training is relevant, to better addresses youth skill deficiencies and is responsive to changing youth perceptions about career opportunities. • Increase gender inclusiveness by training females as trainers and providing additional support to women as they run small businesses or take on their first jobs. Be ready to address related challenges, such as the need to: provide additional support including awareness raising with families of young women and surrounding communities in order to increase retention; create safe classroom environments that are gender-friendly and culturally acceptable; and offer mentoring to newly employed women facing disparities in pay or treatment in the workplace. • Assess the capacity of implementing partners dealing with entrepreneurship interventions at the program outset to determine where they need assistance. If there is no in-house expertise in a particular industry area, engage outside experts who can provide substantial mentoring support to young entrepreneurs at least for the first six months of establishing enterprises. Set program benchmarks to monitor the progress of new businesses. Be prepared for external, unanticipated events that can seriously affect business operations. Part 2: Internships/Apprenticeships Demand-driven skills training and internships/apprenticeships are essential steps to introduce young people to the job market, give them tangible experience as they start on their career paths and bring them into the continuum of job placement and career development. In terms of internships and apprenticeships, most students reported having positive experiences, which was helpful to ensure many of them to obtain jobs. Recommendations: • Ensure that internships and on-the-job training are present in virtually all job training programs for young people. However, the experience of youth in the workforce should be systematically and strategically monitored to ensure that youth receive the necessary support and also fair compensation where possible. Regular feedback meetings with internship providers are helpful because it helps youth improve their performance and increases their chances of getting jobs. In particular, EEA’s experience demonstrates that providing follow-up support to out-of￾school youth (who are employed or running businesses for the first time) is important, and that this element should be fully incorporated into program designs so that youth can make a successful transition into the labor market. As a corollary, projects should work closely with employers also to ensure they create a nurturing environment for young people starting their transition into the workforce, providing sufficient opportunities for mentorship, growth and reflection on challenges. • Help youth beneficiaries to negotiate with internship providers for some form of compensation. If this is not an option, projects should optimally allocate sufficient resources to provide some stipends to youth so they can participate in internships and gain useful experience. • Revisit internship and training periods to ensure youth are equipped with sufficient knowledge, skills and practical experience. Internship durations that are too short will likely provide little experience or time to adjust to a new working environment, while internships that are too long may discourage employers from taking the step toward hiring for full time employment. • Design and provide tailored comprehensive skills training based on the needs of youth. This helps graduates to be fully prepared to undertake internships and subsequently enter the job market or start new businesses. Solicit feedback from youth throughout implementation. Those projects that have done so appear to have performed better. 46 Part 3: Job Placement Services Overall, EEA was able to provide job placement and entrepreneurship support that enabled 56% of its target youth to become employed or self-employed. EEA also helped create over 500 additional jobs for communities in Indonesia, the only country where this element was systematically monitored. Employer satisfaction rates are also high and on average sixty percent of employers interviewed reported that EEA graduates were better workers than other employees of similar age groups. However, EEA programs often did not reach their specific targets as it appears these targets were frequently set unrealistically high for some untested pilot activities being undertaken by new and nascent partnerships. Similarly, the challenging financial climate during this period became worse toward the end of program and affected newly established businesses and available job opportunities. Finally and most importantly, EEA designed and offered shorter technical training to help youth gain marketable skills, get jobs, and earn income immediately. While training programs were in large part reasonably successful in spite of the practical time limitations of training and internships, EEA-graduates faced fierce competition from graduates of longer two to four-year technical, vocational training programs for jobs. Recommendations: • Supporting disadvantaged youth in obtaining jobs begins with creating favorable conditions for job placement throughout the design and training process. Recommendations in this area include: o Conduct market assessments and monitor labor market assessments on a regular basis. Targeted, non-academic market assessments with a substantial sample of employers help ensure that the training provided is demand-driven and enable program designers to make adjustments in order to avoid saturating markets. o Engage potential employers from target industries to develop relevant and quality training models. Where possible, engage business leaders as resource people or guest speakers throughout implementation, not only to help students acquire practical knowledge but also to help them make contacts which can lead to internships and jobs. Secondly, when deciding training durations, be mindful of employer needs as well as the situations of disadvantaged youth who may have financial constraints. Also, consider offering transportation and meal stipends so that youth can travel to find jobs. o Carefully decide on training duration. Program designers should revisit and adjust training and internship durations based upon previous program experiences, particularly given the need to provide post-training follow-up support to graduates. Lengthy design times for interventions are necessary to ensure proper engagement of stakeholders. However, this should not come at the expense of sufficient implementation time to enable job training, placement support, and long term integration into the job market. • Establish a strong advocacy network to change the attitudes of businesses toward hiring former out-of-school, inexperienced youth compared to those who have undergone more formal training. Help them understand the business benefits of training youth in their communities, both in investing in human capital and stimulating local economies. Broadcast success stories to show employers that this “risk” is worth it. • Provide regular feedback and consultative meetings as the first cohort class graduates and enters the workforce. This helps ensure real time improvements are made in areas where training is weak. Additionally, in order to produce graduates who meet the expectations of employers, it is important to create organized local or national level platforms for employers from target industries to gather and provide feedback and recommendations on a regular basis. • Increase program effectiveness by soliciting feedback from youth and integrating post￾training follow-up support as a key component of program designs. A key finding of this evaluation was that the transition for disadvantaged youth into a formal working environment poses both new and exciting opportunities, but also frequent challenges in terms of adjusting to new norms, expectations, and often critical feedback. As such, mentorship for youth – and if possible allowing them to use training facilities to strengthen their technical skills – should be provided at least for the first six months as youth enter the labor market. Consider utilizing technology, such as mobile phones or online platforms, to systematically track graduates who are either looking for a job or just beginning one, and provide the job counseling support they need to be able make a full transition to productive adulthood. • Promote entrepreneurship as a way to help further expand job opportunities for trainees and increase the overall market for jobs for young people. Pilot projects in EEA that 47 incorporated entrepreneurship tracks as a back-up plan or part of the overall plan appear to have performed better – as they were able to help youth set up small businesses where jobs were scarce. Develop clear strategies at the outset to help them develop viable business plans and link them with financial institutions. However, EEA learned that implementing partners and youth need more hands-on coaching and mentoring by experts at later stages. As such, to support the sustainability and success of these businesses, similar projects need to allocate enough resources and time to provide such support. Part 4: Sustainability and Scalability of Employability Projects As noted under IR1, sustainability can be evaluated at multiple levels, including at the country and project levels. As discussed, the development of long term sustainable country alliances at the national level appears to have been a largely unrealistic goal given the relative size of the EEA program in each country and the presence of a wide array of stakeholders among multilateral and bilateral donors, government ministries, trade associations, educational institutions and businesses of all types that would need to be fully engaged in such an effort. Similarly, as the program focused predominantly on developing alliances around project-level activities, broader advocacy efforts by necessity were not emphasized as frequently. Nonetheless, the EEA alliance-based framework had some success in sustaining project-level activities and positioning them for scale through the multitude of partners involved in the design, support, implementation and evaluation of projects. This has demonstrated the utility of using an alliance based approach, providing a broader array of avenues and supporters to build on projects that work and expand them collectively and through their own networks for the long term. Recommendations: • Start the process of alliance building from the outset of needs assessment and design. Engaging stakeholders from all sectors in assessing challenges related to youth development and in determining what is currently and successfully being done is a powerful first step in building their interest in supporting youth programs at a later date. • Think about scale and sustainability from the very outset of discussions around design. It should be clear to all parties that initial pilots are being put forward with the expectation that stakeholders will utilize their networks and resources to expand activities if they prove successful. The design should also embed strategies for scale in the pilots, for instance, working with local organizations or training institutes that have proven systems for training of trainers and creating local cadres of master trainers. • Think carefully about creating rigorous but user-friendly systems for monitoring and evaluation, so that partners can be kept easily apprised of progress and clear successes can generate momentum around the prospects of scale and long term sustainability. Similarly, find ways to ensure regular check-in systems and visits to projects by partners that are inclined to support a second and third phase of the project. In all discussions around the project, emphasize the need for rigor in employment approaches and transparency in assessing outcomes, building confidence in the accuracy of project results. • Engage local governments and support them to sustain project activities beyond EEA. Such activities appear to have worked well – especially if the model and methodologies have proven successful and that governments are engaged early as strong internal champions. Local governments have ample resources and also a direct need to serve constituencies of parents and youth. However, it is important also to fully assess capacity constraints and incentive challenges in the system with respect to government employees. Finding champions in the system can help find innovative solutions to such challenges, but this is a fundamental challenge of working successfully with the public sector. • Encourage corporate partners to adopt the program as part of their broader CSR initiatives so that they can help secure strategic financial and material support for the program. The presence of respected corporate stakeholders also helps to increase needed visibility to project efforts for stakeholders such as government and their constituencies. Local corporate partners are often the strongest ally of initiatives and can serve as important catalysts to generate initial interest in the program as well as the scale up of projects later on. Often, corporate leaders are themselves from disadvantaged communities, and are eagerly seeking ways of increasing the local development of these areas. 48 Conclusion This evaluation was conducted to address two fundamental questions about the utility of alliance-based interventions in EEA’s 35 projects. First and foremost, the evaluation sought to determine whether the use of a multi-sector approach in designing and implementing employability programs for youth delivered unique benefits in terms of education and employability outcomes for young people. This was a particularly important question to ask given the increased time and efforts that appear to have been necessary in developing such programs as compared to more traditional development models that place less emphasis on engagement of partners across sectors. While a fully comparative evaluation is difficult to make regarding these larger questions, it is clear that the EEA approach fostered significant and direct contacts with the private sector throughout EEA programs, which helped ensure the relevance and ultimate effectiveness of training programs. As noted in this evaluation, the focus on private sector engagement helped to secure strong job placement and internship rates, high employer satisfaction with trainee qualifications, and the high enthusiasm of the trainees about their prospects for success in the marketplace. Similarly, private sector support of the program helped nurture and support entrepreneurship programs, with young entrepreneurs able to leverage this support to help set up new businesses and create new jobs. The alliance-based approach also appears to have had an important side benefit of demonstrating to companies the importance of supporting disadvantaged youth – support that could be provided as both part of their corporate social responsibility commitment and for direct business benefit. Secondly, the evaluation sought to better understand whether the EEA alliance-building approach and the use of multi-stakeholder partnerships helped promote the sustainability and scalability of interventions. Here too, EEA’s multi-sector approach brought forward concrete benefits by involving a broad array of partners who were significantly invested in program activities through the dedication of time, energy and resources. USAID funds, used to incentivize other prospective partners, helped bring in many nontraditional groups to work with USAID and created a framework to test and expand upon project-based partnerships to reach greater numbers of youth in second and third phases of activities. This framework was also instrumental in widening the circle of partners and supporters to help continue projects after USAID support ceased. As a result, a significant number of EEA projects show prospects for long term sustainability and increased cost efficiency over time. It is our hope that taken together, the experiences, learnings and recommendations in this report will be useful to those in the public, private and NGO sectors who are either planning or already engaged in an alliance-building strategy to improve outcomes for youth worldwide. ©2010 International Youth Foundation. All rights reserved. No portion of this report may be reproduced in any form without the written permission of IYF. This project was made possible by the generous support of the American people through the United States Agency for International Development (USAID) under grant agreement no. RAN-A-00-04-00043-00 This report was funded in part under grant agreement no. EPP-A-00-08-00006-00. The International Youth Foundation (IYF) invests in the extraordinary potential of young people. Founded in 1990, IYF builds and maintains a worldwide community of businesses, governments, and civil-society organizations committed to empowering youth to be healthy, productive, and engaged citizens. IYF programs are catalysts of change that help young people obtain a quality education, gain employability skills, make healthy choices, and improve their communities. To learn more, visit www.iyfnet.org ANNEX A: Project-level Sustainability Status Education and Employment Alliance: Project-level Sustainability Status EEA Projects Project Summary Partners Sustainability Status Egypt Ain Shams University Career Development Center To create a Career Development Office at Ain Shams University to provide 1,050 students access to job training, placement, and mentorship activities and to place 200 graduating students in jobs Egyptian Junior Business Association, Ain Shams University, EJB Members, Microsoft and Industrial Modernization Council 12 MOUs signed with faculties and four universities to replicate the career development model pending available resources Cairo University Career Development Office To create a Career Development Office at Cairo University providing 2,600 students access to job training, placement, mentorship and placing 400 in jobs Nahdet El Mahrousa, Seweidy Family Foundation, Cairo University, TopNotch HR Company, BP, I￾Source and Training Providers Mind Profit - Egyptian Youth Center One Stop Shop To develop a self sustainable one stop shop for youth employability at Ain el Sira Youth Center in Cairo that provides training programs for 1,500 youth and links 540 to jobs Alashanek Ya Balady, Vodafone, German University in Cairo, American University in Cairo and Mirqah Partnership developed with the US-based National Career Development Association (NCDA) to expand the one stop shop model pending available resources. Revenue generating mechanisms have been put in place Maadi El Gadeeda Youth Center One Stop Shop To develop a self sustainable one stop shop for youth employability at the Maadi Gadeeda Youth Center, providing employability training in environmentally friendly businesses and placing 300 youth in jobs Sekem Development Foundation and Egyptian Biodynamic Association Fayoum and Minia Youth Center One Stop Shops To develop a self sustainable one stop shop for youth employability at two youth centers in Minia and Fayoum that improves vocational training and life skills programs for 1,300 youth and links 400 of them to jobs BEST Foundation, Catholic Relief Organization, Axon International Studies Network, Ministry of ICT, Cisco and Shell Indonesia Livelihood Recovery through Tempe Production and Marketing Project To support 17 local entrepreneurs in Yogyakarta in producing and marketing modern snack foods from traditional products, and to create 148 additional jobs for the community KAKAK, GAP, Local Governments and the Center of Nutrition and Food Business association established as a legal entity and new businesses are anticipated to sustain themselves and expand their product lines under with help from EEA partners EEA Projects Project Summary Partners Sustainability Status Economics for Life in Vocational High Schools Entrepreneur￾ship Project To support the teaching of Economics for Life lessons to 2,900 youth in Indonesian vocational high schools, to provide 250 students support in starting new companies, and to train 24 teachers in Oracle's Think.com platform for schools Prestasi Junior Indonesia, Oracle Corporation and Newmont Corporation Phase II of this project is currently being implemented with the same partners and estimated to reach an additional 3,000 high school students West Java Coastal Community Empowerment Project To strengthen the cooperative fish filet industry in Eretan Kulon in West Java and creating 200 new jobs by providing equipment and training for villagers in better processing, packaging and marketing KUD Mina Bahari, BP Indonesia and Local Government The filet industry is anticipated to sustain itself with support from the local Committee established with EEA multi-sectoral partners Bina Swadaya Catfish Farming Project To provide entrepreneurship skills to twenty youth in Eretan Wetan village in West Java that assist them in starting up catfish farming businesses that create 95 additional jobs LSM Bina Swadaya, BP Indonesia, Local Government of Indramayu Kabupaten and PT. LS Aquaculture The catfish businesses are anticipated to sustain themselves and continue expanding with support from the local Committee established with EEA multi-sectoral partners Cepat Tepat Youth Vocational Training and Entrepreneur￾ship Project To equip 140 youth with vocational skills in the areas of motorbike repair and sewing and assist them in getting jobs or establishing small-scale businesses PKBM Cepat Tepat, BP Indonesia, the Local Government of District Karawang, PKBM Aditya, Bina Mitra, Toyota Astra and local automotive garages New businesses are anticipated to sustain themselves with help from EEA partners. The project is anticipated to continue with support from the local education office and reach an additional fifty youth Factory Employee Life Skills Training Project To train 16 trainers and 40,000 operators in eight factories supplying products to Nike in Jakarta, and improve life skills and positive attitudes in the work environment Business Dynamics and Nike Nike factories are implementing this project on their own using in￾house trainers EEA trained, and estimated to reach over 30,000 new operators CREATE Chicken Broiler Training Project To train 150 youth in broiler chicken operations, and of successful trainees about 90% will be employed in existing broiler operations and 10% will be assisted to start new small operations The Center for Human Resource Development and Applied Technology (CREATE), Charoen Phokpand, Kurnia Poultry Farm, Perdana Putra, Medion and Vaksindo The curriculum developed with EEA funds is still being used by CREATE. Corporate partners still supporting entrepreneurs and EEA partners. Project unable to continue for lack of funding for the next phase. Proposals being developed based on the tested model of EEA EEA Projects Project Summary Partners Sustainability Status Improving Livelihoods of Youth in Surabaya, East Java To improve the economic conditions of 260 disadvantaged youth in Surabaya by equipping them with necessary skills and helping them secure better jobs or establish sustainable businesses Pupuk Surabaya, ALIT and YP3IS New businesses are anticipated to sustain themselves with continued support from the partners SEKAR Youth Vocational Training and Entrepreneur￾ship Project To train 77 disadvantaged youth in Jakarta in waste recycling production of art paper and handicrafts and to start their own businesses SEKAR , PT. Astra International, Gallery Q'ta, Kurnia KAPPIJA￾21 and PT. Suhuf Art Paper New businesses are anticipated to sustain themselves with help from EEA partners. The project is anticipated to continue with support from new funders in different target areas Assisting Young T-Shirt Entrepreneurs in Bandung To assist 1,500 youth in obtaining jobs in the local T￾shirt industry or starting their own franchises Jendela Budaya Lingkungan, Local Bandung Government, C-59, PT. Bio Farma and PT. Nu Art Consultant Company New franchises are continuing to receive help from C-59 and JBL. Unable to continue due to lack of funding Supporting Community Ecotourism in Bali To strengthen the ecotourism industry in Bali by training 100 local villagers in village-based local food production, tourism services and handicraft production Kehati Foundation, ACCOR Indonesia, Village Ecotourism Network and Wisnu Foundation The villages are anticipated to continue using newly acquired skills as they run village tourism businesses Morocco Digital Opportunities Workshop To provide 30 unemployed young people with training in computer installation and repair and to provide 50 schools and 25,000 students access to computers they refurbish Al Jisr, Microsoft and Moroccan Ministry of Education Al Jisr is in the process of opening a third workshop in Agadir in partnership with with Banque Populaire, Regional Academy of Agadir and Ateliers Sans Frontieres Emploi Habilite To improve the employability of 1,244 unemployed youth in Casablanca through life skills and IT training provided through Ministry of Education Vocational Schools and Youth Centers Mjid Foundation, Al Jisr, GE Foundation, Microsoft and Moroccan Ministry of Education Emploi Habilite is being expanded through additional funding from Qatari Foundation, Silatech, EFE-Maroc and Finnish Children and Youth Foundation (FCYF) Workplace Success To provide 230 youth with life skills training, placing 150 of them in jobs, and developing strategies for institutionalizing this life skills curriculum in the Hassan II University offerings Education for Employment Foundation, McGraw Hill and Hassan II University EFE signed an MOU with Hassan II University to integrate the project into the university’s curriculum through select departments. The program is being expanded with additional funding from FCYF EEA Projects Project Summary Partners Sustainability Status School for Entrepreneurs To assist in the creation of a School for Entrepreneurs, where 80 students will receive entrepreneurship training and 80 will be mentored by successful entrepreneurs in starting up new enterprises CJD, Federation of Small Industries and Hassan II University Replication of the project in the Southern city of Agadir with non-MEEA funding Reprofilage To provide 55 unemployed young women with training relevant to the employment needs of Moroccan companies and to place them in internships and/or jobs Moroccan Association of Women Entrepreneurs and its member companies Received additional funding from a Canadian donor to replicate the model in Marrakesh and continue in Casablanca Point Jeune To offer training, job placement and other support to 1,220 young, disadvantaged slum residents (400 of whom are expected to be placed in jobs) through the creation of a new Community Center Job Point and Digital Opportunity Workshop ENDA Maghreb, Fondation Air France, Holding Al Omrane and the National Initiative for Human Development Project ended due to management issues in the third quarter of 2008 Preparing Disadvantaged Young Women for Apparel Sector Employment To provide life and technical skills training to 130 disadvantaged Moroccan women in the apparel and textile industries, with 53 of them targeted to secure jobs in this sector L'heure Joyeuse and Nike The project did not continue due to lack of funding. L'heure Joyeuse gained accreditation as a government training center and will be able to provide government certified training Pakistan Microsoft/ Employability for IT Trainees To Improve IT training in 90 Vocational Training Institutes, used by 250 students and reaching 9,000 students annually Punjab Vocational Training Council and Microsoft The improved IT curriculum and teacher training benefits 10,000 PVTC trainees annually Hospitality Sector Skills Development To improve the skills of 400 young men and women for work in small and medium size hotels and restaurants and to promote the overall climate for tourism by Pakistanis and foreigners Aga Khan Foundation, Serena Hotels and Chefs Association of Pakistan Training program unable to continue due to lack of funding for the next phase Trainings in Vital Service￾Related Job Skills To provide 300 disadvantaged young women and men in three districts with training in service industry-related skills and to help place 250 into jobs ASK Development, DOVE and SUKHI Phase II under implementation with a $100,000 grant from the National Vocational & Technical Education Commission (NAVTEC) to reach an additional 580 youth EEA Projects Project Summary Partners Sustainability Status Sahara Art and Craft Center To provide skills training in arts and crafts for 250 young women and to help them generate income by selling goods at local Arts and Crafts centers Sahara Welfare Organization, Engro Chemicals and Ali Education Institute of Lahore Project unable to continue due to shift in programmatic focus by Sahara Welfare Association. Home-based businesses are anticipated to continue providing income generating opportunities for women Safe and Steady Employment at CNG Stations To provide certified training and job placements for 300 youth seeking to become CNG service station operators and to thereby increase safety conditions and station management capabilities Sabawon, Pakistan, Pakistan CNG Association, Civil Junction, Oil and Gas Regulatory Authority and Hydrocarbon Development Institute Project unable to continue due to lack of funding for the next phase Life Skills for Employability for Pakistani Youth To adapt, implement, certify and widely disseminate a comprehensive life skills program to benefit approximately 25,000 Pakistani youth annually following an initial pilot phase training 180 youth Rural Support Programs Network, Institute for Rural Management and NAVTEC Integration of life skills programming into implementing partner’s technical training program anticipated but not final at time of evaluation Philippines Civil Trades Training for Ex￾Combatants To equip 900 out of school youth of ex-combatants in skilled construction trades such as carpentry, plumbing and masonry and life skills, 754 of these trainees to be placed in jobs ALTERPLAN, Habitat for Humanity￾Philippines, UNDP Act for Peace and local employers Project continues – because of the need for construction workers and also a core project of Alterplan MIAP Plus To train 240 out of school youth with academic, life and technical skills required for employment in Metalworks and welding jobs in Metalworks Industry Association member companies, and to place 194 of them in jobs Metalworks Industry Association of Philippines MIAP, Cagayan de Oro Chamber of Commerce and the City Government of Cagayan de Oro Project being expanded and launched as Youth Productivity Services (YPS) project reaching an estimated 450 youth in nine municipalities of Misamis Oriental and is a core activity of MIAP Integrated Small Engine Repair in ARMM: KAPATID Program To provide 130 rehabilitated juvenile offenders with basic welding, small engine and automotive repair training at three training “service” centers, with 94 trainees expected to secure jobs after training Marcellin Foundation, Chevron, TESDA and ARMM Chamber of Commerce Project being expanded to communities and operating in General Santos City with Chevron funds EEA Projects Project Summary Partners Sustainability Status Integrated Agri-Fisheries Entrepreneurs hip Training for Rural Youth To provide 150 marginalized out of school youth, the majority of them women, with technical and entrepreneurship training in agri-fisheries production and, in turn, to create 50 social enterprises and 300 new jobs Kasanyangan Foundation, Local Government Units, TESDA and Zamboanga Colleges Project stopped due to lack of funds for the next phase Enhancing the Capacity of OSY in Agriculture Production - Mag-Agri To equip 180 out of school youth with demand-driven skills in food production and processing, fruit production, and organic vegetable production, with 113 to gain employment Davao City Chamber of Commerce and Industry, Mango Industry, University of Southeastern Philippines Project continues as Davao Chamber CSR program and in the University of Southern Philippines community outreach program Empowering Youth to Modernize the Seaweed Industry in Tawi-Tawi To support integrated technical training education projects for 1,040 in- and out-of-school youth in seaweed production, with 839 to find jobs Philippines Business for Social Progress, the Philippines Development Assistance Program and Mindanao State University Project continues with new donor funds and loan program for seaweed farmers from Development Bank of the Philippines ©2010 International Youth Foundation. All rights reserved. No portion of this report may be reproduced in any form without the written permission of IYF. This project was made possible by the generous support of the American people through the United States Agency for International Development (USAID) under grant agreement no. RAN-A-00-04-00043-00 This report was funded in part under grant agreement no. EPP-A-00-08-00006-00. The International Youth Foundation (IYF) invests in the extraordinary potential of young people. Founded in 1990, IYF builds and maintains a worldwide community of businesses, governments, and civil-society organizations committed to empowering youth to be healthy, productive, and engaged citizens. IYF programs are catalysts of change that help young people obtain a quality education, gain employability skills, make healthy choices, and improve their communities. To learn more, visit www.iyfnet.org ANNEX B: Typology of EEA Partners and Leverage Contributions by Country Typology of EEA Partners and Leverage Contributions by Country Egypt Partner Name Partner Role and Contributions Corporations Abercrombie and Kent Cash contribution to Ain El Sira Youth Center The Academy of Engineering Sciences Mobile maintenance training for Minia Youth Center; also employment opportunities for youth from the Center Berlitz Training for Ain Shams CDC BP Trainers/speakers/career coaching for Ain Shams CDC and Cairo CEDO Cameron Petroleum Company Internships opportunities for Cairo CEDO students Center for International Private Enterprise Entrepreneurship program curriculum for Cairo CEDO; Training of trainers CISCO Trainers for the project training activities (IT essentials) for Minia Youth Center CompuMe Training costs for Ain Shams CDC Dale Carnegie Training and internships for Ain Shams CDC Diamler Chrysler Internship program offered through Young Arab Leaders (YAL) Egyptian Holding Company Internship opportunities for Cairo CEDO students Etisalat Employment opportunities Ain El Sira Youth Center Exxon Mobil Trainers for Cairo CEDO, Ain Shams CDC, and Ain El Sira Youth Center Hands On Trainers/speakers for Ain Shams CDC HP One high-tech computer lab for Cairo CEDO IBM Trainers/speakers for Ain Shams CDC Leoni AG Hard skills training and internship program at Ain Shams CDC LINKdotNET Job placement, internships and participation in employment fairs at Cairo CEDO and Ain Shams CDC Logic Training for Ain Shams CDC MAC Carpet Trainers/speakers/career coaching for Ain Shams CDC and Cairo CEDO Manaret El Mostakbal Hired CEDO graduate Mansour Company Internship opportunities for Cairo CEDO students Microsoft Training and internship opportunities for Cairo CEDO and Ain Shams CDC; training and Unlimited Potential curricula for all four pilot youth centers MILE Marketing support to corporate sector for Cairo CEDO Mobinil Trainers for skill development programs with Cairo CEDO Modern Motors Egypt Training and internships for Cairo CEDO Mog Company Training costs for Ain Shams CDC Olympic Group Training costs for Ain Shams CDC Pachin Upgrades to the Ain El Sira Youth Center Pepsico Potential support for Cairo CEDO; offers internships opportunities Pepsi Cola Employment/internship opportunities for Ain Shams CDC Premira Employment opportunities for Cairo CEDO Proctor & Gamble Trainers/speakers/career coaching for Ain Shams CDC ProService Internships opportunities for Cairo CEDO Samsung Funding for additional year of EEEA activities at Cairo CEDO, Ain Shams CDC, Ain El Sira Youth Center and Minia Youth Center; offered one CEDO student scholarships for post￾graduate studies Seweidy Electric Meter Training costs for Ain Shams CDC Seweidy Foundation Financial contribution to Cairo CEDO for renovation Smart Village Employment/internship opportunities for Ain Shams CDC Egypt Partner Name Partner Role and Contributions SODIC Provided Cairo CEDO with an advanced soft skills curriculum implemented by Hands-on/YAL; will provide Cairo CEDO and Ain Shams CDC with all needed equipment to establish a computer lab Shell Trainers/speakers for Ain Shams CDC Talaat Moustafa Group Employment opportunities for Cairo CEDO Team Misr Training for Ain Shams CDC Top Notch HR Company Conducted HR manager survey Vodafone Mentorship and support for Cairo CEDO and Ain Shams CDC Wazif Masr Employment opportunities Winning Academy Training (Montage) for Minia Youth Center Zaafrana Factory Employment opportunities sewing trainees at Maadi El Gadeeda Youth Center Government Cairo and Ain Shams Universities Space and personnel costs for hosting of Cairo CEDO and Ain Shams CDC infrastructures The faculties of Mass Communication, Art, Computer Science, Urban Planning at Cairo University MOUs signed to extend project activities to other faculties The faculties of Science and Agriculture at Fayoum University Trainers for the project training activities at Fayoum Youth Center Governorate of Fayoum Provided support to Fayoum Youth Center National Youth Council Approval of decentralized youth center operations by EEEA Youth centers of Ein El Sera (Cairo), El Maadi El Gedeeda (Cairo), El Madina (Minia) and Saha El Shabia (Fayoum) Implementation of EEEA activities at these youth centers Minia Youth Directorate Facilitation of project activities inside the Minia Youth Center Ministries of ICT, Transportation, Finance, Tourism, and Culture Possible job opportunities for Ain Shams CDC graduates Industrial Modernization Council/Industrial Training Council Cash contributions for training at Ain Shams CDC Non-governmental Organizations Aboubakr EL Sedik Trainees for Maadi El Gadeeda Youth Center AIESEC Training provider for Ain Shams CDC Alashanek Ya Balady Ain Shams CDO/Ain El Sira Youth Center implementing partner, training provider, financial contributor American University in Cairo￾School of Continuing Education (AUC/SCE) Training for Cairo CEDO Bashier El Khir Provided children with special needs to participate in SEKEM recycling training at the Maadi El Gadeeda Youth Center AMIDEAST Training provider for Ain El Sira Youth Center and Cairo CEDO BEST Implementing partner for Fayoum and Minia Youth Centers Education For Employment Foundation Training provider for Cairo CEDO; life skills curricula for Ain El Sira Youth Center The Egyptian Association for Biological Planting Training and supervision for SEKEM’s nursery at Maadi El Gadeeda Youth Center Egyptian Scouts Association Assisted in organization of, and provided Scout volunteers for, a youth camp in which participants renovated the Minia Youth Center; provided space and Scouts trainees for training in mushroom cultivation; potential establishment of a selling point for the mushroom on Scouts’ premises Egypt Partner Name Partner Role and Contributions EJB Implementing partner for Ain Shams CDC Injaaz Training provider for Cairo CEDO Institute of International Education (IIE) Training for Cairo CEDO and Ain Shams CDC Near East Foundation TOT training for AYB-SB volunteers Nahdet El Mahrousa EEEA National Secretariat, implementing partner for Cairo CEDO Rotaract clubs Provided services to target groups at Fayoum Youth Center SEKEM Implementing partner for Maadi El Gadeeda Youth Center SEKEM School and Academy Provided trainers for the project training and culture activities (Art activities, scouts, recycling and nursery plant activities) at Maadi El Gadeeda Youth Center Synergos Cash contribution, capacity building and training activities Young Arab Leaders Financial contributions, technical assistance, mentorships, internships, and training curricula for Cairo CEDO and Ain Shams CDC Overseas Development Organizations USAID Egypt Programmatic guidance and collaboration UNDP Possible partnership under UNDP/IBLF Global Company program UNESCO/ICDL Training for Ain Shams CDC Indonesia Partner Name Partner Role and Contributions Corporations BP Board member and resource partner; provided funding for West Java community economic empowerment projects GAP Board member and resource partner; provided funding for Yogyakarta entrepreneurship project GE Board member Nike Resource partner; provided funding for the Nike life skills training Oracle Resource partner for the Junior Achievement project; provided Think.com software Newmont Indonesia Resource partner; provided funding and business volunteers for the Junior Achievement project Rio Tinto Indonesia Board member ACCOR Indonesia Resource partner; provided training on hotel hospitality and management and food processing for Bali ecotourism projects Medion Resource partner for the CREATE project; provided training supplies for the project Vaksindo Satwa Nusantara Resource partner for the CREATE project; provided training supplies for the project Kurnia Poultry Farm Groups Resource partner for the CREATE project; provided training supplies for the project Perdana Putra Chicken Resource partner for the CREATE project; provided training facilities and internship opportunities Charoen Phokpand Resource partner; provided funding for the CREATE project PEAC Bromo Resource partner for East Java projects; provided business consultancies to the newly established business ventures C-59 Resource partner for the JBL project; provided instructors and facilities for t-shirt making training and t-shirt franchise training Telkom – Regional 3 Resource partner for JBL project; provided training facilities for the project TA TV Solo Resource partner for KAKAK tempe chips project; supported the project and marketing of tempe chips Yamaha Provided training instructors, internships and employment opportunities for the Cepat Tepat project Suzuki Provided training instructors, internships and employment opportunities for the Cepat Tepat project Toyota Astra Provided training instructors, internships and employment opportunities for the Cepat Tepat project Honda Provided internships and employment opportunities for the Cepat Tepat project Daihatsu Provided internships and employment opportunities for the Cepat Tepat project Government West Java Provincial and Kabupaten governments Contributed land, equipments and technical assistance for the Cepat Tepat, BSI and KUD projects Local government of Pati Contributed land and meal stipends for the CREATE project Local government of Bandung Provided training supplies and facilities for the JBL project Local Kabupaten governments in Bali Contributed equipment and facilities to support community product development Non-governmental Organizations Indonesia Business Links National Secretariat and Implementing Partner Ford Foundation Board member Kehati Foundation Lead implementing partner; contributed technical assistance and cash leverage Indonesia Partner Name Partner Role and Contributions Wisnu Foundation Implementing partner; contributed staff time and office facilities for Kehati project TIFA Foundation Board member Junior Achievement Implementing partner; provided staff time and office facilities KUD Mina Bahari Implementing partner; provided funding, staff time, office facilities, and factory land Bina Swadaya Implementing partner; provided staff time, office facilities and vehicles Jendela Budaya Lingkungan Implementing partner; provided staff time, and training materials SMK Farming Pati Implementing partner; provided staff time, training venues and facilities Business Dynamics Training provider for the Nike life skills training SEKAR Implementing partner; provided staff time, training venues and facilities PUPUK Surabaya Implementing partner; provided staff time and office facilities ALIT Implementing partner; provided staff time and office facilities YP3IS Implementing partner; provided staff time, training venues and facilities PKBM Cepat Tepat Implementing partner; provided auto mechanics training facilities and instructors PKBM Aditya Implementing partner; provided sewing training facilities and instructors Bina Mitra Provided supervision to the newly created sewing business group under the BP West Java project in Tanjung Sedari Yayasan Kurnia Resource partner for the SEKAR project; provided training equipment and facilities Gallery K’Qta Resource partner for the SEKAR project; provided training equipment and facilities and product promotion support ROMPI Resource partner for the SEKAR project; provided training venues and equipment KAPPIJA-21 Resource partner for the SEKAR project; provided training equipment and facilities and stipends for trainers Overseas Development Organizations USAID/Indonesia Board observer, program contact, strategic guidance and principal funding source International Labor Organization Board member Morocco Partner Name Partner Role and Contributions Corporations Microsoft IT curriculum; financial support; employee volunteers; staff time; software and certifications for computer refurbishment training; computer donations General Electric GE life skills curriculum Munisys Technical assistance in computer refurbishment; internships; computer donations Attijariwafabank Computer donations, cash grant, personnel AFEM Steering Committee member; staff , facilities, volunteers, cash CGEM Steering Committee member Federation PME/PMI Steering Committee member American Chamber of Commerce Steering Committee member Moroccan American Trade and Investment Council Assistance with identifying Steering Committee members Vigeo Assistance in initiatives to raise CSR awareness SOMACA Computer donations, technical assistance, staff time WAFASALAF Equipment, computers, facilities and furnishings Richbond Placement of youth graduates BULL Placement of youth graduates Cisco Systems Computer and other equipment donations Banque Populaire Technical collaboration McDonalds Placement of youth graduates Accor/Ibis Placement of youth graduates Riad Mogador Placement of youth graduates BMCE Bank Placement of youth graduates Nike Financial support Government Ministry of National Education Administrative staff time, workshop director, trainers, translators, training facilities, operating expenses, pilot sites AREF (Academie Régionale pour l’Education et la Formation de Grand Casablanca) Administrative staff time, Workshop director, trainers, translators, training facilities, operating expenses, pilot sites Hassan II University Administrative staff time, computer equipped classrooms, other facilities, trainers Agence Nationale pour la Promotion de l’Emploi et des Compétences (ANAPEC) Use of database to identify beneficiaries for training; reimbursement of training costs Agence Nationale pour la Promotion de la Petite et Moyenne Entreprise (ANPME) Reimbursement of training costs Holding Al Omrane Construction of community center Entraide Nationale Management of community center National Initiative for Human Development (INDH) Funding of management of center and certification of training programs Department of Investment Assistance with Investment Conference, MEEA stand, MEEA Roundtable, press, facilities Non-governmental Organizations Al Jisr Administrative staff time, technical assistance, equipment, training of trainers, facilities, publishing and distributing manual Education for Employment Foundation Administrative staff time, training of trainers, baseline curriculum, facilities, operating expenses Ateliers Sans Frontières Spare parts, training, staff time Morocco Partner Name Partner Role and Contributions Fondation M.J.I.D Administrative staff time, volunteers, facilities Mohammed V Foundation Facilities Centre de Jeunes Dirigeants (CJD) Administrative staff time, technical assistance, trainers, facilities L’Heure Joyeuse Staff time, facilities, training curriculum, contacts with factory employers ENDA Maghreb Staff time, technical assistance Project ALEF Trainers, training curriculum PlaNET Finance Training curriculum Association Essalam Facilities Fondation Auteuil Financial support Near East Foundation Financial support GTZ Technical collaboration Overseas Development Organizations USAID/Morocco Programmatic guidance and collaboration; Steering Committee member Pakistan Partner Name Partner Role and Contributions Corporations DHL Pakistan Steering Committee member Microsoft Providing cash and in-kind assistance / Steering Committee member Engro Chemicals Pakistan Limited Steering Committee member and donor Serena Hotels / Tourism Promotion Services Resource partner providing training venues and other support Chef’s Association of Pakistan Resource partner during training curriculum development CNG Association of Pakistan Donor and resource partner Thardeep Rural Development Program Steering Committee member Government National Vocational and Technical Education Commission (NAVTEC) Providing cash contribution for continuation of training activities Hydrocarbon Development Institute of Pakistan (HDIP) Resource partner during training curriculum development College of Tourism & Hotel Management (COTHM) Resource partner Oil & Gas Regulatory Authority Resource partner during training curriculum development Federation of the Chambers of Commerce Pakistan Potential partner in creating linkages with private sector partners, particularly on job placement issues Pakistan Institute of Tourism & Hotel Management (PITHM) Resource partner Pak Austrian Institute of Tourism & Hotel Management (PAITHOM) Resource partner Non-governmental Organizations Rural Support Programmes Network National Secretariat Punjab Vocational Training Council Steering Committee member; project partner Society for Upgradation of Knowledge, Health, and Infrastructure (SUKHI) Resource partner – mobilizing youth and helping place training participants into internships and jobs Civil Junction Resource partner Aga Khan Foundation Implementing partner ASK Development Implementing partner Sahara Welfare Society Implementing partner SABAWON Implementing partner Institute for Rural Management Training provider for life skills project Pakistan Centre for Philanthropy Steering Committee member Develop Our Valuable Earth (DOVE) Resource partner – mobilizing youth and helping place training participants into internships and jobs Innovation Alliance Training provider for PVTC project Rotary International (Rawalpindi Club) Steering Committee member Ali Institute for Education Technical resource partner for Sahara Arts and Crafts teacher training The Philippines Partner Name Partner Role and Contributions Non Government Organizations ALTERPLAN Implementing partner Asia Pacific Prosperity Co-implementing partner Fisheries Improved for Sustainable Harvest (FISH), Bongao, Tawi-Tawi Educational field visit for the youth Civil Society Network on Education Partnership in developing Entrepreneurship Courses for out-of￾school (OSY) that can be mainstreamed with the Bureau of Alternative Learning Systems Of the Department of Education Davao City Chamber (DCCCI) Implementing partner and project-level Secretariat Habitat for Humanity Philippines (HFHP) – Philippine National Office (PNO) and Habitat Building Resource Center (HBRC) Secured Habitat project sites as venue for OJT component of skills training; Upgraded the quality of Habitat workers; Linked training graduates to employment in Habitat projects; Coordinated skills training with day-to-day project management (in HBRC-managed projects) Habitat for Humanity Chapters, General Santos City (5 chapters) Integrate skills training with day-to-day Habitat project management; opened access to LGUs Kasanyangan Foundation Inc. Assistance in identifying and mobilizing OSY for the training KFI Center for Community Development Foundation Inc. Assistance in identifying and mobilizing OSY for the training Marcellin Foundation Implementing partner Metal Working Association of the Philippines/ Cagayan de Oro Chapter Implementing partner and project-level Alliance Secretariat Mindanao Business Council Provided access to jobs Mindanao Center for Research & Development Coop Assistance with micro-enterprises Philippine Development Assistance Program Implementing partner Ploughshares Implementing partner Southeast Asian Fisheries Development Center (SEAFDEC), Tigbauan, Iloilo Shared technology on seaweed farming Tawi-Tawi Family Life Foundation (TFLF), Bongao, Tawi-Tawi Arranged educational field visits for the youth Private Sector Asia-PRO Cooperative Manpower services and placement of graduates Ateneo de Zamboanga University Part of the implementation team and provided a resource person during training Bloomingdale Store Part of the implementation team, provided a resource person for the training, and provided space for the retail store for OSY products Cagayan de Oro Steel Center, Inc., Tagoloan, Misamis Oriental Provided on-the-job training (OJT) and employment opportunities Edisons Machinery Works, Corp., Cagayan de Oro City Provided OJT and employment opportunities Chevron Philippines Funds for training and materials Davao Region Mango Contractors Association Apprenticeships and access to companies for OJT of youth Durian Industry Council Apprenticeships and access to companies for OJT of youth EEI Construction Corp., Villanueva, Misamis Oriental OJT and employment opportunities Elegant Chemical Alloy Corp., Villanueva, Misamis Oriental OJT and employment opportunities The Philippines Partner Name Partner Role and Contributions Fitweld Limited, Inc., Cagayan de Oro City OJT and employment opportunities Food Processors Association of Davao Apprenticeships, access to companies for OJT of youth Goldtown Industrial Sales Corp., Cagayan de Oro City OJT and employment opportunities Hanjin Heavy Equipment Corp., Villanueva, Misamis Oriental OJT and employment opportunities JENARAs Manpower Services Manpower services and placement of youth graduates Lapanday Agricultural Development Corporation Apprenticeships, access to companies for OJT of youth MENZI Farms OJT of youth Mindanao Fruit Council Apprenticeships, access to companies for OJT of youth Mindanao Oriental Builders, Inc., Villanueva, Misamis Oriental OJT and employment opportunities Orophil Industries, Cagayan de Oro City OJT and employment opportunities Philippine Green Farm Employment and training in the Banana Sector Phividec Industrial Authority Member of Alliance and access to OJT opportunities Private Individuals/Professionals Design of Youth Entrepreneurship Curriculum and training of youth in remote villages Sikulan Seaweeds Farmers’ Association, Tawi-Tawi Mobilization and monitoring of participating youth Sipangkot Seaweeds Farmers’ Association Mobilization and monitoring of participating youth Sitangkai Seaweed LMC Corporation Additional market for young people’s seaweed harvest and a potential source of employment for youth graduates for quality control of seaweeds Sanga-Sanga, Bongao, Tawi-Tawi; Kasangyangan Nursery Seaweed Enterprise (KNSE), Tigtabon Island, Zamboanga City Educational field visit for the youth SITEXPORTS Seaweed market for the youth’s raw dried seaweeds produce Southern Mindanao Mango Industry Development Council Apprenticeships, access to companies for OJT of youth STI School in Zamboanga City Co implementer of the modules, provider of classroom, training for the STI teachers on doing OSY training SUMIFRU Corporation Apprenticeships, access to companies for OJT of youth TADECO Apprenticeships, access to companies for OJT of youth Tinambak Lugus Seaweeds Farmers’ Association Mobilization and monitoring of participating youth Vegetable Industry Council of Mindanao Apprenticeships, access to companies for OJT of youth Vercide Engineering Services, Inc., Tagoloan, Misamis Oriental OJT and employment opportunities VM Paras, Tagoloan, Misamis Oriental OJT and employment opportunities Government: National & Local Government Units (LGU) Bureau of Alternative Education Mentoring and follow up of youth for equivalency for High School City Government of General Santos City Policy support for Marcellin program City Social Services Development Office, Davao City Microfinance loans and access to the city government services Davao City Barangays (six) Community support in recruitment and monitoring of Youth The Philippines Partner Name Partner Role and Contributions Department of Agriculture, Davao City Development of curriculum Department of Agriculture Region 9 Technical assistance in agricultural training Department of Education through their BALS representative for the Municipality, Misamis Oriental Mobilizer of OSY, Assistance in Implementation of Program Department of Education (DepEd), Sitangkai District Functional education curriculum Department of Labor and Employment, Davao City Participation in the Youth Labor Study for the City Government Department of Social Welfare & Comm Dev, National Office Funds for Youth Productivity Service Model Department of Science and Technology Development of curriculum Department of Trade, Davao Technical training and seed capital for livelihood loans ILO/City Government Project, Davao City Youth entrepreneurship training LGU of Davao del Norte, Provincial Department of Agriculture Technical assistance in Hog raising and other agricultural training; provider of facilities for training based on government fees Local Government Units of Misamis Oriental participating in YPS program Development of operating plans for the Youth Productivity Services project Local Government of Davao Norte Technical and market access assistance to OSY enterprise products Mindanao State University - TCTO Technical support for seaweed tissue culture that will help propagate good and productive seaweed seedlings Municipality of Opol, Misamis Oriental Funds, materials and training venue for training of youth under YPS model Municipal Departments of Social Welfare and Development, Zamboanga Sibugay and Zamboanga City Provision for technical assistance; mobilizing out of school youth and co-building enterprises for the youth Municipal Local Government Units of Sitangkai and Sibutu; Barangay Abinsanga (Sipangkot), Tawi-tawi Development of a seaweed industry master plan Local Government of Tagoloan Additional resources and venue for the training of youth Local Government of Villanueva Additional resources and venue for the training of youth Philippine Rice Research Institute Technical Assistance in securing copyright Provincial Department of Labor and Employment Office Technical Assistance in Building Rural Organizations for OSY and provision of funding social enterprises Provincial Government of Misamis Oriental Funds for Youth Productivity Service model and mobilization Region 9 Department of Labor and Employment Technical assistance in building rural organizations for OSY and provision of funding social enterprises Regional Department of Agriculture in Davao Provinces Technical assistance in agricultural training Technical Education and Skills Development Authority (TESDA) in Zamboanga, Davao del Norte, Davao City and Zamboanga Sibugay Doing the OSY training, provider of funds for OSY course TESDA, Misamis Oriental Review and approval of adopted curriculum TESDA & Bureau of Fisheries & Aquatic Resources, Tawi-Tawi Accreditation of youth trainers and recognition of training initiated by PDAP; short listing of the youth graduates for future TESDA training The Philippines Partner Name Partner Role and Contributions TESDA Kor-phil (Korea - Philippines Facility), Davao City Training of youth University of Southern Philippines, Davao City Co-implementing partner WNAS (Wangan National Agricultural School) in Davao City Training of youth Zamboanga City Department of Social welfare and the Department of Agriculture Mobilization of OSY; Assistance in program implementation Zamboanga Sibuguey Provincial Local Government Unit through the offices Governor George Hoffer and the Provincial Cooperative Development Office Provision for technical assistance, funding mobilizing OSY and co-building enterprises for the youth International Organizations Act for Peace Programme and World Food Programme Enhanced access to target beneficiaries in Mindanao; Enhanced access to LGUs with Peace and Development Communities; Resources for additional support and services to beneficiaries Building and Woodworkers Int’l (BWI) and National Union of Building and Construction Workers (NUBCW) Linked training graduates to industry formation and potential benefits of membership in workers organization; Additional resources for training and certification of trainers-assessors; Linkages to industry association International Labor Organization (ILO) Enhanced access to target beneficiaries and LGUs in Guimaras; Resources for training of additional beneficiaries in distressed economic conditions ©2010 International Youth Foundation. All rights reserved. No portion of this report may be reproduced in any form without the written permission of IYF. This project was made possible by the generous support of the American people through the United States Agency for International Development (USAID) under grant agreement no. RAN-A-00-04-00043-00 This report was funded in part under grant agreement no. EPP-A-00-08-00006-00. The International Youth Foundation (IYF) invests in the extraordinary potential of young people. Founded in 1990, IYF builds and maintains a worldwide community of businesses, governments, and civil-society organizations committed to empowering youth to be healthy, productive, and engaged citizens. IYF programs are catalysts of change that help young people obtain a quality education, gain employability skills, make healthy choices, and improve their communities. To learn more, visit www.iyfnet.org ANNEX C: EEA Egypt Evaluation Report CONTENTS Executive Summary Section I: Introduction Section II: EEEA Results Framework Section III: Findings and Analytical Review of Program Achievements Section IV: Challenges, Lessons Learned and Recommendations Conclusion FIGURES Figure 1: List of EEEA Subprojects and Implementing Partners Figure 2: Seed Funds and Leverage Contributions Figure 3: Typology of EEEA Partners and Their Roles Figure 4: Sustainability Status by Subgrant Project Figure 5: Project-level Costs Figure 6: Alliance Building and Program Support Costs Figure 7: Cost per Beneficiary Figure 8: Number of Participants and Completers by Subproject Figure 9: Number of Completers against Targets Figure 10: Percentage of Youth Demonstrating Improved Capabilities by Subproject Figure 11: Youth Satisfaction with the EEEA Training Figure 12: Number and Percentage of Graduates Employed or Self-employed Figure 13: Youth Satisfaction with the Quality of Job Placement Services Figure 14: Percentage of Youth Willing to Pursue Further Education Figure 15: Youth-led Businesses Started by Subproject Figure 16: Youth Satisfaction with the EEEA Entrepreneurship Training Figure 17: Percentage of Surveyed Youth-led Businesses Making Profits Figure 18: Youth Satisfaction with Internships Figure 19: Job Satisfaction Rates ANNEXES Annex A: List of Stakeholders Participated in the Data Collection Process Annex B: Data Collection Process and Instruments Executive Summary The goal of the Egyptian Education and Employment Alliance program (EEEA) was to develop partnerships among the public, private and NGO communities to pilot and expand innovative, new approaches that would address critical education and employment gaps for young Egyptians. Public sector involvement was a key component of the program in order to promote scale and sustainability. Overall, this study, based on interviews and surveys of 230 participants, partners and employers in the EEEA program, together with analysis of evaluation data from each of EEEA’s projects, finds that the alliance approach adopted by EEEA was effective in building partnerships and promoting innovative scalable youth employability programs. Although some challenges are important to continue to address, as described in this report, the program’s two pilot programs – the University Career Development Centers and Youth Center One Stop Shops – created implementing models that show new innovations in the Egyptian context and have the potential to be widely replicable. The university pilots at the Faculties of Engineering at Ain Shams and Cairo universities represented the first ever career centers in the Egyptian public university system. The flexibility of the model and array of career services and solutions offered through the Centers have received strong support from senior levels of the university administration, setting the stage for their models to be replicated to other faculties and public universities. In this respect, MOUs have been signed to replicate the CDC model in at least twelve new faculties as well as a number of other public universities including Alexandria, Tanta and Fayoum Universities. In addition, EEEA developed a strategic partnership with the U.S. based National Career Development Association (NCDA) to provide career facilitation and counseling at EEEA’s career centers which has the potential to create a ripple effect if this partnership can be expanded to reach students at new faculties and universities where the MOUs have been signed. The program’s strategy to partner with formal, public sector institutions to build the foundation for this expansion was the key ingredient to this scalability. At the outset, partnering with the government was perceived to be both a priority and quite challenging. Notwithstanding bureaucratic challenges along the way, the program was successful in penetrating Egypt’s challenging public sector to directly reach youth with needed services. Through its alliance building model, the program built an extensive network of private sector and NGO partners that leveraged almost $3 million in cash and in-kind resources to increase service offerings and reach a larger number of participants. However, while the program was able to leverage significant contributions from the private sector, the relatively short duration of the projects – particularly for the One Stop Shop pilots which were only one year interventions – did not allow enough time to refine program models, improve effectiveness in key areas, and build business models that were robust enough for long term sustainability. Building trust and good relations with the youth center managers was a necessary step to executing on program deliverables, but this was time consuming and left little time for identifying the most successful programmatic interventions, addressing any challenges, and advocating for replication. Evaluation findings also show that the program was effective in improving access to and quality of education and employability outcomes for youth. Both the university career development centers and the One Stop Shop youth center models have evolved from simple job matching programs in 2006 to career development centers at program end. At the program’s closure, the number of youth reached through EEEA activities was 17,6441 – fifty percent above the original target number of 11,750 youth. The university Centers delivered a variety of career development services for students in areas such as career exploration and counseling, soft skills and language training, and student assessment. Through the Youth Center One Stop Shops, the program also targeted out of school youth in the community as well as in-school secondary and technical institute students who were eager to supplement their formal training at school with workplace readiness skills in an afterschool setting. Notwithstanding large participation rates in the program, direct employment through the EEEA program proved to be very challenging. As will be detailed further in this evaluation, low job placement rates were the result of a number of factors including: 1) EEEA programs at the university level were largely reaching students in their first, second, and third years of school (and thus not yet ready to enter the job market) as opposed to only in their final years as originally conceived; 2) grantees were overly optimistic about the number of job placements their new career development center model could provide and set original targets too high; and 3) because projects were piloting new, untested models for youth employment, they 1 With grant support from Samsung Electronics to continue EEEA activities at Cairo and Ain Shams Universities as well as Ain El Sira and Minia youth centers and to expand career development activities to Fayoum University, the number of beneficiaries served by the EEEA framework has now reached to over 20,000 young people. lacked a comprehensive job placement strategy and system to guide the placement process after participants received training. It should be noted, however, that EEEA was relatively more successful at placing students in internships and entrepreneurship opportunities than in formal sector jobs. EEEA has taken steps to improve job placement efforts directly, but strategies in this area still need refinement, particularly in light of the continuing economic crisis which makes job placement even more difficult. There is also a need to further customize EEEA’s current job placement strategy to include outcomes in job preparation for young people that are still in school and not looking for immediate employment or income generating opportunities. In this respect, the valuable services that are provided through the EEEA programs – job counseling, skill building, as well as practical experience building opportunities through internships and volunteer opportunities – merit further attention and support to help young people at various levels along the continuum of training to employment. Given the strong potential for scalability of EEEA initiatives, such a comprehensive strategy would position EEEA more strategically for success in scaling its programs and for responding to future job market and training needs for youth in Egypt. Section I: Introduction Project Evaluation Objectives, Design and Methodology A. Evaluation Objectives This final evaluation report provides an end of project assessment of the Egyptian Education and Employment Alliance (EEEA) – a program of the International Youth Foundation (IYF) carried out in partnership with Egyptian NGO Nahdet El Mahrousa. The report discusses the effectiveness of the Education and Employment Alliance in promoting youth employability, and in particular evaluates the effectiveness, relevance and sustainability of the Alliance model it used in this pursuit. In so doing, the evaluation report also evaluates the program effect on direct beneficiaries, examining to what extent EEEA’s project approaches were successful in enhancing youth employability skills and placing targeted youth in internships, formal employment or entrepreneurship opportunities. The actual evaluation was conducted from January through April 2009 alongside four other assessments conducted by IYF for EEA programs in Indonesia, Morocco, Pakistan and the Philippines. As such, the information in this report not only informs partners and key stakeholders in Egypt about the effectiveness of programmatic activities, but hopes also to directly inform the evaluation of the Global Education and Employment Alliance managed by IYF. In conducting the study, the evaluation aims to respond to three core questions, each corresponding to one of the EEEA Intermediate Results (IRs) set forth the overall EEEA Results Framework set forward early in the program and outlined in Section II of this report: 1) To what extent was the Alliance approach effective in building partnerships, promoting sustainable and scalable youth employability programs? Answers to this question focused particularly on: - Assessment of the overall value and utility of the Alliance approach versus traditional development approaches which depend mainly on contributions from a single donor - The extent and effectiveness of alliance building in raising contributions for EEEA programs - Analysis of partnerships that have formed as a result of the innovative programs technically or financially supported by EEEA - Identification of new partnerships, mechanisms and approaches created as a rippling effect of working with EEEA to support youth employability - Identification of non-target institutions that request to adopt EEEA promoted programs, models and tools as part of their activities - The effectiveness and efficacy of EEEA’s unique management structure of global and national secretariats, an EEEA advisory board, subgrantees and leverage partners to successfully implement and monitor program activities and deliverables - Identification of lessons learned and best practices for future programming efforts 2) How successful was the program in improving access to and quality of education and employability opportunities for youth through EEEA’s designed activities? Focus areas include: - Analysis of nature and extent of program activities and initiatives and their ability to reach out to target groups - Analysis of various target groups (age, gender, socio-economic status, etc) - Analysis of the effectiveness of EEEA training and employability programs in better preparing young people to improve their personal lives, obtain quality employment and/or start entrepreneurial ventures, and become positively engaged and productive in community - Identification of lessons learned and best practices for future programming efforts 3) How successful was the program in improving prospects for employment and entrepreneurship through EEEA’s designed activities? Focus areas here include: - Analysis of the effectiveness of EEEA in placing young people in internships, jobs and entrepreneurship opportunities - Extent of satisfaction among youth and employers with the services received with a highlight on best practices and other activities/approaches could be undertaken to further strengthen program effects B. Evaluation Design and Methodology The evaluation study was designed to be both summative and formative in nature, in order to adequately assess program achievements and weaknesses. The summative evaluation examined target group response to the program activities and services, while the formative evaluation explored potential improvements to best practice approaches as EEEA pilot programs continue beyond initial USAID support through the EEA program and look to scale and replicate at a national level. The report closes with recommendations and lessons learned through the evaluation process. It also includes direct input from stakeholders and their respective analyses of project successes and areas for improvement. In addition, the end of this evaluation provides a series of conclusions regarding practice, process, and methodology. The evaluation of Egypt’s alliance program was overseen by Ms. Shereen Nasef, IYF consultant and Monitoring and Evaluation (M&E) Specialist for the EEEA program. She was assisted in this effort by IYF personnel who have been responsible for overall management of the Egypt EEA program. Ms. Nasef also served as one of four core members on the Global Evaluation team which was formed to plan the evaluation process for the overall global program to ensure consistency of approach and results across the five EEA countries. In conducting the evaluation, 230 individuals from six different stakeholder groups were asked to provide insights into how the project was implemented. These stakeholders included young people participating in the program, implementing partners, the national secretariat, government partners, employers and internship providers, and trainers. A total of 156 youth or roughly one percent of the 14,759 graduates participated in the evaluation. A complete list of stakeholders that participated in the data gathering process is provided in Annex A. The evaluation employed a variety of data collection techniques including a desk review of program reports and other secondary data, place-based focus group discussions, survey questionnaires and semi-structured interviews. In most cases, purposeful sampling was used to collect data. The selection criteria for respondents were primarily based on their availability and accessibility to the field team. Almost all data collection was conducted in Arabic and then translated into English and input into Survey Monkey, EEA’s main evaluation survey tool, with the help of a data entry clerk. C. Evaluation Limitation and Challenges In conducting the evaluation, there were several limitations that could have affected the outcomes of the findings. First, there was insufficient time, resources and staffing capacity to conduct an exhaustive study. It was quite challenging for one evaluator to carry out all the research required given the large size of the Egypt program, coupled with the long distances involved to conduct place based surveys in several governorates. In addition, the evaluator experienced some challenges related to the availability of informants. Because the evaluation was conducted during Egypt’s scholastic examination period, many of the students participating in the program were not responsive to the surveys or were difficult to contact. In general, youth beneficiaries who started their own businesses were easier to reach. Because of these difficulties, the evaluation relied on triangulation of data to verify the results and to ensure that the findings portrayed a sufficiently realistic view of program accomplishments. Additionally, as discussed in detail in the overall global evaluation, this study is inherently limited by the lack of a fully external evaluation of program outcomes. As noted in the global introduction, given both resource constraints and the desire to leverage the broad experience of program participants to fully compare and contrast approaches used in the different countries for governance, program design, leveraging of resources and other elements, a strategic decision was made to utilize participants in the overall program to support this evaluation. In the interest of promoting learning and improvement, the EEA teams have done their best to fully express challenges and problems within programs and the alliance structure, grounded in the reality of their experience in the program; however the study does have this inherent limitation, which is important to disclose fully. Program summary, problem statement and EEEA’s response A. Summary In Egypt, EEA operated under the leadership of Egyptian NGO, Nahdet El Mahrousa through a grant from IYF of USAID funds totaling $742,544. This total grant amount was comprised of $565,500 dedicated to seed funding for EEEA projects with the remainder dedicated to management or “Secretariat” support provided by Nahdet El Mahrousa (NM) to the overall alliance, including the conduct of seed funded activities. In recognition of the enormous employment challenge faced by Egyptian youth, the Egyptian Alliance provided grants and in-kind donations to spur new action-oriented public-private partnerships to carry out what were designed to be innovative education and employment pilot programs for youth. Egypt’s program began in February 2006 and ended on April 30, 2009. B. The Problem Statement Egyptian youth deserve a chance to succeed in today’s economy. Yet many are denied that opportunity because they do not have the skills to compete in the employment market. Egyptian employers have noted consistently that the growth and vitality of their businesses are hampered by the lack of highly skilled graduates. At the same time, young people in Egypt have little access to practical job and entrepreneurship training. The Egyptian public education system in particular is driven by theoretical training and lacks the practical training necessary to better prepare young people for today’s competitive job market. This problem gets deeper in poor urban areas and in governorates outside Cairo and Alexandria where young people lack even the most basic workplace skills, rarely have English and/or computer skills, and have limited access to good educational opportunities that will further education or lead to employment. At the same time, the employment situation in Egypt is changing rapidly. By 2015, government employment is expected to drop rapidly from 43% to 24%. Furthermore, eighty percent of private employment outside of agriculture is expected to be in small and medium enterprises, making entrepreneurial skills a priority. C. EEEA’s Response In an effort to support youth livelihoods in Egypt, the goal of the Egyptian Education and Employment Alliance was to foster innovative, scalable programs that would better prepare unemployable graduates of Egyptian universities and other disadvantaged youth for quality employment. This focus was in direct recognition of the very difficult employment situation faced by the Egyptian economy. To help bridge the gap, EEEA sought to support and develop demand-driven skill building programs to enhance the employability of Egyptian youth through two program interventions – building career development centers in Egyptian public universities and creating “One Stop Shops” for youth development in Egypt’s national youth centers. EEEA programs focused on providing demand driven trainings in the fields of basic life skills, employability skills, information technology, business English and specific technical skills that are requested by the employers. EEEA also developed and supported activities that assisted young Egyptians in developing small and medium enterprises that can serve as engines for economic growth, enhance Egyptian productivity and create new jobs. The Egyptian Alliance was managed by NM as the National Secretariat which supported alliance building and leveraging activities for the program and ensured that EEEA was recognized and supported at the national and governmental levels. NM also identified resources - financial and in kind - that would contribute to the sustainability of the pilot programs and broaden program potential. NM also provided important capacity building and other technical assistance to subgrantees or project implementers. Background on Pilot Projects In close collaboration with IYF, NM organized two solicitation rounds for the award of grants. Through a request for proposals (RFP) solicitation disseminated through NM’s network of partners in 2006, EEA Egypt’s first set of programs created two “Career Development Centers” at the Faculties of Engineering at Cairo and Ain Shams Universities. In a second phase of subgranting in July 2007, the Secretariat awarded three pilot programs for the development and implementation of “One Stop Shops” for employment training and youth engagement in four Egyptian youth centers. EEEA subgrantees are listed in Figure 1 below: Figure 1: List of EEEA Subprojects and Implementing Partners NGO Pilot Program Nahdet El Mahrousa (as an implementing partner) Career Development Office at Cairo University2 Egyptian Junior Business Association (EJB) Career Development Center at Ain Shams University Alashanek Ya Balady for Sustainable Development (AYB) One Stop Shop at Ain El Sira Youth Center, Misr Qadeema, Cairo Business Enterprise Support Tools (BEST) One Stop Shop Youth Center – Minia 2 Now called the Career and Entrepreneurship Development Office (CEDO). NGO Pilot Program One Stop Shop Youth Center – Fayoum SEKEM One Stop Shop Youth Center – Maadi El Gadeeda, Cairo NM and EJB’s pilot projects were located on campus at the Faculties of Engineering at Cairo and Ain Shams Universities respectively and targeted university students. The other three subgrantees worked directly with youth centers targeting marginalized youth in Cairo, Minia and Fayoum governorates. By April 30, 2009 when EEEA officially came to an end, 17,644 young people had participated in the program, with 14,759 completing program activities. Although EEEA program participation rates significantly exceeded targets, significantly fewer program beneficiaries were placed in internships and jobs at the conclusion of their programs than originally targeted. At the same time, the number of young people starting their own businesses exceeded targets. The evaluation points to a number of reasons for why job placement rates were low which are explained below. Of particular note is the fairly significant number of young people who were trained who were not yet eligible for full time employment given their continued schooling. Of those who did find jobs, internships, or started their own businesses, they expressed satisfaction with their new positions and the income earned by their new businesses. A summary description of EEEA’s five pilot programs is provided below. University Career Development Centers Cairo University Career Entrepreneurship and Development Office (CEDO): As an implementing partner, Nahdet El Mahrousa launched a new career development office at Cairo University’s Faculty of Engineering for new graduates and private sector employees. The office was structured to operate on two complementary dimensions: career counseling and job matching. To complement academic work with practical and more in-depth experience, the Center linked engineering and other students to training courses, mentorship opportunities, workshops, career counseling services, student activities, internships and business competitions – all with the aim of producing students and graduates, fit for employment. To ensure students gained skills, most desired by employers, the CEDO built an important network of corporate partnerships and training providers, which included the British Engineering Institute, Diamler, Vodafone and Proctor & Gamble, AMIDEAST and Injaz, among others. The overall program targeted the provision of services to 5,450 youth with 1,565 job placements. IYF provided NM with a $190,380 grant and the CEDO received substantial cash and in-kind leverage from other partners. NM recently received a grant from Samsung (via IYF) to continue some program activities at Cairo University for another year. Ain Shams Career Development Center (CDC): In January 2007, a second grant of $130,300 in seed funds was awarded to the Egyptian Junior Business Association (EJB) to develop a campus-based Career Development Center at the Faculty of Engineering in Ain Shams University. The program also secured considerable leverage from EJB’s extensive network of business partners. Similar to Cairo University’s CEDO, the Ain Shams program provided direct access to students for job training and placement services, offering training courses, mentorship opportunities, internships, career development opportunities and increased job placements for students. Business partners and training providers at EJB’s Center included INJAZ, Proctor and Gamble, Vodafone, ExxonMobil and Mobinil, among others. It targeted 1,935 students for participation in the program and 242 job placements. EJB recently received a grant from Samsung (via IYF) to continue some program activities at Ain Shams University for another year. “One Stop Shop” Youth Centers: EEEA’s second set of programs piloted “One Stop Shops” for youth development in Egyptian National Youth Centers. Working in close partnership with the National Youth Council, EEEA piloted new approaches to better utilize the over 4,500 National Youth Centers located across Egypt to provide life skills and employability training, health awareness and cultural and artistic enrichment activities targeted at disadvantaged youth not traditionally served with extra-curricular skills development activities. Three pilot projects were implemented by Alashanek Ya Balady (AYB), Business Enterprise Support Tools (BEST), and Sekem Development Foundation to recreate Youth Centers as “One Stop Shops” that provide local youth with critical employment and life skills, job placement services and social and cultural enrichment activities. Programs lasted approximately thirteen months. A total of 4,365 youth were targeted for participation in youth center activities during the pilot phase. A brief description of partner programs is provided below: • Ain El Sira “Mind Profit” One Stop Shop Youth Center: Through a grant of $83,000 Egyptian NGO Alashanek Ya Balady Association for Sustainable Development (AYB-SD) expanded and improved the quality of education and employment programs it had been providing to youth at the Ain El Sira Youth Center in Cairo. Building on six years of experience working with the center, AYB-SD provided much needed organizational capacity to upgrade the Center’s existing service offerings through the establishment of a new program called “Mind Profit”– an effort that provided demand-driven soft skills and technical training, employment services and volunteer training to disadvantaged youth and young women, ages 12-30. The program targeted 2,565 youth for participation in the youth center activities and 325 to be placed in jobs. AYB has worked closely with Center management and staff to help them to continue to provide EEA-supported training programs beyond the grant. AYB recently received a grant from Samsung (via IYF) to continue some program activities at the Ain El Sira youth center through December 2009. • Fayoum and Minia One Stop Shop Youth Centers: Business Enterprise Support Tools (BEST) NGO started as an initiative under the cooperation of UNDP/Egypt and The Social Funds for Development (SFD) with the purpose of supporting micro and medium enterprises. BEST has five offices in different governorates. EEEA Program operated in two of BEST’s offices in Minia governorate (Upper Egypt) and Fayoum governorate. With a budget of $83,350 BEST aimed to fundamentally transform the Fayoum and Minia youth centers into productive, sustainable centers through a comprehensive package of life skills and vocational training, health and physical development activities, cultural development activities, volunteerism and community development initiatives. Activities were designed for young people ages 18-30, who are largely unemployed and with limited income. Students of secondary schools, technical institutes and universities were also targeted, as well as rural women, particularly those who are low income and reside at home. The program also focused on improving management and operational capabilities of Center staffs, in hopes of enabling the Centers to continue these programs after the EEA intervention concludes. The program targeted 1,000 youth for participation in Center activities with 390 youth obtaining jobs or starting their own businesses. BEST recently received a grant from Samsung (via IYF) to continue some program activities at Minia youth center through December 2009. • Maadi El Gadeeda Youth Center One Stop Shop: In partnership with the Egyptian Biodynamic Association, grantee Sekem Development Association designed and implemented a series of innovative activities that transformed the Maadi El Gadeeda Youth Center into a “One Stop Shop” that managed effective employment training programs and more fully engaged youth in productive, civic activities. The program specifically targeted disadvantaged youth from surrounding areas, including special needs and handicapped youth. As the program was being crafted, it was clear the Center faced many challenges, particularly in designing and implementing programs aimed at youth development in the areas of vocational and technical skills. A major focus of activities therefore was the establishment of a green house and plant nursery for raising seedlings and training youth on cultivating and selling organic horticulture products. The program targeted 800 youth for participation in Center activities, and 100 to obtain jobs in the formal sector or start their own businesses. Sekem received a grant of $78,470 from EEEA to administer the One Stop Shop at the Maadi El Gadeeda youth center. Section II: EEEA Results Framework Egyptian Education & Employment Alliance Goal: To identify and give young men and women opportunities to build a better future Intermediate Result 1: Alliance building and leveraging resources Intermediate Result 2: Improved Access to and Relevance of Education and Employability Training Intermediate Result 3: Improved Prospects for Employment and Successful Entrepreneurship Indicator 1.1: In kind and cash resources leveraged for the project through private sector, government, civil society and other partners Indicator 1.2:* The effectiveness of the EEA Alliance Approach in building partnerships, and promoting sustainable and scalable youth employability programs Indicator 1.3: Number of private sector, government and civil society partnerships formed through the project to jointly offer employment and business development training activities in targeted areas Indicator 1.4:* New partnerships, mechanisms and approaches created as rippling effect Indicator 2.1: Number of young men and women participating in EEA projects Indicator 2.2: Number and percentage of young men and women completing EEA supported programs Indicator 2.3: Number of teachers, trainers, facilitators and counselors trained under EEA supported programs Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in EEA supported programs Indicator 2.5: Level of satisfaction of young trainees with the quality and relevance of the training, counseling and services received through EEA programs (for both employability and entrepreneurship development) Indicator 2.6: Number and percentage of youth who completed EEA supported projects who report that they have referred at least another youth to the project Indicator 3.1: Number and percentage of trainees who get a job within six months of completing program. Indicator 3.2: Number and percentage of trainees who continue education, obtain an internship or engage in further professional training within six months of completing program. Indicator 3.3: Number and percentage of EEA trainees starting their own businesses within six months of completing training Indicator 3.4:* Number and percentage of youth with satisfactory internships or apprenticeships Indicator 3.5:* Number and percentage of placed youth who declare having a satisfying and quality job *Indicators 1.2, 1.4, 3.4, and 3.5 represent indicators that were added for EEA’s global evaluation study and are not part of the Egypt’s country level Results Framework. Several of these indicators were also tracked at tracked the subgrantee level. Section III: Findings and Analytical Review of Program Achievements Intermediate Result I: Alliance Building and Leveraging Resources Summary of Intermediate Result One Findings Research conducted during the evaluation study shows that overall, EEEA’s Alliance building approach was an effective model for Egypt’s National Secretariat and its implementing partners to build partnerships and promote sustainable and scalable youth employability programs. The approach taken by the Egyptian program appears to be successful in several ways. First, EEEA engaged Egypt’s public sector as strategic partners in the program model laying the foundation for scaling successful pilot projects at the national level. Partnerships with Egypt’s two largest public universities, Cairo and Ain Shams, allowed for the creation of the first ever Career Development Centers in Egypt’s public universities, setting the stage for the replication of their model to Egypt’s other public universities. At the writing of this report, EEEA partners had signed MOUs with twelve new faculties at Cairo University and three new public universities including Alexandria University, Tanta University and Fayoum Universities for the replication of the program, assuming sufficient resources are available. Similarly, a strategic partnership with the National Youth Council enabled the delivery of extracurricular employability training and social and cultural enrichment activities directly on the premises of four national youth centers. In recent years, the National Youth Council has made significant efforts to reform Egypt’s 4,500 plus youth centers, many of which are in disarray and are sorely in need for new youth friendly services that better respond to the needs of local youth. EEEA’s “One Stop Shop” solution directly complimented the NYC’s desire transform these centers into more meaningful community gathering points. Although less tested and less mature than the university level career centers, the One Stop Shop model has been replicated at one new youth center (Abou El Souood youth center in Misr El Qadeema) and all EEEA One Stop Shop implementers said they had been approached by other youth center managers who asked how the One Stop Shop services could be replicated to their youth centers. In addition to strategic alliances with Egypt’s public sector institutions, the EEEA program also greatly benefited from robust partnerships with corporations, training companies, and non-governmental organizations (NGOs) which contributed leverage that allowed for improved training content, increased number and range of courses offered, and a much greater number of beneficiaries to be reached than originally targeted. Despite strong success in this area, one area of challenge noted by the study is the high level of focused time and effort required to effectively build partnerships during program design and implementation. This area of challenge was pointed out by survey participants, particularly implementing partners. Views in this area are counterbalanced by a feeling that there was significant value in developing such partnerships, although support and resources for such efforts were considered insufficient overall. Further analysis of EEEA’s alliance building approach and leverage results as well as the effectiveness of EEEA’s management structures, including the National Secretariat and Advisory Board is provided below. Indicator 1.1: In kind and cash resources leveraged for EEEA through private sector, government, civil society and other partners EEEA’s demand driven employment models attracted strong leverage support from individuals, institutions, and organizations in public, private and civil society sectors. Government partners donated valuable and highly sought after office and training spaces directly on university campuses and at youth centers. The donation of strategically located, onsite space created high visibility for EEEA’s programs helping to attract cash and in-kind resources from a range of civil society and private sector partners, including education foundations, training providers, private businesses and multinational companies. The strategic location of the pilot programs, which attracted large cohorts of young people, fresh graduates and first time job seekers, also created a “win-win” situation as both EEEA and leverage partners aimed to reach the same target audience. Education foundations, NGOs and private companies provided cash and in kind resources including cash contributions, equipment, newspaper and media coverage, training curricula, training providers, office and training space, consulting services and more. With an initial investment of $565,500 in seed money for the implementation of five pilot projects, EEEA raised a total $2,985,246 in leveraged contributions. This represents a 5:1 leverage ratio, far exceeding the 1:1 leverage ratio required by USAID for EEA programs. Figure 2 outlines the dollar amount of contributions (cash and in-kind) the Secretariat and each of the five subgrantees were able to leverage: Figure 2: Seed Funds and Leverage Contributions Implementing Partner/Project Seed Funds Leverage Total Budget Leverage Ratio NM/Cairo University Career and Entrepreneurship Development Office $190,380 $1,109,7483 $1,300,128 5.83 EJB/Ain Shams University Career Development Center $130,300 $1,260,923 $1,391,223 9.68 AYB/Mind Profit - Egyptian Youth Center One Stop Shop $83,000 $350,581 $433,581 4.22 SEKEM/ Maadi El Gadeeda Youth Center One Stop Shop $78,470 $105,885 $184,355 1.35 BEST/Fayoum and Minia Youth Center One Stop Shops $83,350 $158,109 $241,459 1.90 Total $565,500 $2,985,246 $3,550,746 5.28 Indicator 1.2: The number of private sector, government and civil society partnerships formed The Egyptian Alliance created approximately eighty public and private sector partnerships at the Secretariat and subgrantee level. These partnerships enabled the diversification and expansion of program offerings, guided the localization of training to meet market needs, and helped EEEA to reach more program beneficiaries than targeted. Because EEEA’s pilot programs received only seed funding from IYF, the alliance building model of continuously building a network of leverage partners was an essential step towards successful implementation of the program and building its long term sustainability. The figure below illustrates the most significant partnerships achieved at the Secretariat and subgrantee level: Figure 3: Typology of EEEA Partners and Their Roles Partner Name Partner Role Corporate 1 Abercrombie and Kent Cash contribution to Ain El Sira Youth Center 2 The Academy of Engineering Sciences Mobile maintenance training for Minia Youth Center; also employment opportunities for youth from the Center 3 Berlitz Training for Ain Shams CDC 4 BP Trainers/speakers/career coaching for Ain Shams CDC and Cairo CEDO 3 This figure includes $569,507 in leverage generated to support the National Secretariat (NM) in its role to provide cross￾cutting assistance and support to all EEA projects in Egypt. Partner Name Partner Role 5 Cameron Petroleum Company Internships opportunities for Cairo CEDO students 6 Center for International Private Enterprise Entrepreneurship program curriculum for Cairo CEDO; TOT training 7 CISCO Trainers for the project training activities (IT essentials) for Minia Youth Center 8 CompuMe Training costs for Aim Shams CDC 9 Dale Carnegie Training and internships for Ain Shams CDC 10 Diamler Chrysler Internship program offered through Young Arab Leaders (YAL) 11 Egyptian Holding Company Internship opportunities for Cairo CEDO students 12 Etisalat Employment opportunities Ain El Sira Youth Center 13 Exxon Mobil Trainers for Cairo CEDO, Ain Shams CDC, and Ain El Sira Youth Center 14 Hands On Trainers/speakers for Ain Shams CDC 15 HP One high-tech computer lab for Cairo CEDO 16 IBM Trainers/speakers for Ain Shams CDC 17 Leoni AG Likely hard skills training and internship program at Ain Shams CDC 18 LINKdotNET Job placement, internships and participation in employment fairs at Cairo CEDO and Ain Shams CDC 19 Logic Training for Ain Shams CDC 20 MAC Carpet Trainers/speakers/career coaching for Ain Shams CDC and Cairo CEDO 21 Manaret El Mostakbal Hired CEDO graduate 22 Mansour Company Internship opportunities for Cairo CEDO students 23 Microsoft Training and internship opportunities for Cairo CEDO and Ain Shams CDC; training and Unlimited Potential curricula for all four pilot youth centers 24 MILE Marketing support to corporate sector for Cairo CEDO 25 Mobinil Trainers for skill development programs with Cairo CEDO 26 Modern Motors Egypt Training and internships for Cairo CEDO 27 Mog Company Training costs for Ain Shams CDC 28 Olympic Group Training costs for Ain Shams CDC 29 Pachin Upgrades to the Ain El Sira Youth Center 30 Pepsico Potential support for Cairo CEDO; offers internships Partner Name Partner Role opportunities 31 Pepsi Cola Employment/internship opportunities for Ain Shams CDC 32 Premira Employment opportunities for Cairo CEDO 33 Proctor & Gamble Trainers/speakers/career coaching for Ain Shams CDC 34 ProService Internships opportunities for Cairo CEDO 35 Samsung Funding for additional year of EEEA activities at Cairo CEDO, Ain Shams CDC, Ain El Sira Youth Center and Minia Youth Center; offered one CEDO student scholarships for post￾graduate studies 36 Seweidy Electric Meter Training costs for Ain Shams CDC 37 Seweidy Foundation Financial contribution to Cairo CEDO for renovation 38 Smart Village Employment/internship opportunities for Ain Shams CDC 39 SODIC Provided Cairo CEDO with an advanced soft skills curriculum implemented by Hands-on/YAL; will provide Cairo CEDO and Ain Shams CDC with all needed equipment to establish a computer lab 40 Shell Trainers/speakers for Ain Shams CDC 41 Talaat Moustafa Group Employment opportunities for Cairo CEDO 42 Team Misr Training for Ain Shams CDC 43 Top Notch HR Company Conducted HR manager survey 44 Vodafone Mentorship, and support for Cairo CEDO and Ain Shams CDC 45 Wazif Masr Employment opportunities 46 Winning Academy Training (Montage) for Minia Youth Center 47 Zaafrana Factory Employment opportunities sewing trainees for Maadi El Gadeeda Youth Center Government 48 Cairo and Ain Shams Universities Space and personnel costs for hosting of Cairo CEDO and Ain Shams CDC infrastructures 49 The faculties of Mass Communication, Art, Computer Science, Urban Planning at Cairo University MOUs signed extend project activities to other faculties 50 The faculties of Science and Agriculture at Fayoum University Trainers for the project training activities at Fayoum Youth Center 51 Governorate of Fayoum Provided support to Fayoum Youth Center 52 National Youth Council Approval of decentralized youth center operations by EEEA Partner Name Partner Role 53 Youth centers of Ein El Sera (Cairo), El Maadi El Gedeeda (Cairo), El Madina (Minia) and Saha El Shabia (Fayoum) Implementation of EEEA activities at these youth centers 54 Minia Youth Directorate Facilitation of project activities inside the Minia Youth Center 55 Ministries of ICT, Transportation, Finance, Tourism, and Culture Possible job opportunities for Ain Shams CDC graduates 56 Industrial Modernization Council/Industrial Training Council Cash contributions for training at Ain Shams CDC Non-governmental organizations 57 Aboubakr EL Sedik Trainees for Maadi El Gadeeda Youth Center 58 AIESEC Training provider for Ain Shams CDC 59 Alashanek Ya Balady Ain Shams CDO/Ain El Sira Youth Center implementing partner, training provider, financial contributor 60 American University in Cairo-School of Continuing Education (AUC/SCE) Training for Cairo CEDO 61 Bashier El Khir Provided children with special needs to participate in SEKEM recycling training at the Maadi El Gadeeda Youth Center 62 AMIDEAST Training provider for Ain El Sira Youth Center and Cairo CEDO 63 BEST Implementing partner for Fayoum and Minia Youth Centers 64 Education For Employment Foundation Training provider for Cairo CEDO; life skills curricula for Ain El Sira Youth Center 65 The Egyptian Association for Biological Planting Training and supervision for SEKEM’s nursery at Maadi El Gadeeda Youth Center 66 Egyptian Scouts Association Assisted in organization of, and provided Scout volunteers for, a youth camp in which participants renovated the Minia Youth Center; provided space and Scouts trainees for training in mushroom cultivation; potential establishment of a selling point for the mushroom on Scouts’ premises. 67 El Mahrousa Balady Provided a number of trainees for SEKEM training in Maadi El Gadeeda Youth Center 68 EJB Implementing partner for Ain Shams CDC 69 Injaaz Training provider for Cairo CEDO 70 Institute of International Education (IIE) Training for Cairo CEDO and Ain Shams CDC 71 Near East Foundation TOT training for AYB-SB volunteers 72 Nahdet El Mahrousa EEEA National Secretariat, implementing partner for Cairo CEDO 73 Rotaract clubs Provided services to target groups at Fayoum Youth Center Partner Name Partner Role 74 SEKEM Implementing partner for Maadi El Gadeeda Youth Center 75 SEKEM School and Academy Provided trainers for the project training and culture activities (Art activities, scouts, recycling and nursery plant activities) at Maadi El Gadeeda Youth Center 76 Synergos Cash contribution, capacity building and training activities 77 Young Arab Leaders Financial contributions, technical assistance, mentorships, internships, and training curricula for Cairo CEDO and Ain Shams CDC Overseas Development Organizations 78 USAID Washington and Egypt Funding, programmatic guidance and collaboration 79 UNDP Possible partnership under UNDP/IBLF Global Company program 80 UNESCO/ICDL Training for Ain Shams CDC Indicator 1.3: The effectiveness of EEEA alliance approach in building partnerships, and promoting sustainable and scalable youth employability programs Overall Effectiveness The alliance building approach was effective in bringing together diverse partners from the public, private and NGO sectors to support EEEA’s programs. Without this, the program would not have been able to reach as many youth as it did, particularly in the case of the university career centers where fifty percent more than the number of students targeted benefitted from the services offered as a result of leverage in the form of cash contributions, additional training providers and curricula, donated equipment, volunteers and other in-kind resources. While the Alliance building approach was new for NM and most of the subgrantees, the implementing partners felt the alliance approach strengthened their programs and built unity across the EEEA program. EEEA’s Secretariat NM and the subgrantees all reported gaining valuable experience and know-how building public private partnerships. One subgrantee reported transferring the knowledge gained from building partners and leverage for EEEA to another youth-focused program. Equally as important, the strategic partnership with Egypt’s formal sector institutions (i.e. Egyptian public universities and the National Youth Council) created a natural platform for scaling and replicating the program in public universities and youth centers across Egypt. The success of EEEA partnership model is evidenced by the fact that all implementing partners have been asked to expand their programming to new locations. While the Alliance building model of having multiple partners created larger effect than a single partner intervention, it also created some challenges in efficiently managing activities and contributions of the program’s multiple leverage partners. Several subgrantees interviewed for the evaluation said that the time involved with creating new partnerships and meeting IYF leverage requirements at times took focus away from the delivery of core implementation activities. Building partnerships with public sector institutions was especially time-consuming. First, a level of trust had to be cultivated with public sector officials unfamiliar or uninterested in working with NGOs. Once trust was developed, the program experienced governmental and bureaucratic delays in the setting up of MOUs required to begin implementation at Cairo and Ain Shams Universities and the National Youth Council. In the case of the One Stop Shop pilot initiatives – which were launched in a second round of funding after the launch of the university career development centers – it took approximately six months for the signing of an MOU between the NM Secretariat and the National Youth Council. This delay meant that grants to EEEA’s youth center implementers could only be given for the duration of one year which was a time frame too short to ensure adequate self sustainability of program activities and to maximize the full potential of the pilot intervention. Other challenges were related to the youth centers’ inexperience working with NGOs and on donor funded development projects. Youth centers were not generally accustomed to tailoring programming to donor wishes and were also not used to justifying and reporting on program activities to outside organizations. Finally, although subgrantees expressed overall satisfaction with the technical assistance and quality of communication with the Secretariat, implementing partners said that they would have preferred more site visits and capacity-building workshops, as well as exchange visits between different project sites. All told, however, the alliance approach was effective in that it 1) enabled EEEA to serve more beneficiaries with varied and high quality curricula than a single donor program would have been able to, and 2) created a framework and elements of self-sustainability and replication, particularly through public sector interface, that have outlived the life of the EEEA project. Effectiveness of the Secretariat Nahdet el Mahrousa, an Egyptian NGO founded in 2003 to promote Egypt’s development through engaging Egyptian youth as active, empowered citizens in the public arena, served as the EEEA program secretariat. As an NGO led by young Egyptian business leaders with a strong network of business and development contacts, Nahdet El Mahrousa was assessed by IYF as well positioned to use its contacts with the business sector and youth to serve in the role of the Secretariat. In addition, NM’s working culture had a strong entrepreneurial ethos which made it a natural broker connecting business and leverage opportunities to EEEA’s partners. When it came to the time consuming and often challenging process of building partnerships with EEEA’s public sector partners, NM’s reputation as a strong youth￾led NGO, combined with guidance and intervention from EEEA’s influential Advisory Board members (discussed below), appears to have been essential in developing successful partnership arrangements. For example, it would not have been possible for EEEA’s One Stop Shop implementing partners to sign individual MOUs with the management of the youth centers at Ain El Sira, Maadi, Minia and Fayoum without a strategic MOU between NM and the National Youth Council signed in November 2008 which demonstrated NYC’s support and backing for the One Stop Shop model. Once EEEA had brokered overall partnerships with public sector universities and youth centers, EEEA was then able to bring new partners from civil society and business that helped strengthen projects and support long term efforts to promote sustainability. The evaluation shows that NM’s wide network of organizations and individuals working in the field of youth development was of great benefit to implementing partners and the Secretariat played an important role in introducing subgrantees to potential training partners, volunteers, as well as qualified staff to hire. Some important partnerships that the Secretariat successfully brokered to support EEEA’s pilot interventions include: • El Sewedy for a substantial grant to fully renovate the CEDO office at Cairo University • SODIC for an advanced soft skills curriculum • British Petroleum for a cash donation to support training activities • The Center for International Private Enterprises (CIPE) for an entrepreneurship curriculum and Training of Trainers (ToT) training • UNIDO/HP for a high tech computer labs and cash grant • Samsung (through the International Youth Foundation) for a large grant to sustain EEEA activities at Cairo University, Ain Shams University, Ain El Sira and Minia Youth Centers and to expand the CDC model to Fayoum University In addition to supporting the creation of partnerships under the EEEA framework, the Secretariat played an important role in building the capacity of implementing partners to implement USAID funded projects. NM, with IYF support, conducted several workshops for subgrantees in the areas of monitoring and evaluation, leverage and financial reporting and good governance. Two planned workshops on program report writing and job placement strategies did not end up taking place which was unfortunate given that these were both subjects which partners could have benefited from. Workshops were typically limited in space and as such, they were only attended by the project director or focal point of the project. Several implementing partners said that it would have been beneficial to also include participation of other project staff members who were directly involved in the day to day management of program activities in the field. Effectiveness of the Advisory Board One of the first steps in creating the EEEA program was the formation of an Advisory Board of excellent stature and reputation within Egypt on issues of educational reform and employability needs. Core board members included: • Amr Salama, former Minister of Higher Education and Scientific Research (EEEA Advisory Board Co-Chair) • Iman El Kaffas, Associate Dean and Executive Director of Diversity Scholarships and the Students Leadership Development Program at AUC (EEEA Advisory Board Co-Chair) • Alaa Hashim, MAC • Helmy Abouleish, Director of Industrial Modernization Center • Sahar El Sallab, Vice Chairman, Commercial International Bank At the outset of the program, it was envisioned that EEEA’s Advisory Board would convene on a quarterly basis. In reality, it proved challenging to convene all members together particularly given the demanding schedules of such prominent members of Egypt’s education reform landscape. Adapting to this reality, the Secretariat instead met with board members on an individual basis to provide updates and seek guidance. This approach appears to have been quite effective in maintaining the active engagement of at least several individual board members. Board members opened up relationships with public institutions, negotiated commitments from Cairo University, helped design and launch the program and gained a formal agreement with the National Youth Council. In particular, certain board members with previous experience in government and strong reputations across Egypt for engaging in constructive and successful efforts at reform helped directly negotiate partnership commitments with Cairo University as well as the National Youth Council. This “door opening” and subsequent support in troubleshooting and addressing challenges in relationships proved critical to the ultimate integration of EEEA models into the challenging Egyptian public sector environment. Effectiveness of the Alliance Approach from the Subgrantees Perspective The evaluation study found that subgrantees were generally pleased with the alliance approach and rated it reasonably highly (as a three or four on a scale of one to five). Subgrantees credited the alliance approach with creating opportunities to cooperate with partners—namely public institutions—that they would not have had the opportunity to partner with otherwise. Subgrantees also felt that the alliance approach strengthened their programs by helping them to leverage additional resources and share best￾practices and lessons learned among fellow EEEA subgrantees. At the same time, subgrantees found the alliance approach challenging at times. In focus group interviews, subgrantees indicated that it was sometimes difficult to work with public institutions because they were unaccustomed to partnering with NGOs. Subgrantees also mentioned that they needed to set aside significant time in the beginning of the programs to build trust and credibility with their public sector partners before program activities could commence and that the time involved with building new partnership and raising leverage sometimes distracted their attention from focusing on program activities and outcomes. Recommendations for how to better balance time between partnership and leverage building and program activities can be found in the recommendation section. Finally, although subgrantees were in general very pleased with the level and quality of communication between the Secretariat and themselves, they wished there had been more site visits, capacity building workshops, and exchange visits among sub grantees. In addition, they wished there had been more opportunities for subgrantees to convene regularly as a group to discuss challenges and exchange learnings, particularly for the One Stop Shop youth center implementers. Effectiveness of the Alliance Approach in Promoting Program Sustainability Sustainability of alliance programs can be viewed from multiple angles. At the highest level, EEEA could envision the creation of a national public-private alliance of stakeholders to nurture and expand current efforts and create the framework for new programs to be generated. This sustainability strategy would also by necessity require continued support of the National Secretariat, through some form of donor contribution or revenue generation strategy. Such a long term goal for the program was not set explicitly and was likely unrealistic given the fairly small investment of resources to the project and the enormous landscape of actors – including government, civil society and private companies – that have a stake in broader efforts to support more effective youth employability programming. Nonetheless, in this area, the EEEA program appears to have made significant progress, gathering a prominent group of advisors that have first supported the integration of EEEA models into the public sector and are now advocating for their broad replication throughout public systems. Moreover, should additional resources come available, this platform is ready for long term expansion and, over time, government ownership of successful models. In this respect, on Advisory Committee member, former Minister of Higher Education Dr. Amr Salama, has advocated for new policies that make mandatory the creation and support of career development frameworks in all public universities in Egypt. It is particularly interesting to note that EEEA has already secured MOUs with twelve different university faculties and three additional universities to replicate the program should resources become available. As such, a framework for national policy dialogue to meet in the middle with on the ground tested models and interest is ready to be activated. On a different level of sustainability, the alliance approach is becoming increasingly project-focused, seeking avenues to continue support for project activities that have proven successful. In this respect, because the alliance based approach always emphasized the search for new partners and leverage opportunities, EEEA had some success mobilizing grants from new and existing partners. For example, a $340,000 grant from Samsung enabled EEEA to continue to support the Secretariat and carry out some training activities at the Career Centers at Cairo and Ain Shams Universities and at the Ain El Sira youth center, albeit at a more limited level. This leverage also enabled EEEA to replicate the career development center model to a third public university - Fayoum University. Additionally, a number of the subgrantees undertook activities to generate revenue as a means of promoting self sustainability over time. For example, at the university level, CEDO and CDC organized employment fairs in which participating companies were required to pay a sponsorship fee in order to participate. This leverage was used to cover the costs involved in organizing the fair and to fund a limited number of each center’s training activities. The centers also introduced fee based training for many of their courses, although the fees charged were very small and typically covered only immediate expenses related to the costs of delivering the course such as course material supplies. The CDC at Ain Shams University also began charging several companies for the successful job placement of CDC’s network of graduates, however to date this has only been done on a limited scale. The Centers’ management felt that in the start-up phase of the Career Centers it was particularly important to engage as many businesses as possible in support of their activities and that it was important to first earn the trust and respect of companies before charging them placement service fees. At the youth center level, a number of the subgrantees charged a nominal fee to students for services and activities. The funds collected were then used to help support the costs of new activities for the beneficiaries. Another means by which the youth centers helped support themselves was by selling goods produced under the various vocational track components. For example, the leather goods produced by youth trained at AYB’s Ain El Sira youth center are sold to cover the costs of production, pay those who have worked to produce them and to buy new material for the production of new goods. In another example, Sekem established a greenhouse and a plant nursery for decorative plants that will be sold at the Maadi El Gadeeda Youth Center as an income-generating activity. Despite such efforts, it is clear that EEEA’s pilot programs were too short in duration to develop successful mechanisms for long-term sustainability. Even subgrantees at the university level, which had higher capacities for fundraising because of their association with reputable faculties at Cairo and Ain Shams Universities, needed more time to generate funds from the activities and develop effective, sustainable business models. Both subgrantees working with the One Stop Shop youth centers and youth center management themselves required further capacity-building training in resources mobility and fundraising. An additional consideration must also be paid to the fact that such public sector programs are rarely able to self-perpetuate through revenue generation, and will often require some form of public subsidy. Nonetheless, through concerted efforts built up over time, EEEA projects can create more robust revenue streams that reduce the level of such public subsidy. Figure 4 summarizes the sustainability status of each subgrant project: Figure 4: Sustainability Status by Subgrant Project EEA Projects Project Summary Partners Sustainability Status Ain Shams University Career Development Center To create a Career Development Office at Ain Shams University to provide 1,050 students access to job training, placement, and mentorship activities and to place two hundred graduating students in jobs Egyptian Junior Business Association, Ain Shams University; EJB Members, Industrial Modernization Council, Microsoft • Twelve MOUs signed with faculties and three universities to replicate the career development model pending available resources, • A Samsung grant enabled EEEA to continue to support the Secretariat and carry out some training activities at the Career Centers at Cairo and Ain Shams Universities as well as to replicate the career development center model at Fayoum University. • Some limited revenue generating mechanisms have been put in place. Cairo University Career Development Office To create a Career Development Office at Cairo University providing 2,600 students access to job training, placement, mentorship and placing four hundred in jobs Nahdet El Mahrousa, Seweidy Family Foundation, Cairo University, TopNotch HR Company, BP, I￾Source, Training Providers Mind Profit - Egyptian Youth Center One Stop Shop To develop a self sustainable one stop shop for youth employability at Ain el Sira Youth Center in Cairo that provides training programs for 1,500 youth and links 540 to jobs Alashanek Ya Balady, Vodafone, German University in Cairo, American University in Cairo and Mirqah • Samsung grant enabled EEEA to continue to support some “One Stop Shop” Activities at AYB Mind Profit Youth Center and BEST Minia Youth Center. • Some revenue generating mechanisms have been put in Maadi El place. Gadeeda Youth Center One Stop Shop To develop a self sustainable one stop shop for youth employability at the Maadi Gadeeda Youth Center, providing employability training in environmentally friendly businesses and placing three hundred youth in jobs Sekem Development Foundation, Egyptian Biodynamic Association Fayoum and Minia Youth Center One Stop Shops To develop a self sustainable one stop shop for youth employability at two youth centers in Minia and Fayoum that improves vocational training and life skills programs for 1,300 youth and links four hundred of them to jobs BEST Foundation, Catholic Relief Organization, Axon International Studies Network, Ministry of ICT, Cisco, Shell Cost Effectiveness: Under EEA’s global framework, which included a series of both country and global level activities, the overall cost effectiveness of EEEA program has been assessed at two levels. The first level of such costs are directed related to country program subgrants, which include both seed grants administered by USAID and leverage contributions contributed by other partners to support EEA country programs. The second level of costs includes Alliance building and program support costs, which include costs related to the functioning of the EEEA National Secretariat – NM, and local consultants in Egypt, as well as IYF direct and indirect expenditures attributable to the EEEA program. Each of these cost categories are discussed in more detail below. Subgrant Project-level Costs: As noted above, subgrant project-level costs include seed funds and leverage contributions invested toward individual projects which typically covered curriculum design or improvement, training, job placement, enterprise development costs, and project specific costs of NGO subgrantees. EEEA provided complementary two to five-day job readiness training together with job matching services at public universities. Through Youth Centers, EEEA provided longer training programs (sometimes one month long) which consisted of providing vocational, technical and life skills training together with job placement services and enterprise development support. The amount of seed funding support from USAID toward individual projects is a useful metric to assess cost effectiveness given that such funds are directly focused on implementation of projects, are directly leveraged by investments from other parties, and are best positioned for increased efficiency when additional beneficiaries are added to successful pilot programs. In Egypt, the USAID support of approximately $565,500 in seed funds that were coupled with leverage investments of $2,415,739 reached a total of 17,644 beneficiaries. If all project-level costs are taken in account, this brings the cost to $169 per beneficiary, of which $32 was borne directly by USAID and $137 was shouldered by the alliance partners as presented in Figure 5 below. Figure 5: Project-level Costs Subgrant Project Name Number of Beneficiaries USAID Funds Leverage Total Project Budget Cost Per Beneficiary Cairo University CDC 2,917 $ 190,380 $ 540,241 $ 730,621 $ 250 Ain Shams CDC 9,962 $ 130,300 $ 1,260,923 $ 1,391,223 $ 140 Mind Profit One Stop Shop 1,276 $ 83,000 $ 350,581 $ 433,581 $ 340 SEKEM One Stop Shop 1,921 $ 78,470 $ 105,885 $ 184,355 $ 96 BEST One Stop Shop 1,568 $ 83,350 $ 158,109 $ 241,459 $ 154 TOTAL 17,644 $ 565,500 $ 2,415,739 $ 2,981,239 $ 169 Alliance Building and Program Support Costs: A more expansive examination of costs by necessity also includes those alliance building and program support costs related to the EEEA program. These costs supported significant, collective efforts by IYF, NM as National Secretariat, and program consultants to: manage the design and implementation of alliance based interventions; provide international technical expertise on successful employability approaches; assist in leveraging of resources; provide capacity building support to implementing partners in skills training, job matching and post training support to entrepreneurs; provide financial oversight, compliance, and direct monitoring and evaluation support. Upon examination of IYF accounting records, these costs taken together with indirect costs attributable to the EEEA program, totaled $846,7564. This USAID-borne cost adds approximately $48 on a per beneficiary basis as presented in Figure 6. Figure 6: Alliance Building and Program Support Costs Subgrant Name Number of Beneficiaries Alliance Building and Program Support Costs Cost Per Beneficiary Aim Shams CDC 9,962 $ 169,351 $ 17 Cairo University CDC 2,917 $ 169,351 $ 58 Mind Profit One Stop Shop 1,276 $ 169,351 $ 133 SEKEM One Stop Shop 1,921 $ 169,351 $ 88 BEST One Stop Shop 1,568 $ 169,351 $ 108 TOTAL 17,644 $ 846,756 $ 48 4 In order to ensure comparability of data across the EEA program, this figure is exclusive of leverage raised to support NM in its role as EEEA Secretariat. Taken together, these overall costs of both seed funds and Alliance building program support costs total approximately $217 per beneficiary. Of the cost of $217 per beneficiary, $80 was borne by USAID and $137 was contributed by alliance partners as calculated in Figure 7 below. Figure 7: Cost per Beneficiary Cost per beneficiary USAID Alliance Contributions Total Cost Per Beneficiary Project-level $ 32 $ 137 $ 169 Alliance building and program support $ 48 $ - $ 48 Total $ 80 $ 137 $ 217 Finally, and perhaps most significantly, a full assessment of cost efficiency would also require a more thorough assessment of the implication of alliance building activities on long term sustainability and scalability of proven practice programs. For example, should the new efforts to substantially expand the EEEA model in Egypt be successful, the cost effectiveness of initial EEA investments could be dramatically expanded. While the longitudinal study required for this type of evaluation goes beyond the scope and resources available for this study, such factors should be addressed more directly in similar employability projects in the future. Indicator 1.4: New partnerships, mechanisms and approaches created as rippling effects EEEA’s alliance based approach - where leverage opportunities are continually sought after, public sector integration and policy issues are at the forefront, and advisory committee members bring strong networks – carries with it a wide set of expansion possibilities. As mentioned earlier in this report, at the university level, MOUs have been signed with twelve additional university faculties and three new public universities to bring the career development model to their campuses pending available funding. The One Stop Shop model is primed for expansion given the successes of the initial pilot projects and the partnership with the National Youth Council which oversees a network of 4,500 youth centers across Egypt. Consider, for example, the ripple effect of the success of the Ain El Sira Youth Center, located in Cairo’s Misr El Qadeema district and managed by NGO Alashanek Ya Baladay (AYB). At the Ain El Sira Youth Center, out-of-school young women from the neighborhood receive vocational training in the production of high-quality handicrafts. AYB sells their handicrafts at cultural exhibitions, hotel chains and clothing stores. Program participants can choose to work on a full time basis and receive a regular salary, work part-time and be paid on a per piece system, or alternatively receive a loan through AYB’s partner network to start their own business. The overall proceeds from the sales of the handicrafts are reinvested into the Center to help sustain other non-revenue-generating programs. Profit margins from the sales have been robust enough that AYB has been able to expand some of its training activities to Abou El Souood Youth Center. If revenue continues to be strong, AYB could also expand the One Stop Shop services to other four youth centers in the Misr El Qadeema district which have expressed interest in such activities. Beyond the actual replication of the career development and One Stop Shop models, EEEA developed an important strategic partnership with the US-based National Career Development Association (NCDA) which, pending funding availability has the potential to create a positive ripple effect in the Egyptian public university system. Through the partnership with NCDA, a number of staff members from the career centers received certified training to become career development facilitators, setting the stage for a first generation of internationally certified career counselors in Egypt’s tertiary education system. Four of these trainers have now achieved Global Career Development Facilitator (GCDF) certification. Since the close of EEEA, one of these certified career facilitators continued his training and has recently been certified as an “instructor” which means he can now train other Egyptians to be career facilitators. Career facilitation and counseling is a novel idea in Egypt and one that did not exist in public universities before EEEA’s career centers introduced it among its activities. There is a tremendous need for career counseling services because most university students do not have a broad understanding of the various career tracks available to them and end up selecting their career as a default – either because their high grades necessitate that they study medicine or engineering or because their parents push them into a particular field of study. Career facilitation and counseling gives students the opportunity to think carefully about the kinds of career options available to them and how to pursue a track that they are both most interested in and is relevant to the local job market. EEEA’s partnership with NCDA was highly strategic and has the potential to create a ripple effect if this partnership can be expanded to reach more students at more public universities in Egypt with needed services. Intermediate Result II: Improved Access to and Relevance of Employability Training Summary of Intermediate Result Two Findings By April 30, 2009 when EEEA officially came to an end, 17,644 young people had participated in the program, with 14,759 completing program activities. Most of these youth were students in their first, second or third year of university or secondary school who were able to benefit from skills development trainings which were originally conceived for final year students or recent graduates. As such, while EEA’s job placement targets were developed assuming target beneficiaries would be entering the job market, this factor reduced the program’s focus on job placement in favor of a greater focus on job-readiness related training for young people moving into the job market in the coming years. While moving away from this original focus, the flexibility of the program, with diverse training options for underserved and disadvantaged young people at both the university and youth center levels, met student needs and served more students than originally targeted. EEEA’s alliance based approach, leveraged training space, training curricula and training providers and improved both the access and relevance of EEEA’s education and employment training. Students and employers said they found the program to be effective in providing quality and relevant employability skills training to underserved Egyptian youth. Introduction At both the university and youth center level, training topics included soft skills, IT, technical vocational subjects, and entrepreneurship. Although all EEEA programs contained a mix of technical and soft skills, each subgrantee had a considerable amount of freedom and flexibility to tailor their program activities to meet the specific needs of the target groups they were serving. At the university level, the Career and Entrepreneurship Development Center (CEDO) and the Career Development Office (CDO) were housed in Faculties of Engineering on the campuses of Cairo and Ain Shams Universities respectively. Technical training largely targeted engineering topics; the centers also provided soft skills and work place readiness training in a variety of topics including: effective communication, team work, decision-making, leadership, success in the workplace, CV writing, interviewing techniques, and effective job-hunting. As the university programs matured, entrepreneurship components as well as group and individual career exploration and counseling sessions were added. Generally speaking, compared to the students served through the CEDO and CDC, participants at the youth center levels are less educated and from lower socio-economic backgrounds. Activities at the youth center level included soft skills training at a more introductory level, language, IT and entrepreneurship training as well as vocational training in such areas as handicrafts, sewing, mobile maintenance, mushroom cultivation, and fishing, among others. Indicator 2.1: Number of young men and women participating in EEEA projects An aggregated 17,644 young people participated in EEEA supported programs, exceeding the original target of 11,750.5 The evaluation found that program flexibility was one of the factors that enabled the subgrantees to reach many beneficiaries. Although all programs delivered soft skills training and technical training aligned to market needs, programs at both the university and youth center level were flexible and offered courses targeted to different age and education levels, and were varied in duration and level of difficulty. This flexibility enabled the projects to meet the needs of a diverse group of beneficiaries including university students, students with only a middle school-level education, illiterate youth, very poor youth, young widows, and divorcees. The evaluation also found that “word of mouth” played a major role in promoting the programs and recruiting new student trainees with 55% of surveyed youth stating they heard about the program activities through a friend or a family member who had participated in the program. Furthermore, it became clear in discussions with youth in focus groups that they trusted other students’ opinions more than they might other “more official” sources of information. 5 During the course of the pilot programs, EEEA implementing partners tallied “service counts” rather than participant or head counts because many youth, at both the youth center and university level, enrolled in multiple courses. While it is not feasible to determine actual head counts in this study, we have examined records related to new activities being undertaken by the project partners with Samsung support in which the projects have tallied both service counts and individual head counts. Under these new programs, we have found that individual headcounts are roughly equal to 60% of “service counts.” Therefore, given 29,407 service hits were recorded in EEEA, it was estimated that 17,644 individuals were served. Gender Distribution of Program Participants: As noted in the chart below, the total number of males participating in EEEA training in the last five quarters (when complete gender statistics were maintained) exceeded the number of female participants. A major reason for this disparity is that the university career development centers – where the vast majority of EEEA beneficiaries were trained – were housed at the Faculties of Engineering at Cairo and Ain Shams Universities which traditionally enroll many more male students than female students. Nevertheless, it should be noted that women accessed the youth center programs at higher rates than the university level programs. In the case of AYB’s One Stop Shop Youth Center at Ain El Sira Youth Center, 66% of program participants were women and only 34% were men, reflecting the deliberate strategy of this youth center to target women and girls from the community by offering vocational programming – such as sewing and craft-making – which tended to attract more women than men. Figure 8: Number of Participants and Completers by Subproject Project Total number of youth participated Total number of youth participated in last five quarters Total number of male participants in last five quarters Total number of female participants in last five quarters NM/Cairo University Career and Entrepreneurship Development Office 2,917 2,337 1,600 (68%) 737 (32%) EJB/Ain Shams University Career Development Center 9,962 9,275 6,931 (75%) 2,344 (25%) AYB/Mind Profit – Ain El Sira Youth Center One Stop Shop 1,276 1,276 428 (34%) 848 (66%) BEST /Fayoum and Minia Youth Center One Stop Shop 1,921 1,921 928 (59%) 993 (41%) SEKEM/ Maadi El Gadeeda Youth Center One Stop Shop 1,568 1,568 1,122 (58%) 446 (42%) Total 17,644 16,377 11,009 (67%) 5,368 (33%) Indicator 2.2: Number and percentage of young men and women completing EEEA supported projects At the conclusion of the EEEA program, 14,759 men and women completed training, far exceeding the original target of 8,252 as detailed in Figure 9 below. In fact, trainees completing the program constituted 84% of those that commenced programs. Figure 9: Number of Completers against Targets Project Total Target Total Actual Percentage of Target NM/Cairo University Career and Entrepreneurship Development Office 4,085 2,495 61% EJB/Ain Shams University Career Development Center 894 8,096 906% AYB/Mind Profit – Ain al Sira Youth Center One Stop Shop 1,923 690 36% BEST /Fayoum and Minia Youth Center One Stop Shop 750 1,558 208% SEKEM/ Maadi El Gadeeda Youth Center One Stop Shop 600 1920 320% Total 8,252 14,759 179% Indicator 2.3: Number of teachers, trainers, facilitators and counselors trained under EEEA supported programs Subgrantees recruited program trainers through requests for proposals, utilizing their own networks, or from training company alliance partners who supplied their trainers as in-kind contributions. Trainers were regularly evaluated on the bases of trainee feedback as well as the observations of subgrantee partners. The project supported a total of 111 trainers against a target of 102 (109% of the target). Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in EEEA supported programs Generally, students’ impressions of the training were very positive. In fact, 82% of youth surveyed indicated that the training they received under the EEEA program gave them the necessary knowledge and skills to find a job and/or start their own business. Percentages of young people demonstrating improved capabilities through the program as measured through post-training assessment tests and self￾evaluations, exceeded or met targets in all cases. Figure 10: Percentage of Youth Demonstrating Improved Capabilities by Subproject Project Target Actual NM/Cairo University Career and Entrepreneurship Development Office 80% 85% EJB/Ain Shams University Career Development Center 80% 82% AYB/Mind Profit – Ain al Sira Youth Center One Stop Shop 75% 87% BEST /Fayoum and Minia Youth Center One Stop Shop 75% 75% SEKEM/ Maadi El Gadeeda Youth Center One Stop Shop 75% 87% Indicator 2.5: Level of satisfaction of young trainees with the quality and relevance of the training, counseling and services received through EEEA programs Trainees were asked to rate the quality and relevance of the training as very poor, poor, fair, good or excellent. Survey results indicate that program graduates were generally happy with the quality and relevance of training provided. • Ninety-one percent of program beneficiaries said that the overall content of the training program was either good or excellent. Only two percent ranked the training poor. • In terms of relevance of training, 34% of trainees felt that the training relevance was excellent, 43% as good, and 15% as fair. • Eighty-seven percent of program beneficiaries rated the career counseling and mentorship they received in the program as excellent or good. • Ninety-three percent of trainees felt that the technical knowledge of trainers/counselors/facilitators was excellent or good. • Finally 82% of respondents indicated that they gained the necessary knowledge and skills that they needed to find a job or start their own businesses. With respect to youth satisfaction with EEEA entrepreneurship training, of eight entrepreneurs surveyed, all felt that the EEEA program gave them the entrepreneurial knowledge and skills to start a program and all of those surveyed felt “prepared and empowered” as a young entrepreneur following the program. Indicator 2.6: Number and percentage of youth who completed EEEA training and report that they have referred to at least another youth to the project The assessment survey asked whether graduates have referred other youth to EEEA projects and received responses from 132 youth. A total of 122 (92% of respondents) indicated that they had referred other youth to the program. This data reinforce the conclusion above that youth were satisfied with the quality and relevance of these training they received. Indicator 2.7: Level of employers’ satisfaction with the soft skills and/or technical skills of EEEA trained youth Only four employers were able to be contacted under this survey6, but all employers surveyed indicated that they were pleased with the EEEA graduates whom they employed. In this respect, all employers surveyed indicated that the overall performance of EEEA graduates was either good or excellent. Employers rated the technical skills of employees as either good or excellent and the soft skills of their new employees as good. Finally, all surveyed said that they found EEA graduates to be much better than employees of similar age range and educational background, and they would definitely considering offering jobs, internships or apprenticeships to EEA graduates in the future. 6 A total of 156 student participants (representing approximately one percent of the 14,759 who completed training) were interviewed for this evaluation. The evaluator, however, had an especially difficult time eliciting responses from employers of program participants and only four employers agreed to be interviewed for the evaluation. A number of employers indicated that they were too busy to respond to questions about an individual employee and others indicated that company policy prohibited them from revealing information about their employees’ performances. The evaluator suggests that in future iterations of the program the right to evaluate program participants’ performance in their job placements be included in MOUs that employers sign when they hire program graduates. Intermediate Result III: Improved Prospects for Employment and Successful Entrepreneurship Summary of Intermediate Result Three Findings Although EEEA program participation rates exceeded targets, significantly fewer program beneficiaries were placed in internships and jobs at the conclusion of their programs than originally targeted. At the same time, the number of young people starting their own businesses exceeded targets. The evaluation points to a number of reasons for why fewer internship and job placements were made than originally targeted which are explained below. Of particular note is the fairly significant number of young people who were trained who were not yet eligible for full time employment given their continued schooling. Of those who did find jobs, internships, or started their own businesses, they expressed satisfaction with their new positions and the income earned by their new businesses. Among 14,759 youth who completed the EEEA training, 2,750 youth (1,708 from universities7 and 1,042 from youth centers8) were estimated to be ready for employment or self-employment at the conclusion of the program. Of these 2,750 graduates ready to work, 32% obtained jobs or set up small businesses. Broken down by program-type: • Seventeen percent of 1,708 university-level trainees obtained jobs (234) or set up small businesses (50). • Fifty-eight percent of 1,042 youth center trainees obtained jobs (250) or established small businesses (352). Figure 12: Number and Percentage of EEA Graduates Employed or Self-employed Number of graduates ready for employment or self-employment Number of employed youth Number of self￾employed youth Total number of graduates who are employed or self-employed Total percentage of graduates who are employed or self-employed Universities 1708 234 50 284 17% Youth Centers 1042 250 352 602 58% Total 2750 484 402 886 32% 7 At the university level, only those trainees who were in the final year of university (as opposed to the significant number of students in their first, second and third years of study who ended up accessing EEEA training) were considered eligible to work immediately post-program. Only 14% (1,134) of those completing training at Ain Shams University’s Career Development Center and 23% (574) at Cairo University’s Career and Entrepreneurship Development Office (CEDO) were in their final year of university. 8 The EEEA evaluator has indicated, based on sample of databases recording enrollment at the youth center level (not all youth center databases provided complete data on educational level/current position of trainees) that 15% of those who enrolled in training were eligible to work post-training and the remaining 85% were estimated to be either in their first, second or third year of secondary or tertiary school or were homemakers unable to hold a permanent job due to their role as primary family caretakers. Because this estimate was based only on a sample of the youth enrolled, the evaluation team decided to increase the estimation of eligible to work to 25% (or 1,042 individuals) in order to be err on the side of conservativism in assessing employment outcomes. Indicator 3.1: Number and percentage of trainees who get a job within six months of completing program As noted above, an aggregated 2,750 graduates from universities and youth centers were determined to be ready to be employed or self-employed and 484 were placed in jobs (compared to a target of 2,497) within six months of completing the training. While overall EEEA program participation rates were high, the number of job placements was considerably lower than originally targeted. It appears that this is due to five conditions as elaborated below: 1) As mentioned above, EEEA programs at the university level were largely reaching students in the first, second, and third years of school (as opposed to only their final years as originally conceived) and those students still in school were not yet eligible for job placements. Similarly, a large percentage of individuals trained at the youth center lever were also not in a position to obtain jobs after the training because they were still in school or the primary care takers of their children. 2) During the latter half of the program, employers expressed less interest in expanding and taking on new employees due to the downturn in the global economic climate. Participation of employers at job fairs that were hosted by the university programs, for example, dropped dramatically as the full breadth of the economic crisis became apparent. 3) It appears the original job placement targets set were unrealistically high. The high targets were primarily the result of three phenomena: • grantees did not realize how much interest their programs would garner from those individuals still in school (and thus not eligible to work immediately post-training) • grantees did not conduct significant and rigorous labor market assessments at the beginning of the program and as such they were overly optimistic about the number of job placements their new career development center model could provide • because projects were piloting new, untested training models for youth employment, they may have over-estimated the possibilities of placement through short term training activities. In fact, in EEEA’s career development center model, the substantial bulk of beneficiaries were participants in training courses lasting anywhere from two-days to a week, and not the kind of months long comprehensive training programs (integrating life skills, targeted technical and vocational skills, and internship/job placements) that other EEA programs engaged in and were able to attain much higher job placement rates with. 4) It is difficult to draw a direct connection between valuable career guidance support provided to students and job placement. Future training may need to be more intensive to achieve better job placement rates, or different outcomes should be determined for such short term training and career guidance. 5) Finally, also because projects were piloting new, untested models, they lacked a comprehensive job placement strategy. Rather, program partners – particularly at the university level – engaged in short term training followed by “job matching” rather than intensive months long targeted training geared to specific job openings or sectors experiencing economic growth. In this area, it is clear that EEEA partners need to focus more attention to refine existing job placement strategies, and perhaps, as mentioned above, the length and relevance of training in order to build more robust outcomes in this important area. Given the economic downturn, such emphasis and refinement will become even more important for the future. One important and positive indicator in this area, illustrated in the two charts below, is that those students who were in position to obtain jobs post graduation (from the program and from university) did so relatively quickly and were happy with the job placement services they received. Ninety-two percent of the surveyed youth found jobs in less than three months, and the rest within four to six months of completing the training. At the university level, most job placements were in the engineering field; a number of other placements were in marketing or office administration. At the youth center level, job placements were in the following fields: marketing, office administration, sales, factory work, quality control, and hotels and tourism. Figure 13: Youth Satisfaction with the Quality of Job Placement Services Indicator 3.2: Number and percentage of trainees who continue education, obtain an internship or engage in further professional training within six months of completing the program The number of EEEA trainees who continued their education, obtained an internship or engaged in further professional training (outside of EEEA) at the conclusion of their programs was also much lower than expected: an aggregated 161 youth as opposed to 1,292 targeted. The evaluation finds that internship placement rates were lower than anticipated for a number of reasons that became clear in focus group discussions: • Faculty of Engineering students accessing EEEA training at CEDO and CDC indicated in focus groups that they were “stressed out” at the end of the school year and wanted to spend their summers “relaxing” instead of obtaining experience on the job. • Youth were not always ready to intern at a company far away from where they live. Many Egyptian factories are located in 6th of October and 10th of Ramadan areas which are far away from central Cairo and from most residential areas. • Many companies were not interested in paying stipends to interns thus discouraging trainees from taking internships. • Many Egyptian companies are not yet aware of the importance of hiring interns; subgrantees had to devote additional time and awareness to encourage them to advocate for the importance of internships. The low numbers of internships notwithstanding, the evaluation did find that the vast majority of youth surveyed expressed an interest in continuing their educations or were already registered to pursue further education. Figure 14: Percentage of Youth Willing to Pursue Further Education Indicator 3.3: Number and percentage of EEEA trainees starting their own businesses within six months of completing training A total of 402 participants started businesses within six months of training, well exceeding the target set of 153. Business start-ups aligned with local market needs and included mushroom cultivation, rabbit breeding, the production of leather goods, and advertising and tourism among other enterprises. It is likely that the relatively high rates of new business start-ups by participants (as opposed to the lower job and internship placement rates) can be attributed to the extra effort the EEEA team committed to publicizing both entrepreneurship and soft skills training in university and youth center programs: courses were heavily advertised in special recruiting seminars where speakers emphasized the importance of such training for young people in order to compete in a tight job market. That said, however, participants at the youth center level were found to be more active in starting their own businesses than university students (with the one exception of the Sekem Youth Center in Maadi el Gadeeda). The evaluation team finds that especially high entrepreneurship rates at the youth center level can likely be attributed to the careful thought the EEEA team placed into entrepreneurship training – linking the entrepreneurship track with complementary soft skills courses and a specific vocational track, thus exposing beneficiaries to a targeted and integrated skill set they would need to start their own businesses and hopefully succeed at them. Figure 15: Youth-led Businesses Started by Subproject Project Target Actual NM/Cairo University Career and Entrepreneurship Development Office - 45 EJB/Ain Shams University Career Development Center - 25 AYB/Mind Profit – Ain al Sira Youth Center One Stop Shop 28 99 BEST /Fayoum and Minia Youth Center One Stop Shop 75 224 SEKEM/ Maadi El Gadeeda Youth Center One Stop Shop 50 9 Total 153 402 Securing funds for participants who wanted to start their own businesses was a main challenge, despite the fact that subgrantees at the youth center already provide loans to youth through other programs. Evaluation focus groups found that youth were reluctant to take loans because they feared that they would not be able to pay them back. Figure 16: Youth Satisfaction with the EEEA Entrepreneurship Training Figure 17: Percentage of Surveyed Youth-led Businesses Making Profits Indicator 3.4: Number and percentage of EEEA-trained youth with satisfactory internships or apprenticeships Although few students engaged in internships, those who did, were generally very pleased with their experiences as outlined in the chart below. For example, 67% of participants who received internships evaluated the technical skills that they gained through the internship as excellent, 22% as good, and 11% as poor. Additionally, participants rated the relevance of training assignments and coaching received during the internship as mostly excellent or good. Finally, in focus groups, youth indicated that they appreciated having the opportunity to work in real world business environments and reported gaining practical knowledge related to organizational behavior, work ethics, time management and project management. Figure 18: Youth Satisfaction with Internships Indicator 3.5: Number and percentage of placed youth who declare having a satisfying and quality job Some EEEA participants were employed at the time of their training or had been employed in the past. The training helped them to obtain better offers and/or provided them with upgraded skills to improve their on the job performance. While only a limited number of survey participants fit into these criteria, according to the survey, participants who started in a new job after enrolling in the program rated their new jobs as better of much better than their old jobs. Figure 19: Job Satisfaction Rates Section IV: Challenges, Lessons Learned and Recommendations The EEEA program reached a wide range of young people and had many accomplishments. Notwithstanding these successes, NM, subgrantees, youth participants, and other program stakeholders have identified a number of lessons learned as well as recommendations for improvements. It is important to note that some of the recommendations suggested below were not actually possible because of the pilot nature of EEEA’s interventions and the limited grant amounts for each project which restricted both the scope of activities and staff capacity to implement these activities. Nonetheless, these recommendations are important considerations in planning future interventions that build on EEEA’s models and activities. This is particularly the case for EEEA which has proven quite successful in penetrating public sector programs to enable for direct and significant access to young people for training, as well as in leveraging new resources to expand training efforts. The recommendations presented below are applicable to both the university and youth center programs unless specified. Part I: Lessons Learned and Recommendations Corresponding to Intermediate Result One: Alliance Building Approach and Leveraging Resources Building sustainable alliance programs requires more time and funding in order to be most effective Both NM and subgrantees needed more time to activate the partnerships built through the project and to raise leverage. The youth center One Stop Shops programs were particularly short in duration (at one year each) and there was barely sufficient time to build partnerships and even less time to activate them. Each subgrantee indicated that they needed a minimum of five years to build strong alliance partnerships. Similarly, EEEA’s pilot programs were too short in duration to develop successful mechanisms for long￾term sustainability. Even subgrantees at the university level, which had higher capacities for fundraising because of their association with reputable faculties at Cairo and Ain Shams Universities, needed more time to generate funds from the activities and self sustain their business models. Both subgrantees working with the One Stop Shop youth centers and youth center management themselves required further capacity-building training in resources mobility and fundraising. At the same time, it should also be noted that such public sector programs are rarely able to self-perpetuate through revenue generation, and will often require some form of public subsidy. Nonetheless, through concerted efforts built up over time, EEEA projects can create more robust revenue streams that reduce the level of such public subsidy. In addition to implementing and sustaining the program activities inside the youth centers, more funds should have been dedicated to upgrading the youth center’s infrastructure and equipment in order to engage youth center management and build their trust and in order to facilitate the continuation of One Stop Shop activities for the long-term. Furthermore, youth center managers were frequently more keen to sustain their sports programs rather than the other One Stop Shop services which often targeted the wider community; therefore more effort should be directed to training the youth center management and governing boards about the value of non sports-focused activities. In light of these different factors, future donor-funded youth center activities should think seriously about including incentive structures for youth center management, and should frame programs to help guide youth center development toward the areas most essential for long term development of Egyptian youth. In conclusion, necessary factors for long-term sustainability include: sufficient time to develop and implement programming as well as to cultivate partnerships/alliances; capacity-building workshops at the subgrantee level in fundraising/leveraging resources; funds dedicated for the upgrading of youth center infrastructure to enable the continuation of activities for the long-term; and a greater degree of attention paid to developing income-generating plans/activities in program design and from the outset of program implementation, including plans for reinvestment of revenues and scaling up of those activities that are more successful at generating revenue. Building relations with government partners takes time but yields positive results in the long term EEEA’s National Secretariat and implementing partners spent considerable effort to build relations and earn the trust of its government partners, particularly in the early phase of program implementation. While time consuming and sometimes challenging, these efforts paid off. Having the support and buy-in of government partners was not only essential for the successful delivery of pilot activities but has meant that EEEA is well positioned for significant replication of its models in the future. It should be noted that three years after the conception of the career development centers, there are efforts by Egypt’s education reform leaders, including EEEA Advisory Board Co-Chair Dr. Amr Salama, to lobby for the creation and passing of a government bill that would require all Egyptian public universities to offer career services for its students and recent graduates. Success in this area could be enormously important for Egyptian youth, and the models developed through EEEA, refined to ensure maximum potential, could be dramatically scaled with appropriate levels of support. Early program-funded small-scale renovation of government premises helps earn the trust and engagement of government partners Early program-funded small-scale renovations in the form of freshly painted classrooms, upgraded sports equipment and improved landscaping went a long way to earning the initial trust and support of government partners for the EEEA program, particularly at the youth center level. It is recommended that these types of small scale infrastructure investments be part of the “implementation package” in future expansion or replication of EEEA’s pilot models. Encouraging subgrantees to collect leverage is an innovative strategy and was very successful under the EEEA program Securing leverage to broaden program effectiveness was highly successful for the Egyptian Alliance. At the same time, implementing partners indicated that the level of effort expended in securing new leverage partners and documenting/calculating leverage amounts could be time consuming and draw important resources away from program administration. It is recommended that future programs with significant leverage requirements plan for sufficient levels of support and resources to devote to the leverage effort. If additional support and resources are not readily available, more realistic leverage outcomes - ones that partners are able to easily reach with the resources available to them - should be established at the outset. Sign longer MOUs with focal alliances (three to five years at least) with clear definitions of responsibilities and expectations Subgrantees at the university level should sign long-term agreements with the universities to guarantee the on-campus location of their centers for the longer-term. Longer term agreements will avoid the time, hassle and effort involved in renegotiating MOUs after one or two years. To overcome any potential conflicts between the program and government bodies, a program should adopt certain procedures to ensure the success of the relationship. For example, programs should plan ahead for expected delays and wait times, document all correspondences and decisions, and keep government partners involved and updated. Carefully consider the selection of university faculty and youth center partners for program implementation The reason for selecting the Faculties of Engineering – as opposed to other university faculties – to pilot the university career development centers was largely because the early champions of the CDC model came from the Faculties of Engineering at Cairo and Ain Shams Universities. That said, one of the challenges of rolling out training programs for engineering students is that they have very high expectations for internship and employment outcomes while at the same time have little time to invest in training because their workloads are so heavy. While working with engineering students was not a serious handicap to the program, if the program had been designed at a Faculty of Engineering in addition to other university faculties, we may have been able to draw conclusions about which implementation experiences were common to more than one faculty and which were unique to the Faculty of Engineering. The main lesson learned therefore is to think carefully about prospective partners at the university level and potentially engage a variety of university faculty partners in order to determine best practices for a broader range of Egyptian university students. In the same vein, careful pre-program evaluation of potential youth center partners is also important and selection should ideally be made on the basis of a center’s forward thinking management, capacity and the level of community support for that center. Ensure knowledge transfer by university career development center staff and One Stop Shop youth center project directors of best practice models and learnings related to setting up new centers at the university and youth center levels The staff and management of EEEA’s university career centers and One Stop Shop youth centers have learned a number of best practices and valuable lessons related to the design and delivery of employability-based training and job matching services to youth in Egypt’s formal sector. These trained team members carry an enormous amount of institutional knowledge with them, and in the case of the university programs, career center staff represent a first generation of career development specialists in Egypt’s tertiary public education system. It is recommended that their knowledge be shared and transferred to other individuals, and that best practices be documented – both to ensure continuation of good practices as models are replicated as well as to ensure that, if current management decides to leave, they will be replaced with knowledgeable individuals. One particular area to be built upon is the nascent effort at developing a skilled cadre of career counselors in Egypt certified by the U.S.-based career developed accrediting agency, the National Career Development Association (NCDA). Using NCDA’s platform of certified and tested curricula and instructor-level training certifications, future efforts can significantly expand the accessibility of career counseling services available to Egyptian youth. Egyptian public university students, who are often tracked early into career paths that are unsuitable for their interests and skills, will benefit greatly from greater exposure to career counseling; they will be more likely to be employed in jobs that suit their skills and interests and less directed toward bottlenecks in Egyptian labor market. Regular meetings of Alliance members When a program involves multiple subgrantees, networking facilitates cross fertilization and sharing of best practices and ultimately improves outcomes for all parties. EEEA’s National Secretariat organized marketing and capacity-building events but there were too few alliance meetings that involved all subgrantees. Regularity of meetings is beneficial so that subgrantees can learn from and share experiences with one another throughout all stages of program development. In addition, inclusion of all staff involved in the project (beyond just the project director level) should be considered in future interventions both to deepen local capacity building and build a deeper level staff motivation vis-à-vis the projects. Suggestions going forward would be: 1) to have monthly partner meetings with an opportunity for all staff to convene every second month for example or 2) a rotation system which ensures that each member of the subgrantees’ staff are at least present in some meetings even if it is difficult to include all of them every time. Allow subgrantees to conduct exchange visits to one another Since the program activities are consistent in themes and, to some extent, approach, it would have been a very good learning opportunity to organize exchange visits between various programs where subgrantees could learn from each others’ activities and brainstorm new innovative approaches to program delivery and monitoring and evaluation together. Conduct more field visits Although subgrantees were divided in their opinions about the adequacy of the number of visits of NM staff to the program, all expressed the importance of visits and the importance of maintaining a consistent level of communication with the Secretariat. Include all subgrantee project staff in capacity building training as much as possible As mentioned above, the inclusion of involved wider selection of staff members is recommended and would be appreciated by subgrantees. In addition, a thorough assessment of staff’s capacity needs and work toward filling these needs would have made the program even more successful. Similarly, training programs for youth center staff focusing on the subjects of development, mobilizing resources, fundraising, managing and sustaining income-generating activities would have been useful. Add qualitative approaches to M&E system Most of the indicators of the EEEA M&E system were quantitative. Subgrantees expressed their interest in evaluating the quality of the pilot projects as well as keeping track of their quantifiable achievements. Simplified Monitoring and Evaluation (M&E) tools for subgrantees EEEA’s M&E system framework (ranging from five to nine tracking tools) was perceived as cumbersome by some subgrantees. While it is important to monitor and evaluate all aspects of a program, it is recommended to decrease the number and complexity of tools designed to capture program achievements; this is particularly important when working with implementing partners who have had minimal experience with monitoring and evaluation systems prior to EEEA and/or have minimum capacity to implement M&E activities. Part II: Lessons Learned and Recommendations corresponding to Intermediate Result Two: Improved Access to and Relevance of Education and Employability Training Design and implement effective awareness campaigns at project launch Since the launch is the most important phase of a program, a program should design and implement effective awareness campaigns at the project launch that will target the largest possible number of prospective participants and potential partners. Provide advanced courses to participants Participants have indicated that it would have been helpful if the program provided advanced-level training courses in addition to introductory, beginner and intermediate level courses. Many trainees expressed interest in pursuing additional, advanced level courses not offered by the EEEA program and were forced to enroll in such courses outside of EEEA. While EEEA’s interventions represented initial activities, as programs move beyond the pilot phase, offering more advanced employability courses would prepare students even better for the job market. Require soft/life skills training as a core component of all programs Some students did not realize the importance of soft skills and/or that they lacked certain soft skills until they actually enrolled in soft skills training classes. Post-enrollment, students from a number of different educational and socio-economic backgrounds indicated that they appreciated the soft skills training. In fact, many participants recommended the program provide additional soft skills training topics. Because the communication skills of their prospective trainees were so low, one youth center subgrantee required that life skills training be taken before participants were allowed to enroll in other vocational or technical training courses. Such a strategy proved to be successful and it is recommended that other projects consider adopting a similar strategy. Engage parents, young children and demographic target groups from the community in One Stop Shop youth center activities The One Stop Shop youth center program was originally designed to serve youth aged 18 to 30. However, many other demographic groups from the community also made use of the the youth centers including young children, parents of youth, and those above thirty. The evaluation shows that in cases where these groups were indirectly integrated into program activities, it was very helpful in galvanizing community and/or parental support for the program. It is recommended that the One Stop Shop activities actively target and engage a variety of different age groups to make sure no one community group feels “left out” or marginalized when accessing services. At the same time, job placement targets should be further adjusted to account for those trainees who will be engaged in training but not necessarily eligible to work post-training. Customize program career development center interventions for different age cohorts Given the especially large participation in the university career development center programs of age cohorts not originally targeted for training programs, the EEEA program should consider customizing interventions for different age groups based on their eligibility to work post-program. For example, students in their first, second and third years of university should be offered career counseling and guidance to help them understand the range of career options available to them. Students should also be placed in summer internships to help them explore different career sectors and to give them the kind of practical experience employers look for when they apply for jobs post-graduation. Students in their fourth year, on the other hand, should be given training that complements their studies and directly relates to areas of economic growth and, if possible, existing job openings; such training should then be followed by intensive job placement and matching services. Ensure ongoing communication with former participants and knowledge transfer The program has built a strong base of constituencies among participants. It is very important for program implementers to maintain active communication with their trainees. Subgrantees working with universities have some mechanisms in place to ensure continuous communication with CDC alumni (i.e., Yahoo groups, websites, SMS). Universities should broaden these practices and youth center partners should learn from and adopt similar technologies. Both universities and youth centers should also host occasional networking events between alumni and current trainees. Make more explicit EEEA’s definition of “marginalized youth” and set indicators and tools to monitor the selection of youth meeting the definition The meaning of “marginalized” was not clearly defined at the outset of program. Early on, new iterations of the program should clearly define what “marginalized” constitutes for EEEA, and then develop performance indicators and monitoring tools around this definition. Pay special attention to gender considerations in upper Egypt (e.g., girls have to be home before sunset) One of the subgrantees implemented project activities in upper Egypt where there are some social restrictions on girls’ leaving their homes in the afternoons. The program should address such community concerns by offering more training sessions earlier in the day or establish special courses for females only. Part III: Lessons Learned and Recommendations Corresponding to Intermediate Result Three: Improved Prospects for Employment and Successful Entrepreneurship Conduct comprehensive market studies before launching program activities Allocating a portion of program budgets to conduct at least a rapid labor market assessment prior to the project launch is essential in an employability focused project. More comprehensive research would enable program partners to design training programs which are specifically geared towards areas of economic growth as well as to those sectors actively engaged in employing marginalized youth. Compiling information on the specific skills demanded by employers and on those geographical areas with increased employer demand would have been helpful in securing employment or internship commitments in advance of program activities and improving employment outcomes. Set realistic targets for employability activities Securing first time employment is extraordinarily difficult in Egypt where unemployment is largely a problem of educated youth graduating from public university or secondary technical institutes. In addition, fostering a culture of entrepreneurship is not an easy task in a country that has traditionally guaranteed employment through the public sector. As such, the program should set realistic job placement targets based on an assessment of labor market demands instead of setting unrealistically high targets which are very hard to achieve. In addition, these targets should be reviewed on a one to two￾year basis based on changing conditions of the market. Set clear strategies for job placement and employability activities The program should have developed a clear employment strategy from the beginning on how to get youth beneficiaries employed. Recommendations include: • Assigning higher level and increasingly market driven training to senior students • Securing employment or internship agreements with companies and/or factories in advance of program activities • Developing clear marketing plans for the centers, particularly geared toward employers • Having clear strategies for corporate outreach and building employer relations based on successful models from human resource companies, job placement agencies and others • Putting in place systematic monitoring and support mechanisms with clear indicators to track each participant’s path to employment including surveying participants’ satisfaction after they get hired/receive internships and keeping in contact with them • Building a strong internship strategy for first, second and third year students which is increasingly important to successful job placement upon graduation • Implementing training activities in parallel with employment activities, such as resume development, practice interviews, etc., in order to show the connectivity between training and ultimate efforts at securing a job • Being aware of the limitations of short-term training courses for successful job placement. Although short-term employability-related training courses are important for rounding out the formal education young people receive in the university, the EEEA program showed that such courses cannot always directly correlate to job placement. • Implementing training programs which target smaller numbers of participants with a more intensive employability training. Such an approach will likely yield higher outcomes per participant versus the current EEEA model which targets larger cohorts of participants in order to scale the program for larger demographic reach. Adopt follow-up activities for participants who start their own business (during and after start-up) Although following up with participants who started their own projects was not an initial goal of the program, many participants identified mentoring as a critical need. Participants needed both mentoring and support to follow up with their registration procedures and start up of their business. They also need advisory schemes along the way to guide them on business adjustments needed and to ensure that they were on the right track. Involve youth beneficiaries in program design Involving youth beneficiaries in the program design and implementation activities such as program administration, event organizing, fundraising, public relations, and outreach yields positive results – it both ensures that programming is relevant to youth and their needs and also gives youth additional experience, skills and competencies that will be helpful on the job. Conclusion As a pilot program, the Egyptian Education and Employment Alliance was innovative and produced strong results. The program adopted several innovative approaches that ensured its success including, first and foremost, the involvement of local Egyptian leaders on EEEA’s Advisory Board who, through their strategic connections, formalized early program partnerships with Egyptian public institutions. The involvement of the formal/governmental sector in the youth employability models created a natural platform for program expansion at both the university and youth center level and broad scale outcomes. In this respect, MOUs have been signed to replicate the career development center model in at least twelve new faculties as well as on the campuses of Alexandria, Tanta and Fayoum Universities. In addition, EEEA’s partnership with the U.S. based National Career Development Association (NCDA) to provide career facilitation and counseling at EEEA’s career centers was very strategic and has the potential to create a ripple effect if this partnership can be expanded to reach students at new faculties and universities where the MOUs have been signed. The EEEA program also succeeded in establishing a broad network of alliance members and partners which bought close to $3 million in cash and in-kind resources to the program enabling the program to reach many more young people than originally conceived. As EEEA is sustained and expanded over time, it will be important to heed attention to the recommendations and lessons learned from the pilot phase including the need for a more comprehensive job placement strategy for the career development centers and One Stop Shops that is demand driven and reflective of current labor market realities and making this strategy a core element of all future training and implementation activities. In addition, future expansion investments should consider more funding and longer implementation times for replicating the model; upgrading the youth center facilities upon program launch to ensure sustainability of the program; and strengthening the income generating component of activities to ensure the long-term sustainability of the program. Annex A: List of Stakeholders Participated in the Data Collection Process February-April 2009, Egypt Interview code Name Stakeholder Type/position Subgrantees (affiliation) Gender Project Alliance members: Semi-Structured Interview Cairo, Fayoum, Survey Monkey Dr. Mohamed Helal Government/focal appointed person from FOE in CEDO office CEDO M Survey Monkey Dr. Samir Ahmed Seif Dean of FOA/Fayoum University BEST Fayoum M Survey Monkey Dr. Mohamed Abdulwahab Morsy Deputy Dean of FOHT/Fayoum University and Manager of Faculty Center for Services and Training BEST Fayoum M Nahdet El Mahrouse: Semi-Structured Interview Cairo, NM Office NM-I-1 Jackline Kameel and Karine Zabal EEEA Secretariat NM 2 F Sub – grantees and YC Staff: Semi-Structured Interviews Cairo, Fayoum and Minia BM-I-1 Shoukry Mahmoud Mohamed YC Manager – Madina A - Minia BEST Minia M BM-I-2 Waleid Hussien BEST Minia Manager BEST Minia M BM-I-3 Mohamed Kamel, Ahmed Ibrahim and Mohamed Hamed Project Volunteers BEST Minia 3 M BM-I-4 Suziuet Samir, Badr Essam and Hassan Project Manger and two coordinators BEST Minia 1 F and 2 M BF-I-1 Karem and Lamyaa El Saiad BEST Fayoum Office Manager and project coordinator BEST Fayoum 1 M and 1 F AYB-I-1 Ibrahim Ahmed Abd El Halim YC Manager – Ein El Sera YC AYB M AYB-I-2 Raghda Mahmoud Abdulnabi, Shimaa Fayed Gabullah, Nihal Nashat and Radwa Fayez Khairy AYB project staff AYB 4 F S-I-1 Salah Abdulatif YC Manager – El Maadi El SEKEM M Gedeeda S-I-2 Ahmed Aboulgheit Project coordinator SEKEM M S-I-3 Dr. Mahassen Hassanin Program Director SEKEM F CDC-I-1 Seham Office Coordinator CDC F CDC-I-2 Ines Houzein Office Manager CDC F CDO-I-1 Hesham Nassar, Wafaa Bekhit, Amnai Ali, Arrej Moustafa, Ahmed Ali, Ahela, Riham Office team CEDO 2 M and 5 F CDO-I-2 Salma Elbahrawy Office Manager CEDO F Employers: Semi-Structured Interviews Cairo, Fayoum and Minia Survey Monkey Dr. Mohamed Abdulwahab Morsy Deputy Dean of Faculty of Hotel and Tourism/Fayoum University and Manager of Faculty Center for Services and Training BEST Fayoum M Survey Monkey Dr. Magda Soliman Abdullah, Deputy Dean of the Faculty of Agriculture BEST Fayoum F Survey Monkey Mr. Mohamed Taha Abdulhalim The International Academy for Teaching Mobile Maintenance BEST Minia M Mr. Ayman Mohamed El Forkan Company for manufactured food products BEST Minia M Trainers: Semi-Structured Interviews Cairo, Fayoum and Minia Survey Monkey Haitham Abdulhamid Mohamed Trainer on human development courses- Best Minia BEST Minia M Survey Monkey Mohamed Ewis Mohamed Hassan In- House professional trainer on Information systems BEST Fayoum M Survey Monkey Dr. Ashraf Abdul Maaboud Trainer on hospitality and rooms division BEST Fayoum M Survey Monkey Mahmoud El Seid Abdulaziz Trainer in the kitchen division of Faculty of Hotel and Tourism/Fayoum in University BEST Fayoum M Survey Monkey Dr. Adly Thomas Professional trainer CDC M Survey Monkey Hesham Nassar In-House professional trainer CEDO M Survey Monkey Asmaa Barakat In-House trainer SEKEM F Survey Monkey Nagah Abdulhadi In-House trainer (YC staff) SEKEM F Survey Monkey Moustafa Mohamed Eissa In- House professional trainer (SEKEM) SEKEM M Project beneficiaries: Interviews, Random Samples and Focus Groups Cairo, Fayoum and Minia Survey Monkey 156 participants who were interviewed, part of FGs or randomly selected 156 participants who were interviewed, participated in FGs or randomly selected All subgrantees 83 M and 73 F 11 FGs 11 FGs conducted that involved 116 participants 116 participants participated in FGs discussions All subgrantees 48 M and 68 F Annex B: Data Collection Process and Instruments Collecting data in the field lasted for 17 days among different subgrantees and different project locations and sites: BEST Minia (3 days) in Minia governorate at different locations, BEST Fayoum (2 days) in Fayoum governorate at different locations, CDC (2 days) at CDC office in FOE, Ein Shams University, CEDO (3 days) at CEDO office in FOE, Cairo University, AYB (2 days) at AYB office and target YC, SEKEM (2 days) at SEKEM office and target YC and NM (2 days) at NM office. All interviews, surveys and FGs were conducted in native language (Arabic) and then translated to English in the minutes reporting and data entry process. Eleven focus groups were conducted that involved a total of 116 youth, 48 males and 68 females, from various education and social backgrounds. Each FG had from 7- 12 participants. FGs were conducted at different places and premises as detailed in the figure below: FGs locations Premises Number of FG conducted Cairo University CEDO office 2 Ain Shams CDC office 1 BEST Minia office 1 YC/BEST Minia 1 BEST Fayoum office 1 Faculty of Hotel and Tourism, Fayoum University/ BEST Fayoum 1 YC/AYB 2 YC/SEKEM 2 Total 11 Each FG lasted between half an hour to one and half hours depending on the information, size of each FG and informant’s availability. FG participants also filled a survey questionnaire prior to starting the FG. Similarly, the interviews with subgrantees, participants and other stakeholders were conducted at various locations. Each interview lasted between one to two and half hours depending on the information and informant’s availability. ©2010 International Youth Foundation. All rights reserved. No portion of this report may be reproduced in any form without the written permission of IYF. This project was made possible by the generous support of the American people through the United States Agency for International Development (USAID) under grant agreement no. RAN-A-00-04-00043-00 This report was funded in part under grant agreement no. EPP-A-00-08-00006-00. The International Youth Foundation (IYF) invests in the extraordinary potential of young people. Founded in 1990, IYF builds and maintains a worldwide community of businesses, governments, and civil-society organizations committed to empowering youth to be healthy, productive, and engaged citizens. IYF programs are catalysts of change that help young people obtain a quality education, gain employability skills, make healthy choices, and improve their communities. To learn more, visit www.iyfnet.org ANNEX D: EEA Indonesia Evaluation Report CONTENTS Executive Summary ................................................................................................................................. 3 Section I: Introduction ............................................................................................................................. 6 Section II: YEEI Results Framework ....................................................................................................... 11 Section III: Findings and Analysis of Program Achievements................................................................... 12 Intermediate Result I: Alliance Building and Leveraging Resources Intermediate Result II: Improved Access to and Relevance of Education and Employability Training Intermediate Result III: Improved Prospects for Employment and Successful Entrepreneurship Intermediate Result IV: Positive Indirect Effect on Families and Communities of Youth Section IV: Challenges, Lessons Learned and Recommendations ............................................................ 39 FIGURES Figure 1: Youth Unemployment Rates in Select Asian Countries .............................................................. 7 Figure 2: Seed Funds and Leverage Contributions.................................................................................. 13 Figure 3: Inventory of In-kind Leverage Resources.................................................................................. 14 Figure 4: Typology of YEEI Partners and Leverage Contributions ............................................................ 14 Figure 5: Sustainability Status of YEEI Projects ..................................................................................... 18 Figure 6: Project-level Costs ................................................................................................................... 20 Figure 7: Alliance Building and Program Support Costs ......................................................................... 20 Figure 8: Number of Direct and Indirect Beneficiaries ............................................................................ 23 Figure 9: Direct Beneficiaries (Participated and Completed) by Gender and against Targets .................... 24 Figure 10: Program Graduates by Gender and Region ............................................................................ 24 Figure 11: Participants - Target vs. Achieved .......................................................................................... 25 Figure 12: Program Graduates by Region ............................................................................................... 25 Figure 13: Level of Change in Youth’s Self-confidence ............................................................................ 27 Figure 14: Youth Satisfaction with YEEI Employment Training .............................................................. 27 Figure 15: Youth Satisfaction with YEEI Entrepreneurship Training ....................................................... 28 Figure 16: Numbers and Rates of Graduates Employed and Self-Employed Against Targets ................... 29 Figure 17: Entrepreneurs by Projects and Gender .................................................................................. 31 Figure 18: Employers’ Satisfaction with YEEI Graduates ........................................................................ 33 Figure 19: Job Satisfaction of YEEI Graduates ....................................................................................... 34 Figure 20: Income Levels of YEEI Graduates .......................................................................................... 35 Figure 21: How Youth Use Their Earnings ............................................................................................. 37 ANNEXES Annex A: Summary of YEEI Projects ...................................................................................................... 41 Annex B: Nike “Champion Worker” Life Skills Training Evaluation ......................................................... 43 Executive Summary Rising unemployment among today’s youth is a pressing global challenge. Among the most at-risk are youth in Southeast Asia, who are generally five times more likely than adults to be unemployed. In Indonesia, six million youth — nearly one in three — are unemployed – and these youth face enormous challenges as they attempt to enter the workforce, due to a weak support system that is frequently unable to help them either gain employability skills or launch their own businesses. In response to these challenges, EEA’s program in Indonesia, locally branded as the Youth Employment and Entrepreneurship Initiative (YEEI), was launched in June 2006 to establish a multi-sector group of public and private sector leaders who would work together to prepare underserved youth in Indonesia to become productive employees and entrepreneurs. It was intended to support five to six innovative workforce development activities, and help at least sixty percent of youth trained get jobs. Similarly, it aimed to have at least fifty percent of youth provided with entrepreneurship training successfully launch their own enterprises. In turn, this summative evaluation, consistent with overall EEA global evaluation protocols, assessed the following: 1) the overall value and utility of the alliance building approach used by YEEI; 2) the effect of YEEI on direct beneficiaries; and 3) the indirect effect of the program on families and communities of youth. Nearly 300 youth, 22 employers, and 16 alliance partners participated in the study through surveys, interviews, and focus group discussions. Part I: Overall Value and Utility of the Alliance Building Approach The evaluation study finds that YEEI was effective in forging meaningful public-private alliances which were critical in generating new leverage resources to support and expand program, and to drive a partnership model that would engender greater innovation and broader support among relevant stakeholders. Having overcome challenges during its initial trust-building and negotiation phase, YEEI partnered with 65 public-private entities during the course of the program, including local and multinational corporations, non-governmental organizations (NGOs), and local governments which have together supported eleven projects benefitting youth. The leverage requirement has encouraged lead implementing partners to look for corporate and NGO partners they would not have otherwise have found and this focused effort has brought forward a 2:1 leverage ratio against program seed funds, and generated nearly $1.3 million in leverage contributions. Interviews undertaken as part of this evaluation indicate that a robust process put in place for the Board and alliance participation proved to be valuable in encouraging public-private sector collaboration, which in turn increased perceptions of overall program quality and local ownership. In this regard, findings suggest that the adherence to alliance protocols encouraged active participation from the private sector and meaningful collaboration between NGOs and businesses resulting in strengthened capacity of NGOs to implement alliance-based projects and strong job placement rates. Most fundamentally, even though YEEI has ended, a number of local alliances generated through the program are continuing to provide support to newly created businesses, graduates who are employed across Java, and in some cases, phase two of the projects. The Alliance building experience, however, was not without challenges. While in a macro sense, alliances generated seem to yield valuable benefits, the process of Alliance development for partners can appear unwieldy and challenging. For example, while YEEI was effective in attracting local businesses, USAID due diligence protocols for prospective company partners needed to be undertaken for over thirty corporations, which in turn caused delays and likely affected the reach of some projects because of the need to shorten program durations. Additionally, some partners expressed their reservations about implementing similar alliance projects in the future because the complexities involved, such as the need to take a participatory approach in design; to invest more time and resources in engaging a broad segment of partners with a broad diversity of experience in such efforts; to focus consistently on finding new leverage contributions, and to negotiate with various parties in the case of conflicts. Such efforts, in addition to the rigorous financial and program management standards put forward by USAID, IYF and the Alliance Steering Committee, often left partners somewhat overwhelmed. In the end, however, the benefits of developing such alliances in support of programs are noteworthy in terms of generating new resources, ensuring local ownership and helping to promote a holistic approach to program design. As such, our findings here suggest that new efforts in the area of alliance building should focus on ensuring refined operational protocols for program design, approval and implementation through alliance structures and also a more realistic understanding of the “alliance management” component of activities as projects are designed and budgeted. Part II: Effectiveness of the YEEI Program Surveys conducted through this evaluation found that the program overall was able to provide quality employability skills training to underprivileged youth groups including orphans, street and rural youth. Through three intervention types (integrated employability, entrepreneurship, and life skills projects), YEEI benefitted a total of 13,718 people of whom 13,202 are direct youth beneficiaries and 516 are indirect beneficiaries who have taken additional jobs created. Of 13,202 direct beneficiaries, 5,547 participated in an average of six-month integrated employability skills training and 7,655 participated in a two-day life skills training. An aggregated 1,286 youth were determined to be eligible and ready for salaried or self-employment. Of these, 1,074 youth (or 84% of those ready or eligible for work) have been placed in jobs or set up small businesses within the six months of completion. As detailed in the sections below, the quality and relevance of training was rated high by graduates who also provided suggestions to further strengthen the program. Suggestions included longer training periods for select projects, improving training materials, and the need to continue providing follow-up support to those who have started their jobs or enterprises. The study also finds substantial improvements in capabilities of graduates in both technical and life skills based on triangulation results. However, it is also noted that the partners’ use of different life skills curricula made it difficult to systematically measure learning outcomes. Effectiveness of Employment Projects and Employer Satisfaction: YEEI has successfully placed 714 youth or 84% of 850 vocational skills training graduates who were eligible to get jobs. This represents 88% of the original employment target of 810 youth to be placed in jobs. The study finds that its job placement successes can be attributed to the following factors, each relevant to an alliance based program with rigorous focus on inclusion of the private sector in particular: 1) Industrial knowledge of implementing partners; 2) Ability to respond to local markets by conducting high quality and relevant training (combining technical, life skills, equivalency education, and on-the-job training) followed by job placement services and follow-up support; 3) Ability to develop and sustain good partnerships with local businesses, and involve them in curriculum review, internship and job opportunities; and 4) Fostering a culture of working effectively and collegially. Strong findings are particularly seen in the reaction of employers to graduates. More than half of the employers interviewed stated that YEEI graduates are “much better” than other graduates from similar backgrounds. Several employers also mentioned that YEEI graduates exhibit strong technical skills, work ethic, and willingness to learn. Over ninety percent of employers felt that YEEI was able to provide qualified graduates for the business sector, thereby closing the gap between employers’ needs and employees’ qualifications. Suggestions include further strengthening technical and communication skills of some graduates and familiarity with more sophisticated equipment. Effectiveness of Entrepreneurship Projects: YEEI enabled 360 youth or 83% of 436 entrepreneurship training graduates to successfully launch and sustain their own businesses. This represents ninety percent of the original self-employment target of 399 youth to become entrepreneurs and establish small businesses. Our study finds that YEEI at first struggled to implement entrepreneurship projects due to cultural norms, external shocks (natural disasters and sharp increase in consumer prices) and local capacity issues. The study also finds that the following remedial actions helped YEEI overcome these challenges and contributed to the success of newly established businesses run by 360 youth: 1) Providing substantial mentoring support throughout the project and strategically using the power of alliance to help young entrepreneurs stay motivated and focused; 2) Mobilizing additional resources to counterbalance the shocks and reinstate project activities; and 3) Hiring business coaches to strengthen and sustain businesses that were in precarious situations. More generally, the assessment has concluded that youth businesses met and surpassed benchmarks established such as production and sales, maintaining an appropriate bookkeeping system, making profits, and sustaining after six months of establishment. Part III: Positive Indirect Effect on Families and Communities of Youth The study also shows that YEEI demonstrated to have positive if indirect effects on families and communities of youth served. Using new incomes, graduates are financially helping their parents, siblings, and spouses/children for daily consumption needs, education, and health expenses. They are also saving part of their income for future investments, business expansions, and as emergency funds. More than half of the graduates surveyed reported that their financial support helped improve the financial and social situations of their families. YEEI-supported enterprises have also created 516 additional jobs that have generated primary or secondary incomes for the communities of youth. Lessons Learned and Recommendations Building and Reinforcing Alliances: Establishing a robust process for Board and Alliance participation proved to be worthwhile as it positively contributed to the program quality and effectiveness. However, it also demanded considerable resources in terms of time and human resources for coordination, negotiation and sometimes conflict resolution. This relatively new and intensive process of building and reinforcing alliances overwhelmed some Indonesian NGO partners, with some suggesting that the alliance process was too difficult compared to traditional development projects. Additionally, in terms of engaging the private sector, YEEI’s ability to engage local businesses for program design and employment support is crucial for the program success. Having potential employers review the curriculum and giving them access to graduates have resulted in the delivery of demand-driven training, high employer satisfaction and job placement rates. As such, recommendations are as follows: • Provide strong capacity building support to local staff at secretariats and within implementing partners on how to forge and reinforce vibrant alliances in appropriate cultural contexts. Knowledge transfer to local staff helps promote the scalability and sustainability of alliance-based projects. Dedicate sufficient program management support staff to fill the various roles that alliance programs demand. • Clearly define and ensure a reasonably even distribution of roles and responsibilities of each alliance member. Establish mechanisms to hold each other accountable and responsible throughout the project. • Develop alliances based on mutual respect and understanding. Leverage local cultural approaches that encourage caring for and partnering with local communities to achieve social good. • Find innovative ways to mobilize and recruit the private sector. Tap into their knowledge and expertise beyond financial resources and involve them in curriculum design and pilot activities. • Give the private sector access to program graduates for internships and job opportunities. Ask for regular feedback regarding the performance of graduates and improve the quality and relevance of the training. Implementing Effective Alliance-Driven Employability Programs: YEEI’s approach to review initial concept notes and co-create projects with local NGOs and sometimes community development experts worked well. Involving all stakeholders in the design and implementation stages encouraged local implementers to take ownership and also helped increase program effectiveness. Furthermore, YEEI’s strategy to design and provide tailored comprehensive skills training based on the needs of youth contributed to youth making a successful transition into the labor market. Regular feedback meetings with internship providers were helpful to improve youth’s performance and increase their chances of getting jobs. YEEI has learned that providing follow-up post-training support to youth is critical, and the YEEI projects that provided follow-up counseling support produced better results. Recommendations are as follows: • Engage all key stakeholders during the design stage – particularly training providers, prospective employers and local experts – to increase the quality, relevance and effectiveness of the training. • Tailor comprehensive skills training activities based on local needs, followed by internships, job placement and post-training support. Integrate follow-up support as a key component of program designs and provide the follow-up counseling support at least for the first six months as youth enter the labor market. This can also help correct cases of skills mismatches. • Engage youth in meaningful ways. Solicit feedback and consult with them on a regular basis to ensure programs are relevant to them and address the reality of their educational experience, capabilities, family and community constraints, and other issues of importance in ensuring quality training and successful outcomes. Section I: Introduction A. Evaluation Objectives, Design and Methodology This evaluation study aims to assess the following: 1) overall value and utility of the EEA alliance approach; 2) YEEI’s program effect on direct beneficiaries focusing on skills acquisitions and change in employment status; and 3) indirect effect on beneficiaries’ families and communities. The country evaluation team was led by Muchlis Ali, a former USAID employee who served as the M&E consultant for EEA Indonesia, with support from Delly Paramita (IBL Senior Program Officer), Arfandi Rifai (IBL Finance Officer) and Lin Lin Aung (IYF Program Manager). A local survey assistant was also hired to help with data entry, aggregation and analysis using Survey Monkey software. YEEI initiated its country-specific summative evaluation process in October 2008 by adapting the EEA’s globally evaluation framework that included standard outcome indicators, information gathering tools and protocols. Three information gathering instruments - surveys, interviews and focus group discussions (FGDs) – were used to gather the necessary data. A total of five surveys were developed and tested for various stakeholders: • A youth follow-up survey to measure the quality and relevance of skills training, effectiveness of job placement strategies or sustainability/viability of new enterprises, and indirect effect on their families and communities • An employer survey to assess their satisfaction with EEA training programs • Three surveys for implementing partners, alliance members and Secretariats to assess the overall utility and effectiveness of the EEA alliance approach This summative evaluation is built upon interim evaluations conducted in the summer of 2008. Nearly 300 youth, 22 employers, nine implementing partners, and seven alliance partners participated in the overall evaluation. In addition to three projects assessed during interim evaluation, the team visited seven projects in November to conduct evaluations involving youth and all major stakeholders. The eleventh and last project, Nike “Champion Worker” life skills training, was evaluated separately by Tibies Sinergi to measure knowledge gain and behavior changes involving 150 operators (see Annex B of this country evaluation report). B. Evaluation Limitations and Challenges In undertaking the evaluation, four challenges were faced, resulting in the following limitations as detailed below: 1) geographic and programmatic diversity of projects; 2) resource constraints; 3) challenges associated with scheduling graduates to participate in the evaluation; and 4) independence of evaluators. Geographic and Programmatic Diversity of Projects: When YEEI was soliciting proposals in 2006, prospective implementing partners had concerns as to whether they could meet leverage requirements of a minimum 1:1 leverage contribution. This resulted in proposals for smaller projects, and YEEI ended up awarding eleven subgrants valued between $33,000 and $85,000. The fact that these projects were geographically diverse – with locations across Java and Bali - proved to be a challenge for conducting evaluations. In addition, between 2006 and 2009, YEEI benefitted over 13,000 people of which 7,655 were Nike factory operators who received the “Champion Workers” life skills training. Because of its specialized programmatic focus, its evaluation was conducted separately and the findings were not incorporated into the overall analysis of the program – see Annex B of this report for detailed findings. Resource Constraints: Budgetary and time constraints made it difficult for the team to travel to all eleven project sites spread out across Java and Bali for evaluation. To overcome this challenge, interim evaluation results of three projects were incorporated into this summative evaluation study, specifically Kakak tempe chips production in Yogyakarta, Bina Swadaya catfish farming and KUD fish filet projects in West Java. The evaluation team felt that interim evaluation results were relevant and sufficient for two reasons: 1) Being the first three projects implemented by YEEI, these projects ended in mid-2008 and interim evaluations were conducted in July 2008, essentially upon their conclusion. 2) Indonesian youth follow-up and employer surveys were later adapted by the global team, and the global surveys are by and large the same with an additional section on changes in participants’ capabilities and attitudes. Scheduling Graduates: As YEEI subgrants ended and graduates began working towards the end of 2008, the third challenge was to have all youth gather in one venue for surveys and focus group discussions because most of them ended up working in towns far away from initial project sites or were unable to participate during work hours. The team was not able to randomly select graduates and largely used the purposive sampling method for this evaluation. As an attempt to overcome this challenge, surveys were also mailed out to graduates working far away from project sites followed by telephone interviews to achieve good representation of all youth involved. For those who were unable to participate because of work, special meetings were arranged after working hours to administer surveys and conduct focus groups. The team felt that the sample - while not randomly selected – was adequate because it included responses from nearly 300 youth for the overall evaluation. Independence of Evaluators: As discussed in detail in the overall global evaluation, this study is inherently limited by the lack of a fully external evaluation of program outcomes. As noted in the global introduction, given both resource constraints and the desire to leverage the broad experience of program participants to fully compare and contrast approaches used in the different countries for governance, program design, leveraging of resources and other elements, a strategic decision was made to utilize participants in the overall program to support this evaluation. In the interest of promoting learning and improvement, the EEA teams have done their best to express challenges and problems within programs and the alliance structure, grounded in the reality of their experience in the program; however, the study does have this inherent limitation, which is important to disclose fully. C. Program Summary The YEEI is the Indonesian program of the USAID-funded Education and Employment Alliance program implemented in six Asian and Middle Eastern countries. The program was branded locally as the Youth Employment and Entrepreneurship Initiative (YEEI) to emphasize that it aimed to help prepare youth to be both productive employees and when so inclined, productive entrepreneurs. Problem Statement and Program Background Rising unemployment among today’s youth is an increasingly pressing global challenge. According to the International Labor Organization, approximately 88 million youth throughout the world are without jobs. Among the most at-risk are youth in Southeast Asia, who are generally five times more likely than adults to be unemployed. In Indonesia, six million youth – nearly one in three - unemployed, grown from 1.4 million recorded in 1990. Indonesia’s current annual economic growth of six percent is too low to absorb the annual increase in the labor force, adding 300,000 to 500,000 more to the total unemployed each year. Unemployed youth represent more than sixty percent of the total unemployed. Compared to other Asian countries, Indonesia had an alarmingly high youth unemployment rate of 30.6% in 2006 as indicated in the figure below. Figure 1: Youth Unemployment Rates in Select Asian Countries Youth unemployment rate (%) Ratio of youth to adult unemployment rates Share of youth unemployed in total unemployed (%) Cambodia (2004) 1.5 2.5 61.0 Indonesia (2006) 30.6 6.0 61.6 Malaysia (2004) 11.7 8.4 68.2 Philippines (2006) 16.9 3.6 49.5 Vietnam (2004) 4.6 3.2 46.2 India (2004) 10.5 3.0 45.8 Source: ILO, Key Indicators of the Labor Market (KILM) 4th Edition; National Secretarial Office data Some Indonesian economists and businessmen argue that the issue is not a lack of job opportunities but low quality of human resources. Regardless, young men and women in Indonesia in fact face enormous challenges as they attempt to enter the workforce, due to a weak support system that is unable to help them either gain employability skills or launch their own businesses. Some job seekers voluntary accept low salaries or take part-time, informal or seasonal jobs. As a result, unemployed and underemployed youth have become increasingly frustrated from not being able to take their places in the market as respected, productive adults. Leaving the youth as unemployed will not only exacerbate prevailing socioeconomic problems but also affect the overall human development now and for future generations. The Youth Employment and Entrepreneurship Initiative (YEEI) In response to local challenges, the Youth Employment and Entrepreneurship Initiative (YEEI) program was initiated by the International Youth Foundation (IYF) and Indonesia Business Links (IBL) in mid￾2006 to increase employment and private enterprise opportunities for unemployed and underemployed. With an initial US$600,000 in seed funds, YEEI sought to achieve two fundamental goals: 1) To establish a multi-sector group of public and private sector leaders (the Steering Board) who provide oversight and actively engage in activities that directly train or prepare youth for jobs or enterprise startups, resulting in an increased number of Indonesian youth finding jobs or becoming productive entrepreneurs, and 2) To support five or six innovative activities promoting support for demand linked employment training and entrepreneurship with an average seed funding of $100,000 per activity. Specific objectives of YEEI include: 1) To improve the skills of over 11,110 youth beneficiaries for better employability and enterprise development;1 2) At least sixty percent of 810 youth trained will get jobs following their vocational/technical training; and 3) At least fifty percent of 399 youth provided with entrepreneur training and other supports will be able to successfully launch their intended enterprises and sustain and grow their businesses within one year after launching. IBL was selected to serve as the National Secretariat - responsible for working closely with IYF in awarding subgrants, implementing the program, providing direct management support and ensuring monitoring, reporting and evaluation on funded activities. The Board was established in mid-2006 with representatives from corporations, civil society groups and international organizations such as GAP, BP Indonesia, TIFA, Ford Foundation, GE Foundation and International Labor Organization. YEEI faced no major challenges recruiting board members as IBL and IYF used their own networks to recruit practitioners from both private and public sectors. With funded oversight, YEEI put in place a robust process for Board participation throughout program implementation to encourage meaningful collaboration. Even though YEEI initially planned to support five or six projects, it ended up supporting eleven projects because of the high enthusiasm of more than thirty NGOs that submitted their concept notes. In general, the budget requested per project ranges between $30,000 and $80,000 for one-year projects.2 While there was no intention to geographically focus or limit the project initially, YEEI decided to limit the activities to be funded to Java and Bali islands at the later stages due to concerns on logistics and monitoring costs. After dissemination of Requests for Proposals, IYF, IBL and the YEEI board reviewed concept papers and provided significant recommendations to improve the design and viability of worthy concepts presented to the Alliance. Secondly, in a more opportunistic fashion, when the Alliance was approached by corporations with specific geographic focus such as GAP and BP, YEEI collaboratively designed programs with them and hired community development consultants to recommend locally relevant, viable activities for hard-to-reach, remote areas in West Java. Using these processes, a few partners received subgrants as early as March 2007. Others were awarded much later in the year because of the delays associated with conducting the USAID-required due diligence process for over thirty corporate supporters. All subgrants were awarded from March through December 2007 and implementation took place through March 2009 for subgrant projects. The Secretariat operations ended in mid-July of 2009. Summary of YEEI Project Outcomes Below are brief descriptions of all YEEI projects which are divided into four categories: a) entrepreneurship projects; b) integrated employment and entrepreneurship projects; c) community empowerment and employment generation projects; and d) life skills project. 1 a) 810 for employment, b) 3,300 for entrepreneurship of which 399 for enterprise development and 2,900 for vocational high school entrepreneurship training, and c) life skills training for over 7,000 additional youth. 2 For more information on leverage and grant amounts, see Indicator 1.1. a) Entrepreneurship Projects Economics for Life and Student Company Program in Vocational High Schools in Jakarta: The Economics for Life entrepreneurship program was implemented by Prestasi Junior Indonesia in Jakarta from June 2007 through late-2008. It trained 26 teachers and 3,232 vocational students on the EFL curriculum and the use of Oracle’s Think.com software, enabling more effective project-based learning and collaboration among students and teachers. An aggregated 250 selected students established student companies and gained first-hand experience managing their own companies. Catfish Farming Entrepreneurship and Economic Empowerment Project in West Java: This project, implemented by Bina Swadaya, trained twenty youth to become skilled and productive entrepreneurs and provided financial and business support to youth in establishing two enterprises. These enterprises have created 140 additional jobs. A local committee has been established with existing YEEI partners to provide continued support to young entrepreneurs. Livelihood Recovery and Promotion for Earthquake Affected Communities in Yogyakarta: This project was implemented by Kakak from March 2007 through November 2008. It has helped seventeen youth in Yogyakarta establish small businesses producing tempe snack food and generated new employment for 58 additional people. YEEI also helped youth establish a legal business association called “Happy Group” that coordinates the supply of raw materials, sales of tempe chips and provision of working capital loans. Supporting Vulnerable Youth in North Jakarta: This project, implemented by Sekar, offered comprehensive training (technical, equivalency education, computer literacy and life skills) to 77 street youth in North Jakarta and enabled them to establish six successful business groups from December 2007-08. They are now working with fifty businesses for the supply of raw materials and ten local distributors/stores to market their products. b) Integrated Employment and Entrepreneurship Projects Car and Motorcycle Engine Repair Training in Karawang: This project, implemented by Cepat Tepat, has provided the technical skills training in sewing and engine repair to 105 youth from October 2007-08. It has placed 57% or sixty youth in jobs with local business partners (such as Toyota and Honda), and enabled 43% or 45 youth to set up business groups, thereby achieving a 100% employment rate. Training Broiler Chicken Operators and Technicians in Pati: This project, implemented by CREATE, has trained 150 youth to become broiler chicken farm operators from October 2007-08. It has placed 143 graduates (95%) in jobs with local business partners in East, West, and Central Java, and provided financial support to five additional graduates set up their own poultry runs, thereby achieving a nearly 100% employment rate. Two graduates decided to pursue further education. Providing Employment and Entrepreneurship Training for Surabaya youth: This program, implemented by the East Java Consortium, has helped 282 street youth and orphans in Surabaya gain entrepreneurial and technical skills (sewing, handicrafts, and computer literacy) from November 2007-08. Sixty percent of youth have set up group businesses and another 35% secured jobs in computer-related fields. Assisting Young T-shirt Entrepreneurs in Bandung: This project, implemented by JBL, has trained 575 T￾shirt entrepreneurs in T-shirt design and production, and placed 423 youth (74%) in jobs with local businesses. It has also helped 983 more youth learn about obtaining and managing C-59 T-shirt franchises of which twenty qualified youth have received financial assistance to set up their own franchises. c) Community Empowerment and Employment Generation Projects Capacity Building and Sustainable Economic Empowerment Project in West Java: This project, implemented by KUD Mina Bahari, has helped strengthen a local fish filet industry by providing the necessary training, equipment and financial support from June 2007-November 2008. It has allowed the industry to undertake additional processing, packaging and marketing of fish filet products, and increased income of an estimated 250-350 local residents. Twenty youth have also been trained to work as machine operators. Supporting Community Ecotourism in Bali: This ecotourism project, implemented by Kehati and Wisnu Foundations, has provided technical assistance and other support to the Village Ecotourism Network - made up of four villages in southeastern Bali from November 2007-08. It has also trained 154 village members of which 86 are youth - in ecotourism, hospitality/hotel management services and improving their local production of foodstuffs and handicrafts. d) Life Skills Training “Champion Workers” Life Skills Training for Nike Factories: This employee life skills training was implemented by Business Dynamics in ten Nike contractor factories in the Jakarta region from October 2007-June 2008. During its initial phase, the project trained twenty trainers of which sixteen have been certified to become in-house trainers who will then train an estimated 40,000 operators in the next few years. As of March 2009, an aggregated 7,655 operators have received this life skills training. Section II: YEEI Results Framework Indonesian Youth Employment and Entrepreneurship Initiative Goal: Improving the skills of young people (ages 18-24) for better employment and successful entrepreneurship Intermediate Result 1: Alliance Building and Leveraging Resources Intermediate Result 2: Improved Access to and Relevance of Education and Em plo yabilit y Trainin g Intermediate Result 4: Positive Indirect Effect on Families and Communities Intermediate Result 3: Improved Prospects for Employment and Successful Entre preneurshi p Indicator 1.1: In-kind and cash resources leveraged for the project through private sector, government, civil society and other partners Indicator 1.2: The effectiveness of the YEEI alliance approach in building partnerships, and promoting sustainable and scalable youth employability programs Indicator 1.3: Number of private sector, government and civil society partnerships formed through the project to jointly offer employment and business development training activities in targeted areas Indicator 1.4: New partnerships, mechanisms and approaches created as a rippling effect of partners working with YEEI to promote youth employability Indicator 2.1: Number of young men and women participating in YEEI programs Indicator 2.2: Number and percentage of young men and women completing YEEI supported programs Indicator 2.3: Number of teachers, trainers, facilitators and counselors trained under YEEI supported programs Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in YEEI supported programs Indicator 2.5: Level of satisfaction of young trainees with the quality and relevance of the training, counseling and services received through YEEI programs Indicator 2.6: Number and percentage of youth who completed YEEI supported projects who report that they have referred at least another youth Indicator 3.1: Number and percentage of trainees who get a job within six months of completing the program Indicator 3.2: Number and percentage of trainees who continue education or engage in further professional training within six months of completing the program Indicator 3.3.: Number and percentage of YEEI trainees starting their own businesses within six months of completing the training Indicator 3.4: Level of employers’ satisfaction with the soft skills and/or technical skills of YEEI trained youth Indicator 3.5: Number and percentage of youth with satisfactory internships or apprenticeships Indicator 3.6: Number and percentage of placed youth who declare having a satisfying and quality job Indicator 3.7: Percentage of youth (who had prior jobs) earning more income as a result of participation in the YEEI program and net salary increase Indicator 4.1: Number and percentage of employed youth financially helping their families with household, health and education expenses Indicator 4.2: Number and percentage of youth who have reported improved financial and social conditions of their families as a result of participation in the YEEI programs Indicator 4.3: Number of jobs created for community as indirect beneficiaries of YEEI programs Section III: Findings and Analysis of Program Achievements Intermediate Result I: Alliance Building and Leveraging Resources Summary of Intermediate Result One Findings This evaluation suggests that YEEI was effective in forging meaningful public-private alliances that were able to: 1) generate leverage resources necessary to maximize program potential and engender local ownership; and 2) drive multi-sectoral approaches to program design that helped identify prospective partners particularly in the private sector for stronger outcomes. In terms of leverage, while overcoming challenges faced during its initial trust-building and negotiation phase in which prospective partners had challenges understanding the alliance program, YEEI successfully partnered with 65 public-private entities including local and multinational corporations, NGOs and local governments who have supported eleven projects. In turn, despite its initial concerns to meet leverage requirements, YEEI has achieved a leverage ratio of 2.2:1 exceeding the initial target of a 1:1 leverage ratio against project seed funds of approximately $600,000. Secondly, the study shows that YEEI strategically used the collective power of alliances to drive its projects to achieve optimal results. A robust process put in place for the Board and alliance participation has proven valuable in encouraging public-private sector collaboration in the designing, implementation and evaluation of projects, which in turn appears to have improved overall program quality and increased local ownership and sustainability of subgrant activities. The Alliance building experience, however, was not without challenges. While in a macro sense, alliances generated seem to yield valuable benefits, the process of Alliance development for partners can appear unwieldy and challenging. For example, while YEEI was effective in attracting local businesses, USAID due diligence protocols for prospective company partners needed to be undertaken for over thirty corporations, which in turn caused delays and likely affected the reach of some projects because of the need to shorten program durations. Additionally, some partners expressed their reservations about implementing similar alliance projects in the future because the complexities involved, such as the need to take a participatory approach in design; to invest more time and resources in engaging a broad segment of partners with a broad diversity of experience in such efforts; to focus consistently on finding new leverage contributions, and to negotiate with various parties in the case of conflicts. Such efforts, in addition to the rigorous financial and program management standards put forward by USAID, IYF and the Alliance Steering Committee, often left partners somewhat overwhelmed. In the end, however, the benefits of developing such alliances in support of programs are noteworthy in terms of generating new resources, ensuring local ownership and helping to promote a holistic approach to program design. As such, our findings here suggest that new efforts in the area of alliance building should focus on ensuring refined operational protocols for program design, approval and implementation through alliance structures and also a more realistic understanding of the “alliance management” component of activities as projects are designed and budgeted. Finally of interest to future efforts, the study sees an evolutionary change in terms of the relations between NGOs and businesses and despite the notion that fundamentally, behavior change will take time. In this respect, the study acknowledges that YEEI has encouraged local NGOs to reach out to corporations despite a sense of mistrust between two sectors and seeded a culture of public-private partnerships with a shared goal to promote youth employability. The study finds strong evidence that partnerships established between the two sectors are likely to be sustainable beyond YEEI for a majority of projects, for example, local businesses in Pati are continuing to work with the implementing partner and young entrepreneurs, and BP Indonesia has continued to provide support to young entrepreneurs as part of the local Committee modeled on YEEI (see sustainability status chart under Indicator 1.3 for details on sustainability of each subgrant project). The study does not find, however, specific evidence that other corporations have reached out to YEEI NGO partners to support similar interventions at this time. While this was not a main focus of the study, it can be surmised that partnership initiation around corporate social responsibility (CSR) may be more driven by civil society, who can in turn provide direct avenues for corporate support and participation in initiatives that are of interest to them. Indicator 1.1: In kind and cash resources leveraged for YEEI through private sector, government, civil society and other partners YEEI has successfully mobilized 65 public and private sector organizations in Indonesia with a shared sense of purpose to promote youth employability. They provided both cash and in-kind contributions including staff time, trainers and technical assistance, training venues and facilities, land, equipment and internships. With an initial investment of $581,589 of seeding funding for eleven projects, YEEI raised $1,272,310 in leverage contributions – thirty percent cash and seventy percent in-kind contributions. YEEI received 56% of total leverage from its NGO partners, 26% from its multinational and local corporate partners and 18% from the local governments. The figures below present seed funding and leverage contributions for each YEEI activity, and the classification of in-kind leverage contributions. In most activities, leverage was considered as cost sharing by project partners. Significant efforts were made by the Secretariat to ensure that leverage contributions were made as pledged so that all activities had the necessary resources to achieve desired program outcomes, particularly internship and job placement activities. The Secretariat specifically facilitated the signing of MOUs for each project to properly document and monitor contributions from all partners involved. Even though MOUs were not legally binding, most partners honored their commitments and contributed both cash and in-kind contributions. Two projects in particular faced difficulties securing leverage mainly because of budgetary constraints of and challenging relationships with corporate partners. The total leverage amount does not include in-kind contributions (mainly time spent) from Steering Board members. Figure 2: Seed Funds and Leverage Contributions Implementing Partner/Project Seed Funds Leverage Total Budget Leverage Ratio KAKAK $38,680 $77,596 $116,278 2.01 Prestasi Junior Indo $38,415 $121,605 $160,021 3.17 KUD Mina Bahari $47,515 $146,969 $194,484 3.09 Bina Swadaya $33,632 $51,492 $85,124 1.50 Kehati/Wisnu $48,851 $101,263 $150,114 2.07 CREATE $71,086 $203,086 $203,086 2.86 East Java Consortium $74,360 $182,812 $257,172 2.46 JBL $84,850 $126,822 $211,672 1.50 SEKAR $41,400 $49,827 $91,227 1.20 PKBM Cepat Tepat $68,800 $95,239 $164,039 1.38 Business Dynamics $34,000 $115,600 $150,600 3.4 Total $581,589 $1,272,310 $1,854,902 2.19 Figure 3: Inventory of In-kind Leverage Resources Indicator 1.2: The number of private sector, government and civil society partnerships formed through YEEI to jointly offer employability skills training Of YEEI’s 65 alliance partners, 49% represent international and local corporations, 31% local NGO implementing partners, 14% local government units, and 6% local and international donor organizations. A total of 26 project partners shared the reasons why they participated in the alliance during the evaluation process and results are as follows: • To help youth acquire relevant employability skills (13% of respondents) • Youth is part of our institutional concerns (19%) • Corporate social responsibility (6%) • To recruit qualified people for our businesses by providing internships and hiring skilled graduates of YEEI (62%) The figure below presents all YEEI alliance partners by types of organizations, showcasing an array of partners that YEEI attracted and worked with to improve the livelihoods of youth and their families: Figure 4: Typology of YEEI Partners and Leverage Contributions Partner Name Partner Role and Contributions Corporations BP Board member and resource partner; provided funding for West Java community economic empowerment projects GAP Board member and resource partner; provided funding for Yogyakarta entrepreneurship project GE Board member Nike Resource partner; provided funding for the Nike life skills training Oracle Resource partner for the Junior Achievement project; provided Think.com software Newmont Indonesia Resource partner; provided funding and business volunteers for the Junior Achievement project Rio Tinto Indonesia Board member ACCOR Indonesia Resource partner; provided training on hotel hospitality and management and food processing for Bali ecotourism projects Medion Resource partner for the CREATE project; provided training supplies for the project Vaksindo Satwa Nusantara Resource partner for the CREATE project; provided training supplies for the project Kurnia Poultry Farm Groups Resource partner for the CREATE project; provided training supplies for the project Perdana Putra Chicken Resource partner for the CREATE project; provided training facilities and internship opportunities Partner Name Partner Role and Contributions Charoen Phokpand Resource partner; provided funding for the CREATE project PEAC Bromo Resource partner for East Java projects; provided business consultancies to the newly established business ventures C-59 Resource partner for the JBL project; provided instructors and facilities for t-shirt making training and t-shirt franchise training Telkom – Regional 3 Resource partner for JBL project; provided training facilities for the project TA TV Solo Resource partner for KAKAK tempe chips project; supported the project and marketing of tempe chips Yamaha Provided training instructors, internships and employment opportunities for the Cepat Tepat project Suzuki Provided training instructors, internships and employment opportunities for the Cepat Tepat project Toyota Astra Provided training instructors, internships and employment opportunities for the Cepat Tepat project Honda Provided internships and employment opportunities for the Cepat Tepat project Daihatsu Provided internships and employment opportunities for the Cepat Tepat project Government West Java Provincial and Kabupaten governments Contributed land, equipments and technical assistance for the Cepat Tepat, BSI and KUD projects Local government of Pati Contributed land and meal stipends for the CREATE project Local government of Bandung Provided training supplies and facilities for the JBL project Local Kabupaten governments in Bali Contributed equipment and facilities to support community product development Non-governmental Organizations Indonesia Business Links National Secretariat and Implementing Partner Ford Foundation Board member Kehati Foundation Lead implementing partner; contributed technical assistance and cash leverage Wisnu Foundation Implementing partner; contributed staff time and office facilities for Kehati project TIFA Foundation Board member Junior Achievement Implementing partner; provided staff time and office facilities KUD Mina Bahari Implementing partner; provided funding, staff time, office facilities, and factory land Bina Swadaya Implementing partner; provided staff time, office facilities and vehicles Jendela Budaya Lingkungan Implementing partner; provided staff time, and training materials SMK Farming Pati Implementing partner; provided staff time, training venues and facilities Business Dynamics Training provider for the Nike life skills training SEKAR Implementing partner; provided staff time, training venues and facilities PUPUK Surabaya Implementing partner; provided staff time and office facilities ALIT Implementing partner; provided staff time and office facilities YP3IS Implementing partner; provided staff time, training venues and facilities PKBM Cepat Tepat Implementing partner; provided auto mechanics training facilities and instructors PKBM Aditya Implementing partner; provided sewing training facilities and instructors Bina Mitra Provided supervision to the newly created sewing business group under the BP West Java project in Tanjung Sedari Partner Name Partner Role and Contributions Yayasan Kurnia Resource partner for the SEKAR project; provided training equipment and facilities Gallery K’Qta Resource partner for the SEKAR project; provided training equipment and facilities and product promotion support ROMPI Resource partner for the SEKAR project; provided training venues and equipment KAPPIJA-21 Resource partner for the SEKAR project; provided training equipment and facilities and stipends for trainers Overseas Development Organizations USAID/Indonesia Board observer, program contact, strategic guidance and principal funding source International Labor Organization Board member Indicator 1.3: The effectiveness of the overall alliance approach in building partnerships, and promoting sustainable and scalable youth employability programs The study finds that the development and use of alliance-based approach to guide implementation of projects required ample time and significant resources and attention from all YEEI partners during the first year start-up phase. Such efforts included -- creating and developing a common understanding among alliance partners, building and reinforcing initial alliance partnerships, establishing a Steering Board, conducting due diligence for local businesses, and providing capacity building for the Secretariat and implementing partners. One particular challenge in this respect was a requirement to conduct due diligence for over thirty corporations, which significantly delayed the subgrant process and affected the quality of training for subsequent projects because of the need to shorten the duration of training. Nonetheless, survey responses and analysis indicate that this time taken at the outset of the program helped ensure efforts were comprehensive, effective, generally cost-effective and reasonably sustainable. In particular, findings suggested the alliance structure was able to help strengthen the capacity of local NGOs to implement alliance-based projects, secure active participation from the private sector, foster meaningful collaboration between NGOs and businesses, and achieve high job placement rates. Additionally, all partners surveyed felt that YEEI improved their capabilities to work with youth, and that their YEEI experience positively changed the way their organizations work. This section focuses on the effectiveness of the YEEI program model, specifically the value and effectiveness of the Secretariat, Steering Board and overall alliance approach together with the cost effectiveness of the model, in promoting sustainable youth development programs. Effectiveness of the Secretariat During the first year, IYF invested significant resources in putting in place the necessary sub-award monitoring system, and building the capacity of the National Secretariat - particularly in areas of building and reinforcing alliances, leverage raising, conducting due diligence, financial management and overall program management (design, implementation, monitoring and evaluation). IBL subsequently shared its acquired knowledge and skills in these abovementioned areas with all implementing partners through capacity building workshops and regular technical support visits depending on areas where partners needed help. In terms of corporate outreach, the Secretariat also played a critical role in establishing linkages with local businesses for some projects with funded oversight. The ability to attract local businesses to join the alliance and play a substantive role has been noted as one of the major achievements of the program. Furthermore, the Secretariat learned how to negotiate and mediate in case of conflicts as often demanded by alliance programs. During this evaluation, partners were asked to assess the technical assistance received from YEEI and results are presented below: ƒ Eleven percent of partners reported receiving technical assistance from the Secretariat in program management; another 11% in technical training; 22% in monitoring and evaluation; and 56% in financial management. ƒ Thirteen percent of partners rated the quality of technical assistance from IBL as fair, 63% as good and 25% as excellent. ƒ All partners have implemented traditional development projects that do not use the alliance approach in the past. They were asked to describe their alliance experience in comparison with traditional development models, and their responses are presented below: o Forty-four percent said it is similar; o Thirty-three percent said it is better because of an increased ability of the alliance to: ƒ Leverage more resources and provide more comprehensive support to youth; ƒ Build and maintain relationships at the project-level which are positioned for scale-up should additional resources become available; and ƒ Sustain program outcomes on a greater level, by enhancing the support and mentorship to project participants, particularly to new enterprises that have been established which continue to receive support either on an individual basis or as part of a local Committee established based on YEEI model; o Twenty-two percent said it is worse because of complexities involved such as the need to sustain alliances, raise leverage contributions, take a participatory approach demanding more resources, engage in conflict resolution, and to comply with reporting requirements. Value and Effectiveness of the Steering Board Before presenting the findings of the Board effectiveness, it is important to note that some YEEI resource partners such as GAP and BP volunteered to serve on the Steering Board based on their institutional interests. Development practitioners from international organizations also served on the Board as observers. The study shows that the Board made contributions to the program in three significant ways. First of all, a robust process put in place to encourage Board participation has resulted in meaningful collaboration among Board members and alliance partners with YEEI team playing a facilitation role. At the program outset, the Board met bi-monthly to review proposals, provided suggestions to strengthen program designs, and approved subgrant activities that would best support YEEI program goals. Secondly, after projects were initiated, the Board monitored program developments and provided strategic guidance to strengthen the program in subsequent Board meetings convened quarterly. Those who served on the Board and also provided financial support to projects were closely involved in program implementation and helped brainstorm ideas to strengthen select activities based on mid-term evaluation results in July 2008. Finally, it became an effective mechanism to hold both IBL and IYF accountable throughout the implementation, given the interest in multiple parties to program outcomes. It was also noted however that YEEI struggled to sustain active participation of some Board members after seed funds were allocated. With less focused activity to undertake in this respect, Board participation naturally waned. The Board, however, may not be dissolved after YEEI ends because several Board members have expressed interest in continuing to act as a force to promote youth employment in Indonesia, albeit in somewhat limited capacity. Value and Effectiveness of the YEEI Alliance Approach For a number of implementing partners, it was the first time establishing and working with alliances. Some projects initially faced challenges realizing leverage commitments, maintaining healthy working relationships and undertaking shared responsibilities because of YEEI’s push to forge vibrant alliances. Largely, with guidance from YEEI, the partners and programs appeared to have overcome these problems, with specific outcomes noteworthy below: • Mobilizing Local Resources: YEEI’s leverage requirement has encouraged lead implementing partners to look for corporate and NGO partners that they would not have found or partnered with otherwise. YEEI was also able to raise nearly $1.3 million in leverage which is two times more than the initial seed funding amount of $600,000 provided by USAID. • Communities Taking Ownership Leading to Active Participation: YEEI’s decision and efforts to encourage active participation of all partners resulted in partners engaging through several innovative avenues – such as collaboratively designing projects they supported, reviewing training curricula, monitoring project developments with YEEI and engaging in regular dialogues, and incorporating mid-term evaluation learnings to improve ongoing projects. • Engaging the Private Sector: YEEI engaged its corporate partners and prospective employers in the curriculum development stage for select projects which is found to be effective for ensuring that the training was relevant and demand-driven. Partnership with local corporations was critical in ensuring that skills training programs match the needs of the industry. Regular consultative and feedback meetings with internship providers allowed YEEI to monitor and improve the performance of trainees and increased their chances for job offers. In some cases, partners made arrangements with corporate partners to give competency exams to YEEI graduates and recruit them at the training site. These steps taken resulted in high job placement rates for most projects. • Sense of Community: YEEI encouraged all partners to work together and support each other. For example, in Indonesia, the cultural norm is such that it is more prestigious to have a stable job as civil servants or private sector employee than to become an entrepreneur. When the catfish farming project was setback by an initial period of drought followed by a flood resulting in significant damage and the loss of small catfish, young entrepreneurs were discouraged and faced with criticism in their communities. To the alliance’s credit, partners responded immediately by providing the necessary technical, and additional financial support to repair fishponds and buy new catfish. Interviews with young people indicated that such collaboration led the youth participants to realize they could indeed find support in times of need, which in turn increased their self-confidence and motivation. These twenty entrepreneurs have today created 140 additional jobs for the community. Additionally, YEEI has learned that conflict resolution and negotiations become more effective and manageable once partners are instilled in this sense of kinship and responsibility. While findings beyond leverage calculations are by necessity qualitative, they suggest capacity building efforts of the program around leverage and alliance building are successful and added value to the program. In addition, the requirement for proposals to have at least one corporate and one NGO partner subsequently helped nurture a culture of working in multi-sectoral partnerships. This has later evolved into a greater partnership of 65 public-private organizations working together to promote youth employability, and seems to provide a strong platform for continued collaboration and expansion of successful efforts in support of youth employability needs. The overall approach appears to have contributed to promoting long-term sustainability of program outcomes. Figure 5 below summarizes the sustainability status of each subgrant project: Figure 5: Sustainability Status of YEEI Projects Project Project Objective Partners Sustainability Status Livelihood Recovery through Tempe Production and Marketing To support 17 local entrepreneurs in Yogyakarta in producing and marketing modern snack foods from traditional products, creating 148 jobs KAKAK, GAP Corporation, Local Governments, the Center of Nutrition and Food Business association established as a legal entity and new businesses are anticipated to sustain themselves and expand their product lines under with help from YEEI partners Economics for Life in Vocational High Schools To support the teaching of Economics for Life lessons to 2,900 youth in Indonesian vocational high schools, to provide 250 students support in starting new companies, and to train 24 teachers in Oracle's Think.com platform for schools Prestasi Junior, Oracle Corporation, Newmont Corporation Phase 2 of this project is currently being implemented with the same partners and estimated to reach an additional 3,000 high school students West Java Coastal Community Empowerment To strengthen the cooperative Fish Filet industry in Eratan Kulon in West Java and creating 200 new jobs by providing equipment and training for villagers in better processing, packaging and marketing KUD Mina Bahari, BP Indonesia, Local Government The filet industry is anticipated to sustain itself with support from the local Committee established with YEEI multi-sectoral partners Project Project Objective Partners Sustainability Status Bina Swadaya Catfish Farming To provide entrepreneurship skills to twenty youth in Eretan Wetan village in West Java that assist them in starting up catfish farming businesses that create 95 additional jobs LSM Bina Swadaya, BP Indonesia, Local Government of Indramayu Kabupaten, PT. LS Aquaculture (LSA) The catfish businesses are anticipated to sustain themselves and continue expanding with support from the local Committee established with YEEI multi￾sectoral partners Cepat Tepat Youth Vocational Training and Entrepreneurs hip Project To equip 140 youth with vocational skills in the areas of motorbike repair and sewing and assist them in getting jobs or establishing small-scale businesses PKBM Cepat Tepat, BP Indonesia, the Local Government of District Karawang, PKBM Aditya, Bina Mitra, Toyota Astra, local automotive garages New businesses are anticipated to sustain themselves with help from YEEI partners. The project is anticipated to continue with support from the local education office and reach an additional 50 youth. Factory Employee Life Skills Training Project To train 16 trainers and 40,000 operators in eight factories supplying products to Nike in Jakarta, and improve life skills and positive attitudes in the work environment Business Dynamics, Nike Nike factories are implementing this project on their own using in￾house trainers YEEI trained, and estimated to reach over 30,000 new operators. CREATE Chicken Broiler Training Project To train 150 youth in broiler chicken operations, and of successful trainees about 90% will be employed in existing broiler operations and 10% will be assisted to start new small operations Center for Human Resource Development and Applied Technology (CREATE), Charoen Phokpand, Kurnia Poultry Farm, Perdana Putra, Medion, Vaksindo The curriculum developed with YEEI funds is still being used by CREATE. Corporate partners still supporting entrepreneurs and YEEI partners. Project unable to continue for lack of funding for the next phase. Proposals being developed based on the tested model of YEEI. Improving Livelihoods of Youth in Surabaya, East Java To improve the economic conditions of 260 disadvantaged youth in Surabaya by equipping them with necessary skills and helping them secure better jobs or establish sustainable businesses Pupuk Surabaya, ALIT, YP3IS New businesses are anticipated to sustain themselves with continued support from the partners. SEKAR Youth Vocational Training and Entrepreneurs hip Project To train 77 disadvantaged youth in Jakarta in waste recycling production of art paper and handicrafts and to start their own businesses SEKAR, PT. Astra International, Gallery Q'ta, Kurnia KAPPIJA-21, PT. Suhuf Art Paper New businesses are anticipated to sustain themselves with help from YEEI partners. The project is anticipated to continue with support from new funders in different target areas. Assisting Young T-Shirt Entrepreneurs in Bandung To assist 1,500 youth in obtaining jobs in the local T-shirt industry or starting their own franchises Jendela Budaya Lingkungan, Local Bandung Government, C-59, PT. Bio Farma, PT. Nu Art Consultant Company New franchises are continuing to receive help from C-59 and JBL. Unable to continue due to lack of funding. Supporting Community Ecotourism in Bali To strengthen the ecotourism industry in Bali by training 100 local villagers in village-based local food production, tourism services and handicraft production Kehati Foundation, ACCOR Indonesia, Village Ecotourism Network, Wisnu Foundation The villages are anticipated to continue using newly acquired skills as they run village tourism businesses. Cost Effectiveness: The overall cost effectiveness of the EEA Pakistan country program has been assessed at two levels. The first level is related to country program subgrants, which include both seed grants administered by USAID and cash and in-kind leverage contributions contributed by other partners to support EEA country programs. The second level more broadly includes costs classified as alliance building and program support costs. This second level includes costs outside of those directly incurred by project implementers. They accordingly include costs related to the functioning of the YEEI National Secretariat – IBL, local consultants and country advisor in Indonesia, as well as IYF direct and indirect expenditures attributable to the Indonesia program. Each of these cost categories are discussed in more detail below. Subgrant Project-level Costs: As noted above, subgrant project-level costs include seed funds and leverage contributions invested toward individual projects which typically covered curriculum design or improvement, training, job placement costs, and project specific costs of NGO subgrantees. The amount of seed funding support from USAID toward individual projects is a useful metric to assess cost effectiveness given that such funds are directly focused on implementation of projects, are directly leveraged by investments from other parties, and are best positioned for increased efficiency when additional beneficiaries are added to successful pilot programs. In Indonesia, the USAID support of approximately $581,590 in seed funds that were coupled with leverage investments of $1,272,311 reached a total of 13,202 beneficiaries. This results in a cost-per beneficiary of approximately $140 per beneficiary, of which $44 was borne directly by USAID and $96 was shouldered by the alliance partners. Figure 6: Project-level Costs Subgrant Project Name Number of Beneficiaries USAID Funds Leverage Total Project Budget Cost Per Beneficiary Kakak Entrepreneurship 17 $ 38,680 $ 77,596 $ 116,276 $ 6,840 BSI Catfish Farming 20 $ 33,632 $ 51,492 $ 85,124 $ 4,256 KUD Fish Filet Industry 20 $ 47,515 $ 146,969 $ 194,484 $ 9,724 Junior Achievement Vocational Training 3,232 $ 38,416 $ 121,605 $ 160,021 $ 50 Wisnu Ecotourism Training 86 $ 48,851 $ 101,263 $ 150,114 $ 1,746 CREATE Chicken Farm Operator Training 150 $ 71,086 $ 203,086 $ 274,172 $ 1,828 East Java Vocational Training 282 $ 74,360 $ 182,812 $ 257,172 $ 912 JBL T-shirt Training 1,558 $ 84,850 $ 126,822 $ 211,672 $ 136 Sekar Vocational Training 77 $ 41,400 $ 49,827 $ 91,227 $ 1,185 Cepat Tepat Engine Repair Training 105 $ 68,800 $ 95,239 $ 164,039 $ 1,562 Nike Life Skills Training 7,655 $ 34,000 $ 115,600 $ 149,600 $ 20 TOTAL 13,202 $ 581,590 $ 1,272,311 $ 1,853,901 $ 140 Alliance Building and Program Support Costs: A more expansive examination of costs by necessity also includes those alliance building and program support costs related to the YEEI program. These costs supported significant, collective efforts by IYF and IBL as National Secretariat, as well as local program consultants. Efforts by these partners included managing the design and implementation of alliance based interventions; providing international technical expertise on successful employability approaches; assisting in leveraging of resources; providing capacity building support to implementing partners in skills training, job placement and post training support to entrepreneurs; and providing financial oversight, compliance, and direct monitoring and evaluation support. Upon examination of IYF accounting records, these costs taken together with indirect costs attributable to the Indonesia program, totaled $1,083,896. This USAID-borne cost adds approximately $82 on a per beneficiary basis. Figure 7: Alliance Building and Program Support Costs Subgrant Project Name Number of Beneficiaries Alliance Building and Program Support Costs Cost Per Beneficiary Kakak Entrepreneurship 17 $ 98,627 $ 5,802 BSI Catfish Farming 20 $ 98,627 $ 4,931 KUD Fish Filet Industry 20 $ 98,627 $ 4,931 Junior Achievement Vocational Training 3,232 $ 98,627 $ 31 Wisnu Ecotourism Training 86 $ 98,627 $ 1,147 CREATE Chicken Farm Operator Training 150 $ 98,627 $ 658 East Java Vocational Training 282 $ 98,627 $ 350 JBL T-shirt Training 1,558 $ 98,627 $ 63 Sekar Vocational Training 77 $ 98,627 $ 1,281 Cepat Tepat Engine Repair Training 105 $ 98,627 $ 939 Nike Life Skills Training 7,655 $ 98,627 $ 13 TOTAL 13,202 $ 1,084,897 $ 82 Taken together, these overall costs of both seed funds and Alliance building program support costs total approximately $ 222 per beneficiary. These costs appear quite modest in relation to other good practice country projects of similar scope. Significant investments were also made to provide equipment and financial support to youth in setting up small businesses and in West Java, support a fish filet industry that created additional jobs for the community. Of the cost of $222 per beneficiary, $126 (57%) was borne by USAID and $96 (43%) was contributed by alliance partners. These cost calculations include eleven projects which provided job skills, entrepreneurship and life skills training. It is useful to note that one of these projects supported by Nike presented a different type training, focusing on providing life skills through short term training efforts to 7,655 beneficiaries employed at Nike contractor factories in Indonesia. Should this program be excluded from the calculation, the cost per beneficiary becomes $530 per beneficiary, of which $301 (57%) was borne directly by USAID and $229 (43%) was covered by alliance contributions. Finally, and perhaps most significantly, a full assessment of cost efficiency would also require a more thorough assessment of the consequences of alliance building activities on long term sustainability and scalability of proven practice programs. For example, should the new efforts to substantially expand the YEEI model be successful, the cost effectiveness of initial YEEI investments could be dramatically expanded. A modest investment in the initial life skills training project reached over 7,000 Nike factory operators after being scaled up and adopted by local factories, and it is expected to continue beyond YEEI. In this respect, as noted in the section on sustainability and section 1.4 below, should the new efforts to substantially expand YEEI in Java and Bali, the cost effectiveness of initial YEEI investments could be significantly expanded with a culture of working in alliances established and vibrant local alliances in place. Other activities that continue beyond YEEI support including the Junior Achievement and activities in Java built on the YEEI model are also expected to bring additional value and increased cost efficiencies as the necessary infrastructure and curricula developed and proven successful through YEEI. While the longitudinal study required for this type of evaluation goes beyond the scope and resources available for this study, such factors should be addressed more directly in similar employability projects in the future. Indicator 1.4: New partnerships, mechanisms and approaches created as a rippling effect of partners working with YEEI to promote youth employability While YEEI has not been able to acquire new funding to replicate the whole YEEI structure in Indonesia, sustainability prospects on the project activity level are promising and appear to have been enhanced by the development of alliance based approaches in support of program design, development and implementation. As an overseer of implementation and alliance program design, IBL is continuing to utilize and expand the alliance building capacity it has developed under YEEI in many of its programs, and has a number of staff who are involved in implementing and supporting the YEEI program. Similarly, IBL continues to use the subgrant monitoring system put in place for other projects they are implementing, and is well-positioned to implement future alliance-based projects. In addition, organizations such as World Vision Indonesia approached YEEI to learn how to implement alliance-driven vocational training programs with strong support from the private sector. More specifically, however, a number of partners remain enthusiastic about continuing alliance programs that were piloted and tested during YEEI. Such interest and activity in support of program continuation is among the strongest rationales for alliance building approaches to be utilized in such programs, given the promotion of avenues for local communities and their supporters to take ownership for a number of projects. Below are illustrative examples: • Junior Achievement project is continuing with support from YEEI partners – Oracle and Newmont – with plans to reach 3,000 additional youth in vocational schools. • A local steering committee has been established based on the EEA model to provide continued support to youth-led businesses in West Java with active participation from YEEI partners (BP Indonesia, local governments and KUD Mina Bahari) after the program ended. BP and the local government recently provided additional land and financial support for young entrepreneurs to expand their business operations and establish a new factory producing fish feed. The factory is quickly growing and producing fifty kilogram of fish food per day with help from three additional employees. • A joint proposal has been agreed upon in principle to replicate and expand the chicken operators training program to other promising areas in Indonesia. Partial support from at least one location has already been obtained by CREATE, and IBL is assisting CREATE in obtaining broader alliance support. • The Champion Workers life skills program continues in Nike factories with training workshops carried out by Nike in-house trainers trained by YEEI. It is anticipated to reach over 30,000 additional operators in the coming years. • Bali eco-tourism activities will continue in four target villages with people using skills provided and systems put in place by YEEI. Implementing partners, Wisnu Foundation and JED, are well positioned to continue providing support to these four villages. ACCOR Indonesia has also expressed interest in continuing to support the activity, though likely to be in somewhat limited capacity. With the financial crisis, villagers have been prepared not to rely heavily on ecotourism businesses in the coming months but focus on producing and marketing wine, rice, and coffee based on their natural resources with continued support from Wisnu Foundation and JED. • Across East, West and Central Java, most newly created businesses are anticipated to grow and sustain themselves creating additional jobs for their communities under leadership of YEEI graduates years after YEEI ends. Intermediate Result II: Improved Access to and Relevance of Education and Employability Training Summary of Intermediate Result Two Findings Survey findings suggest that YEEI was successful in mobilizing local resources and using these local alliance to provide quality employability skills training to underprivileged youth groups including orphans, street and rural youth. Through three intervention types (integrated employability, entrepreneurship, and life skills projects), YEEI benefitted a total of 13,718 people of whom 13,202 are direct youth beneficiaries and 516 are indirect beneficiaries who have taken additional jobs created. Of 13,202 direct beneficiaries, an aggregated 5,547 participated in an average of six-month integrated employability skills training and 7,655 participated in a two-day life skills training. Figure 8: Number of Direct and Indirect Beneficiaries Number of direct beneficiaries participating in employability skills training Number of direct beneficiaries participating in life skills training Total Number of Direct Beneficiaries Number of Indirect Beneficiaries Total number of both direct and indirect beneficiaries 5,547 7,655 13,202 516 13,718 The quality and relevance of training was rated high by graduates who also provided suggestions to further strengthen the program. Suggestions include longer training periods for select projects, improving some training materials, and the need to continue providing follow-up support to those who have started their jobs or enterprises. The study also finds significant improvements in capabilities of graduates in both technical and life skills areas based on triangulation results. Survey findings also find improvements in participants’ sense of identity, future aspirations, and the way they relate to the society. Indicator 2.1: Number of disadvantaged youth who participated in the YEEI-supported programs YEEI was mandated to serve underprivileged youth between the ages of 18 – 24, with some flexibility for slightly higher or lower age groups and gender considerations. All projects used YEEI selection criteria to screen and select trainees during the program orientation and recruitment process. Selection criteria used were as follows: 1) youth between 18-24 years of age; 2) youth from economically disadvantaged families; and 3) youth who demonstrated aptitude, commitment and potential. The YEEI program was introduced through local government offices, youth organizations or implementing partner network organizations in order to reach underserved youth from different backgrounds and geographic areas. According to survey responses, 54% of participants heard about the program from their friends or family members who were YEEI alumni; 28% from youth associations; ten percent from local governments; and six percent from their former schools. Following the application process, implementing partners interviewed prospective candidates to assess their interests, motivation, education and other background information. Interviews found that there were no major challenges in the recruitment of the participants, except that each project struggled to limit the number of trainees based on proposed activities and budget. This was particularly because half of YEEI projects were designed to serve rural youth based on local needs resulting in high interest and more applications than anticipated from unemployed youth in target areas. The decision to support eleven projects also affected the number of beneficiaries reached and some YEEI projects reached less than 150 youth given substantial start-up costs including curriculum development. The surveys found that about half of YEEI beneficiaries are within the desired age group of 18-24; 34% are slightly below 18; and 18% are slightly above 24. Reasons noted were focused on the need to balance between reaching those in the target age group and helping those from economically disadvantaged families that both demonstrated interest and aptitude. A quarter of youth served were working in the informal sector prior to joining YEEI, and 75% of youth were neither in school nor working, meaning a majority of youth had no opportunity to engage in productive activities previously. YEEI benefitted a total of 13,718 people of whom 13,202 are direct youth beneficiaries and 516 are indirect beneficiaries who have taken additional jobs created. Of 13,202 direct beneficiaries, a total of 7,655 participated in a two-day life skills training and 5,547 (vs. 5,079 targeted) participated in an average of six-month integrated employability skills training. Of these 5,547 youth who participated in employability skills training, 52% are young women and 48% are young men, thereby achieving a roughly equal gender balance. Additionally, an aggregated 3,232 out of 5,547 were the first and second year vocational high school students who received entrepreneurship training, and are now continuing education. Indicator 2.2: Number of disadvantaged youth who completed the YEEI-supported projects Of 5,547 youth who participated in various employability interventions, 5,504 completed the training and subsequently received support from YEEI to find jobs or become entrepreneurs. YEEI experienced 45 drop-outs from four projects because they decided to pursue other educational opportunities, move or get married. In a few cases, trainees dropped out because felt that the training would not support their long￾term goals. All Nike factory workers who received two-day intensive life skills training graduated; however, the study was unable to get statistics by gender. The total number of youth participated and completed for all projects except Nike life skills project are presented in Figure 9 below: Figure 9: Direct Beneficiaries (Participated and Completed) by Gender and against Targets Target Total Youth Participating Total Youth Completing Number of Dropouts Number Percent Number Percent Number Percent Number Percent Women 2,507 49% 2,872 52% 2,844 52% 28 0.97% Men 2,572 51% 2,675 48% 2,660 48% 17 0.64% Total 5,079 100% 5,547 100% 5,504 100% 45 0.81% Figure 10: Program Graduates by Gender and Region Target Actual Number Percentage Number Percentage By Gender Women 2,507 49.4% 2,844 51.7% Men 2,572 50.6% 2,660 48.3% By Region Jakarta 2,977 58.6% 3,309 60.1% West Java 1,645 32.4% 1,669 30.3% Central Java 162 3.2% 163 3.0% East Java 260 5.1% 277 5.0% Bali 35 0.7% 86 1.6% As indicated above, YEEI was generally successful in achieving gender equality as envisioned – 52% of total trainees are young men and 48% are young men. Additionally, 49.4% or 2,507 of total program graduates are women and 50.6% or 2,572 are men. There were two projects (PKBM Cepat Tepat and CREATE) that were not able to achieve their target number of female participants. Explanations provided for this challenge included social norms and the nature of vocational/technical training provided such as automotive repair and chicken farm operators training. Figure 11: Participants - Target vs. Achieved Figure 12: Program Graduates by Region Indicator 2.3: Number of teachers and trainers trained under YEEI programs YEEI did not focus heavily on teacher training because most implementing partners were direct training providers that appeared to have a cadre of experienced trainers. As such, YEEI generally provided funds only to refine and improve existing curricula. One YEEI project that provided training for teachers was Prestasi Junior Achievement Entrepreneurship and Student Company project that trained 26 teachers (vs. 24 targeted) at five vocational schools in Jakarta on the Economics for Life (EFL) curriculum and the use of Oracle’s Think.com software. EFL is an activity-based entrepreneurship lesson plans integrated within the normal curriculum. Oracle’s Think.com intranet facilitates collaboration between students and teachers and promotes student company’s products and services. These teachers subsequently trained over 3,000 vocational students. One issue noted was that the Oracle training was delivered in one day and the materials were in English. PJI subsequently translated the materials into Bahasa Indonesia to help teachers better understand the training content. The assessment found that the Oracle training could be improved in two areas to make it more effectiveness: training duration and monitoring the use of Think.com software on the school level. The training should be made longer in the future and that there should be more frequent monitoring visits to ensure the successful implementation of Think.com on the school level. Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in YEEI-supported programs YEEI projects were designed to help youth gain practical knowledge and skills in order for them to find quality jobs or establish small businesses. Since YEEI’s mandate was to serve underprivileged youth such as out –of-school, rural and street youth, projects provided non-formal technical training which provided employability skills training to youth rather than engaging in formal educational activities. Training activities were designed based on local needs and market demands. Even though all training activities were varied in nature, they all entailed two fundamental areas: hard/technical skills and soft/life skills – followed by on-the-job training and job placement services or enterprise development support. Technical skills training entailed two components -- in-class and on-the-job training. In-class training activities lasted anywhere from three to six months followed by an average three-month on-the-job training. On the other hand, soft skills training was designed to complement the technical skills training so that graduates would know how to work in teams, solve problems and adopt good work ethics when they start working. Life skills topics range from time management, team work to communication skills and self confidence. Implementing partners used their own systems to track the performance of trainees over the project life and measure competency gains such as tracking test scores, performance-based scores based on field work and feedback provided by internship providers, or the number of graduates who received government certificates. For entrepreneurship programs, partners used observation and tracked the success of youth-led businesses to monitor their performance and competency gains. Survey results suggest that there are substantial improvements in capabilities of graduates across YEEI projects in areas of technical knowledge and skills, leadership and life skills. Using triangulation, test scores and overall patterns were also scrutinized for select projects to validate this finding. For instance, in the CREATE training program, average pre-test and post-test scores for the same group of beneficiaries were calculated for comparison purposes and significant improvements were observed – an average score of 4.5 before the training and 7.3 after the training. Another example is the Cepat Tepat motorcycle and car engine repair training project in Karawang – where all graduates passed the examination with high average scores (6.85 for car engine repair and 7.4 for motorcycle repair on a scale of one to ten) and have been certified by the local Education Office in Karawang. Similarly, 95% of graduates agreed that they have acquired the necessary non-technical or life skills to gain jobs and do their jobs well. Ninety-two percent of youth reported feeling more positive about their future as a result of participation in the YEEI program. Major improvements are seen in the following domains: communication skills, listening skills, self-confidence and sense of initiative, and conflict resolution. However, the study discovers that not all projects placed an emphasis on providing life skills and therefore a small percentage of students did not fully feel that they have substantially improved their life skills. The assessment also finds that some trainees did not fully understand what life skills meant. This is because several projects delivered life skills lessons informally while students were undergoing in￾class and on-the-job training, and did not explicitly explain why it was an important component of the training. As noted previously, YEEI allowed partners to use their own curricula for both technical and life skills training which made it challenging to systematically measure competency gains. While there were no issues with the technical component, life skills training could be improved in terms of the content, delivery and performance monitoring. This might have affected the quality of life skills training delivered. Given the overall importance and value of life skills, future YEEI-like projects should standardize and deliver life skills lessons more seriously and systematically. YEEI also intended to prepare youth to take their place in the market as responsible, confident and productive adults with strong leadership and civic engagement skills. Therefore, the study looked at what effect the training has had on the way youth feel about themselves and their future aspirations. Of particular note, 98% of youth, who would have continued to work in the informal sector or look for opportunities, felt that they now have confidence and a marketable skill set, and overall felt more positive about their future. The study sees substantial improvements in their sense of identity and the way these out-of-school youth relate to their communities and vice versa. Having an alliance that supports them becomes a source of pride and inspiration for them to work hard and make a successful transition to responsible adulthood. This has become a driving force for them to look for more opportunities to strengthen their education and practical skills. • Fifty four percent of youth responded said that they wanted to pursue further education and work at the same time. • Eighteen percent of youth responded said they did not want to pursue higher education because they wished to work to earn money immediately. Figure 13: Level of Change in Youth’s Self-confidence Indicator 2.5: Level of satisfaction of young trainees with the quality and relevance of the training, counseling and services received through YEEI programs Survey results indicate that program graduates are generally happy with the quality and relevance of training provided as detailed below: • Ninety-four percent of graduates said the quality of training materials was either good or excellent. Only one percent of the surveyed was not satisfied with the training materials. • In terms of relevance of the training, 15% of graduates felt the training was excellent, 67% as good and 18% as fair. • Ninety-two percent of graduates received counseling, mentoring and job placement support from YEEI upon graduation. This appears closely linked with overall job placement success of projects. For example, four employment projects (CREATE, Cepat Tepat, JBL and YP3IS) achieved high placement rates between 79% and 90%. Figure 14: Youth Satisfaction with YEEI Employment Training Of 98 entrepreneurs surveyed, 25% received financial and business support from YEEI to establish their own businesses and 75% became part of youth-led group businesses supported by YEEI. A majority of YEEI entrepreneurship projects were designed to help youth form group businesses offering motorcycle and car engine repair services, making handicrafts, snack food, organic fertilizer, carpets and organic farming based on local demands. CREATE chicken farm operators training, Kakak tempe chips production and Bina Swadaya catfish farming projects in particular provided financial and business support to youth to set up small businesses on their own. With regard to youth satisfaction with YEEI entrepreneurship projects, when asked to rate the quality of business support received, 19% of participants said it was excellent, 72% as good and 9% as fair. Additionally, 97% of entrepreneurs surveyed felt that they had gained the necessary entrepreneurial skills to start a business upon graduation. Figure 15: Youth Satisfaction with YEEI Entrepreneurship Training Some youth also provided additional comments on the quality and relevance of overall YEEI training as follows: 1. YEEI has good training materials (reported by 100 respondents) 2. YEEI training has helped us get jobs (by 85 respondents) 3. YEEI training materials are easy to understand (by 97 respondents) While generally positive, a number of youth participants felt that the training materials and facilities could be improved. In particular, it was noted that the training was not long enough for them to acquire optimal jobs. Based on these comments, the team had discussions with implementing partners in order to investigate how these projects were initially designed and provide recommendations for improvement for future programs. The study found that most training programs are designed as basic, short courses rather than in-depth courses to help underserved youth get jobs right after the training. In some cases, the jobs that youth were able to find did not fully match the training they had undergone because not many jobs were available in the market and young people took what was available to them. The duration of the training was another reason stated because some participants felt that it should be made longer in order for them to further strengthn their skills. In this respect, the CREATE chicken farm operators training participants admitted that the overall training was very good; however, they felt that there were technical skills areas that were not covered by the condensed version of the program offered by SMK Farming Pati because of time constraints. Furthermore, projects such as Cepat Tepat, purchased basic tools for the training and field work, and some youth had to adjust using more sophisticated tools at workplace. In addition, for projects such as CREATE and Cepat Tepat, YEEI graduates had to compete with regular vocational students who typically undergo a much longer training. Indicator 2.6: Number and percentage of youth who completed YEEI training and report that they have referred to at least another youth to the program In order to determine the level of satisfaction of youth with YEEI projects, the assessment also asked whether graduates would refer other youth to YEEI projects and received responses from 85 youth. A total of 63 youth (84% of respondents) said they would be willing to refer the training to other youth. Nearly forty youth surveyed said that they had in fact referred this program to others. This also reinforces the earlier conclusion that a majority of participants had positive experiences with the program and were satisfied with the quality and relevance of training they received. Twelve youth (16% of respondents) said they would not because of their disappointment with the salaries received at entry-level positions. Of particular note in this respect, some of these respondents were vocational high school students from Junior Achievement Program and did not have an opportunity to apply their knowledge at actual work places, which should be considered for future applications of this program. Intermediate Result III: Improved Prospects for Employment and Successful Entrepreneurship Summary of Intermediate Result Three Findings YEEI has successfully placed 714 youth (84%) out of 850 vocational training graduates who were ready to get jobs, which represents 88% of the program target of 810 youth to be placed in jobs. Additionally, YEEI enabled 360 youth (83%) out of 436 entrepreneurship training graduates to successfully launch and sustain their own businesses. This represents ninety percent of the target of helping 399 youth become entrepreneurs and launch small businesses as presented in Figure 16 below.3 Figure 16: Overall Numbers and Rates of Graduates Employed and Self-Employed Against Targets Number of youth targeted to be placed in jobs Actual number of youth placed in jobs Job placement rate against the target Number of youth targeted to become entrepreneurs Actual number of youth who became entrepreneurs Entrepreneurship Success rate against the target Female 419 149 88% 148 123 90% Male 391 565 251 237 Total 810 714 399 360 More than half of the employers interviewed stated that YEEI graduates are “much better” than other graduates from similar backgrounds. Several employers also mentioned that YEEI graduates exhibit strong technical skills, work ethic, and willingness to learn. Over ninety percent of employers think that YEEI has been able to provide qualified graduates for the business sector, thereby closing the gap between employers’ needs and employees’ qualifications. Suggestions include further strengthening the technical and communication skills of some graduates and familiarity with more sophisticated equipment. Indicator 3.1: Number and percentage of YEEI trainees who get a job within six months of completing the program YEEI provided two types of employability training – employment and entrepreneurship based on local needs and interests. Through its employability interventions, YEEI intended to improve the employment status of 810 youth by helping at least sixty percent of them get quality jobs. Over the three-year program period, YEEI has successfully placed 714 youth in jobs, which represents 88% of the program target of 810 youth to be placed in jobs, and 84% of the 850 vocational training graduates who were ready to get jobs. This does not include 516 jobs that YEEI created through enterprise development. Effectiveness of Job Placement Services Ninety-two percent of survey respondents said they received counseling, mentoring and/or job placement services from YEEI. Program graduates who are working were surveyed to determine how long it has taken them to find jobs - 88% secured jobs within the first three months upon graduation and six percent received jobs within four to six months after graduation. This demonstrates the success of partners in placing youth in jobs in a relatively short period of time. Of all graduates who got jobs within the first six months, 77% are male and 23% are female. Only three people got jobs after 6 months upon graduation. As also discussed in Indicator 2.6, YEEI graduates are working in the following sectors: automotive and motorcycle repair (10.4%), chicken farming (10.4%), computer assembly, financing and seller (47.9%), T-shirt industry (12.5%), youth-led enterprises in Northern Jakarta (6.3%) and fish fillet industry (12.5%). 3 It is important to note that a good majority of the individuals benefited by the program were not eligible for job placement, either through the Nike Champion workers life skills training project for current workers or the Junior Achievement project for current students, which together numbered 10,887 beneficiaries. Geographic areas include: • West Java Province: Bogor, Karawang, Bandung • Central Java Province: Pati • East Java Province: Surabaya, Pasuruan, Sidoarjo, Jember, Sumenep, Gresik, Tulung Agung, Jombang • Northern part of Greater Jakarta (DKI Jaya) Capabilities of implementing partners were varied and the study notes that YEEI team made necessary adjustments to provide mentoring and other support effectively based on local needs. Some partners have come in with long institutional track records and their own networks, for example, YP3IS has an organization network of over 30,000 Zakat donors who are business owners or managers in Surabaya. The project strategically used the network to place youth in computer-related jobs. On the other hand, other partners requested more technical assistance and mentoring support from YEEI to establish alliances and improve program effectiveness. Overall, they needed more support with the job placement component which was relatively new to them. YEEI took a hands-on approach during the job placement period and invested significant resources establishing partnerships with local businesses aiming for high job placement rates. As previously mentioned, most participants were channeled into the job market by implementing partners through a mechanism that allowed partners to access, test and employ YEEI graduates. YEEI’s job placement success can be attributed to the following: 1. Industrial knowledge of implementing partners 2. Ability to respond to local markets by conducting high quality and relevant training (combining technical, life skills, equivalency education, and on-the-job training) followed by job placement services and follow-up support 3. Ability to develop and sustain good partnerships with local businesses that were experiencing strong growth, and engage them in curriculum review, internships and job opportunities 4. YEEI’s success in creating a culture of working effectively and collegially in alliances Change in Employment Status: Among graduates surveyed, 37% had jobs prior to joining the program and 63% of youth did not have jobs. YEEI placed an emphasis on helping youth acquire decent jobs and better their future and this result suggests that YEEI has achieved this goal: 85% of youth who were previously working reported that they are now earning more income as a result of participation in the YEEI program. Indicator 3.2: Number and percentage of YEEI trainees who continue education, obtain an internship or engage in further professional training within six months of completing program Ninety-seven percent of the respondents underwent internships while three percent were not able to find appropriate internships. The internship period was typically two months and afterwards graduates received support from YEEI to acquire a stable job with either internship providers or new companies. Substantial negotiations took place at this stage and this was the most stressful and busiest time of the project. Prestasi Junior Indonesia is the only project whose beneficiaries continue their studies after the training completes. The project provided entrepreneurship knowledge and opportunity to run school companies to vocational high school students in grades ten and eleven. Sekar street youth project also provided support to twenty of its beneficiaries to receive education equivalency program (i.e. elementary and secondary level) because most of them are street youth who dropped out of school. There are two projects (CREATE and YP3IS) where six graduates decided to continue further studies upon completion of the training. Indicator 3.3: Number and percentage of YEEI trainees starting their own businesses within six months of completing the training YEEI enabled 360 youth (83%) out of 436 entrepreneurship training graduates to successfully launch and sustain their own businesses. This represents ninety percent of the target of helping 399 youth become entrepreneurs and launch small businesses.4 Specifically, 35% of 360 entrepreneurs were women. They 4 It is important to note that a good majority of the individuals benefited by the program were not eligible for job placement, either through the Nike Champion workers life skills training project for current workers or the Junior Achievement project for current students, which together numbered 10,887 beneficiaries. formed 32 business groups of which each group typically consisted of five to ten people. A small portion of these start up entrepreneurs originally graduated from the employment training projects. It is part of the YEEI initial design with the implementing partners of employment projects in particular to facilitate training graduates who demonstrate aptitude and interest to start their own businesses or those who are unable to find jobs in areas where jobs are extremely scarce. Figure 17: Entrepreneurs by Projects and Gender Female Male Total Entrepreneurship Projects KAKAK 7 5 12 Bina Swadaya Indramayu 9 11 20 Wisnu 21 9 30 PUPUK Surabaya 15 33 48 ALIT 39 85 124 SEKAR 14 61 75 Employment Projects CREATE - 5 5 Jendela Budaya Lingkungan 8 3 11 PKBM Cepat Tepat/Aditya 10 25 35 Total 123 237 360 Upon completion of the vocational training, graduates received financial, business and other support from implementing partners to set up small businesses or business groups depending on initial project design. In some projects, they were provided with working capital such as raw materials and production tools. Besides working capital, the implementing partners continued to give technical supervision and business management training to the youth, such as marketing, bookkeeping and business development. YEEI struggled to implement entrepreneurship projects in the beginning due to three reasons: 1) cultural norms that discourage youth to become entrepreneurs; 2) external shocks such as natural disasters and sharp increase in consumer prices that threaten the existence of enterprises; and 3) lack of the same level of sufficient experience of some partners to implement entrepreneurship components of their overall employability interventions. The study finds that the following remedial actions have helped YEEI overcome these challenges and contributed to the success of newly established businesses: 1) Providing hands-on support and strategically using the power of alliance to keep young entrepreneurs stay motivated and focused; 2) Mobilizing additional resources to counterbalance these shocks and reinstate project activities; and 3) Hiring business coaches to strengthen and sustain businesses in precarious situations. Meeting the need for substantial, hands-on mentoring support During its mid-term evaluation, YEEI learned that capacity building workshops alone were not enough and that partners needed more hands-on mentoring support in the first six months after businesses have been established. Therefore, it took an important step of hiring local business coaches to help businesses that were struggling in Yogyakarta, Jakarta and West Java and improve their product development, marketing and sales. They included tempe chips production businesses (affected by recent increase in consumer prices such as cooking oil and tempe), organic fertilizer business run by former street youth, and fish filet industry in West Java. YEEI business coaches specifically helped the fish filet industry develop a new product (crispy fish snacks) and market it effectively with support from the local government. The coach also helped tempe chips entrepreneurs diversify their products and establish as a legal cooperative so they could get access to credit. Sekar youth engaging in organic fertilizer business have decided to also make recycled art paper to fulfill overwhelming demand that other groups cannot meet in North Jakarta. The coach also helped this business group to get access to credit so that they could increase their capacity and acquire new tools to make recycled art paper. Sustaining new enterprises and attitudinal changes The study concluded that ninety percent of newly created businesses met and surpassed benchmarks established including carrying out production and sales, maintaining an appropriate bookkeeping system, making profits, and sustaining after six months of establishment. Bina Swadaya project in particular has become the most successful entrepreneurship project model and youth-led businesses been able to create 140 additional jobs for the community. They expanded their business operations with continued support from YEEI partners - added six additional catfish ponds and established a small fish feed production businesses. During the project implementation, YEEI noted that creation of sustainable, successful new businesses needs more time than one year. It entailed not only equipping the youth with technical and entrepreneurial skills but also changing their mentality to become successful, confident decision makers and entrepreneurs. Despite this challenge, YEEI has reached agreements with several implementing partners that they will continue providing informal assistance to these entrepreneurs to the extent necessary after YEEI ends, which will hopefully continue to nurture the development of these young entrepreneurs. Indicator 3.4: Level of employers’ satisfaction with the soft skills and/or technical skills of YEEI trained youth A total of 19 employers participated in this evaluation representing the following sectors: automotive and motorcycle repair shops (31.6%), chicken farming or poultry industry (31.6%), computer assembly and seller (5.2%), T-shirt industry (15.8%), handicraft business (5.2%), training institutions (5.2%) and financial sector (5.2%). The study also attempted to determine how employers came to know about the program through the employer survey: 63% were approached directly by partners, 13% learned about the program from program brochures and 25% said the program was introduced and referred to them by their contacts. In terms of employers’ satisfaction, 69% of employers rated the overall performance of YEEI graduates as good and 31% as fair. In terms of performance in this area, as mentioned previously, a number of projects provided basic, short-courses preparing out-of-school youth for work rather than 2-or 4-year vocational school programs. As such, while some employers use quite sophisticated technology and equipments and the evaluation suggests that youth did not have such experience nor the time to necessarily adjust the skills they had to support them in this area. In turn, YEEI has made adjustments in their process, with arrangements made with implementing partners so that graduates are given an option to continue to receive some additional training to further refine their skills during on-the-job training and when they start working. In addition, employers were asked to compare YEEI graduates with other employees from similar age range and backgrounds. 56% of employers think that YEEI graduates are much better and 44% think they are similar. Several employers also mentioned that strong technical skills and their willingness to learn and work are strengths of YEEI graduates. Other employers, however, noted that weaknesses include the need to improve technical skills of some graduates they have hired and to further enhance communication skills to work effectively in professional settings. Some survey results worth highlighting are as follows: • Ninety percent of employers think that YEEI has been able to provide qualified graduates for the business sector and it is closing the gap between employers’ needs and the employees’ qualifications. • Ninety-three percent of employers said they would consider offering jobs, internships or apprenticeships to YEEI graduates in the future. • In terms of how companies recruited youth, 63% of businesses interviewed were approached directly by YEEI partners and 13% learned about the program through brochures and posters. Twenty-five percent of employers said youth were referred by their contacts. • The study also assessed how these companies recruited YEEI graduates for internships and/or jobs and results are as follows: 78% through YEEI partner NGOs that submitted resumes on behalf of graduates, 11% through company outreach and competency tests given at the training site and 11% through internship or job applications sent directly by graduates. • It also attempted to determine whether YEEI program has had any effect on the human resource management practices of companies. Sixty-three percent of employers said that their companies plan to become more involved in YEEI-like alliance-based programs focusing on youth employability. Twenty-five percent said that they are more open to the idea of providing internships and mentorship support to youth in general. Thirteen percent stated that they are likely to recruit youth through YEEI and other similar programs as a result of participation in the YEEI program. Employers were also asked to provide suggestions and comments on the overall training - 57% of employers stated that soft skills and some areas of technical skills can be further strengthened. Twenty￾nine percent said the training should be made longer so youth can acquire more experience and confidence. Fourteen percent said that the training is already very good and that it should be continued. Employers of youth graduated from engine repair training and chicken farm operators training added that they should develop familiarity with the most advanced technology and tools particularly for the car and motorcycle engine repair and chicken farming. Figure 18: Employers’ Satisfaction with YEEI Graduates Indicator 3.5: Number and Percentage of Youth with Internships and Apprenticeships Implementing partners’ good relationships with companies have resulted in a high internship placement rate of 97% for YEEI. These are small to medium businesses such as chicken farms, auto-repair shops and other industries. During the internship period, YEEI partners met with employers on a regular basis to monitor the progress and performance of the interns and the trainees to provide counseling and plan for job placements after the internship. Below are some survey results worth noting in terms of internships on the program: • When asked to rate their internship experience, 29% of youth surveyed said it was excellent, 53% as good and 18% as fair. Seventy-three percent were male and 27% were female. • In terms of the relevance of assignments to the training received, 53% said it was good, 12% as excellent while only 1.5% said it was poor. • As for technical skills learned, 56.7% surveyed stated good, 6.9% excellent and another 30% stated fair. • In terms of coaching received during internships, 75% mentioned coaching was good, another 10% stated that it was excellent, and the remaining 9% said it was fair. What they found most useful was the opportunity to learn new technical skills or refine their skills learned in classrooms. Below are some additional comments from the graduates: • It provided the participants with knowledge and skills they have not learned during the training (reported by 16 people). • It was an opportunity to practice skills they have learned in classrooms (reported by 15 people). • It provided additional experience (reported by 12 people). • The internship does not fit with the training (reported by 1 person) because he had different expectations after the internship period to be directly employed at the firm and was disappointed when he was not. Indicator 3.6: Number and percentage of placed youth who declare having a satisfying and quality job Most of the graduates were channeled for jobs with assistance from the implementing partners who had made agreements with businesses to offer internships and jobs to graduates. Existing relationships based on mutual understanding enabled implementing partners to monitor the performance of the graduates regularly. The monitoring also became a good strategy for the implementing partners to discuss the progress with the graduates: whether they were any technical or non-technical issues at work and how they could best handle them. Job evaluation survey results are presented below: • Eighty-eight percent of them reported that they are happy with their overall work environment. Twelve percent of graduates said they were unhappy because they had hoped for better-paid jobs or because they had to relocate to places far from home. • In term of relevance of training, 75% are happy and 13% are very happy with their field of study in relation to their current jobs. Only 12% are unhappy because their current jobs are not in the same field they focused on or lack of adequate equipment in the workplace. Some felt that the training could be extended to provide them with more experience. • Fifty-six percent are happy with the support they are receiving from their supervisors and/or peers whereas 38% are unhappy. Figure 19: Job Satisfaction of YEEI Graduates . Those who had previous jobs provided the following responses: • Fifty percent of them said that the opportunities for professional growth are better now; 30% stated that they are much better; and 20% claimed that the professional growth opportunities were the same compared to their previous jobs. • Eleven respondents participated in the salary level survey reported that 64% of them had similar salary levels, and 38% of them said the current salary is better or much better. This indicated that most of the graduates are satisfied and happy with their current jobs. Those who are not satisfied with their current jobs (12% of the respondents) reported their reasons as follows: • Disappointment with lower salaries than expected; • Current jobs not being in their interests; and • Current jobs not fitting with the training. Those who had previous jobs were asked to compare their former jobs with current jobs received after graduating from the YEEI program. Seventy-three percent of youth reported that their current jobs are either better or much better while 27% said they are similar. This further supports the conclusion that YEEI helped youth land better jobs and improve their livelihoods. Indicator 3.7: Number and percent of youth (who had prior jobs) earning more income as a result of participation in the YEEI program and net salary increase of youth Most of YEEI participants did not have a job before joining the program. Those who lived in rural areas were mainly doing unpaid work by helping run their parent businesses or small farms. Meanwhile those who lived in the large city like Jakarta and Surabaya were doing nothing or surviving as street singers. Among 115 respondents who provided their answers on the related job evaluation questions in the survey, 37.4% of them already had jobs before the training and 62.6% of them were unemployed before the training. Among graduates who reported their salaries: • Twenty-seven percent of youth are earning below IDR 500,000 (US$ 50) per month; • Fifty-two percent of youth are earning IDR500,000 – 1,000,000 (US$50-100) per month; and • Six percent of youth are earning above IDR 1,000,000 (US$100) per month. Figure 20: Income Levels of YEEI Graduates Participants receiving less than IDR 500,000 (US$50) are currently working in various sectors including T-shirt and fish filet industry. The level of income is mostly determined by their experience and scale of the businesses they work for. Six percent of respondents reported their previous salaries as uncertain because they were doing seasonable jobs in Bali. Most of these respondents are still attending schools and not fully involved in the ecotourism business managed by the village community. For surveyed respondents who already had jobs before the training program: • Eighty five percent of them reported that they are earning more money after completing the program. The salary increase depends on the sectors that they are in. For instance, one participant from Cepat Tepat project was able to get 45% salary increase. Meanwhile, a participant from JBL project was able to get 33% salary increase. • Graduates in Karawang and Pati are earning much more than the minimum wage in these areas. Those in Surabaya are earning close to the minimum wage. In many cases, employers also provide meals and housing to YEEI graduates/employees. • Fifteen percent of graduates surveyed reported that they did not earn more money after graduating from the program. There are two reasons: some of the graduates are switching from their old jobs to new sectors for them with minimal experience in the new area, so that they had no bargaining position in getting income increment. Another reason is that some decided to remain in the same field after joining the program to enhance their knowledge and skills. • Of 74 young entrepreneurs surveyed, 45% had jobs prior to joining the program and 55% did not. Those who had jobs before were asked whether they were earning more income with newly established businesses. Of 36 who responded, 42% said yes and 59% said no. • Eighty percent of surveyed youth-led businesses reported making profits. Sixty-eight percent of entrepreneurs reported that they would not have been able to establish businesses without YEEI support. Thirty-two percent of graduates said they would have been able to anyway with financial support from their parents and relatives. • Thirty-three percent of entrepreneurs reported that they have hired new employees for their new businesses. These new jobs created are for a mix of both young people and adults in the community. Some of them are part-time or seasonal jobs to help with tempe chips production or catfish harvesting. Jobs created in West Java’s fish filet industry are based on daily demand for Kuniran fish. Intermediate Result IV: Positive Indirect Effect on Families and Communities of Youth Summary of Intermediate Result Four Findings The study shows that the program is successful in helping most youth successfully enter the labor market and start earning regular incomes through new jobs or enterprises. With new incomes, graduates are beginning to financially help their parents, siblings and spouses/children for daily consumption needs, education and health expenses. They are spending their income wisely and saving part of their income for future investments, business expansions, and as emergency funds. More than half of the graduates surveyed reported that the financial and social situations of their families. YEEI-supported enterprises have also created more than 500 additional jobs that have generated primary or secondary incomes for the communities. Indicator 4.1: Number and percentage of employed youth financially helping their families for household, health and education expenses The study found that most of the youth share their income with their parents and siblings now that they are able to work and earn regular income through jobs or new businesses – as a way to repay parents’ gratitude and an act of becoming a responsible adult. Out-of-school youth in rural parts of Indonesia tend to get married early and a number of youth YEEI served are in fact married and have children. They tried to support their families by doing informal jobs prior to joining the YEEI program. Now with improved employment status, most of them are able to hold stable jobs with regular incomes that they can begin saving for their future, invest in their children’s education and cover health expenses. Figure 21: How Youth Use Their Earnings This assessment also studies how youth use their earnings and results are as follows: • Twelve percent of youth who are married and have their own families said they use their earnings to support their spouses and children; • Fifty-nine percent of youth are financially helping their parents and siblings to cover basic needs, education and health expenses; • Forty-six percent use their earnings for daily personal needs such as paying for rent, food, clothing and cell phone bills; and • Fifty-nine percent are saving part of their income for the future. Indicator 4.2: Number and percentage of youth who have reported improved financial and social conditions of their families The study seeks to determine not only how many youth are helping their families but how many feel like they now have ability to improve financial and social conditions of their families. • Fifty-nine percent of graduates stated that that their financial support (with the income they shared) helped improve their family’s financial and social situation. • Thirty-one percent claimed that their support somewhat improved the situation. • Ten percent said there was no change because their income is not enough to support their families yet after spending it for personal needs. The study also examines how the families are using the money shared by youth: • Eighty-four percent of families use the money primarily for basic needs and household expenditures. • Sixteen percent for other uses such as health and education for children in the family. Indicator 4.3: Number of jobs created for community as indirect beneficiaries of YEEI projects YEEI-supported enterprises have also created 516 additional jobs that have generated primary or secondary incomes for the communities around the youth. These jobs are typically part-time jobs depending on where these enterprises need help. For example, in the BSI cat fish farming project, additional workers are engaged in areas of the construction of catfish ponds, harvesting, sorting and marketing fish. This provides a good opportunity for the communities who are already in the same job market and want to take second jobs to meet their basic consumption needs or earn extra income. Businesses like BSI catfish farms – which were able to expand with continued support from the local steering committee - have demonstrated that they will be able to grow further and employ more youth from the community with continued support from the local alliance that YEEI established in West Java. Similarly, in Ereton Kulon village, the YEEI-supported fish filet industry has created additional jobs for the community of which an estimated thirty percent of indirect beneficiaries are women. These people supply fish fillets to the industry and earn more than Rp.1 million ($100) per month depending on the demand and their production capacity. Overall these jobs appear to be sustainable even after the program ended in 2008. Section IV: Challenges, Lessons Learned and Recommendations Part I: Building and Reinforcing Alliances The study finds that putting in place a robust process for Board and Alliance participation has encouraged meaningful collaboration at various stages of implementation, and improved the program design, quality and effectiveness (manifested in high job placements and successful enterprises). However, the establishment of such process requires considerable resources in terms of time, human resources for coordination efforts and others. In this respect, despite overall success, particularly in terms of sustainability through new partnerships, the intensive process of partnership development appears to have taken a toll on partners, with a number suggesting the process was too difficult. In addressing this basic challenge, it appears that one important strategy is to normalize roles and expectations within partnerships as quickly as possible. In this respect, as activities are implemented, YEEI learnings suggest that roles and responsibilities of each partner have to be clearly defined and agreed to achieve optimal results. Moreover, it is important to ensure a reasonably even distribution of roles and responsibilities, both in terms of fairness and also broadening the platform for expansion of initial program successes through multiple channels, In this respect, relying heavily on one player and implementing partner not having sufficient market information or expertise can have detrimental effects on the project if challenges arise, and, alternatively, will prove more difficult in ensuring sustainability or expansion if programs are ultimately proven successful. Additionally, YEEI has learned the importance of putting in place some measures that hold partners accountable and responsible – similar to how YEEI was held accountable by the Steering Board. The cultural context and multiple partnership arrangements often makes it uncomfortable for people to share unpleasant news, and in a few cases this affected the quality of projects delivered. Importantly, YEEI took remedial actions as issues surfaced during mid-term evaluations, and since then, has paved ways for open and honest communication channels. Recommendations • Train local staff on how to generate public interest and forge public-private alliances; reinforce alliances in appropriate cultural contexts; and negotiate and resolve conflicts if crisis arises. Knowledge transfer to local staff helps promote the scalability and sustainability of alliance-based projects. • Dedicate sufficient program management support staff to fill the various roles that alliance programs demand. • Recruit a group of like-minded civil society and private sector leaders to serve as champions and a collective force to promote youth employability. • Establish a process where both Board and Alliance members are given an opportunity to engage and contribute to the program. Soliciting their feedback is critical for all stages of the implementation, particularly program design and post-midterm evaluations. In case Board or alliance members have time constraints to attend regular meetings, make an effort to have a telephone conversation or individual meeting to solicit their feedback and guidance. • Develop alliances based on mutual respect and understanding. If possible, go beyond legalistic, formal partnerships and build vibrant alliances with a shared sense of purpose and responsibility. To the greatest extent possible, leverage local cultural approaches that encourage caring for and partnering with local communities to achieve social good. • Clearly define roles and responsibilities of each alliance member and keep each other accountable and responsible throughout the project. • Encourage knowledge sharing and open communication among partners. Part II: Co- Designing and Implementing Effective Alliance-driven Employability Programs YEEI’s approach to review initial concept notes and co-create projects (such as brainstorming ideas to improve the design and seeking corporate support if absent) with local NGOs has worked well. For CSR community empowerment projects in hard-to-reach areas, conducting community research and then co￾designing the project with corporate partners and local community development experts has proved to be helpful in ensuring that economic activities are locally driven, relevant and viable. Involving all stakeholders in the design and implementation stages has encouraged them to take ownership and also has helped increase program potential. YEEI’s strategy to design and provide tailored comprehensive skills training based on the needs of youth has helped graduates are fully prepared to enter the job market or start new businesses. Soliciting feedback from youth is imperative and projects that have done so appear to have performed better. Regular feedback meetings with internship providers are helpful because it helps youth improve their performance and increase their chances of getting jobs. YEEI has however learned that providing the follow-up support to out-of-youth (who are employed or running businesses for the first time) is important, and therefore, incorporated this into program designs so that youth can make a successful transition into the labor market. YEEI learned halfway that workshops alone were not sufficient to build the capacity of local implementing partners and substantial mentoring was required for entrepreneurship activities because of varied degrees of capacity on the ground. Retrospectively, the program should have closely assessed the capacity of each implementing partner to determine areas where they need assistance at the program outset. Hiring local business coaches have enabled the projects to take corrective actions for businesses that were in precious situations, and strengthened their business activities. Recommendations • Engage all key stakeholders during the design stage – particularly training providers, prospective employers and local experts – to increase the quality, relevance and effectiveness of the training. • Tailor comprehensive skills training activities (combining life and technical skills, computer literacy, equivalency education and on-the-job training) based on local needs, followed by internship and job placement support. • Integrate the follow-up support as a key component of program designs and provide the follow-up support at least for the first six months as youth enter the labor market. This can also help correct cases of skills mismatches. • Engage youth in meaningful ways. Solicit feedback and consult with them on a regular basis. • Assess the capacity of implementing partners at the program outset to see where they need assistance. If there is no in-house expertise in a particular industry area, engage outside experts who can provide substantial mentoring support to young entrepreneurs at least for the first six months of establishing enterprises. Set program benchmarks to monitor the progress of new businesses. And be prepared for external, unanticipated shocks that can seriously affect business operations. Part III: Engaging the Private Sector YEEI’s ability to engage the private sector for program design and employment support is absolutely crucial for the program success. Having potential employers review and approve the curriculum materials confirms that the skills training provided are demand-driven and relevant. Giving local business partners access to a pool of qualified graduates and ability to test and recruit them through competency exams or internships have led to high job placement rates. Regular feedback meetings with employers have also enabled the program to improve the quality of graduates produced. Recommendations • Find innovative ways to mobilize and recruit local businesses – such as outreach visits and calls presenting the program and benefits of joining the alliance. • Engage them meaningfully tapping into their knowledge and expertise, and involve them in curriculum design and pilot activities. • Give them access to program graduates for internships and job opportunities. Ask for regular feedback regarding the performance of graduates and improve the quality and relevance of the training. Annex A: Summary of YEEI Projects The following table provides information on main activities, total direct youth beneficiaries (both target and achieved), and location of each YEEI project: No Implementing Partner Main Objective Location Target Achieved 1 Yayasan KAKAK To help youth and farmers in earthquake affected areas near Yogyakarta to generate additional income and new employment in the local food snacks (tempe) industry Klaten (Central Java) 12 12 2 Bina Swadaya Indramayu To provide youth on the northern coast with access to entrepreneurial education and financial support to establish small catfish enterprises through better use of sustainable local resources Eretan Wetan Village, Indramayu (West Java) 20 20 3 KUD Mina Bahari To support the establishment of a local fish fillet industry in the village to expand local youth employment opportunities and increase income through the better use of sustainable resources in coastal communities in West Java Eretan Kulon Village, Indramayu (West Java) 20 20 4 Prestasi Junior Indonesia To provide high vocational school students with essential knowledge and confidence to undertake structured entrepreneurship in the future and improve their employability prospects Jakarta 2,900 3,232 5 Yayasan WISNU To provide ecotourism and management training to youth to make them become productive and engaged in the local ecotourism business and reduce urban migration Four Villages in Bali, namely Tenganan Pegiringsingan, Sibetan, Nusa Ceningan and Kiadan-Pelaga 37 86 6 CREATE To equip rural youth with the necessary skills to take advantage of career opportunities in the broiler chicken industry Pati (Central Java) 150 150 7 East Java Consortium (PUPUK Surabaya, ALIT, YP3IS) To invest in the neediest youth, particularly orphans and street youth, and help improve their education and skills to break into the job market or start their own enterprises, and begin productive working lives Surabaya (East Java) 260 282 8 Jendela Budaya Lingkungan To equip disadvantaged youth with essential knowledge and technical skills to obtain gainful jobs in T-shirt industry Bandung (West Java) 1,500 1,558 9 Yayasan SEKAR To empower the neediest youth in North Jakarta to become successful entrepreneurs with business focus on organic composting, making handicrafts, carpets and recycled art paper through the provision of necessary knowledge, skills and financial support North Jakarta 77 77 No Implementing Partner Main Objective Location Target Achieved 10 PKBM Cepat Tepat To empower underserved local youth by providing essential skills to become employed or self-employed in the automotive and motorcycle repair industry Karawang (West Java) 105 105 11 Business Dynamics To provide “Champion Worker” training programs to factory operators in the Nike suppliers to improve their life skills and positive attitudes in the work environment Jakarta, Banten (West Java) 1,250 7,655 Annex B: Nike “Champion Worker” Life Skills Training Evaluation Background Information on the Champion Workers Training YEEI provided life skills training to 7,655 operators at Nike factories in Indonesia aiming to help them improve their life skills, become more professional, affect attitudinal change in order to achieve a balanced life between work and home. Life skills areas included improving communication towards positive relationships at home or work, listening skills, seeking support from others to create win-win situations, managing stress and creating a balanced life. Evaluation Methodology and Objectives This project evaluation was conducted separately by PT. Tibies at three factories because of its specialized programmatic focus. This study was conducted to measure knowledge gains, behavior changes and challenges that operators faced in applying newly acquired skills and overall program effect. A different set of surveys and protocols were developed and used to gather information from direct beneficiaries, a control group and managers of Nike factories. Evaluation Findings A total of 150 operators (for both control group and YEEI-trained groups), supervisors and managers participated in the evaluation conducted in January through March 2009. Respondents were randomly selected from the group trained directly by YEEI implementing partner Business Dynamics and another group trained by factory in-house trainers (trained by Business Dynamics). Eighty-eight percent of survey respondents were female and 12% were male. Most were high school graduates in their twenties with the level of work experience around one to three years (32%) and seven to ten years (59%). The study focused on assessing three areas: learning acquisition, behavior change and overall program effect. 1. Learning Acquisition In terms of learning acquisition, trained groups had better scores than control groups in two out of three factories as the charts below show. The first chart compares the scores among three groups: the control group (CG), those trained by Business Dynamics (BD) and those trained by in-house trainers at Nike factories (Int). The second chart compares scores across three companies in comparison with the control group. The training appeared to be most helpful in such domains as stress management and communication. Training sessions helped them enhance their knowledge and inspired them to use some new techniques to de-stress themselves in everyday life. There were no significant differences in other domains. CG – Control Group BD – Operators trained by Business Dynamics Int – Operators trained by in-house trainers (trained by Business Dynamics during Trainer of Trainers in 2007) Note: Ratama, Dayup and Tuntex are three factories participated in the impact study. 2. Behavior Change In terms of behavior change, there were no significant score differences between main groups and the control groups’ behavior based on survey data and analysis. However, in-depth interviews with both groups suggested that trained operators were able to manage their stress better, sustain positive attitudes, and work well in teams. Trained operators who were interviewed reported that they experienced significant changes in their life after participating in the training, particularly with managing their emotions, team work, self-discipline, improving communication skills both at work and home. Survey responses and interviews suggested that trained operators had a broader understanding of what assertive communication means compared to the control group. For example, they understand that assertive communication does not mean tendency and willingness to disagree with everybody leading to arguments, but rather striving for a ‘win-win’ situation by being tactful. This was manifested in their responses stating that they would not readily turn to arguments, but instead look for ways to communicate effectively and reach ‘win-win’ situations. Some respondents reported that they were not able to fully apply assertive communication at the workplace, citing that their workplace was not conducive enough to do so and they were having issues with supervisors. Professionalism 1 2 3 4 Positive Attitude 1 2 3 4 Assertive Communication 1 2 3 4 Emphatic Listening 1 2 3 4 Persuasion Skill 1 2 3 4 `Stress Level 1 2 3 4 3.1 3.1 Control Group Trained by BD Trained by in-house trainers POOR EXCELLENT LOW HIGH 3.4 2.9 2.0 3.5 3.1 3.2 3.0 2.0 3.5 3.1 3.0 2.8 2.0 3. Overall Program Effect In terms of the program effect, significant changes were noted in the communication aspects by supervisors and the management as operators developed friendlier attitudes in communicating with their colleague while it is unclear whether it has any effect on the quality of work and productivity levels. According to the supervisors and upper management, among all trainings given to operators within the past year, the Champion Worker Training has the most significant effect on the operators as many of them apply their newly acquired knowledge about communication and stress management to their daily lives. The Champion Worker Training is also expected to be useful in transforming operators to be better workers and individuals, as it has given them new outlooks in terms of the way they manage their lives. Furthermore, participants were asked if they noticed any significant changes in their interpersonal skills and a majority of participants said yes. Changes were manifested in improved communication skills (i.e. male operators know how to treat their female co-workers, operators in general know how to treat others such as colleagues, spouse and family members), and stress management skills (i.e. how to handle their problems). The team dynamics are improving as relationships are more open and work is conducted in an environment built upon mutual trust and respect. In one factory, it was mentioned that they have better relationships within the group, as they are more confident in persuading friends to be better or warning them when they make mistakes. In the future, for this kind of training to be successful, most respondents suggested that it adds modules that are simple for operators to understand, and elaborate more about noble values (virtues) and responsibility. Training activities have to be refreshed at certain periods, in order to help participants remember all materials covered in the training. ©2010 International Youth Foundation. All rights reserved. No portion of this report may be reproduced in any form without the written permission of IYF. This project was made possible by the generous support of the American people through the United States Agency for International Development (USAID) under grant agreement no. RAN-A-00-04-00043-00 This report was funded in part under grant agreement no. EPP-A-00-08-00006-00. The International Youth Foundation (IYF) invests in the extraordinary potential of young people. Founded in 1990, IYF builds and maintains a worldwide community of businesses, governments, and civil-society organizations committed to empowering youth to be healthy, productive, and engaged citizens. IYF programs are catalysts of change that help young people obtain a quality education, gain employability skills, make healthy choices, and improve their communities. To learn more, visit www.iyfnet.org ANNEX E: EEA Morocco Evaluation Report CONTENTS Executive Summary ................................................................................................... 4 Section I: Introduction ............................................................................................... 7 Section II: MEEA Results Framework.. ..................................................................... 14 Section III: Findings and Analytical Review of Project Achievements ......................... 15 Section IV: Challenges, Lessons Learned and Recommendations .............................. 42 FIGURES Figure 1: Data Collection Mechanisms and Sources ................................................................. 8 Figure 2: Resources Leveraged by Project ............................................................................... 15 Figure 3: Leverage Resources by Year..................................................................................... 16 Figure 4: Intra-MEEA Sub-Grantees’ Partnerships ................................................................. 18 Figure 5: Classification of Technical Assistance Provided by the Secretariat ........................... 20 Figure 6: Sustainability Status by Subproject ........................................................................ 21 Figure 7: Project-level Costs ................................................................................................... 22 Figure 8: Alliance Building and Program Support Costs ......................................................... 22 Figure 9: Cost per Beneficiary ................................................................................................ 23 Figure 10: Number of Partnerships Formed by Subproject ..................................................... 23 Figure 11: New partnerships to Continue, Expand, or Replicate MEEA Projects ..................... 24 Figure 12: Number of Youth Participated, Completed and Placed in Internships and Jobs .... 26 Figure 13: Beneficiaries by Age .............................................................................................. 27 Figure 14: Classification of Graduates by Gender and Project ................................................ 27 Figure 15: Number of Teachers, Trainers and Counselors Trained by Project ......................... 29 Figure 16: Youth Satisfaction with Trainers and Facilitators .................................................. 30 Figure 17: Youth Satisfaction Rates ....................................................................................... 32 Figure 18: MEEA Job Placement Rates by Project .................................................................. 35 Figure 19: Salary Ranges of Employed Youth ......................................................................... 39 Figure 20: Classification of How Youth Use Earnings ............................................................. 41 ANNEX Annex A: List of MEEA Partners and Leverage Contributions ................................................. 44 Acronyms AFEM Association des Femmes Entrepreneurs Marocaines (Association of Moroccan women business leaders) AREF Academie Régionale d’Education et de Fomation Professionnelle (Regional Education and Vocational Training Department) CFAs Centre de Formation par Apprentissage (Vocational Training Centers) CJD Centre des Jeunes Dirigeants (Center of Young Leaders) EFE Education for Employment Foundation FGD FPME Focus Group Discussion (FGD) Fondation des Petites et Moyennes Entreprises (Small and Medium Enterprises Foundation) HJ Heure Joyeuse Association IT Information Technology IYF International Youth Foundation LS Life Skills LOP Life of Project MEEA Morocco Education and Employment Alliance ToMT Training of Master Trainers PTS Passport to Success ToT Training of Trainers USAID United States Agency for International Development WS Workplace Success training program Executive Summary This final evaluation report aims to assess the immediate and mid-term results of the Morocco Education and Employment Alliance (MEEA) program that comprises seven projects, six of which were based in the poor neighborhoods of the Casablanca region of Morocco, and one in the city of Salé, outside of Rabat. The report was developed based on the EEA’s Results Framework (see Section II) designed by the EEA global evaluation team and vetted by the International Youth Foundation (IYF). The report attempts to respond to four main evaluation questions: 1. To what extent did the Morocco Education and Employment Alliance approach succeed in building partnerships and promoting sustainable and scalable youth employability programs? (Intermediate Result One) 2. How successful was the MEEA program in improving access to and quality of education and employability opportunities for youth through the program interventions? (Intermediate Result Two) 3. How much did the MEEA program contribute to improving prospects for employment and successful entrepreneurship? (Intermediate Result Three) 4. Was there any indirect effect on families and communities of youth and if so, to what extent? (Intermediate Result Four) This report is structured into four main sections. Section I outlines the objectives of the evaluation together with the evaluation questions the study aims to address, as well as the research design, evaluation methodology and limitations of this study. Section I also provides a project summary of the EEEA program and describes the “Development Problem” and MEEA’s response while explaining the proposed intervention and its relevance to the challenges youth face in Morocco. Section II presents the MEEA Results Framework, followed by Section III which presents specific findings related to the four intermediate results or evaluation questions. Each sub-section under the Findings Section addresses a specific indicator by providing a finding statement, illustrative data supporting the statement, and an analysis of the finding. Section IV concludes the report, presenting challenges and lessons learned focusing on leveraging and alliance building, scalability, and sustainability. Study Methodology This evaluation is based on a mix of qualitative and quantitative methods: a) focus group discussions with the youth; b) the administration of survey tools; and c) key informant interviews. The survey tools have been developed to allow for measurements of how the seven projects performed against defined indicators, as well as to generate a free discussion that provided information that can lead to better understanding of the motivations, behavior and perspectives of MEEA partners, employers and the youth. Out of the 2,238 participants in MEEA-supported training programs,1 159 have been surveyed and 106 have attended focus group meetings organized by the M&E advisor. Additionally, 18 employers of MEEA trainees were interviewed for the purposes of this study. These responses provided numerous examples of skills, knowledge and attitudes acquired by the youth through the different projects implemented under the MEEA program between 2006 and 2009. Findings The evaluation findings support the position that MEEA was an effective mechanism for fostering new partnerships between diverse institutions to further youth employment. MEEA generated $2,415,240 in leverage funds and forged over forty formal partnerships and over two hundred public/private collaborations.2 All six MEEA subgrantees found that the Alliance experience taught them how to create new partnerships with diverse types of partners and access new funding to contribute to program sustainability. All subgrantees also said that the Alliance helped them improve their work with youth by strengthening their technical expertise and capacity. Moreover, the findings show MEEA’s concrete results in terms of improving employment opportunities, skills and attitudes of young people. To date, an aggregated 1,688 youth have completed training offered under the MEEA programs and of this number, 1,195 were placed in internships, 300 have been placed in jobs, and 15 youth have created their own small business. Key findings include: • Fifty-seven percent of youth respondents said their internships were either good or excellent. 1 An overall total of 1,688 youth completed MEEA-supported training programs. It should be noted that an additional 34,777 school children were indirect beneficiaries of one MEEA program, the Al Jisr Digital Opportunities Workshop project, because their schools received refurbished computers. 2 IYF distinguishes between formal partnerships and public/private collaborations by defining formal partnerships as those in which there was either a written agreement or memorandum of understanding. • Of employed youth respondents, 57% said they were happy with their work as a result of participation in the MEEA program. • Seventy-five percent of youth respondents said they had referred another youth to MEEA￾supported training program and 89% said they had acquired necessary technical skills needed to find a job. • The evaluation findings along with the MEEA testing for technical and life skills competencies showed youth improved their skills in listening, conflict resolution, confidence, and felt a greater sense of responsibility as a result of their training. • Sixty-nine percent of employers responded rated the technical skills of MEEA-trained youth as good. In terms of life or professional skills that were a significant focus of MEEA training programs, 18% of employer respondents rated capabilities of MEEA-trained youth as excellent and 47% as good. • All employers surveyed said they would offer jobs to MEEA graduates in the future. Challenges, Lessons Learned and Recommendations Part I: Challenges Given that MEEA was one of the pioneer programs in Morocco whose operational framework was based on public-private partnerships focusing on employment for out-of-school youth, the program experienced several noteworthy challenges. Leverage and Alliance Building • Changing the attitude of businesses toward youth education and employment requires the development of a strong advocacy network. The MEEA’s Steering Committee, while comprised of several private sector representatives, did not consistently advocate for the Alliance with other businesses. This was in part because Steering Committee members were too senior to have time to devote to aggressive, regular outreach with employers and partly due to the Alliance’s inability to sufficiently motivate them to play this role. To address this challenge, the Alliance would likely need to expand Steering Committee membership from among targeted communities and industries, as well as refine roles in order to maximize opportunities for outreach with businesses for job placement. On the other hand, collaboration and sharing of lessons learned among project partners within MEEA in developing effective placement strategies was quite successful, suggesting a more bottom-up and community based approach in this area is worthy of investment. • Introducing the “Alliance” concept was challenging in the context of a Moroccan society where community spirit has been traditionally focused more on the immediate family circle and where civil society and the private sector actors played a very minor role (if at all) in the education of Moroccan youth. • Fundraising to sustain project activities was initially a major challenge for most of the partners. Most of the implementing partners have benefited from foreign donor grants in the past, but their capacity to develop efficient strategies to raise leverage funds for their projects was very limited when MEEA was launched. Scalability • Scaling up MEEA projects requires building new regional and local alliances in each region where a given MEEA project would be expanded or replicated. This in itself represents a major challenge, and requires a strong and local Steering Committee that would be fully committed to brokering partnerships, building partners’ capacity in developing partnerships and leveraging resources, and in monitoring project quality. • Finding jobs for the newly trained youth was a challenge for the EEA implementing partners, from the small to large scale projects. In order to achieve job placement on a larger scale, further efforts should build on job placement activities to develop a comprehensive job placement strategy that involves various stakeholders and coordinated outreach activities with businesses. This would go beyond the reasonably successful but more “opportunistic” and door-to-door approach undertaken by the MEEA program for job placement. • Scaling programs is often best achieved through integration into existing institutions with broad reach, such as government. And yet partnering with government requires the government have a certain level of capacity and “buy in” to the project, as well as having a conducive policy environment. Emploi Habilité, MEEA’s project which has best positioned itself for scale through significant integration within government institutions, was challenged by these issues and as a result, the project experienced some setbacks in terms of enrollment and job placement. Efforts have been made to improve such integration, and it will be important in future analyses to identify best approaches in this area as the program continues to expand with USAID and other funding support. Program Sustainability • Sustainability can be viewed on multiple levels, at both the project and broader alliance levels. While significant progress was made in project based sustainability efforts, the overall Morocco Alliance’s lack of an institutional “home” in Morocco and maturation beyond the intensive activity required for designing and distributing seed funds negatively impacted the ability of the Alliance to be sustainable on a macro level beyond IYF’s direct role. Furthermore, while the prominent role of the private sector in the Alliance helped it to broker new partnerships, the private sector alone was not suitable to take on the role of leading the Alliance, and there were not civil society organizations positioned to lead it. Further efforts in this regard would likely require substantial resources given the array of institutional interests across all sectors to be represented, additional time and focus on the advocacy role beyond project specific activities, and a proper institutional framework to lead such efforts. Part II: Lessons Learned and Recommendations The final evaluation reveals several lessons learned that could be applied in implementation and management of similar projects. Leverage and Alliance Building • Alliance development efforts are more effective when private service actors and private businesses stop looking at their work with the alliance as mere charitable contributions, but because it makes good business sense. In this respect, MEEA programs that provide them with quality employees, which will ultimately contribute to the businesses’ higher productivity, are the most likely to be valued and show program success. • Regular follow up on youth and monitoring of their performance in training and their experience in internships is essential to ensure quality. Moreover, the monitoring visits are a good opportunity for the implementing partners to advocate for the MEEA graduates and to build a strong relationship with the private sector partners. • In order to work effectively with government, the program has learned through MEEA to ensure project implementation and institutionalization, the program has to work simultaneously at the local, regional and central levels. Scalability • Capacity building of implementing partners in partnership development, project monitoring, reporting, and financial management contributed to making MEEA partners more capable of effectively designing and implementing alliance-based workforce development programs tailored to meet the needs of both youth and employers. • Good trainers and a sound follow up system are the sine qua non conditions to scale up MEEA projects. Sufficient resources to cover trainers’ salaries, and transport for follow up with trainees is essential. Program Sustainability • MEEA was quite successful in equipping its NGO partners with capacity to leverage resources and conduct effective outreach with donors and private sector, so that most of the seven partners have been able to sustain their programs with new funding and in new regions. • Integration of life skills and language skills training was greatly valued by employers and youth themselves. More efforts by MEEA to pursue partnerships to facilitate language training (French and English) were needed, and employers and youth also expressed a need in this area. • A system for tracking youth graduates and soliciting feedback such as mobile-based or online that youth can access easily would have been valuable for improving the responses from youth in the evaluation report and to track employment. • Providing more systematic job counseling and guidance services to youth is a critically important component to ensure success in job searches, job placement and successful integration into the workforce. Section I: Introduction A. Evaluation Objectives and Scope The goal of this final MEEA program evaluation is to examine the effectiveness, relevance and sustainability of the program; and to identify lessons learned and recommendations on program management and implementation approaches in order to strengthen impact of future youth employability development initiatives. The evaluation focused on the activities of the MEEA program undertaken during the period between January 2005 and May 2009. The MEEA final country report aims to highlight the immediate and mid￾term results of the Morocco Education and Employment Alliance (MEEA) program comprised of seven projects based in the regions of Casablanca and Rabat. To achieve the evaluation objectives stated above, the report seeks to address four main evaluation questions based on EEA’s four Intermediate Results (IRs) - see Section II: 1. To what extent did the Morocco Education and Employment Alliance approach succeed in building partnerships and promoting sustainable and scalable youth employability programs? (Intermediate Result I) 2. How successful was the MEEA program in improving access to and quality of education and employability opportunities for youth? (Intermediate Result II) 3. How much did the MEEA program contribute to improving prospects for employment and successful entrepreneurship? (Intermediate Result III) 4. Was there any indirect effect on families and communities of youth and if so, to what extent? (Intermediate Result IV) The evaluation report is organized in four main sections: Section I: Outlines the objectives of the evaluation and the evaluation questions the study aims to address, the research design and evaluation methodology applied, including the applied methodology for sampling, informants profile, data collection tools, data collection process and data analysis; as well as the limitations of this study. It provides a project summary of the EEEA program, describes the “Development Problem” and IYF’s response while explaining the proposed intervention and its relevance to the challenges and needs youth face in Morocco. Section II: Presents the EEA’s Result Framework. Section III: Presents specific findings according to four IRs or evaluation questions. Each sub-section under the Findings Section addresses a specific indicator by providing a finding statement, illustrative data supporting the statement, and an analysis of the findings. Section IV: Summarizes challenges and lessons learned focusing on leveraging and alliance building, scalability and sustainability. B. Research Design and Evaluation Methodology The evaluation’s data collection tools were designed by the EEA global evaluation team and adapted to the Moroccan context to identify factors critical to the current context analysis of youth education and employment in Morocco. Indicators were defined through a participatory process involving five EEA countries, and the framework includes both quantitative and qualitative indicators organized under four intermediate results – see Section II. Evaluation data were collected in the field over a period of three months from February to May 2009. Interviews, surveys and focus group discussions (FGDs) were conducted in French and/or Arabic. Survey data collected were then translated into English and entered in the Survey Monkey online database. The EEA Global Evaluation team was composed of: - Mrs. Mariam Britel Swift, Evaluation Coordinator for Global EEA Program and the MEEA Monitoring and Evaluation (M&E) Team Leader - Mrs. Marcia Miranda, Evaluation Coordinator for the EEA-Philippines program - Ms. Shereen Nasef, M&E Consultant for the EEA-Egypt program - Ms. Lin Lin Aung, EEA Program Manager The Morocco M&E Team Leader developed the M&E plans for each MEEA project and offered project implementers technical assistance and tools to collect data, monitor project implementation and assess short and mid-term results. She designed the evaluation methodology for the MEEA country evaluation in close collaboration with the rest of the global evaluation team members, Mrs. Swift led the data collection process through the organization of focus group meetings, surveys and interviews, ensured the translation and entry of data into Survey Monkey, interpretation and analysis of collected data; and preparation of the final project and country reports. The following people participated in the data collection process in the field: - Mrs. Noufissa Haidara: A Life Skills trainer in Abou Al Aalaa Training Center, working with youth in the Tourism Sector. She administered beneficiary youth and employer’s questionnaires and final Life Skills acquisition test, and provided observations on youth she has trained and on the functioning of the training center where she worked. - Mrs. Fatima Mouaouya and Mr. Mohamed Dbiri: Life Skills trainers and Master Trainers. Monitored performance of trainers, administered beneficiary youth and employers’ questionnaires and final Life Skills acquisition test, and provided observations on youth she has trained and on the functioning of the training center where she worked. Evaluation Methodology Due to the diversity of activities and complexities associated with the MEEA program, it is obvious that no single source of evidence or analytical design will be able to address all evaluation issues. Multiple lines and levels of evidence (MLLE) have been used for this country level evaluation, because it can infer relationships using various sources of information. Therefore, this country level evaluation of the MEEA program employed a variety of qualitative and quantitative techniques - summarized in Figure 1 below. Figure 1: Data Collection Mechanisms and Sources Evaluation Question/Intermediate Result Data Collection Mechanism Data Sources To what extent did the Morocco Education and Employment Alliance (MEEA) approach succeed in building partnerships and promoting sustainable and scalable youth employability programs? (Intermediate Result I) Interviews Project implementers Alliance members How successful was the MEEA program in improving access to and quality of education and employability opportunities for youth? (Intermediate Result II) Focus groups Questionnaires Interviews Desk review Project participants Employers Trainers How much did the MEEA program contribute to developing improved prospects for employment and successful entrepreneurship? (Intermediate Result III) Focus groups Survey Interviews Employers Project participants To what extent did the MEEA program succeed in improving youth engagement in their families and communities (Indirect effect on communities and families of youth)? (Intermediate Result IV) Focus groups Survey questionnaires Interviews Project participants, and their family members or relatives The team verified the data using triangulation approach based on focus group discussions, surveys, and key informant interviews. Generic survey tools were developed to allow for measurements of how the indicators performed across the different EEA countries, as well as be used to generate a free discussion with the key informants on their EEA experience. The discussion provided information that led to a better understanding of the motivations, behavior and perspectives of our partners, employers and the youth. The evaluation used the following definitions for the three categories of data sources: • Partners: Government, civil society and private sector partners who have directly or indirectly supported EEA programs. These included alliance/steering committee members, project implementers, partners or project implementers. • Beneficiaries: A) Direct: youth who participated in EEA training programs, employability programs, coaching, or other EEA supported programs; B) Indirect: community, trainers, families and friends. • Employers: Private or public sector employers who have offered internships and/or jobs to EEA youth. They may or may not be partners of the EEA alliance. Surveys Five comprehensive questionnaires were developed to gather input from the three groups of data sources described above. Focus group interviews Focus group protocols were developed to involve all youth who have completed a specific program, or just a representative sample thereof. Nine focus group interviews were organized with the participation of 106 youth who graduated from the different projects. Each focus group discussion was attended by four to thirty participants and lasted approximately two hours. Participants were invited to complete the participant survey prior to the start of the focus group discussion. Interviews Interviews with the employers were conducted at their offices, and lasted between thirty minutes to an hour. A questionnaire prepared in collaboration with the EEA global evaluation team was used to guide the discussion. In most cases, these interviews were also used to further sensitize the employers about the MEEA program concept, and to encourage them to hire more youth in the future. Interviews with implementing partners were conducted either through the telephone, the internet, or in person. These interviews were also guided by the survey tool prepared in collaboration with the EEA global evaluation team. Sampling The global EEA evaluation team proposed the following guidelines for sampling to guarantee a full enumeration of partners, and a good sampling of employers and youth: Sample Sizes • Participants: A minimum of twenty youth per project in each country • Employers: 50% of employers per project in each country • Partners: At least 50-100% alliance members; 100% implementing partners Composition of Samples Participants: The samples were composed of youth who have completed EEA programs. The team attempted to have fair representation in terms of gender as well as those employed vs. not yet employed. It became obvious during the course of collecting project-related data that only a very small proportion of the population invited to participate in the focus groups and surveys would respond. Therefore, selection criteria for respondents were mainly based on the availability of youth and employers and their willingness to respond. Out of the 2,238 youth that have been trained under the MEEA-supported training projects, 159 youth have responded to the beneficiary survey questionnaire, and 106 participated in the nine focus groups organized by the M&E Team Leader. Four implementing partners responded to the implementing partner’s survey; and 18 employers responded to the employer’s survey. The implementing partners and employers under the project “Point Jeunes” were not interviewed because the project was terminated before its completion. The “Work Place Success Project” implemented by EFE, was evaluated separately by a New York University evaluation team.3 Desk Review The desk review of secondary data, consisting primarily of project documentation, was conducted prior to on-site data collection in order to identify potential topics for in-depth probing during interviews and focus groups. The following documents were reviewed: 1) EEA program description 2) EEA monitoring and evaluation plan 3) EEA quarterly reports 4) Subgrant project quarterly and final reports 5) USAID evaluation guidelines 6) Youth Exclusion in Morocco: Context, Consequences, and Policies, Brahim Boudrabat, Aziz, Ajbilou, Middle East Youth Initiative Working Paper, Wolfensohn Center for Development, Dubai School of Government. 7) EQUIP3 “Guide to Conducting USAID Cross-Sectoral Youth Assessments” Draft 2/21/08. 8) Evaluating Global Development Alliances: An Analysis of USAID’s Public-Private Partnerships for Development, Tom Dewar (DAI). 3 EFE’s evaluation was conducted to measure knowledge gains, and behavior changes. A different set of surveys and protocols were developed and used to gather information from direct beneficiaries under this particular project. To ensure a minimum synchronization of the EFE evaluation with the overall country level evaluation, the MEEA M&E Team Leader met with the New York University Evaluation Team to give them some guidance on the evaluation tools to be used and the indicators that should be included in their evaluation. A total of 26 youth responded to EFE’s alumni questionnaire and six employers responded to the employer’s questionnaire. EFE’s NYU evaluation was not available to IYF at the time of writing this report. Data Analysis The data collected were analyzed according to specific indicators under each IR, specifying trends emerging from responses and analyzing reasons and justifications behind these trends. Data related to youth acquisition of skills was triangulated using test results, trainers’ observations, and youth self￾assessments. Four instruments were developed to allow for the assessment of youth progress in acquiring the soft skills and technical skills that the MEEA supported training programs sought to develop in target youth. These different instruments served to triangulate the results to ensure data accuracy and validity: 1. Technical and life skills tests: administered to youth throughout the training and at the end of each training program. 2. The youth self assessment questionnaire: this questionnaire asks the youth to assess their own capacities before and after the program. 3. Teacher’s observation: These observations were provided either orally, in writing or using a grid to rate youth progress in relation to each skill that the curriculum sought to develop in youth. 4. Employers’ assessment of the performance of MEEA-trained youth as interns: questionnaire or interview of employers and direct supervisors of youth that can give first hand observation and assessment of the youth performance on the job (see indicator 2.6). C. Limitations and Challenges During the process of data collection, this evaluation faced a number of challenges that affected reporting on some specific indicators, particularly those related to youth employment. Some of these challenges are listed below: - As no systematic data were available on participants’ whereabouts it was not possible to develop a statistical sample, and much time was spent on finding participants to interview. Only 159 out of the 1,688 of the beneficiary youth that have completed MEEA-supported training have responded to the questionnaire and/or attended the focus groups. - The largest cohort of youth trained by Emploi Habilité Casablanca did not complete training until June 2009. These youth were still being surveyed to get their feedback on the quality and relevance of the training they received, but it is still too early to check on their employment status, as this should normally be done three to six months after graduation. - Some of the youth surveyed were semi-illiterate, and in certain cases, were unable to complete the survey or write down their comments. It was necessary during the data collection process to request the help of four to five people in order to provide one-on-one help to those youth who could not complete the questionnaire on their own. In ninety percent of the cases, completing the beneficiary questionnaire took almost two hours. - Surveying MEEA working youth was challenging, due to the fact that most these youth found it difficult to take time out of their work to come to the focus group meetings or to complete the questionnaire. To accommodate these working youth, focus group meetings and surveys were conducted during weekends or after working hours. - Partner employers who recruited MEEA youth were generally too busy to respond to our interview requests. These were heads of small or very small enterprises that were generally performing several functions at the same time, and their availability for interviews was very limited. Additionally, as discussed in detail in the overall global evaluation, this study is inherently limited by the lack of a fully external evaluation of program outcomes. As noted in the global introduction, given both resource constraints and the desire to leverage the broad experience of program participants to fully compare and contrast approaches used in the different countries for governance, program design, leveraging of resources and other elements, a strategic decision was made to utilize participants in the overall program to support in the conduct of this evaluation. In the interest of promoting learning and improvement, the EEA teams have done their best to fully express challenges and problems within programs and the alliance structure, grounded in the reality of their experience in the program; however, the study does have this inherent limitation, which is important to disclose fully. Problem Statement Morocco’s growing youth population needs to be fully engaged and gainfully employed. Although opportunities exist in Morocco’s labor market and private sector, many young people lack the skills demanded by the market. Access to acquiring these skills is very limited for those who have dropped out of the educational system. Those who complete their education often do not have the portable and adaptable skills needed in today’s competitive marketplace. Moroccan youth face a number of challenges as they prepare for the world of work. The official unemployment rate for young people (ages 25-34) is 56.2% in the urban areas and 60.1% in the rural areas.4 While employment opportunities in the private sector are growing, few young people are equipped with the attitudes and skills required to thrive in the workplace. The country’s education system, despite improvements, continues to produce young people who lack relevant skills for the globalizing economy. Even if youth possess relevant technical skills, the employers with which IYF has partnered over the last four years in Morocco note that young people do not join the workforce with the necessary critical thinking, communication, and other “soft” or behavioral skills they need to succeed at a given job. Moreover, youth lack confidence and knowledge of how to navigate the job market, use networks to look for jobs and then apply successfully for available jobs. Program Response The Moroccan Educational and Employment Alliance (MEEA) engaged business, government and NGOs to address the national priority of improving education and training for youth that is relevant to the job market. Through public-private partnerships, MEEA developed and expanded job training, placement and entrepreneurship programs for disadvantaged young women and men. MEEA implemented seven projects benefiting disadvantaged youth aged 16 to 25 in the Grand Casablanca region, and is presently scaling-up one of these projects “Emploi Habilité” in the Tétouan-Tangier region. Four other projects are being replicated in other regions of Morocco (The Digital Opportunity Workshop in Ouajda, and the Reprofilage project in Marakesh, WorkPlace Success in Tangier, School of Entrepreneurs in Agadir). The seven projects focused on different sectors (including IT/Hardware repair, Textile, Tourism, Preschool teaching, services, and other). These projects aimed to reach different categories of disadvantaged youth, ranging from illiterate young women, to out of school street children, to unemployed young female university graduates. Six of the MEEA projects focused primarily on youth training and placement. Training programs lasted from nine months to two years and programs targeted different categories of disadvantaged youth (quasi illiterate young women, university under-graduate young women, school dropouts) aged 16-25. Two out of the seven MEEA subprojects were specifically designed for young women; all five others targeted both young males and females. All training activities included two fundamental areas: 1) hard/technical skills, and 2) soft/life skills. 1) Technical skills training comprised two components: in-class/theory training and practical hands-on training at the enterprise. 2) Life skills training was designed to complement the technical skills training so that graduates know how to work in teams, communicate effectively, solve problems and adopt good work ethics at the workplace. Two life skills development curricula were used for this purpose: a) the Passport for Success curriculum, designed by IYF with funding from the GE Foundation, and adapted to the Moroccan context by a team of Moroccan Emploi Habilité experts; and b) The Workplace Success curriculum, designed by McGraw Hill University, and adapted to the Moroccan context by EFE experts. Only one of the MEEA projects was focused on entrepreneurship development: School of Entrepreneurs Project, implemented by the Center of Young Leaders (CJD). The following are brief descriptions of each of these projects and their immediate results: DIGITAL WORKSHOP OPPORTUNITY: This program, implemented by the NGO Al Jisr, provided disadvantaged youth who have completed their high school education, practical training on computer hardware repair and maintenance, and couples this training with the Emploi Habilité life skills curricula. The program includes four months of computer repair training, basic software training and marketing, followed by two months of internships. Project Results: Eighty-seven out of 92 participants completed the training, 67 completed two- month internships, and forty were placed in permanent employment. Over 2,333 computers were collected, repaired and redistributed to primary schools in poor neighborhoods, benefitting 34,777 students. Computers donated were refurbished by the trainees and distributed for use by 61 schools, seven associations and five training centers (CFAs). Al Jisr has entered into a new partnership with Banque Populaire to donate computers and other equipment for both the Casablanca and Oujda workshops. The Casablanca workshop model is being replicated in Oujda. 4 All statistics in this section are from the National Employment Survey (Direction de la Statistiques), 2006 or the DSSP – Division des études et des statistiques. –2007/2008 data. Emploi Habilité (EH): This program, in its first phase implemented by the Mjid Foundation, sought to train school dropouts from disadvantaged neighbourhoods of Casablanca in the GE/Passport to Success Life Skills and Microsoft Unlimited Potential (UP) programs. Building on the success of its first phase, the project’s second phase took the model of IT and life skills training and integrated this package into the Ministry of Education’s Casablanca Academy’s vocational training centers – Centres de Formation par Apprentissage (CFA). CFAs are based on a training model of twenty percent in-class training and eighty percent hands-on training for youth studying technical trades such as metal work, IT, early childhood education, and tourism/hospitality, over a one to two-year period. In the CFAs, the training package represented a complement to the normal offering of the vocational training centers, enabling graduates to strengthen the soft skills needed to accompany their technical skills during their internships and equip them with a foundation of computer applications that they can apply to their specific trade. Emploi Habilité is presently being replicated in the region of Tétouan-Tangier, as a first step towards scaling up the program nationwide. Project results: This program trained 48 youth in the pilot phase, of whom 44 completed the training and internships. Sixteen graduates from the pilot phase secured jobs. During the phase two, a total of 816 CFA youth participated in three cohort classes focusing on life skills, basic IT and job search preparedness as elaborated below: • Cohort Class I (September 2007 – June 2008): a total of 107 youth participated and completed the one-year track EH training - inclusive of internships, and 54 graduates have been placed in jobs. • Cohort Class II (September 2007 – June 2009): a total of 314 youth participated and completed the two-year degree track training inclusive of internships.5 • Cohort Class III (September 2008 – June 2010): a total of 395 youth are currently undergoing in the two-year track training. Of these 395 youth, 380 have been placed in internships to date. The success of the program has led to a number of new opportunities for expanding the project. MEEA/IYF continues to receive interest in the Life Skills curriculum from new public and private sector partners. Private sector and organizations that expressed interest and with whom IYF/MEEA has met include: ANAPEC, Auteuil Foundation (French NGO), McDonalds, Accor, Federation of Tourism, USAID￾funded Civic Education Center, North-East Foundation, the Finnish Children and Youth Foundation and SILATECH. Through two phases, this project trained a total of 1,237 youth including 373 youth who received life skills training through other MEEA partners. Workplace Success/EFE: This program, implemented by the Education for Employment Foundation (EFE), includes adaptation and translation of EFE’s Workplace Success curriculum for Morocco and the training of 230 youth, including youth beneficiaries from two of MEEA’s other programs – the first group of Al Jisr’s Digital Opportunities Workshop trainees, and AFEM’s Reprofilage program. Training was conducted in partnership and on the campus of University Hassan II, Morocco’s largest public University. EFE’s model entails partnering with companies seeking to train their new employees on Workplace Success, so participants receive the Workplace Success training along with their workplace experience and then can transition into permanent employment. EFE has such agreements in place with BMCE, YNNA Holdings and the Chabi group, and job placements were targeted for approximately 85% of trainees. Project results: An aggregated 182 youth completed the training and 113 (49%) were placed in permanent jobs. The University Hassan II pilot courses were launched with the participation of students from the faculties of Sciences and Literature in March 2008. To ensure the sustainability of the program, EFE trained 1 Master trainer, who will start training the trainers in the University Hassan II. Reprofilage/AFEM (Skills upgrading): The Reprofilage program, implemented by the women’s business association, AFEM, identified jobs that were in demand among AFEM members and trained unemployed women to fill them. The program then secured commitment from the businesses that they would provide internships and hire the women with whom they were matched, following their successful completion of the training. This project helped reduce the number of young women who were unemployed by preparing them to successfully break into the job market while providing businesses that are members of AFEM with qualified employees. Project results: The Reprofilage/AFEM grant drew to a close in June 2008, training a total of 61 women 5 Job placement figures were not available for cohort class II and III because participants were in the process of being placed in jobs or undergoing internships in early-mid 2009 when the evaluation was being completed. of whom 41 completed, placing 38 in internships (target of fifty) and placing 32 of the graduates in jobs or 64% (target of 38). The Reprofilage project generated many lessons of broader interest to MEEA because of its approach of targeting youth unemployed graduates (“diplomés chomeurs”) in Morocco and particularly young women. Learnings from Reprofilage related to the importance of orienting employers as well as employees prior to internships and the need for stronger mentorship among business partners are issues that have emerged across several of the MEEA grants. While the number of women reached remained small, the program was successful in receiving additional funding from a Canadian donor to replicate the program in Tangier. School of Entrepreneurs/CJD: With support from MEEA, the NGO Center for Youth Leaders (or Centre des Jeunes Dirigeants - CJD) offered youth a dynamic training program that would allow them to discover their entrepreneurship capacity and develop enterprise management skills. The program worked through the USAID-funded partner project, ALEF’s “Entrepreneurial Spirit Program” (ESP) and “Entrepreneurial Development Program” (EDP). CJD also offered young entrepreneurs opportunities to participate in other seminars to complement their entrepreneurship training, including “Finance for Non Finance Professionals” and “Negotiating with Your Banker.” Participants who successfully completed the ESP modules could also benefit from an advanced Entrepreneurship Spirit Program (ESP II). Following the first stage training workshops, the program selected a small number of potential entrepreneurs with strong a commitment to receive direct coaching and mentorship from businesses for a period of over 18 months. Additionally, an agreement with Hassan II University in Casablanca will integrate and mainstream the training into their bachelor and masters programs to ensure the sustainability of the project beyond the pilot period. Project results: A total of 433 young men and women have received the Entrepreneurship Spirit (ESP) training, and 13 young men and women have been selected to benefit from coaching services for the creation of small enterprises in Agadir. “Point Jeunes” Digital Workshop / Job Nest – ENDA Maghreb: The “Point Jeunes” project led by ENDA Maghreb aimed to improve the employability of Moroccan youth in Salé through demand-driven training and job placement services through a “Job Nest” or employment center housed at a newly-built Community Center - Dar El Hay. Among the training programs offered were the “Digital Opportunities Workshop” led by Ateliers Sans Frontieres, Life Skills for Employability/Passport to Success training as well as basic IT office skills and job searching techniques training. Project results: Two groups of ten disadvantaged completed internships. In addition, 18 youth were trained on IT and life skills, and 22 youth completed training on micro-enterprise development by the Zakoura Foundation. Despite Point Jeune’s small-scale successes of training youth within the Salé communities, the intractability of the management problems within Enda ultimately led EEA/IYF to decide to terminate the grant a month early in May 2008. Parallel to IYF’s decision to end the grant, IYF learned that the organization’s poor management actually forced Enda to close down its program in Morocco. IYF channeled the remaining funds from the grant to contribute to the GE/Passport to Success Life Skills Master training that IYF conducted in October 2008. Training Young Women to Work in the Apparel Sector/L’Heure Joyeuse: This program, implemented by the NGO l’Heure Joyeuse, and supported by Nike, targeted 130 young women from disadvantaged neighborhoods of Casablanca to improve their skills in clothes-making, as well as their life skills and entrepreneurship skills, with the goal of placing 31 of them in jobs in the apparel sector. A high quality package of training was provided in life skills, basic entrepreneurship skills, job searching techniques, basic literacy and numeracy, as well as language training. Project results: A total of 68 youth out of the 93 participants completed the training, 45 were placed in internships, out of which 31 (target of 31) were placed in permanent employment, and 15 are still in internships at the moment this report was developed. L’Heure Joyeuse also worked with Emploi Habilité to train additional youth in their other programs in Life Skills as part of the “pre-orientation for vocational training” consisting of Life Skills, ESP Basic, and basic literacy and numeracy package. As a result, the program has been able to reach a greater number of disadvantaged young men and women through the life skills program and other basic skills training. They can then be eligible to participate in the MEEA￾funded textile training program, CFA programs, as well as L’Heure Joyeuse’s other vocational training programs. L’Heure Joyeuse has been discussing this new model with GTZ and ESITH, a business association of textile businesses in Morocco, who would assist in promoting program among partner factories and lining up factories to participate in the program. Section II: MEEA Results Framework Education and Employment Alliance - Morocco Goal: Improving the relevance of education to better prepare young people (ages 15-25) for jobs and self-employment Intermediate Result 1: Alliance Building and Leveraging Resources Intermediate Result 2: Improved Access to and Relevance of Education and Employability Training Intermediate Result 4: Positive Indirect Effect on Families and Communities of Youth Intermediate Result 3: Improved Prospects for Employment and Successful Entrepreneurship Indicator 1.1: In-kind and cash resources leveraged for the project through private sector, government, civil society and other partners Indicator 1.2: The effectiveness of the MEEA Alliance Approach in building partnerships, and promoting sustainable and scalable youth employability programs Indicator 1.3: Number of private sector, government and civil society partnerships formed through the project to jointly offer employment and business development training activities in targeted areas Indicator 1.4: New partnerships, mechanisms and approaches created as a rippling effect of MEEA Indicator 2.1: Number of young men and women participating in MEEA programs Indicator 2.2: Number and percentage of young men and women completing MEEA supported programs Indicator 2.3: Number of teachers, trainers, facilitators and counselors trained under MEEA supported programs Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in MEEA supported programs Indicator 2.5: Level of satisfaction of young trainees with the quality and relevance of the training, counseling and services received through MEEA programs Indicator 2.6: Number and percentage of youth who completed MEEA supported projects who report that they have referred at least another youth to the program Indicator 3.1: Number and percentage of trainees who get a job within six months of completing the program Indicator 3.2: Number and percentage of trainees who continue education, obtain an internship or engage in further professional training within six months of completing the program Indicator 3.3: Number and percentage of MEEA trainees starting their own businesses within six months of completing the program Indicator 3.4: Number and percentage of youth with satisfactory internships or apprenticeships Indicator 3.5: Number and percentage of placed youth who declare having a satisfying and quality job Indicator 3.6: Percentage of youth (who had prior jobs) earning more income as a result of participation in the MEEA program and net salary increase Indicator 4.1: Number and percentage of employed youth financially helping their families with household, health and education expenses Indicator 4.2: Number and percentage of youth who have reported improved financial and social conditions of their families as a result of participation in the MEEA program Section III: Findings and Analytical Review of Project Achievements Intermediate Result I: Alliance Building and Leveraging Resources The findings presented in this evaluation are organized by intermediate result presented according to four key evaluation questions. The findings are meant to provide a broad picture of the effectiveness of the MEEA projects and to identify good practices, areas for improvement, challenges, and lessons learned related to the MEEA program implementation. Intermediate Result/Evaluation Question I: To what extent did the MEEA Alliance approach succeed in building partnerships, promoting sustainable and scalable youth employability programs? To respond to this question, the evaluation used the following four indicators: • Indicator 1.1: In-kind and cash resources leveraged for the project through private sector and government partners • Indicator 1.2: The effectiveness of the EEA Alliance approach in building partnerships, and promoting sustainable and scalable youth employability programs • Indicator 1.3: Number of private sector, government and civil society partnerships formed through the project to jointly offer employment and business development training activities in targeted areas • Indicator 1.4: New partnerships, mechanisms and approaches created as a rippling effect of working with EEA programs to support youth employability Summary of Intermediate Result I Findings The Morocco Alliance successfully leveraged over two million dollars and built the capacity of the seven local NGO partners to develop strategies for outreach with donors and partners and who as a result were successful in leveraging additional funding. Moreover, the Alliance succeeded in fostering new and diverse partnerships among NGOs, private and government institutions which contributed to sustainability of most of the seven projects beyond MEEA. At the country level, however, the Alliance was not able to sustain itself beyond the EEA program due in part to factors related to the Steering Committee. Indicator 1.1: In kind and cash resources leveraged for the project through private sector, government, civil society and other partners Contributions leveraged since the MEEA program inception totaled $2,415,240 including $543,718 in cash and $1,871,521 in in-kind contributions. In addition to these funds, the Finnish Children and Youth Foundation (FCYF) and EFE-Maroc approved a $270,000 grant to support adaptation of the PTS program to specific business sectors and train youth for placement in these over the next three years. An additional $310,000 has been provided by SILATECH, a new Qatari initiative, to support the program in training youth in the tourism sector. Illustrated in Figure 2 are the total amounts of cash and in-kind leverage from six out of seven sub-grantees. Figure 2: Resources Leveraged by Project Cash In-kind IYF Secretariat $4,000 $11,350 Digital Opportunities Workshop $88,914 $451,606 Emploi Habilité $33,696 $1,204,328 Workplace Success $266,964 $15,000 School for Entrepreneurs $0 $25,463 Reprofilage $50,144 $163,774 Point Jeune6 $0 $0 Preparing Disadvantaged Young Women for Apparel Sector Employment $100,000 $0 TOTAL $543,718 $1,871,521 $2,415,239 6 Project terminated early and leverage documentation collected by partner was not sufficient to confirm leverage amounts. Leverage contributions by year (2005-09) are presented in Figure 3 below: Figure 3: Leverage Resources by Year Fiscal Year Cash In-kind FY 2005 $ - $ 11,350 FY 2006 $ 170,335 $ 1,405,000 FY 2007 $ 283,992 $ 96,876 FY 2008 $ 33,412 $ 297,732 FY 2009 $ 55,980 $ 60,563 TOTAL $ 543,718 $ 1,871,521 $2,415,239 In 2005, when the MEEA program was launched in Morocco, Moroccan civil society organizations in general, including the six NGOs that implemented MEEA projects, had only a limited capacity to leverage cash contributions to support project implementation. In fact, with the exception of Education for Employment (EFE) --which is an international organization that has a broad experience in leveraging resources and fund raising-- the five Moroccan local NGOs (Al Jisr, AFEM, M’Jid, ENDA, CJD, and l’Heure Joyeuse) did not have a specific plan or strategy to leverage resources. Advocacy and outreach activities were conducted on an ad-hoc basis, without any systematic or rational planning. Most NGOs depended on small cash donations from government institutions or international organizations. Contributions from local businesses were very limited and consisted mostly of in kind contributions such as used furniture and computers. Once these NGOs started working with IYF using the Alliance approach, they realized that they had to develop sound outreach strategies to properly leverage required funding to support their projects. Hence, under the MEEA program, each one of the partner NGOs developed an outreach plan in one form or another, and all of them organized communication and advocacy events, where individual MEEA projects were presented to potential donors. As a result, all MEEA sub-grantees have developed their capacities to leverage resources in a more organized way; and have managed by the end of 2008 to leverage at total of $2,399,889 (excluding $15,350 leverage from IYF Secretariat) in support of their project implementation. In addition, through the experience acquired during their collaboration with MEEA, five out of seven sub￾grantees managed to attract funding to continue, expand, or replicate their projects beyond the MEEA funding (see the figure under indicator 1.4) which helped promote long term program sustainability. Indicator 1.2: The effectiveness of the EEA Alliance Approach in building partnerships, and promoting sustainable and scalable youth employability programs From the outset, the Morocco Alliance emphasized the importance of the private sector’s role in youth employment programs. In fact, the Alliance’s emphasis on the private sector greatly influenced the composition of its Steering Committee which in turn positively affected the program’s effectiveness in building partnerships and eventually its sustainability. The MEEA Steering Committee was originally comprised of the following members: • General Confederation of Moroccan Enterprises (CGEM) • Moroccan American Chamber of Commerce • PME-PMI, the Federation of small and medium enterprises • AFEM, a women’s business association • AL JISR, a Moroccan NGO led by a VP of a major Moroccan bank that fostered links between Moroccan businesses and local public schools • USAID/Morocco • IYF During 2007-2008, as the program’s priority shifted to the sustainability of the Moroccan Alliance, IYF and the Committee decided to expand the membership of the Steering Committee to include representatives of other NGOs partners involved in the Alliance and to identify a few replacements to represent existing member organizations. Therefore, in the spring of 2008, several new members were nominated such as L’Heure Joyeuse and the Center for Young Leaders (CJD). A new representative from CGEM was invited but did not join. The central role of the private sector in MEEA’s Steering Committee contributed to its effectiveness in building partnerships with variously sized-businesses, yet it may have contributed to its difficulties in sustaining itself. Between 2005 and 2008, MEEA awarded seven grants to six local NGOs/associations. Following the award process, the IYF Secretariat, and the Steering Committee to a lesser extent, focused more on program monitoring, technical support around job and internship placements, and fostering partnerships. The Moroccan Alliance used several strategies for building partnerships and capitalizing on its network of partners once they had joined. Firstly, the IYF Secretariat7 drew upon IYF’s global partners to add leverage funding and in-kind resources to the program. IYF brought partnerships with Microsoft, Oracle and Nike to MEEA that enhanced several of the sub grants. The IYF Secretariat organized promotional events such as at Morocco’s annual “Fundamentals of Investment Conference” to raise visibility of the program among businesses. The IYF Secretariat also conducted outreach efforts with businesses through meetings and phone calls to enlist new businesses and to identify internship and job opportunities for MEEA youth. As the training subprojects became more mature and the job placement needs were greater, this one-on-one outreach was less effective and so MEEA turned to other means to promote the program such as direct outreach with the Steering Committee members and organizing the MEEA Forum through which MEEA invited businesses, public sector, and youth to attend the event to learn more about various subprojects and hear testimonials about the program. Finally, during the last year of MEEA, the IYF Secretariat facilitated regular project director meetings to which it invited directors from each of the partner organization. These meetings were quite valuable in allowing the implementing organizations to share lessons and strategies, and to foster further cross-project partnerships. As a result of a combination of the strategies described above, MEEA proved to be effective in fostering new and diverse partnerships and collaboration among public and private sector actors. The MEEA program fostered partnerships in two categories: collaboration among the MEEA sub-grantees themselves and partnerships with other entities. Collaboration among the MEEA sub-grantees themselves: Through the help of the project director meetings, MEEA discouraged its partners from working in isolation, but rather to use their limited resources and capacities to work in partnership with other organizations to leverage resources, share knowledge, and exchange services and expertise. All six implementing partners have built partnerships among themselves and begun helping each other to implement their projects more effectively. Of particular note in this respect are the life skills interventions developed and managed through the Emploi Habilite and Workplace success programs, each targeting a slightly different age and educational attainment level. These two life skills programs were shared across several MEEA projects which strengthened trainer and master training capabilities among the organizations, fostered mentorship of such trainers within Morocco, and allowed partners to share and exchange best practices and lessons learned. The figure below presents the six different subgrantees with arrows to indicate the cross collaboration and synergies. 7 IYF served as the Secretariat institution in Morocco due to difficulty in identifying a Moroccan association that had the appropriate profile and organizational capacity. Figure 4: Intra-MEEA Sub-Grantees’ Partnerships Partnerships with other entities As a result of participation in the MEEA program, all six implementing partners were more capable of forging new effective partnerships with other national and international NGOs, as well as with public and private institutions (Banks, universities, Ministry of Education, Ministry of Labor). These new partnerships helped the grantees access additional funding to replicate their own programs in other regions of the country and therefore contributed toward their programs’ sustainability. To date, over 200 businesses have been directly involved in the different MEEA programs, providing computer equipment, furniture, but most importantly, offering internships and jobs to MEEA trainees and graduates.8 See Annex A for the list of MEEA partners and their contributions. Survey responses from NGO grantees, private sector and public sector partners support these conclusions with specific examples of how they benefitted through MEEA. When asked about their experience in MEEA, the six MEEA grantees responded that being part of an alliance gave them the opportunity to exchange experiences and expertise, and to mutually benefit from other partners’ added advantage. There were indeed several cases where one project would use the training experts or parts of a training program of another project, hence taking advantage of what the others do best, saving much needed financial resources, and avoiding effort duplication. Such examples are: • EFE offered its Workplace Success training curriculum and trainers to train AFEM’s “Reprofilage” young women and to Al Jisr’s Digital Opportunities Workshop trainees. • Emploi Habilité offered its Life Skills training program and its trainers to 3 MEEA projects: l’Heure Joyeuse, Al Jisr, and ENDA. • Al Jisr Digital Opportunity Workshop refurbished computers and sold them at a minimal price to Emploi Habilité to provide basic IT lessons to out-of-school youth. • Heure Joyeuse hired a Reprofilage (AFEM) graduate, and offered training to young women referred by Emploi Habilité, and is now developing a pre-vocational training program that will start preparing out-of-school youth that are below 15 in order to orient them to the Emploi Habilité Vocational Training Centers (CFAs). • AFEM offered internships to three Emploi Habilité youth and is expecting to continue offering paid internship positions to at least three Emploi Habilité youth each year. 8 The evaluator was not able to get a final number of the private sector partnerships developed by the MEEA, as many of these partnerships were informal, and were not documented using an MOU or other forms of cooperation agreement. Emploi Habilite Heure Joyeuse EFE AFEM Al Jisr ENDA Similar to other EEA programs, such as EEA-Egypt, building alliances around the MEEA program not only helped implementers better implement project activities but also enrich individual projects by adding components that were crucial to their successes. Implementing partners felt that the MEEA projects’ cross-collaboration is one of the major achievements of the Alliance, and one that will last and expand far beyond MEEA funding. All implementing partners confirmed that MEEA helped them improve their capacities to work with youth to a great extent. In fact, partners were also asked to assess the quality of the assistance they received from the Steering Committee. Three out of the four implementing partners interviewed rated the technical assistance from the MEEA as good or excellent. One of them stated that what they learned from MEEA continues to help them with the implementation of the subsequent phase of the project. Another partner said that it was because it it allowed the association to re-formulate the content of its training program by adding non-technical components. When implementers were asked what they gained from their collaboration with the Alliance, they cited the following: • Expansion of the associations’ network • Development of new alliances in other regions of Morocco • Reinforced the credibility of the associations as an organizations that helps youth • Other partnerships with current alliance members • Exchange expertise and knowledge • Exchange of services with other MEEA members (EFE, AFEM, ALJISR) All partners have implemented traditional development projects that do not use the alliance approach. When asked to describe their alliance experience in comparison with traditional development models, two said it was similar, while three said it was better because of its innovative approach that fostered synergies among the different actors in the field of youth training and employability development and harnessed what each of these actors does best for greater project effectiveness. When asked what they liked most about MEEA, partners said its quick responsiveness. The IYF Secretariat did indeed respond immediately to implementing partners’ requests for assistance in relation to project monitoring and problem troubleshooting (especially in relation to recruiting youth and trainers as well as reporting). However, all implementing partners said they were expecting more involvement from the Steering Committee in placing youth in jobs and internships. The MEEA Steering Committee members had a wide range of business connections, but the level of integration of MEEA trained youth into these networks was limited. In addition, implementing partners who were not represented on the MEEA Steering Committee commented that they would have liked to have more regular contact with the Steering Committee members, through regular meetings, to discuss project implementation issues. Between 2005 and 2008, only two Steering Committee meetings involved project implementers’ representatives, and those two meetings did not happen until toward the end of the MEEA program. Figure 5 presents the percentage of partners who requested and received certain types of technical assistance from the EEA Secretariat/IYF based on their institutional needs. For example, all partners requested and received assistance from the Secretariat in monitoring and evaluation, whereas roughly 65% of partners needed and received help forming linkages. Nearly forty percent of implementing partners sought technical assistance from the Secretariat in areas of financial management, institutional development, project development and trouble shooting. “Our project coordinator has been trained by AFEM’s Reprofilage project; our trainers were trained by Emploi Habilité Master Trainers, our youth are benefiting from the Life Skills curriculum offered through Emploi Habilité, and now my organization is accredited by the State as an official Vocational Training Center (CFA), and it is all thanks to the experiences we gained through working with the Alliance.” Mrs. Geneviève Berrada, Project Manager, and Treasurer, Heure Joyeuse Nawfal Fassi Fihri, CEO, EFE￾MAROC: “Without the alliance EFE would not have been aware of other organizations working in the youth employability field” Figure 5: Classification of Technical Assistance Provided by the Secretariat Business partners strongly supported the public-private alliance model. MEEA has supported and promoted a new culture based on the principle that the private sector must be directly involved in development, and should be encouraged to stay involved to improve effectiveness and sustainability of vocational training and job readiness programs. The strongest supporters of these types of alliances are those who have direct experience with them, even when alliance outcomes are mixed. There is also recognition that MEEA provided partners with an opportunity to test and validate public-private partnership models. Specific examples include partnerships fostered between local businesses and the Regional Education Authority in Casablanca (AREF), to offer some 700 youth one to two-year internships that will ultimately lead to full time employment. Partner public institutions, such as the University Hassan II and the Regional Education Authority in Casablanca (AREF Casablanca), also expressed satisfaction from partnering with the Alliance, citing that the program has expanded and improved their core curricula through the addition of new courses, particularly the Workplace Success and Passport to Success Life Skills and Microsoft’s Unlimited Potential basic IT curricula. Overall, findings regarding MEEA’s effectiveness in building partnerships indicate its strong success in fostering a new culture of partnerships among public, private and civil society institutions. Findings regarding MEEA’s effectiveness in promoting sustainable youth programs indicate that MEEA was effective in piloting several projects and positioning them to be sustainable and continue beyond MEEA and USAID funding. Yet sustainability on a macro-country level proved more difficult for the Morocco Alliance. Despite some efforts to create a permanent entity that would continue its role of fostering partnerships for youth development, the MEEA Steering Committee was not able to maintain itself once the IYF Secretariat was no longer available to help coordinate it. In this regard, expectations for the development of such an entity may have been too optimistic, given the significant effort required to start up any new enterprise. Moreover, as mentioned above, there was a perception also that members had not been sufficiently proactive in using their networks to support job placement of MEEA youth graduates. These challenges may be attributable to the profile of the Steering Committee members who were fairly high-level private sector leaders who lacked sufficient time for this type of more hands-on support. An additional reason may also have been unrealistic expectations in this regard, with other country alliance programs focusing more on Steering Committee’s roles in program design and formulation, rather than Mr. Yahia Melhaoui, From Hotel Ibis Moussafir, Casablanca (ACCOR Hotel Chain) said when asked to comment on the hotel’s collaboration with the Emploi Habilité Project: “This is truly a win-win situation. The youth gain a lot of knowledge and technical skills through the internships we offer them in our hotel; and the hotel gains from the work the CFA youth perform at the hotel. We contribute to the training of the youth, and we teach them the skills that are useful for our hotel, and of course, our hotel will gain in employing these youth as interns and as staff, because they have exactly the skills we need.” direct support in job placement, which was typically a function of project and secretariat staff. The figure below summarizes the sustainability status of each MEEA project: Figure 6: Sustainability Status by Subproject Project Name Project Objective Partners Sustainability Status Digital Opportunities Workshop To provide fifty unemployed young people with training in computer installation and repair and to provide 50 schools and 25,000 students access to computers they refurbish Al Jisr, Microsoft, Moroccan Ministry of Education Al Jisr is in the process of opening a third workshop in Agadir in partnership with with Banque Populaire, Regional Academy of Agadir and Ateliers Sans Frontieres. Emploi Habilite To improve the employability of 1,244 unemployed youth in Casablanca through life skills and IT training provided through Ministry of Education Vocational Schools and Youth Centers Mjid Foundation, Al Jisr, GE Foundation, Microsoft, Moroccan Ministry of Education Emploi Habilite is being expanded through additional funding from Qatari Foundation, Silatech, EFE￾Maroc and Finnish Children and Youth Foundation (FCYF). Workplace Success To provide 230 youth with life skills training, placing 150 of them in jobs, and developing strategies for institutionalizing this life skills curriculum in the Hassan II University offerings Education for Employment Foundation, McGraw Hill, Hassan II Univ. EFE signed an MOU with Hassan II University to integrate the project into the university’s curriculum through select departments. The program is being expanded with additional funding from FCYF. School for Entrepreneurs To assist in the creation of a School for Entrepreneurs, where 135 students will receive entrepreneurship training and eighty will be mentored by successful entrepreneurs in starting up new enterprises CJD, Federation of Small Industries, Hassan II University CJD received additional support to replicate their program in Agadir. Reprofilage To provide 55 unemployed young women with training relevant to the employment needs of Moroccan companies and to place them in internships and/or jobs Moroccan Association of Women Entrepreneurs, Member companies Received additional funding from a Canadian donor to replicate the model in Marrakesh and continue in Casablanca. Point Jeune To offer training, job placement and other support to 1,220 young, disadvantaged slum residents (400 of whom are expected to be placed in jobs) through the creation of a new Community Center Job Point and Digital Opportunity Workshop ENDA Maghreb, Fondation Air France, Holding Al Omrane, National Initiative for Human Development Project ended due to management issues in the third quarter of 2008. Preparing Disadvantaged Young Women for Apparel Sector Employment To provide life and technical skills training to 130 disadvantaged Moroccan women in the apparel and textile industries, with 53 of them targeted to secure jobs in this sector L'heure Joyeuse, Nike L'heure Joyeuse gained accreditation as a government training center and will be able to provide government certified training. In addition, L’heure Joyeuse is continuing to offer life skills and basic entrepreneurship training as part of its ongoing technical training. Cost Effectiveness: The overall cost effectiveness of the MEEA program has been assessed at two levels. The first level is related to country program subgrants, which include both seed grants administered by USAID and cash and in-kind leverage contributions contributed by other partners to support EEA country programs. The second level more broadly includes costs classified as alliance building and program support costs. This second level includes costs outside of those directly incurred by project implementers. They accordingly include costs related to the functioning of IYF as the Secretariat, local consultants and advisors in Morocco, as well as IYF direct and indirect expenditures attributable to the MEEA program. Each of these cost categories are discussed in more detail below: Subgrant Project-level Costs: As noted above, subgrant project-level costs include seed funds and leverage contributions invested toward individual projects which typically covered curriculum design or improvement, training, job placement costs, and project specific costs of NGO subgrantees. The amount of seed funding support from USAID toward individual projects is a useful metric to assess cost effectiveness given that such funds are directly focused on implementation of projects, are directly leveraged by investments from other parties, and are best positioned for increased efficiency when additional beneficiaries are added to successful pilot programs. In Morocco, the USAID support of $538,825 in seed funds that were coupled with leverage investments of $2,399,889 reached a total of 2,238 beneficiaries. If all project-level costs are taken in account, this brings the cost to $1,313 per beneficiary because of significant leverage raised by MEEA, of which only $241 was borne directly by USAID and $1,072 was shouldered by the alliance partners. Figure 7: Project-level Costs Subgrant Project Name Number of Beneficiaries USAID Funds Leverage Total Project Budget Cost Per Beneficiary Digital Opportunities Workshop 92 $ 147,583 $ 540,520 $ 688,103 $ 1,896 Emploi Habilite (& PTS Life Skills) 1,237 $ 88,105 $ 1,238,024 $ 1,326,129 $ 1,072 Workplace Success 182 $ 75,000 $ 281,964 $ 356,964 $ 1,961 School for Entrepreneurs 433 $ 52,964 $ 25,463 $ 78,427 $ 181 Reprofilage 61 $ 73,772 $ 213,918 $ 287,690 $ 4,716 Point Jeune 140 $ 85,748 $ - $ 85,748 $ 612 Nike Life Skills Training 93 $ 15,653 $ 100,000 $ 115,653 $ 1,244 TOTAL 2,238 $ 538,825 $ 2,399,889 $ 2,938,714 $ 1,313 Alliance Building and Program Support Costs: A more expansive examination of costs by necessity also includes those alliance building and program support costs related to the MEEA program. These costs supported significant, collective efforts by IYF and local program consultants. Overall efforts focused on a number of items related to program activities, including grant administration, managing the design and implementation of alliance based interventions; providing international technical expertise on successful employability approaches; assisting in leveraging of resources; providing capacity building support to implementing partners in skills training, job placement and post training support to entrepreneurs; and providing financial oversight, compliance, and direct monitoring and evaluation support. Upon examination of IYF accounting records, these costs taken together with indirect costs attributable to the MEEA program, totaled $1,227,933. This USAID-borne cost adds approximately $549 on a per beneficiary basis. Figure 8: Alliance Building and Program Support Costs Subgrant Name Number of Beneficiaries Alliance Building and Program Support Costs Cost Per Beneficiary Digital Opportunities Workshop 92 $ 175,419 $ 1,907 Emploi Habilite (& PTS Life Skills) 1,237 $ 175,419 $ 142 Workplace Success 182 $ 175,419 $ 964 School for Entrepreneurs 433 $ 175,419 $ 405 Reprofilage 61 $ 175,419 $ 4,279 Point Jeune 140 $ 175,419 $ 1,253 Nike Life Skills Training 93 $ 175,419 $ 1,886 TOTAL 2,238 $ 1,227,933 $ 549 Taken together, these overall costs of both seed funds and Alliance building program support costs total approximately $1,862 per beneficiary, and appear roughly equivalent compared to other good practice country projects of this duration (one to two years for the bulk of participants) and that include internships, on the job training and international technical assistance. Of the cost of $1,862 per beneficiary, $789 (42%) was borne by USAID and $1,072 (58%) was contributed by alliance partners. Figure 9: Cost per Beneficiary Cost per beneficiary USAID Alliance Contributions Total Cost Per Beneficiary Project-level $ 241 $ 1,072 $ 1,313 Alliance building and program support $ 549 $ - $ 549 Total $ 789 $ 1,072 $ 1,862 Finally, and perhaps most significantly, a full assessment of cost efficiency would also require a more thorough assessment of the effectiveness of alliance building activities on long term sustainability and scalability of proven practice programs. As such, the MEEA model of programming for example demonstrates the significant increase in beneficiaries and outcomes possible through an alliance building approach, particularly with government or quasi-government agencies as partners. Should the new efforts to substantially expand the EEA model be successful, the impact and cost effectiveness of initial EEA investments could be dramatically expanded. While the longitudinal study required for this type of evaluation goes beyond the scope and resources available for this study, such factors should be addressed more directly in similar employability projects in the future. Indicator 1.3: Number of private sector, government and civil society partnerships formed through the project to jointly offer employment and business development training activities in targeted areas Under the MEEA program, the Steering Committee and the MEEA implementing partners formed some forty partnerships that included governmental institutions, private businesses, and local and international NGOs. These partnerships were either formal (based on an MOU or signed agreement), or informal (tacit/oral agreement) – and helped support project implementation and sustainability. The table below illustrates partnerships developed through the Alliance by each implementing partner. An illustrative list of MEEA partnerships by type and role of partner is included in Annex A. Figure 10: Number of Partnerships Formed by Subproject Project Number of partner￾ships NGO Partners Private Sector Partners Public Sector Partners Emploi Habilité Over 200 M’Jid Foundation, Al Jisr, Near East Foundation, Al Karam, Fondation Auteuil, PlaNet Finance, DARNA Microsoft, some 200 private businesses placing youth in internships (including McDonalds, Ibis, Accor, Richbond) Ministry of Education-Regional Academy of Casablanca (AREF), Department of Vocational Training, ANAPEC, OFPPT Workplace Success/EFE 7 AFEM, Al Jisr BMCE bank, Manpower, YNNA holding Hassan II University, ANAPEC Reprofilage/AFE M (Skills upgrading) 3 EFE 200 AFEM member businesses Hassan II University, ANAPEC School of Entrepreneurs/ CJD 5 EFE Maroc, and over 200 CJD network members DELL, PAAP Hassan II University Training Young Women to Work in the Apparel Sector/L’Heure Joyeuse 5 Projet Emploi Habilité, Projet ALEF (AED & MSI), GTZ Nike, ESITH ANAPEC Project Number of partner￾ships NGO Partners Private Sector Partners Public Sector Partners Digital Opportunity Workshop/Al Jisr 16 Al Jisr, Ateliers sans frontières (ASF), EFE. Microsoft Afrique du Nord, Munisys, Attijariwafa bank, Wafa Gestion, Wafa Salaf, Microdata, Wintor Nixdors, Foundation Safrioui Badissi, L’IDEC, La SAMIR, Crédit Suisse, Hundai, Bull AREF Casa, EFE/Hassan II University Indicator 1.4: New partnerships, mechanisms and approaches created as a rippling effect of working with EEA programs to support youth employability Building on their work and experience under MEEA, all implementing partners have developed new partnerships with other local NGOs, international organizations, as well as with public and private entities. These new partnerships contributed to ensuring the continuation, replication, or expansion of individual projects. Figure 11: New Partnerships to Continue, Expand, or Replicate MEEA Projects Project/Grantee Donor Objective Reprofilage/AFEM ACDI Funding used to continue training of young female university undergraduates in Casablanca, and to replicate the project in Marrakesh School of Entrepreneurs/CJD Replication of the project in the Southern city of Agadir Workplace Success/EFE MEPI Continuation and expansion of program nationwide FCYF SILATECH Preparing young women for employment in the Apparel Sector/Heure Joyeuse INDH, Ministry of Labor Continuation of the project in Casablanca, Development of a new pre-vocational training program using the MEEA tools and expanding the training to youth populations that have not been covered by the project before, creation of a CFA within l’HJ to offer youth technical and life skills training Emploi Habilité USAID Expansion of project in the North FCYF Expansion of project in the region of Casablanca and nationwide SILATECH Expansion of project within the tourism sector In addition, other NGOs, donors and foundations within outside of the Casablanca region have approached project implementers to adopt MEEA training programs. These groups included: SILATECH, the Finnish Children and Youth Foundation, PlaNet Finance, Near East Foundation, Fondation Auteuil, and Association Solidarité Féminine. Although MEEA has not managed to establish a permanent Secretariat or legal entity that will continue to directly create new partnerships after the initial seed funding, MEEA provoked a “ripple effect” that led the implementing partners to develop new partnerships. In fact, all seven project implementers have noted that they can no longer continue to work in isolation to achieve their missions, and that the success of their future projects depended on forging effective and efficient partnerships with the private, public and civil society entities. In addition, project implementers have introduced the Alliance approach to their new partners, who in turn have adopted it for the implementation or strengthening of their operations and activities. One such example is PlaNet Finance - a French NGO established in Morocco - that entered in partnership with MEEA’s Emploi Habilité program. It has since benefitted from the life skills training, and started using both the training methodology and alliance model in the city of Tangier North of Morocco, developing new partnerships with other local associations working with youth in the North, such as Darna Association. Intermediate Result II: Improved Access to and Relevance of Education and Employability Skills Training Evaluation Question/Intermediate Result II: How successful was the MEEA in improving access to and quality of education and employability opportunities for youth? The study used seven indicators to assess the Intermediate Result II: • Indicator 2.1: Number of young men and women participating in MEEA programs • Indicator 2.2: Number and percentage of young men and women completing MEEA supported programs • Indicator 2.3: Number of teachers, trainers, facilitators and counselors trained under MEEA supported programs • Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in EEA supported programs • Indicator 2.5: Level of satisfaction of young trainees with the quality and relevance of the training, counseling and services received through EEA programs • Indicator 2.6: Number and percent of youth who completed MEEA supported projects who report that they have referred at least another youth to the program • Indicator 2.7: Level of employers’ satisfaction with the soft skills and/or technical skills of MEEA trained youth Summary of Intermediate Result II Findings An aggregated 2,238 youth participated in the MEEA-supported training and 1,688 successfully graduated, thereby exceeding the original completion target of 1,509. MEEA also helped strengthen the skills of 141 trainers in delivering technical and life skills curricula. Youth reported strong satisfaction with the training and findings point to improved attitudes and skills among youth participants. Employers expressed satisfaction with capabilities and life skills of MEEA interns and employees. Indicators 2.1 and 2.2: Number of young men and women who participated and completed MEEA￾supported training MEEA targeted two broad categories of unemployed youth in Morocco: ¾ Out-of-School Youth: MEEA was one of the rare programs implemented in Morocco that focused mainly on out-of-school youth. Indeed, aware of its weak education system that results in high dropout rates, especially between grade six and grade nine, the Moroccan Government has adopted several reforms to strengthen the formal education system in public schools, and created new technical training tracks at the high school level to provide youth with more diversified venues for their carriers. However, up to 2005, there was not a single comprehensive program targeted to the different categories of school dropouts. In response to this huge gap, four out of seven MEEA projects focusedon out-of-school youth, and attempted to offer these youth comprehensive programs designed to develop and strengthen technical and life skills of these youth. The projects that targeted to out-of-school youth were: • Emploi Habilité • Training Young Women to Work in the Apparel Sector - Heure Joyeuse • “Point Jeunes” Digital Opportunity Workshop – ENDA • Digital Opportunity Workshop - Al Jisr ¾ Unemployed University Graduates: MEEA also attempted to address one of the most challenging problems in Morocco, namely, the problem of unemployed university graduates or “Diplomés Chomeurs.” Morocco is characterized by a large number of unemployed university graduates who have not found jobs after graduation. A small study conducted by IYF on businesses in Casablanca in preparation for the design of the MEEA program showed that the high unemployment rate among graduates was not due to the lack of employment opportunities, but rather on the inadequacy of youth qualifications to the job market demands. To respond to this problem, MEEA used the Alliance approach and mobilized resources from public education institutions and private sector actors to implement two skills upgrading projects for unemployed graduates - Reprofilage/AFEM and Workplace Success/EFE. An aggregated 2,238 participated in the MEEA training, and 1,688 youth completed training supported by MEEA, exceeding the target of 1,509 established for all of MEEA projects. Figure 12 presents the target and actual numbers of youth who participated and completed MEEA training, and subsequently obtained internships and jobs by subproject. Figure 12: Number of Youth Participated, Completed and Placed in Internships and Jobs Project Name Start and end dates Number of youth who participated in the program Number of youth who completed the program Number of youth placed in Internships Number of youth placed in jobs Target Achieved Target Achieved Target Achieved Target Achieved Emploi Habilité Pilot June 1, 2006- May 31, 2007 48 48 48 44 48 44 14 16 Emploi Habilité Phase II Cohort 19 Sept 1, 2007 –June 30, 2008 300 10710 240 107 240 107 240 54 Emploi Habilité Phase II Cohort 211 Sept 1, 2007 –June 30, 2009 450 314 360 314 360 314 360 - Emploi Habilité Phase II Cohort 3 Sept 1, 2008- June 30, 2010 450 395 360 - 360 380 360 Workplace Success/EFE July 1, 2006- Dec 30, 2009 230 182 230 182 196 182 196 113 Reprofilage/ AFEM (Skills upgrading) Oct 1, 2006- June 30, 2008 55 61 50 41 50 38 38 32 School of Entrepreneurs￾CJD January 1, 2007- Dec 2008 80 433 80 433 0 0 0 2 “Point Jeunes” Digital Workshop/ Job Nest – ENDA Maghreb January 1, 2007- Dec 2007 1220 140 20 39 20 18 13 12 9 Each training cohort of the second phase of Emploi Habilité has been presented separately in this table to better assess the effectiveness of the program, given the lack of job placement and completion data available for those that started the training program in September of 2008. In this respect, the first cohort of phase II underwent training (inclusive of internships) that started in September 2007 and completed in June 2008. Accordingly, more complete data is available with respect to this cohort and is presented separately in this table. Thereafter, the second cohort that also started its training in September, 2007 but followed a two year degree track completed its training program in June 2009. Unfortunately, placement was still in process for this cohort as this evaluation was being completed and full tracking was not possible once the program ceased operations in Morocco. The final third cohort of youth started in September, 2008 and they are expected to complete the program in June 2010. As such, for this purpose, for both the second and third cohorts, we have included such trainees as having completed the program but not counted job placement figures in overall totals. 10 It should also be noted that with respect to cohort 1 participation figures, there were an additional 617 youth that dropped out of the program, largely around three months into the program before internships started as a part of their training. This dropout rate was attributable largely due to unrealistic assumptions regarding the absorptive capacity of the government run vocational training centers, as described in more detail in the narrative below. For the purposes of this chart, however, we have not included such dropouts as participants to avoid skewing the impression of significantly more participants that had very limited participation in the program. 11 In dividing data between the three cohorts of the second phase of Emploi Habilite program, certain assumptions were made about target numbers set forth by the program. More specifically, while the program set a target range of 1,000-1,400 participants in the program, for this purpose we have taken the midpoint of 1,200 as the overall target, and divided this amount over the three cohorts taking into account normal startup time. For the purpose of calculating targets for completion and job placement, we followed the original project targets of eighty percent of all participants completing the training, getting internships and securing jobs. As noted in the narrative, these targets appear to have been unrealistically high given the absorptive capacity of the CFA system and lower rates of employment within similarly situated, well regarded vocational training systems. Nonetheless, in this respect, it is of interest to note that job placement among completers during the first cohort was approximately fifty percent, and while job data are unavailable for the second and third cohorts, as the program went into its second year, its rate of internships have steadily increased, reflecting learnings of how to better work within the constraints of this government training system. Project Name Start and end dates Number of youth who participated in the program Number of youth who completed the program Number of youth placed in Internships Number of youth placed in jobs Training Young Women to Work in the Apparel Sector May 1, 2007-July 31 2009 130 93 91 68 91 45 31 31 Al Jisr Digital Opportunity Workshop June 1, 2006-June 30, 2009 30 92 30 87 30 67 24 40 Number of youth who completed Emploi Habilité life skills training12 0 373 0 373 0 NA 0 NA Total 2,993 2,238 1,509 1,688 1,395 1,195 1,276 300 All youth who completed the program were aged between 16 and 25, with very few exceptions (see Figure 13). Most of them were disadvantaged youth from low socio-economic class. But educational background and level of these youth varied because each project was targeted to a different category of out-of-school youth. Figure 13: Beneficiaries by Age Overall, 46% of beneficiaries were young women and 54% were young men. Two projects (Reprofilage – AFEM and Training Young Women for the Apparel Sector - Heure Joyeuse) specifically targeted young women. Figure 14: Classification of Graduates by Gender and Project Project Women Men Total % Women % Men Emploi Habilité Pilot 14 30 44 32% 68% Emploi Habilité Phase II 139 282 421 33% 67% Point Jeunes Digital Workshop – ENDA 13 26 39 33% 67% Digital Workshop – ALJISR 26 61 87 30% 70% Reprofilage – AFEM 41 0 41 100% 0% School of Entrepreneurs – CJD 237 196 433 55% 45% Training young women in the Apparel Sector - Heure Joyeuse 67 1 68 99% 1% Workplace Success – EFE 108 74 182 59% 41% TOTAL13 645 670 1,315 49% 51% 12 Through the Emploi Habilité II subgrant, youth-serving NGOs who were not MEEA subgrantees were trained on the Passport to Success Life Skills curriculum in order to reach greater numbers of youth with the life skills program. Because these NGOs were not MEEA grantees, this data was tracked separately. 13 This total does not include the 373 youth who received life skills training through the EH program because the project was not obligated to track these youth by gender. The IYF Secretariat was not involved in the NGO grantees’ process for recruiting youth, and each project used different recruitment methods. Partners’ methods included: • Announcements in local schools and other education institutions • Government databases of unemployed job seekers • Door to door by local government officials and project outreach people (social assistants, placement agents) • Brochures, posters, announcements placed in locations where youth gather • Seminars and small to large scale communication events, participation in local and national fairs • And finally, word of mouth which was in fact the most effective of all these methods, as over 97% of surveyed youth said that they registered in the various training programs because they were referred by other youth who went through these training programs. Some of the MEEA projects experienced difficulties in recruiting youth during the initial phase. The Reprofilage project implemented by AFEM experienced recruitment challenges because they had had initially relied on a massive government database to identify candidates which ultimately did not help identify motivated candidates with the appropriate profile. Throughout the program, L’Heure Joyeuse was not able to recruit the number of young women they had anticipated due to changed interests among the young women they had been targeting, more options for vocational training for youth to choose from due to recent government investment in job training, and relocation of the apparel factories so that they were not close in proximity to L’Heure Joyeuse or the targeted young women in their neighborhood. Another challenge was the fact that some youth dropped out as the projects were underway. MEEA’s overall dropout rate is quite low compared to dropout rates registered by vocational training programs offered by public institutions (which can reach up to 50%). Overall, excluding the Emploi Habilité Phase II program which is a specific case as discussed below, 155 youth dropped out, bringing the total drop-out rate to an average of approximately ten percent. In this respect, of the non-Emploi Habilite participants, 1,422 youth enrolled in different training programs after the initial registration/orientation phase (which lasted from a week to one month depending on the training program), and 1,267 youth (89%) completed the different courses and internships. Overall, across the MEEA program, there were two stages at which most dropouts occurred: at the registration stage and at the end of the training program. Most youth dropped out at the registration phase, i.e. within the first month of training. Those who remained were generally fully committed and stayed until the end of the training. Typically, at the registration phase, all MEEA projects (with the exception of L’Heure Joyeuse project) received high numbers of candidates who wanted to register in the various training programs offered. The registration lists exceeded the target numbers set up for each project, and reflected the high demand for the kind of training programs offered by MEEA. However, after the first orientation month, a large portion of these youth dropped out for several reasons: ¾ After the orientation phase, youth who came just because they were curious or bored, realized that a high level of commitment was required from their part to continue benefiting from MEEA supported training, and they decide to quit because they were not willing to fully commit their time to the training. ¾ Youth who did not meet the minimum education requirements to allow them to pursue the training quit after attending the first class. ¾ In some cases, the implementing partner did not explain sufficiently the objectives of the training to the youth before selecting them, and in these cases, youth realized after the start of the training that this was not what they were expecting, so they quit. ¾ In other cases, the implementing partners did not set up clear selection criteria based on which the youth would be chosen to benefit from a specific training program, which resulted in a mismatch between the training content and scope and the youth profile and/or expectations. ¾ A few more youth left the program to enroll in formal schooling or to take advantage of employment opportunities. Two projects --Reprofilage/AFEM (49%) and Training Young Women to Work in the Apparel Sector/Heure Joyeuse (37%) -- registered the highest dropout rates because of all the reasons stated above. The Emploi Habilité Phase II project represented unique circumstances in this respect, and had high dropout rates for different reasons. At the outset, given the success of the Phase I pilot, the project kept the registration and selection process open in order to encourage the maximum number of youth to register. This strategy resulted in a massive group of youth registering at the beginning of the school year, reaching over 1,500 in the 2007-08 school year. After the first orientation month, however, almost fifty percent were forced to drop out because there was not sufficient staff capacity in the CFA or the program to place such a large number of youth in internships. Since the training offered by this project was based on a twenty percent in-class and eighty percent enterprise-based training, those youth who did not sign an internship contract had to quit. This showed the project team that: a) although the seating capacity of the training centers working with the project is high and can reach over 1500 in any given year, the capacity of these centers to place youth and follow up on them remains limited, and the maximum they can handle is 700 youth in a given year; and b) the private sector in any given region of Morocco has a limited capacity to provide internships to youth with adequate supervision and coaching. Hence, to ensure proper follow up and quality of training and internships, the decision has been made by the project implementers that in the future, a total not to exceed 700 youth will be admitted in the Casablanca Ministry of Education Vocational Centers (CFAs) during any given year. Indicator 2.3: Number of teachers, trainers, facilitators and counselors trained under EEA supported programs A major strategy of MEEA programs was to build capacity within partner institutions in key content areas, including life skills, technical subjects and Information Technology (IT). As a result, MEEA organized or supported several training of trainers workshops between 2005 and 2008, providing training to some 141 trainers in Life Skills, IT, and technical skills – see Figure 15. Out of the 141 trainers and teachers who were trained, 52 were school teachers trained in IT by the Association AlJisr using the Microsoft training program to allow these teachers to use the computers refurbished by the Digital Opportunity Workshop graduates, and distributed them to over fifty primary schools in Casablanca. An aggregated 74 trainers were trained under the Emploi Habilité and the Workplace Success projects in Life Skills. A total of 15 were trained to deliver Entrepreneurship Spirit training and to coach potential young entrepreneurs. Figure 15: Number of Teachers, Trainers and Counselors Trained by Project Subgrant Project Total Target Actual Al Jisr Digital Opp. Workshop 4 52 MJID Emploi Habilite 19 19 Work Place Success – EFE 35 31 Reprofilage - AFEM - - School of Entrepreneurs – CJD 19 13 ENDA Maghred: Point Jeunes - 2 L'Heure Joyeuse - 4 Al Jisr Emploi Habilite Phase II 20 20 Total 97 141 Most of these trainers were existing staff in MEEA partners (University Hassan II, AREF Casablanca, Ministry of Employment, Al Jisr, Heure Joyeuse, EFE, AFEM network, CJD network. Such kind of human resources leveraging reduced the project cost, facilitated the integration of the program within partner institutions, and contributed to program sustainability. Figure 16 summarizes youth’s ratings of their trainers/facilitators using a five-point scale (1-very poor, 2-poor, 3-fair, 4-good and 5-excellent). Youth ratings of their trainers were above four corresponding to good, which demonstrated that MEEA was successful in improving the capacity of trainers and delivering good quality training. Of 85 youth respondents, 49% rated the overall performance of their trainers as good, and 46% chose excellent. Figure 16: Youth Satisfaction with Trainers and Facilitators Implementing partners considered the training provided by MEEA to their trainers as a value-add not only to their project, but to their organization as a whole. Interviews with trainers indicated that they themselves reported having undergone a great transformation after being trained on the Passport to Success Life Skills Training of Trainers. More specifically, trainers who received training in Life Skills (74) have changed the way they see teaching and interacting with youth and at the teacher-student relationship. All of them said that they changed their teaching methodology and adopted an interactive teaching method and a coaching style that responds better to the needs of youth. Some even reported that the life skills training provoked a change in their personality and in their social life. Two specific things trainers noted of the PTS life skills training were: • Interactive methodology used during the training of trainers (TOT) • Team spirit created among the trainers during and after the TOT Moreover, although they were expert adults who are experienced in working with youth, trainers said that the life skills training of trainers highlighted sides of their personality that they never suspected existed in them before. Two main changes in personality reported by trainers included: • Self-confidence: Trainers reported improvements in self-confidence and capacities after the training of trainers. • Listening skills: Trainers also reported that they used to think that they were good listeners, but during the training, they realized that they actually had to work more on their listening skills. To ensure the sustainability of MEEA supported training programs, some projects offered the training of Master Trainers. For example, Emploi Habilité offered such training to the best ten life skills trainers - seven trainres from CFAs and three from other organizations. Several TOTs have already been delivered by these new Master Trainers (Tetouan, NEF, Planet Finance, Heure Joyeuse). Monitoring of the life skills trainers in particular highlighted a few potential obstacles to the trainers’ work with the target youth: • Ministry of Education trainers who were trained as PTS life skills trainers do not have an officially recognized status by the Ministry as MEEA life skills trainers, which can make it more difficult for the program to be integrated into the Ministry and therefore sustained in the long term. • Trainers need regular coaching and follow up when they start delivering training to the youth, which requires substantial time and resources. Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in EEA supported programs MEEA used pre- and post- self-assessment tests to assess youth participants’ skills following training. Overall, all seven MEEA-supported training projects strengthened the technical and life skills of target youth. A total of 73 youth or 89% of 82 youth surveyed declared that they have acquired the technical skills needed to find a job. Using triangulation, employers were also surveyed and interviewed. Employer survey results also show that 64% of 17 employers surveyed rated the overall performance of MEEA graduates as good and 69% of 17 employers surveyed rated their technical skills as good. Detailed survey results are presented below: • Over eighty percent of responding youth felt that they gained technical knowledge and skills needed for employment. Eighty percent of youth surveyed in fact passed different technical tests and received over ten out of twenty (minimum passing grade) in their practical exercises. • Before the training, 23% of 111 youth who responded to this particular question rated their communication skills as very poor, 22% as poor and 43% as fair before the training on a five-point rating scale (very poor, poor, fair, good and excellent). After the training, 41% of 111 respondents reported the change as following: 27% reported huge improvements, 31% reported some improvements, and only 1 person reported no change. • Fifty-three percent of 111 respondents rated their listening skills prior to the training as very poor, poor or fair using a five-point rating scale. After the training, 29% reported some improvement, 43% reported good improvements and 26% reported huge improvements. • Seventy-four percent of 109 respondents rated their self-confidence as very weak, week or fair using a 5-point rating scale before the training. After the training, 73% of 109 respondents reported good or huge improvements in their self confidence. • Seventy-four percent of 110 respondents reported that their sense of initiative was very poor, poor or fair using a five-point rating scale before the training. After the training 61% of 110 respondents reported good or huge improvements in their sense of initiative. • Seventy-two percent of 108 respondents assessed their sense of responsibility as very poor, poor or fair before the training. After the training, 72% of the respondents reported good or huge improvements. • Seventy-seven percent of 106 respondents considered that their capacity to resolve conflicts was fair or poor before the training. After the training, 64% of 106 respondents reported good or huge improvements, and 26% reported some improvements. • Seventy percent of 105 respondents also reported that their relationship with family and friends improved as a result of participation in the training. Indicator 2.5: Level of satisfaction of young trainees with the quality and relevance of the training, counseling and services received through MEEA MEEA strived to ensure the relevance of all its training programs to the needs of the job market, through the involvement of the private sector, partner NGOs and public institutions in the design of the MEEA training programs. However, MEEA did not involve the youth themselves in program design so their specific input was not considered when planning the training programs. This is a shortfall of the program. In addition, MEEA projects did not offer systematic counseling services, except indirectly through the life skills training programs. The counseling services offered were not sufficient to respond to the needs of the disadvantaged youth population targeted by the MEEA program. Nevertheless, survey results indicate that program graduates were generally satisfied with the quality and relevance of training provided. The table below summarizes the youth’s satisfaction with various aspects of the training, with average ratings between 3.5 and 4.3 on a scale of five. When asked what the youth liked most about the MEEA-supported training, most answered life skills training while many responded that they enjoyed their trainers and the trainings’ hands-on approach. “If I had teachers like these when I was in formal school, and if they used with me the same methodology used in Life Skills lessons, I would not have quit.” - MEEA Youth participant Figure 17: Youth Satisfaction Rates There were several areas for improvement the youth participants recommended in their comments: • Youth from the Digital Workshop Opportunity program and from Emploi Habilité program felt that the technical training materials should be improved. • The duration of training provided by three out of the seven projects was considered too short Reprofilage/AFEM (four-month in-class and two-month internship); Digital Workshop Opportunity/Aljisr and Point Jeunes (ENDA) (six-month in-class and two-month internship). Emploi Habilité with its one to two-year tracks was considered of adequate duration by the youth and trainers. • With the exception of the Workplace Success (EFE) and Reprofilage (AFEM) projects, several youth cited the facilities used for life skills and technical training by the projects to be unconducive learning environment. Participants and trainers complained about broken and unsanitary lavatories, inadequate lunch and rest areas or absence thereof, and inadequate and unequipped classrooms. Participants in the Emploi Habilité project suffered particularly of most of these problems, due to the fact that the facilities were managed by the Ministry of Education, and state budgets were not used adequately to keep these facilities in good standing. Indicator 2.6: Number and percentage of youth who completed MEEA supported programs who report that they have referred at least one other youth to the program This indicator is both a measure of the program’s quality and relevance, as much as it is a measure of the impact that the youth graduates can have on their peers. Out of the 85 youth who responded to this particular question, 74 youth or 87% of the total respondents have reported that they have referred at least one person that has subsequently registered in one of the MEEA supported training programs. Some of them have reported having referred over 10 people. This is confirmed by the fact that 75% (66) out of 88 respondents reported that they have been referred to the program by a friend or a family member that has completed or is completing MEEA supported training program. MEEA graduates are now actively helping other youth to get out of their disadvantaged situation. All respondents stated that they wanted other youth to benefit from the programs as they themselves did. Indicator 2.7: Level of employers’ satisfaction with the soft skills and/or technical skills of MEEA trained youth The ultimate measure of youth’s acquisition of the technical and soft skills through the MEEA supported projects is the level of satisfaction of employers who have hired them either as interns or as permanent employees. This indicator also enables the project to verify the findings of indicator 2.4: percentage of young men and women demonstrating improved capabilities through participation in MEEA supported programs. A total of 18 employers were interviewed, representing the following sectors: tourism, IT, apparel and office administration. 14 14 Broken down by project, 22% of employers interviewed were from the Digital Opportunity Al Jisr Workshop, 39% from Emploi Habilite, 11% from Reprofilage, and 28% from the Nike training project. Answer Options Very poor Poor Fair Good Excellent N/A Rating Average Response Count Overall content of the training program 0 1 11 54 19 0 4.07 85 Quality of training materials 2 2 11 44 26 1 4.06 86 Relevance of content and training materials to employment needs 2 6 26 31 11 2 3.57 78 Interactive/practical exercises and activities (if any) 1 3 17 32 15 13 3.84 81 Life Skills and attitudes learned from training (if any) 0 0 10 30 35 7 4.33 82 Technical skills acquired from training 0 2 21 34 18 8 3.91 83 Detailed employer survey results are presented below: • In terms of employers’ satisfaction, of the 17 employers who responded to this question, two employers or 11% rated the overall performance of the youth as excellent, nine or 53% rated the overall performance of MEEA graduates as good and five or 29% as fair. • In a question asking employers to rate the technical skills of the youth, 16 employers responded and of these, eleven employers or 69% rated the technical capacities of the youth as good, four or 25% as fair, and six percent as poor. • In a question asking employers to rate the life skills of the youth, 17 employers responded, and of these, three employers or 18% rated the MEEA youth’s life skills as excellent, eight or 47% rated them as good, and six or 35% as fair. • All eleven employers who responded thought that MEEA has been able to provide qualified graduates for the business sector and it contributed to closing the gap between employers’ needs and the employees’ qualifications. • All employers who responded said they would definitely offer jobs, internships or apprenticeships to more MEEA graduates in the future. • When asked to compare MEEA graduates with other employees from similar ages and backgrounds, sixteen employers responded to this question. Three or 19% of these reported that MEEA graduates were much better than other employees, five or 31% reported that they are better, and eight or 50% think they are the same. It is important to take into consideration the fact that most projects provided basic, short-courses preparing youth for work. Graduates had to compete with regular vocational students who underwent much longer training programs. What they appreciated most about MEEA interns and graduates were their soft skills, which they consider even more important than the technical skills. Employers have changed their hiring patterns and practices as a result of working with the Alliance. Instead of posting vacancy announcements in the newspapers or on the web, or paying for the services of a placement agency that would not really respond to their specific needs, employers are now getting youth from MEEA projects directly. These employers now consider the “Alliance” as a pedigree that guarantees the quality of the young men and women they hire. There were three kinds of employers that offered internships and jobs to MEEA graduates: • Large international companies and Franchises such as Bull, Munisys (IT), New Rest (textile), Accor (hotels), Fruit of the Loom, McDonalds (restaurants): These large companies hired large numbers of young MEEA interns. • Medium enterprises such as the AFEM network enterprises (small companies run/owned by women), small textile enterprises (first sports), hotels and restaurants, steel and machine shops, auto repair, and construction companies. Each of these medium-sized companies would hire one to five young interns. • Small and micro-enterprises generally run by one or two people, such as auto repair workshops, small service companies. These enterprises would typically hire one or two interns at a time. Businesses represented a range of sectors including tourism, textile, sales, and services (including plumbing, electricity repair, welding, and mechanics), IT, office administration. Most youth were hired within the tourism sector. Typically, these employers hired youth as “on-the-job trainees” and hired those who excelled during their internship. Internships lasted from two to four months for most projects, with the exception of the Emploi Habilité Project that included one to two years of on the job training, depending on the technical training track. The types of employment included: • Full time employment in the formal sector: The tourism sector is the largest employer of MEEA youth, followed by the IT sector and the textile sector. • Part time employment in the informal sector: Most of the youth who received training in welding, plumbing, mechanics, electricity repair and auto repair tend to work as freelancers in the informal sector. They have not been organized in small/micro enterprises, but they have managed to make a living out of their professional activities. This evaluation also sought to assess if working with the Alliance resulted in any impact on the human resource practices of partner companies. Ten or 71% of the businesses surveyed said that they are more open to the idea of providing internships and mentorship support to youth in general, and to disadvantaged out-of￾school youth. Moreover, while at the beginning of the MEEA program, employers accepted MEEA interns as a charity act or as a favor, they now realize that MEEA is providing their companies with young qualified labor for much less. Surveyed employers offered the following recommendations: • Improve the language skills of the youth both in French and in Arabic, and ideally in English • Broaden their horizon through extracurricular activities if possible • Develop more familiarity with windows applications • Provide longer training in order to impart more technical skills • Further develop their sense of responsibility and initiative Mrs. Borie Odile, Director of KAVAA Global Services and member of the AFEM Network who hired two young graduates from the Reprofilage projects: “I congratulate all the young graduates of project Reprofilage, I congratulate all the partners for the success of this project. I would like to stress that the problems related to youth employability cannot be solved without a well targeted vocational training, and without a stronger involvement of all economic and social actors.” Intermediate Result III: Improved Prospects for Employment and Successful Entrepreneurship Evaluation Question/Intermediate Result III: How much did the MEEA program contribute to developing improved prospects for employment and successful entrepreneurship? The study used six indicators to assess the Intermediate Result III: • Indicator 3.1: Number and percentage of MEEA trainees who get a job within six months of completing the program • Indicator 3.2: Number and percentage of MEEA trainees who continue education, obtain an internship or engage in further professional training within six months of completing the program • Indicator 3.3: Number and percentage of MEEA trainees starting their own businesses within six months of completing the program • Indicator 3.4: Number and percentage of MEEA trainees with satisfactory internships or apprenticeships • Indicator 3.5: Number and percentage of employed youth who declare having a satisfying and quality job • Indicator 3.6: Net salary increase of youth employed through MEEA within six months after placement Summary of Intermediate Result III Findings Job placement rates exceeded fifty percent in four projects, although it appears that placement targets were originally set too high which proved unrealistic for pilot programs reaching primarily disadvantaged youth. There were also high internship placement rates reported by MEEA sub-grantees. Those who were working previously reported higher levels of income as a result of participation in the MEEA program. Indicator 3.1: Number and percentage of MEEA trainees who get a job within six months of completing the program Figure 18 presents employment targets and actual employment figures for each of the seven subprojects. Of the seven subprojects, two exceeded or met the original employment targets (AFEM and Emploi Habilité Pilot), and a total of four projects’ job placement rates exceeded fifty percent (Emploi Habilité Phase II - Cohort 1, ENDA, AFEM and EFE). Figure 18: MEEA Job Placement Rates by Project Project Number of program graduates Target employment rate for graduates Actual number of youth employed Actual percentage of graduates employed Emploi Habilité Pilot 44 30% 16 36% Emploi Habilite Phase II – Cohort 1 107 80% 54 50.5% Emploi Habilité Phase II – Cohort 2 31415 80% Data still being collected - Point Jeunes Digital Workshop – ENDA 39 80% 12 60% Digital Workshop – ALJISR 87 80% 40 46% Reprofilage – AFEM 41 75% 32 78% 15 As noted in our discussion of completion rates under indicators 2.1 and 2.2, we have split Empoi Habilite Phase II program participants into three cohorts to reflect the different time periods of participation in the program and the lack of full data for the last 2 cohorts of participants. In this respect, while cohort 1 had completed its training and placement for full measurement, cohort 2 was still in the six month process of placement as defined under this indicator 3.1, and cohort 3 was only midway through their training (inclusive of internships) while the evaluation was being completed. As such, for the purposes of this indicator in the chart above, we have not entered information about cohort 3 and, with respect to cohort 2, we have noted that 314 have graduated from the program which is marked by successfully completing their internships and overall coursework. Of these graduates in cohort 2, preliminary data collected from the evaluation, which occurred during 6 month job placement efforts measured under this indicator, indicates that 54 were currently employed, and the rest were in the process of being placed in jobs. More broadly, while job placement data is not fully available for both cohorts 2 and 3, as noted in indicators 2.1 and 2.2, it can be noted that of the 816 participating in training, 810 of these individuals have been placed in one to two-year internships as a part of their training, Project Number of program graduates Target employment rate for graduates Actual number of youth employed Actual percentage of graduates employed Training young women in the Apparel Sector - Heure Joyeuse 68 70% 31 46% Workplace Success – EFE 182 85% 113 62% Nonetheless, finding jobs for the newly trained youth was a challenge for all MEEA implementing partners. There are several factors that affected job placement of MEEA’s graduates. Firstly, very high targets of 80-85% were set for the projects when the programs were designed which were probably not appropriate for pilot training programs reaching disadvantaged youth. Other factors included: - The capacity of implementing partners to liaise with businesses was limited as was their experience in job placement. - The MEEA Steering Committee did not actively help in the placement process, as expected. Steering Committee members were either without time or not well-positioned to reach out to their business networks to assist with youth job placement. - Some of the Emploi Habilité youth who graduated from the electricity, plumbing, welding, and mechanics tracks are working in the informal sector, either as full time or part time freelancers, but are reluctant to report their employment status due to fear of being reported to the tax authorities or in order to be kept in the list of people that still need to be placed. - Businesses were initially very reluctant to hire out-of-school youth or youth who have not been able to get a job for two years after their graduation from the university. They considered that hiring these youth was very risky for their business productivity. - Employment policies impacted youth graduates’ ability to be hired as permanent employees particularly in the Emploi Habilité Phase 2 project. A policy was put in place that allowed businesses to hire youth as interns and pay them less than Morocco’s minimum wage. This then served as a disincentive to hiring youth as employees because employers did not want to pay those more. Therefore, youth either had to settle for a lower wage or became frustrated and left their internships to find a job elsewhere. - The recent global economic crisis affected certain sectors of the Moroccan economy, especially textile, tourism, construction and IT, causing a slowdown in the hiring of youth in these sectors. Indicator 3.2: Number and percentage of trainees who continue education, obtain an internship or engage in further professional training within six months of completing the program Over eighty percent of the respondents completed internships within small, medium and large enterprises because of the effective use of to the Alliance approach which allowed for the placement of these youth in internships within partner enterprises and companies. The duration of internships varied between two months to two years and most of the interns received small stipends during their internships. Some projects experienced initial constraints in placing youth in internships. Notably with the AFEM Reprofilage project, employers were not sufficiently oriented on the program at first, so they expected the intern to be a “finished product” rather than learning on the job while undergoing parallel classroom training as well. In addition, some employers rejected the intern candidates during the interviews because they were wearing the veil. MEEA supported training programs also assisted youth in continuing their education in the formal sector. Over fifteen youth were assisted by MEEA volunteer trainers to sit for the “Baccalaureat” exam (end of high school exam that allows youth to get access to university studies). Fifty percent out of the 102 respondents said that they would like to get further vocational training after they have finished the MEEA supported program. In addition, eighty percent of the surveyed youth reported that they decided to continue their education or engage in further professional training as a result of participation in the EEA program. Indicator 3.3: Number and percentage of MEEA trainees starting their own businesses within 6 months of completing the program The “School of Entrepreneurs” project led by the Center of Young Leaders (CJD) was the only project of the seven that focused on entrepreneurship development among youth. The project did not meet its target of coaching eighty youth to help them create their own business, but it did provide training to some 344 youth in the Entrepreneurship Spirit Program (ESP), a curriculum that provides an introduction to concepts of entrepreneurship and assessing youth’s entrepreneurship capabilities. After a very slow start, the “School of Entrepreneurs” project managed to help 13 youth to create their businesses in Agadir, a city in the south of Morocco. Because the businesses created are too young, it is not possible to assess their sustainability at this stage. The CJD/ESP program did effect a slow but noticeable change of attitude among youth. A separate evaluation was conducted on these youth by the ESP training provider, Management Systems International (MSI). The CJD subproject’s weak performance in terms of business creation is due to several factors, including: • The he weak management of the implementing partner, which suffered from high turnover of its senior staff and therefore led to delays in program implementation. • The fact that in Moroccan society, becoming an entrepreneur is still not considered as a reliable source of income, and youth still do not believe in their capacity to start and run new businesses. • Recent financial climate in Morocco and worldwide made it very difficult for small businesses to be launched and to survive. Attitudinal change is difficult to achieve, and requires a long time and continuous coaching and mentoring. MEEA projects have not resulted in the creation of a large number of new enterprises, but have managed to foster changed attitudes toward entrepreneurship among Moroccan youth. The School of Entrepreneurs is continuing its activities with the youth under new donor funding, and is expanding to new cities (Agadir). Two other youth through MEEA’s other subprojects (Al Jisr Digital Opportunities Workshop and Emploi Habilité) started their own businesses for a total of 15 youth starting their own businesses (target 121). Indicator 3.4: Number and percentage of youth with satisfactory internships or apprenticeships Internships were an integral part of the MEEA support training programs. The duration of internships generally varied between two to four months. MEEA’s Emploi Habilité subproject used a training model that included a combination of twenty percent classroom-based technical and life skills training and eighty percent internship over the course of one to two years. Trainees received small stipends for their work as interns or apprentices, and in some cases, especially for trainees in the tourism sector, the best performing interns received employment contracts at the end of their internship period. Youth were asked to rate their internship experience using a rating scale of one to five with one corresponding to very poor and five corresponding to excellent. Follow-up questionnaires with youth highlighted the following perspectives on their internship experience: • Twelve or 14% of 114 youth who responded to this question said that their overall internship experience was excellent, 36 or 42.9% said that it was good and 18 or 21% said it was fair. • In terms of the relevance of assignments to the training received, 28% (22) said it was good, 14% (11) as excellent while 14% said it was poor. • Approximately 44% (36) surveyed stated the technical skills learned through their internship were good, 19.5% (16) excellent and 17% (14) stated that they were fair, and 17% (14) stated that they were poor to very poor. • In terms of coaching received during internships, 25% of responding youth (21) mentioned than coaching was good, 21% (18) stated that it was excellent, and 14% (12) said it was fair, however, 31% (26) stated that the coaching they received was poor to very poor. Interns reported that what they appreciated most about their internships was the opportunity to learn new technical skills or refine their skills learned in classrooms. Below are some additional comments from responding graduates: • It provided the participants with knowledge and skills they have not learned during the training • It was an opportunity to practice skills they have learned in classrooms • It provided additional technical and professional experience • The internship does not fit with the training because the training did not prepare them in advance to work on the kind of sophisticated machinery that the factory uses There were some noteworthy gender differences in the youths’ internship experiences. In some cases in the field of computer hardware repair, supervisors refused to let the female interns participate in certain activities because they did not think the tasks were appropriate for females. In other cases, young interns were forced to perform tasks that they felt were not relevant to their internship; this was particularly the case for the young pre-school teachers, who, in most cases worked in small private pre-schools, and were asked to do all the chores that the others did not want to do, such as sweeping the floors, cleaning the lavatories and the kitchen. In addition, there were a few rare cases in which female interns were subject to harassment from their direct supervisors. This was the case for young women who were completing their internship in the hotel industry. These girls have reported that in the past, they would not have been able to protect themselves against this kind of behavior from their supervisor, but because of the life skills courses on sexual harassment, they first managed to recognize that those actions were in fact sexual harassment against them, and second, they became aware that they should not accept these actions just because they were afraid to lose their job/internship. These instances were systematically reported to the training center/trainers, and remedial action was immediately taken. It should be noted that when the implementing partners would meet with employers on a regular basis to monitor the progress and performance of the interns during their internship, a high level of satisfaction with the internship experience was reported by both youth and their supervisors. Where this regular monitoring was lacking or too irregular, interns have reported that there internships were not useful, and both supervisors and youth complained about the lack of follow-up support. Indicator 3.5: Number and percentage of placed youth who declare having a satisfying and quality job Through the Alliance approach, implementing partners have started placing youth using the assistance of partner private sector actors who had made agreements with businesses to offer internships and jobs to graduates. The satisfaction of youth with their jobs is fair to low, with a very small number of employed youth reporting that they are highly satisfied with their jobs. Of the 88 youth who responded to the questions related to their work status: • Forty-nine youth or 56% of respondents reported that they were working. • Twenty-six or 57% reported that they are happy with their work environment. • Thirty-three or 80% reported that their job was relevant to their field of study. • Fifteen or 34% of respondents said they were happy with their salary, and another 34% (15) said that they were unhappy because they had hoped for better salaried employment. • Twenty one or 50% of respondents are happy with the support they are receiving from their supervisors and/or peers, whereas nine or 21% are unhappy to very unhappy because their supervisors did not provide them with the needed support. The evaluation revealed that youth who reported that they were satisfied their jobs, were those working in tourism (cooking, hospitality) and IT (word processing, hardware repair, software installation) sectors. Youth working in construction and welding reported the lowest levels of satisfaction with their jobs; this is mainly due to the fact that these jobs require a great physical effort, and offer very low salaries. This evaluation also revealed that young women working as pre-school teachers (Emploi Habilité graduates) are the least satisfied with their salaries, and reported that their employers are forcing them to sign internship contracts to keep paying them below the minimum wage. To face this situation, affected projects have recently hired placement agents and senior outreach consultants, who are negotiating with the employers new contract terms for the youth. Indicator 3.6: Net salary increase of youth employed through MEEA (within six months after placement) Most MEEA graduates were jobless before joining the program, in fact, that is the main reason why they joined the program. So they went from no salary to a regular salary, which is a significant change in their income. Those who were professionally active before completing an MEEA supported training program did not report a remarkable increase in their income. Among the 69 respondents to this question, 61% (42) were unemployed before the training, and 39% (27) were already employed before the training. Monthly salaries of MEEA working graduates varied between $50 and $300 as presented in the chart below. Some of the youth who were working in the formal sector and were placed in formal jobs through the MEEA projects continued to work as freelance service providers, in addition to their formal jobs. This allowed them to increase their income from 50% to 100%. These youth generally decline to report their informal jobs for fear of taxation. Youth working in both the informal and formal sectors are generally electricians, plumbers, smiths and mechanics. Figure 19: Salary Ranges of Employed Youth 17% 26% 46% 9% 2% 0% Less than $50 $55 ‐ $100 $150 ‐ $300 $350 ‐ $500 $550 ‐ $800 More than $800 Intermediate Result IV: Positive Indirect Effect on Families and Communities of Youth Evaluation Question/Intermediate Result IV: To what extent did the MEEA program succeed in improving youth engagement in their family and community? To assess the success of the MEEA projects to improve youth engagement in their family and community, two proxy indicators were used: • Indicator 4.1: Number and percentage of employed youth financially helping their families with household, health and education expenses • Indicator 4.2: Number and percentage of youth who have reported improved financial and social conditions of their families as a result of participation in the MEEA program The “community spirit” in Moroccan society in general, and among youth in particular has always been very weak. Moroccan society is mostly family oriented, families are generally closed and inward looking. The sense of responsibility towards the outside world (i.e. outside of the family boundaries) was practically nonexistent until recently. It is therefore no surprise that all youth who joined the MEEA supported training programs, came with the same careless attitude toward their community, and did not feel that they had to give back to their society. In fact, all of these youth felt that their community did not provide them with what they needed, and therefore, they did not feel like giving their community anything. Although the youth who joined the MEEA supported training programs wanted, in most part, to help their families, they did not have the capacity to do so before the training simply because they felt that since they did not have any income, they could not be of any help to their parents and siblings. MEEA projects had therefore to “start from scratch” in generating a new sense of community and family involvement among the MEEA youth. MEEA relied largely on IYF’s Passport to Success life skills training program to achieve this. Summary of Intermediate Result IV Findings The study finds that youth are sharing income with families even if they are not earning a large amount yet. Sixty percent of graduates surveyed are using part of their income to financially help their parents and siblings and contributing to cover household and health expenses. Furthermore, 22% of the graduates surveyed reported that their support helped improve the financial and social situations of their families, and 58% claimed that it somewhat improved the situation. Indicator 4.1: Number and percentage of employed youth financially helping their families with household, health and education expenses Survey results show that most of the youth share their income with their parents and siblings once they are working and earning regular income through jobs or self-employment. The figure below shows general trends as to how youth use spend their income (youth could answer yes to all categories that applied). Only 43 of the total 126 respondents answered this question. With the change in their employment status, most employed youth who responded to this question stated that even when their salary is very low, 61% (26) of youth use part of their income to help their parents and siblings, generally to contribute to cover household, education and health expenses. Figure 20: Classification of How Youth Use Earnings Indicator 4.2: Number and percentage of youth who have reported improved financial and social conditions of their families as a result of participation in the MEEA program Due to the low salaries the new MEEA graduates are presently earning from their entry level jobs, only 22% of respondents (45) stated that that their financial support helped improve their family’s financial and social situation; 58% (26) claimed that their support somewhat improved the situation; and 20% (9) said there was no change because their earnings were barely enough to cover their own personal needs. Section IV: Challenges, Lessons Learned and Recommendations This final section highlights the challenges and lessons learned particularly related to leverage and alliance building, scalability of MEEA projects, and sustainability of the MEEA program. Part I: Challenges Given that MEEA was one of the pioneer programs in Morocco which focused on improving employability of out-of-school youth using the public-private partnership model, the program faced several noteworthy challenges. Leverage and Alliance Building • Changing the attitude of businesses toward youth education and employment requires the development of a strong advocacy network. The MEEA’s Steering Committee, while focused heavily on design and approval of projects, did not fully occupy this role and likely would need expansion of membership from targeted communities and industries, as well as a refinement of roles in order to maximize opportunities for outreach with businesses for job placement were not maximized. On the other hand, collaboration and sharing of lessons learned among project partners within MEEA in developing effective placement strategies was successful, suggesting a more bottom-up and community-based approach in this area is worthy of investment. • Introducing the “Alliance” concept was challenging in the context of a Moroccan society where community spirit is traditionally focused more on the immediate family circle and where civil society and the private sector actors played a very minor role (if at all) in the education of Moroccan youth. • Fundraising to sustain project activities was initially a major challenge for most of the partners. Most of the implementing partners have benefited from foreign donor grants in the past, but their capacity to develop efficient strategies to raise leverage funds for their projects was very limited when MEEA was launched. Scalability • Scaling up MEEA projects requires building new regional and local alliances in each region where a given MEEA project would be expanded or replicated. This in itself represents a major challenge, and requires a strong and local steering committee or advisory group that would be fully committed to brokering partnerships, building partners’ capacity in developing partnerships and leveraging resources, and in monitoring project quality. • Finding jobs for the newly trained youth was a challenge for the EEA implementing partners, from the small to large scale projects. In order to achieve job placement on a larger scale, further efforts should build on job placement activities to develop a comprehensive job placement strategy that involves various stakeholders and coordinated outreach activities with businesses. This would go beyond the reasonably successful but more opportunistic “door to door” approach undertaken by the MEEA program for job placements. • Scaling programs is often best achieved through integration into existing institutions with broad public sector reach including the government. However, this approach requires the government to have a certain level of capacity and “buy-in” to the project, as well as having a conducive policy environment. Emploi Habilité, MEEA’s project which has best positioned itself for scale through significant integration within government institutions, was challenged by these issues and as a result, the project experienced some setbacks in terms of enrollment and job placement. Efforts have been made to improve such integration and it will be important in future analyses to determine best approaches in this area as the program continues to expand with USAID and other funding support. Sustainability • Sustainability can be viewed on multiple levels, at both the project and broader alliance levels. While significant progress was made in project based sustainability efforts, the overall Morocco Alliance’s lack of an institutional “home” in Morocco and maturation beyond the intensive activity required for designing and distributing seed funds negatively the ability of the Alliance to be sustainable on a macro level beyond IYF’s direct role. Furthermore, while the prominent role of private sector in the Alliance helped it to broker new partnerships, the private sector alone was not suitable to take on the role of leading the Alliance, and there were not civil society organizations positioned to lead it. Further efforts in this regard would likely require substantial resources given the array of institutional interests across all sectors to be represented, additional time and focus on the advocacy role beyond project specific activities, and a proper institutional framework to lead such efforts. Part II: Lessons Learned and Recommendations The final evaluation reveals several lessons learned that could be applied in implementation and management of similar projects. Leverage and Alliance Building • Alliance development efforts are more effective when private service actors and businesses stop looking at their work with the alliance as mere charitable contributions, but because it makes good business sense. MEEA provides them with quality employees which will contribute to achieving higher productivity. • Regular follow-up on youth and monitoring of their performance in training and their experience in internships is essential to ensure program quality. Moreover, the monitoring visits serve as good opportunities for the implementing partners to advocate for the MEEA graduates and to build a strong relationship with the private sector partners. • In order to work effectively with government, MEEA has learned that in order to ensure project implementation and institutionalization, the program has to work in synchronization at the local, regional and central levels. Scalability • Capacity building of implementing partners in partnership development, project monitoring, reporting, and financial management contributed to making MEEA partners more capable of effectively designing and implementing alliance-based workforce development programs tailored to meet the needs of both youth and employers. • Good trainers and a sound follow-up system are the sine qua non conditions to scale up MEEA projects. It is important to allocate sufficient resources to cover trainers’ salaries and transportation to follow-up with trainees. Sustainability • MEEA was quite successful in equipping its NGO partners with capacity to leverage resources and conduct effective outreach with donors and private sector, so that most of the seven partners have been able to sustain their programs with new funding and in new regions. • Integration of life skills and language skills training was greatly valued by employers and youth. More efforts by MEEA to pursue partnerships to facilitate language training (French and English) were needed and employers and youth expressed a need in this area. • A system for tracking youth graduates and soliciting feedback such as mobile-based or online that youth can access easily would have been valuable for improving the responses from youth in the evaluation report and to track employment. • Providing more systematic job counseling and guidance services to youth is a critically important component to ensure success in job searches, job placement and successful integration into the workforce. Annex A: List of MEEA Partners and Leverage Contributions Partner Name Partner Role and Contributions Corporations Microsoft IT curriculum; financial support; employee volunteers; staff time; software and certifications for computer refurbishment training; computer donations General Electric GE life skills curriculum Munisys Technical assistance in computer refurbishment; internships; computer donations Attijariwafabank Computer donations, cash grant, personnel AFEM Steering Committee member; staff , facilities, volunteers, cash CGEM Steering Committee member Federation PME/PMI Steering Committee member American Chamber of Commerce Steering Committee member Moroccan American Trade and Investment Council Assistance with identifying Steering Committee members Vigeo Assistance in initiatives to raise CSR awareness SOMACA Computer donations, technical assistance, staff time WAFASALAF Equipment, computers, facilities and furnishings Richbond Placement of youth graduates BULL Placement of youth graduates Cisco Systems Computer and other equipment donations Banque Populaire Technical collaboration McDonalds Placement of youth graduates Accor/Ibis Placement of youth graduates Riad Mogador Placement of youth graduates BMCE Bank Placement of youth graduates Nike Financial support Government Ministry of National Education Administrative staff time, workshop director, trainers, translators, training facilities, operating expenses, pilot sites AREF (Academie Régionale pour l’Education et la Formation de Grand Casablanca) Administrative staff time, Workshop director, trainers, translators, training facilities, operating expenses, pilot sites Hassan II University Administrative staff time, computer equipped classrooms, other facilities, trainers Agence Nationale pour la Promotion de l’Emploi et des Compétences (ANAPEC) Use of database to identify beneficiaries for training; reimbursement of training costs Agence Nationale pour la Promotion de la Petite et Moyenne Entreprise (ANPME) Reimbursement of training costs Holding Al Omrane Construction of community center Entraide Nationale Management of community center National Initiative for Human Development (INDH) Funding of management of center and certification of training programs Department of Investment Assistance with Investment Conference, MEEA stand, MEEA Roundtable, press, facilities Non-governmental Organizations Al Jisr Administrative staff time, technical assistance, equipment, training of trainers, facilities, publishing and distributing manual Education for Employment Foundation Administrative staff time, training of trainers, baseline curriculum, facilities, operating expenses Ateliers Sans Frontières Spare parts, training, staff time Partner Name Partner Role and Contributions Fondation M.J.I.D Administrative staff time, volunteers, facilities Mohammed V Foundation Facilities Centre de Jeunes Dirigeants (CJD) Administrative staff time, technical assistance, trainers, facilities L’Heure Joyeuse Staff time, facilities, training curriculum, contacts with factory employers ENDA Maghreb Staff time, technical assistance Project ALEF Trainers, training curriculum PlaNET Finance Training curriculum Association Essalam Facilities Fondation Auteuil Financial support Near East Foundation Financial support GTZ Technical collaboration Overseas Development Organizations USAID/Morocco Programmatic guidance and collaboration; Steering Committee member ©2010 International Youth Foundation. All rights reserved. No portion of this report may be reproduced in any form without the written permission of IYF. This project was made possible by the generous support of the American people through the United States Agency for International Development (USAID) under grant agreement no. RAN-A-00-04-00043-00 This report was funded in part under grant agreement no. EPP-A-00-08-00006-00. The International Youth Foundation (IYF) invests in the extraordinary potential of young people. Founded in 1990, IYF builds and maintains a worldwide community of businesses, governments, and civil-society organizations committed to empowering youth to be healthy, productive, and engaged citizens. IYF programs are catalysts of change that help young people obtain a quality education, gain employability skills, make healthy choices, and improve their communities. To learn more, visit www.iyfnet.org ANNEX F: EEA Pakistan Evaluation Report Page | 2 CONTENTS Executive Summary 4 Section I: Introduction 6 Section II: EEA-Pakistan Results Framework 12 Section III: Findings and Analytical Review of Program Achievements 13 Section IV: Challenges, Lessons Learned and Recommendations 26 Conclusion 27 FIGURES Figure 1: Youth Enrollment in Vocational and Technical Training in Pakistan 7 Figure 2: List of Steering Committee Members 8 Figure 3: Summary of EEA-Pakistan Subgrant Projects 9 Figure 4: Overall Project Budgets and Leverage Contributions by Project 13 Figure 5: Sustainability Status by Project 14 Figure 6: Project-level Costs 15 Figure 7: Alliance Building and Program Support Costs 15 Figure 8: Typology of EEA-P Partners and Their Contributions 16 Figure 9: Number of Youth Participated by Project 18 Figure 10: Number of youth Completed by Project 18 Figure 11: Number of Teachers and Trainers Trained by Project 19 Figure 12: Improvements in Communication Skills 19 Figure 13: Conflict Resolution Skills 19 Figure 14: Youth Satisfaction with Internships and Apprenticeships 21 Figure 15: Job Satisfaction of Youth 22 Figure 16: Job Income Comparison 22 Figure 17: Income Level Comparison 22 Page | 3 Acronyms and Abbreviations AKF Aga Khan Foundation AKRSP Aga Khan Rural Support Programme CNG Compressed Natural Gas COTHM College of Tourism & Hotel Management DOVE Develop Our Valuable Earth EEA Education and Employment Alliance HDIP Hydrocarbon Development Institute of Pakistan IRM Institute for Rural Management IYF International Youth Foundation NAVTEC National Vocational and Technical Education Commission NRSP National Rural Support Programme NGORC NGO-Resource Centre OGRA Oil & Gas Regulatory Authority PAITHOM Pak Austrian Institute of Tourism & Hotel Management PITHM Pakistan Institute of Tourism & Hotel Management RSPN Rural Support Programmes Network TPS Tourism Promotion Services USAID United States Agency for International Development Page | 4 Executive Summary Pakistan has a long history of vocational and technical training programs, but over the years these training programs have become more supply driven with very poor industry linkage. The government bureaucracy has further compounded the problem by hindering the development of new, innovative systems for preparing young people for the workforce. EEA-Pakistan (EEA-P) has been implemented by the International Youth Foundation (IYF) and Rural Support Programmes Network (RSPN) to form an alliance of private and public sector leaders to address the workforce development needs of young people and industry by strengthening employability skills of youth and placing them in quality jobs. Six workforce development projects were implemented by local NGOs under direction of RSPN and the EEA-P Steering Committee from December 2006 to April 2009. The summative evaluation was conducted by an independent evaluator to assess: 1) the overall value and utility of the alliance approach; 2) the effect of EEA-P on direct beneficiaries; and 3) the indirect effect of EEA-P on families and communities of youth. One hundred youth, twenty employers, and six alliance partners participated in the study through surveys, interviews, and focus group discussions. Overall Value and Utility of the Alliance Approach: The study shows that EEA Pakistan effectively formed partnerships with 19 public-private sector organizations to provide demand-driven skills training to marginalized youth and help them find employment. It achieved a 1:1 leverage ratio as targeted, generating $862,742 in leverage contributions. EEA-P strengthened the capacity of local implementing partners by improving their training modules and helping them to implement alliance-based projects and provide job placement services to youth. The training modules developed through these projects will continue to be used by local organizations for future training activities, thereby improving the quality of workforce development training and contributing to the sustainability of the EEA program. One project in particular has received funding of US$100,000 from National Vocational and Technical Education Commission (NAVTEC) to implement the second phase of EEA-P project. Overall, more than twenty businesses, including banks, hotels, mobile phone companies, and restaurants, provided on-site training and/or internships to youth participants, and several of these same companies subsequently hired the participants after they completed the training program. Program Benefit to Direct Beneficiaries: The study shows that EEA-P is successful not only mobilizing local resources but also providing quality employability skills training to marginalized youth focusing on specific industries including compressed natural gas, hospitality and services. With respect to beneficiaries fully tracked, the program trained a total of 1,406 youth (98% of the number of beneficiaries targeted) of which 41% are women; 98.5% of all participants completed their training programs. Additionally, more than 10,000 trainees across one hundred Vocational Training Institutes throughout Punjab benefitted from the basic IT training that was improved through this program.1 The improved curriculum and instructor training is expected to continue to benefit 10,000 Punjab Vocational Training Council (PVTC) trainees annually. The quality and relevance of EEA-P training is rated highly by graduates and employers. The study also finds substantial improvements in life skills particularly in areas of self-confidence and conflict resolution. EEA-P has placed 551 youth (63% of the employment target) within six months upon graduation. Fourteen graduates have decided to establish small businesses with the new skills they acquired. The study found that income levels of most youth who were working previously have increased as a result of participation in EEA-P training programs. In terms of employer satisfaction, 95% of employers surveyed feel that EEA graduates are better than other graduates, illustrating the effectiveness and relevance of EEA-P skills training provided. Additionally, all employers interviewed think that EEA-P was able to provide qualified graduates for the business sector, thereby closing the gap between employers’ needs and employees’ qualifications. Indirect Benefit to Families and Communities: The study also finds positive if indirect benefits for the program on families of youth. Over eighty percent of youth are sharing their incomes with their families for health, education and household expenses. Additionally, seventy percent of youth surveyed reported that their financial support helped improve the financial and social situations of their families. 1 EEA-P’s implementing partner - PVTC was not able to track the job placement data after the training. Therefore, this evaluation does not take into account 10,000 youth trained by PVTC but rather focuses on graduates from five other projects that EEA-Pakistan has been able to track systematically. Target and actual numbers have been presented accordingly. Page | 5 Lessons Learned and Recommendations: EEA-P successfully enhanced the capacity of implementing partners and provided relevant skills training to youth, but it could be further improved based on recommendations provided by youth and other stakeholders through this evaluation study. Recommendations for similar initiatives in the future are as follows: • Effectiveness: One target that was not met in EEA-P was the percentage of female participants. Effectiveness of the program in terms of gender inclusiveness can be increased by training females as Master Trainers which was not specifically targeted in the program. It can also be enhanced by creating innovative avenues to increase accessibility and full participation of young women. • Relevance: The relevance of the program can be further enhanced by increasing the number of exposure visits and providing more opportunities for hands-on training. Moreover, internship periods and training duration can be increased to improve the relevance of the training programs. • Sustainability: The long-term effectiveness of the project could have been even greater if the sub￾grant projects would have been replicated and scaled up immediately following the pilot training phase supported with EEA funding. Unfortunately, as opportunities for scaling up began to arise, the overall EEA program was coming to an end, making it difficult to capitalize on such opportunities. For future efforts, building the alliance structure is hopefully less time intensive, with more time to focus on project implementation and second stage scaling. Also, EEA-P should have made more concerted efforts to engage in policy level discussions with the Government of Pakistan to initiate the EEA approach at its training institutes. It should have also pursued opportunities to work with the private sector to offer these trainings on a profit-oriented basis. Finally, corporate social responsibility departments of various companies could have been approached to generate funding for scholarships for unemployed youth. Page | 6 Section I: Introduction Pakistan has a long history of vocational and technical training programs, but over the years these offerings have become more supply driven with very poor industry linkage; government bureaucracy has further compounded the problem by hindering the development of new, innovative systems for preparing young people for the workforce. The main impediments for the development of training in Pakistan have been the rigid and out dated labor regulations and the lack of a unified national training policy. The direct result of the rigid labor regulations is that firms are weary of expanding their workforce for fear of not being able to retrench people when necessary. This has resulted in the majority of them being employed as contract or temporary workers. The firms therefore have no incentive to train temporary workers as there is high incidence of labor turnover. Over 35% of the labor force in the manufacturing sector is composed of temporary workers. This is a major obstacle to developing the demand for training. One of the problems arising out of this is that there is a very poor level of work ethics. The lack of a well thought-out national policy for training has also taken a toll on the skill levels of the workforce in Pakistan. But it must be noted that the high level of illiteracy is also a serious problem that has to be addressed immediately. There are over 500 training institutions in the country providing training to around 200,000 people every year. The Punjab Vocational Training Council (PVTC) and Technical Education and Vocational Training Authority (TEVTA) are the institutions which have the most extensive reach in the country. The Vocational and Technical Training (VTT) system is administered by a number of federal, provincial and private agencies. These various agencies have little or no knowledge of a national initiative for skills development and their training programs do not match the requirements of the industry. It is not surprising that although there are 546 training institutions, there are very few courses that meet the demands required by industry. Furthermore there is little provision for on-the-job training by most of these institutions. This depicts that there are major structural deficiencies in the VTT system in Pakistan and there is a need for the private sector to intervene to offer demand-driven sustainable training services. Indeed, Pakistan provides a unique blend of challenges and opportunities for initiatives aimed at addressing core economic needs through improved education and employment. While many would argue that the current environment leaves all Pakistani youth under educated, poorly trained, and generally unable to obtain meaningful employment, others can point to large and small programs supported by the public and private sectors that are starting to have positive effect on Pakistani youth. One such initiative is International Youth Foundation’s (IYF) USAID-funded ‘Education Employment Alliance Pakistan’ (EEA-P) program implemented by Rural Support Programmes Network (RSPN) from December 2006 through April 2009. EEA-P was designed to support successful workforce development projects to address employability skills of young men and women in Pakistan, using creative resources in partnership with capable Pakistani private sector and civil society organizations. The summative evaluation was conducted by Anjum Masood, an independent consultant, from November 2008 through February 2009. This study seeks to assess the following: 1) the overall value and utility of the alliance approach, and 2) the effectiveness of EEA employability interventions on direct beneficiaries and their families. Specific objectives are to: 1. Assess the effectiveness, relevance, sustainability and efficiency of the EEA Pakistan approach according to the following intermediate results: a. Alliance Building and Leveraging of Resources b. Improved Access to and Relevance of Education and Employability Training c. Improved Prospects for Employment and Successful Entrepreneurship d. Sustainability of Project Results e. Positive Indirect Effect on Families and Communities of Youth 2. Identify strengths, weaknesses and challenges of the project in the context of current trends in youth employment in Pakistan that would have implications on further strengthening and expanding the project. Page | 7 Problem Statement Pakistan is a country with high incidence of poverty and a growing population that is characterized by low levels of literacy and skills base. The current educational system of the country is a curriculum system rather than the competency based system. Whereas competency based systems impart skills that are both up-to-date and very much related to the industry requirement, the curricula used for the most part in Pakistan were developed many years back, do not include the industry input, and were mostly designed by the public sector. As such, students graduating out of technical and vocational colleges, though trained, do not have the skills required by the industry. This has a two pronged effect on the labor/youth markets. The youth are becoming increasingly unemployed or underemployed in certain sectors; and the industry is getting hungrier for trained workforce. Thus, those who graduate from middle or high school, or even university, do not have the right set of skills to find a job or meet the needs of the job market. The traditional education system does not enable the youth to identify their true potential by analyzing their aptitude. The absence of career guidance combined with a lack of innovative market oriented training program limits the range of career paths followed, resulting in labor surplus in some trades and deficit in others. Second, the country has limited resources and very few facilities for imparting technical and vocational skills; moreover, those facilities that are available are either not properly equipped or not managed. Third, the skills imparted by technical and vocational training institutes fails to ensure employability for their graduates, as it is not demand driven, owing to a lack of effective cooperation between the training institutes and the industry/private sector, which would employ their product, i.e. skilled labor. Fourth, there are very few programs that develop, encourage and promote entrepreneurial talent among young men and women in order to help them find their niche, be self-employed and participate more actively in the growth of their local economies. Last, the specific training needs of and employment opportunities for young women which form a growing section of the workforce do not receive adequate attention by the mainstream skill development programs. Many of these issues were outlined in a recent USAID/Pakistan RFA – Empower Pakistan: Jobs – which references the EEA-P program and is structured to build on the successes of the EEA-P program. Selections from the RFA are included below: Due to short-comings in the formal education system, even secondary school and university graduates tend to lack basic communication skills, self-confidence, English language skills, ICT proficiency, career development information, and job-search skills. Pakistan is typical of many South Asian countries in that its educational pedagogies generally do not cultivate assertive self-presentation skills, independent or collective problem-solving, self-initiative, or results-based productivity beyond test-taking. Yet these are the skills that are in high-demand among employers in a variety of sectors, especially the service sectors where successful interface with customers is critically important. The challenge is even greater for young women graduates who are usually socialized to remain passive in public settings… Employers in Pakistan report that they cannot find qualified workers, while Pakistani youth don’t know where to look for jobs or further training options to increase their employability. There is also little in the way of professional development for workforce development professionals in Pakistan… There are already a significant number of training institutions and poverty alleviation NGOs in Pakistan, though they are inadequately distributed across the country and face challenges achieving outcomes for more disadvantaged population sub-groups. Figure 1 presents the breakdown of male and female enrollment of youth in vocational and technical education training. The figures have been extracted from the Pakistan Education Census of 2005. Figure 1: Youth Enrollment in Vocational and Technical Training in Pakistan Education Type Female Male Total Vocational 76,265 86,672 161,937 Technical (i.e., Bachelor’s degree in Engineering) 6,462 37,339 43,801 Project Summary In December 2006, IYF identified Rural Support Programmes Network (RSPN) as the National Secretariat to implement EEA-P. The Secretariat was set up under the Special Projects Wing of RSPN. RSPN hired a Senior Programme Officer, Finance Manager, and Administrative Assistant to support the overall program. The functions of the Secretariat included the following: a. Develop the grants program, its management system, and criteria for award of sub-grants Page | 8 b. Proactively engage with partners from local and international private sector concerns with the support of IYF’s country representative and home office c. Develop innovative and scalable programs for education and employment of youth, especially girls, with a focus on regions and communities with limited opportunities d. Implement effective leveraging and grant monitoring and evaluation systems and processes e. Ensure compliance with USAID and IYF requirements and guidelines f. Report to RSPN and IYF on project progress and financial progress on a regular basis In accordance with EEA-P’s objectives, RSPN aimed to create and support alliances that enhanced the capacities of disadvantaged young women and men (ages 15-25 years) of Pakistan while ensuring fair income generating activities either through employment opportunities or entrepreneurship. Specific objectives of the program included: • Alliance building and leveraging of resources from the private sector • Improving access, quality and relevance of employability and entrepreneurship training opportunities • Improving opportunities for employment and entrepreneurship While targeting youth groups, with a special focus on women, the project pursued opportunities in the following areas: a. Development of Employability Skills b. Linking Training Facilities with Employment Opportunities c. Entrepreneurship Training and Support d. Career Counseling e. Business Development Services f. Development of Business Clusters The methodology of project implementation is discussed below: National Alliance/Steering Committee and Secretariat: To accomplish program objectives, RSPN formed the Steering Committee comprised of the private sector, civil society, youth groups and representatives of USAID and IYF. This Committee acted as an advisory body for the EEA-P seed grants program. Figure 2 includes the list of the Steering Committee members: Figure 2: List of Steering Committee Members Name Title Affiliation Mr. Faisal Ali Khan Director Commercial DHL Pakistan Pvt. Ltd. Mr. Haider Ali Isani Resident Manager Engro Chemicals Pakistan Ltd. Mr. Imran Ijaz Government Account Manager Microsoft Mr. Waqar A. Anjum Programme Manager Philanthropy Services Pakistan Centre for Philanthropy Mr. Abid Channa Programme Manager Thardeep Rural Development Program Ms. Maria Ahmad Programme Manager Radio 1 FM Mr. Aazar Bhandara Economic Growth Specialist USAID/Pakistan Ms. Savera Hayat Country Advisor IYF Mr. Saeed Khan CEO Punjab Vocational Training Council Ms. Afreina Noor Project Manager The Free and Fair Election Network The private sector representatives contributed their knowledge of the industry and also helped build links with other private sector partners. The civil society members brought their knowledge of disadvantaged communities and issues facing the poor. The youth leaders—Maria Ahmad and Afreina Noor—helped the Steering Committee consider the real needs of the target group. Grants Program Parameters: While working under the guidance of the committee, RSPN managed a competitive grants program, whereby proposals were invited from non-profit organizations, public sector institutions and the private entities. The seed grants were in the range of US $10,000-200,000. In total, 5 grants were given to five implementing partners: Aga Khan Foundation (AKF), Sabawon, ASK Development, Sahara Welfare Society, and National Rural Support Programmes-Institute for Rural Management with a focus on 8 districts: Karachi, Lahore, Gilgit, Peshawar, Rawalpindi, Faisalabad, Sialkot, and Muzafarabad. With the exception of ASK Development, each implementing organization is a non-profit. ASK Development operates as a for-profit training institution, but implemented their EEA-P activity as a non-profit under their Social Sector Program. Page | 9 Figure 3: Summary of EEA-Pakistan Subgrant Projects Sub-Grant 1 Project Title: Livelihood Up-gradation and Development of Youth in the Hospitality Sector of Select Districts Implementing Partner: AKF Strategic Partners: COTHM, PITHM, PAITHOM, NGORC, AKRSP, and TPS Total Number of Alliance Partners: 7 Objectives: Build the capacity of unemployed youth and the junior staff (kitchen & management) of small to medium sized hotels and restaurants paying special attention to quality control, improved hygiene, and better management. Partnership Setup: AKF was the overall project manager while the strategic partners provided support and resources in curriculum development, training, and employability of youth. Sub-Grant 2 Project Title: Training in Vital Job Skills for Disadvantaged Youth in Pakistan Implementing Partner: ASK Development Strategic Partners: DOVE, SUKHI, and Bedari Total Number of Alliance Partners: 4 Objectives: To train disadvantaged youth in professional skills such as customer services skills, articulation, selling skills and corporate ethics to become better quality employees for the services and manufacturing industries. Partnership Setup: ASK Development designed the curriculum, imparted the trainings, and assisted in employability of youth, while the partners identified youth participants for the training program. Sub-Grant 3 Project Title: Innovative Youth Employability Scheme (I-YES) Implementing Partner: Sabawon Strategic Partners: All Pakistan Compressed Natural Gas (CNG) Association, HDIP, and OGRA Total Number of Alliance Partners: 4 Objectives: To provide in-class and on-the job training to youth in various aspects of CNG station management and services including the handling and maintenance of CNG dispensers and book keeping. Partnership Setup: Sabawon managed the overall project while the partners assisted and pooled its resources to develop the curriculum, impart trainings, and facilitate employment. Sub-Grant 4 Project Title: Sahara Art & Craft Center Implementing Partner: Sahara Welfare Society Strategic Partners: Engro Chemicals, Ali Institute of Education Total Number of Alliance Partners: 3 Objectives: To provide training in stitching, drafting, hand embroidery, machine embroidery, knitting work, preparation of home decoration items to 250 out of school girls. Partnership Setup: Sahara Welfare Society imparted the training and identified the youth, Ali Institute of Education provided technical assistance for curriculum development and teacher training, while Engro provided resources in setting up the Center. Sub-Grant 5 Project Title: Passport to Success – Life Skills for Employability Program for Pakistan Implementing Partner: NRSP-IRM Objectives: To pilot the GE Foundation and IYF’s Passport to Success Program with out of school youth engaged with vocational training and those with a Bachelors degree; and measure its effect on the youth. Partnership Setup: RSPN adapted IYF’s Passport to Success curriculum, provided Training-of-Trainers, and carried out a Trainer’s Consultative Session, while IRM imparted the training to youth. In order to support the technical aspects of these activities, RSPN also hired a curriculum expert who assisted sub-grantees in the development of training curricula and training master trainers. This was Page | 10 especially important for the I-YES and Hospitality training programs, which developed new curricula. RSPN’s curriculum expert worked with the implementing partners to ensure the content of the curricula and the quality of the trainers would translate into effective training for the youth participants. Target Beneficiaries: The target group was intended to be youth, particularly women, between the ages of 15-25 years, irrespective of their level of education. However, since the official youth age range is 15-29 year old in Pakistan, the project recruited trainees in this age group. A majority of the young people were at least 18 years old because enrollment in projects required the National ID card. The project focused on the youth from marginalized communities by using community-based NGOs to identify participants. Project Duration: The program was designed and implemented for a period of 21 months from December 2006 to August 30, 2008. It was given a no-cost extension to April 30, 2009. Evaluation Methodology This evaluation is based on a mix of qualitative and quantitative methods and uses a triangulation approach based on a) focus group discussions, b) the administration of survey tools, and c) key informant interviews. The generic survey tools developed by IYF have been utilized to assess the program outcomes in accordance with the EEA Results Framework. The focus group discussions revealed information that subsequently led to a better understanding of the motivations, behavior and perspectives of the project partners, employers and the target beneficiaries-youth. The evaluation targeted three groups of stakeholders: • Partners: Consisting of government, civil society and private sector partners that have directly or indirectly supported EEA-P program. These include Steering Committee members, project implementers and other alliance partners. • Beneficiaries: Consisting of the youth who participated in EEA training programs, employability programs, coaching, or other EEA supported programs and their communities, family and friends. • Employers: These were private or public sector employers who offered internships/and or job to EEA-P trained youth. Formal survey questionnaires: Comprehensive survey questionnaires developed by IYF were used to gather input from a wide spectrum of the total universe of EEA-P direct and indirect beneficiaries, EEA partners, Alliance members, and employers. A total of five main questionnaires were utilized as listed below, and each was designed to target one of the groups stated above: • Partner Survey in three parts: o Survey -- Alliance Members (both members of National or Project-based Alliances) o Survey -- Implementing Partners o Survey -- National EEA Secretariat • Follow-up participant Survey • Employer survey The partners’ and employer’s surveys and youth questionnaires were administrated directly on a one-to￾one basis. Focus group interviews: Focus group discussions were held with youth and employers as these provide a different dynamic and strengthen the case for generalizing findings from smaller samples. Key informant Interviews: Key informant interviews were undertaken to validate findings of the participants’ survey. It targeted trainers, mentors, and supervisors. Sampling: Given the diversity in the profile and size of target population, the following sample size was used: A. Sample Sizes: • Participants: one hundred youth • Employers: twenty employers • Partners: three implementing partners and three alliance members Page | 11 B. Composition of Samples: • Participants: The samples composed of youth who have completed EEA programs and included: o Working and non-working youth o Young men and women o Geographically dispersed locations: Karachi, Lahore, Gilgit, Peshawar, Rawalpindi, Faisalabad, Sialkot, and Muzafarabad. Youth who left the program before completion were not included. V. Findings The EEA-Pakistan Results Framework (see below) is developed based on the global framework, and global indicators applicable to EEA-Pakistan project are assessed. The findings on each of the indicators, applicable to EEA Pakistan project are presented in Section III. Section II: EEA-Pakistan Results Framework Education and Employment Alliance in Pakistan Goal: Improving the skills of young people for better employability and successful entrepreneurship Intermediate Result 1: Alliance building and leveraging resources Intermediate Result 2: Improved Access to and Relevance of Education and Em plo yabilit y Trainin g Intermediate Result 4: Indirect Effect on Communities and Families of Youth Intermediate Result 3: Improved Prospects for Employment and Successful Entre preneurshi p Indicator 1.1: In kind and cash resources leveraged for the project through private sector, government, civil society and other partners Indicator 1.2: The effectiveness of the EEA Alliance Approach in building partnerships, and promoting sustainable and scalable youth employability programs Indicator 1.3: Number of private sector, government and civil society partnerships formed through the project to jointly offer employment and business development training activities in targeted areas Indicator 1.4: New partnerships, mechanisms and approaches created as a rippling effect of working with EEA projects to promote youth employability Indicator 1.5: Number of non￾target institutions that request to adopt EEA promoted programs, models and tools as part of their activities Indicator 2.1: Number of young men and women participating in EEA projects Indicator 2.2: Number and percentage of young men and women completing EEA supported programs. Indicator 2.3: Number of teachers, trainers, facilitators and counselors trained under EEA supported programs Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in EEA supported programs Indicator 2.5: Level of satisfaction of young trainees with the quality and relevance of the training, counseling and services received through EEA programs Indicator 2.6: Number and percent of EEA graduates who report that they have referred or will refer EEA to other youth Indicator 3.1: Number and percentage of trainees who get a job within six months of completing the program Indicator 3.2: Number and percentage of trainees who continue education, obtain an internship or engage in further professional training within six months of completing the program Indicator 3.3: Number and percentage of EEA trainees starting their own businesses within six months of completing the training Indicator 3.4: Number and percentage of youth with satisfactory internships or apprenticeships Indicator 3.5: Number and percentage of placed youth who declare having a satisfying and quality job Indicator 3.6: Net salary increase of youth employed through EEA programs Indicator 3.7: Level of employers’ satisfaction with the soft skills and/or technical skills of EEA trained graduates Indicator 4.1: Number and percentage of employed youth financially helping their families for household, health and education expenses Indicator 4.2: Number and percentage of youth who have reported improved financial and social conditions of their families Page | 13 Section III: Findings and Analytical Review of Program Achievements Intermediate Result I: Alliance Building and Leveraging Resources Summary of Intermediate Result One Findings The study shows that EEA Pakistan has effectively formed partnerships with 19 public-private sector organizations to provide demand-driven skills training to marginalized youth and help them find employment. It has achieved a 1:1 leverage ratio as targeted, generating $862,742 in leverage contributions against direct USAID support. EEA-P has strengthened the capacity of local implementing partners by improving their training modules and training them on implement alliance-based projects and effectively provide job placement services to youth. The training modules developed through these projects will continue to be used by local organizations for future training activities, thereby improving the quality of workforce development training and contributing to the sustainability of EEA program. One project in particular has received funding ($100,000) from NAVTEC to implement the second phase of EEA-P project. Overall, more than twenty businesses, including banks, hotels, mobile phone companies, and restaurants, provided on-site training and/or internships to youth participants, and several of these same companies subsequently hired the participants after they completed the training program. Indicator 1.1: In kind and cash resources leveraged for the project through private sector, government, civil society and other partners The program was able to achieve a 1:1 leverage ratio as targeted under the grant agreements with AKF, ASK Development, Sahara Welfare, and SABAWON. One partner, NRSP-IRM, did not contribute resources toward the implementation of the Passport to Success Program because the bulk of the program consisted of adapting and piloting the life skills program in Pakistan, which was conducted under a tight timeline that could not depend on leveraged resources for implementation. It was envisioned that after the successful pilot training period, leverage resources could be sought to continue and expand the life skills training. Also, for this project, contributions by the GE Foundation in the development of the curriculum were not counted, but rather included in calculations related to other country adaptations of the program. Alliance partners contributed $859,526 toward individual projects against USAID support of $586,248. RSPN also contributed $3,217 toward Secretariat operations, bringing the total leverage to $862,742. Figure 4: Overall Project Budgets and Leverage Contributions by Project Program Partners Leverage Value ($) Grant Value ($) Comment 1. AKF 201,872 200,000 Alliance partners contributed resources by providing trainers, curriculum, staff time, training resources, training venues, etc. 2. ASK Development 249,216 141,531 3. Sahara Welfare 32,208 10,000 4. SABAWON 116,230 81,520 5. NSRP-IRM - 153,197 6. PVTC 260,000 - Total Value 859,526 586,248 Indicator 1.2: The effectiveness of the EEA Alliance Approach in building partnerships, and promoting sustainable and scalable youth employability programs The EEA-P program was able to form partnerships with five local organizations: AKF, ASK Development, Sahara Welfare, SABAWON, and NRSP-IRM. Through these partnerships it developed and implemented youth employability projects and trained 1,250 youth in technical skills and 180 in life skills for employability in eight cities. The training modules developed through these projects have enhanced the capacity of the four partner organizations for further replicating the trainings as these were adopted by the local organizations implementing the projects. Partners have either planned to use the training modules to provide further trainings as in the case of COTHM or utilize them to implement similar projects if funding is made available. However, these projects are not financially independent due to their lack of revenue generation and therefore are dependent on donor financing. Figure 5 provides background information and summarizes the sustainability status of each project: Page | 14 Figure 5: Sustainability Status by Project Project Name Objective Alliance Partners Sustainability Status Microsoft/Emplo y-ability for IT Trainees To Improve IT training in 90 Vocational Training Institutes, used by 250 students and reaching 9,000 students annually Punjab Vocational Training Council, Microsoft The improved IT curriculum and teacher training benefits 10,000 PVTC trainees annually. Hospitality Sector Skills Development To improve the skills of 400 young men and women for work in small and medium size hotels and restaurants and to promote the overall climate for tourism by Pakistanis and foreigners Aga Khan Foundation, Serena Hotels, Chefs Association of Pakistan Training program unable to continue due to lack of funding for the next phase. Trainings in Vital Service￾Related Job Skills To provide 300 disadvantaged young women and men in three districts with training in service industry-related skills and to help place 250 into jobs ASK Development, DOVE, SUKHI Phase 2 under implementation with a $100,000 grant from NAVTEC to reach an additional 580 youth. Sahara Art and Craft Center To provide skills training in arts and crafts for 250 young women and to help them generate income by selling goods at local Arts & Crafts centers Sahara Welfare Organization, Engro Chemicals, Ali Education Institute of Lahore Project unable to continue due to shift in programmatic focus by Sahara Welfare Association. Home-based businesses are anticipated to continue providing income generating opportunities for women. Safe and Steady Employment at CNG Stations To provide certified training and job placements for 300 youth seeking to become CNG service station operators and to thereby increase safety conditions and station management capabilities Sabawon, Pakistan, Pakistan CNG Association, Civil Junction, Oil & Gas Regulatory Auth., Hydrocarbon Dev. Inst. Project unable to continue due to lack of funding for the next phase. Life Skills for Employability for Pakistani Youth To adapt, implement, certify and widely disseminate a comprehensive life skills program to benefit approximately 25,000 Pakistani youth annually following an initial pilot phase training 180 youth. Rural Support Programs Network, Institute for Rural Management, NAVTEC Integration of life skills programming into implementing partner’s technical training program anticipated but not final at time of evaluation. The Steering Committee, which was formed in part to enhance the public-private partnerships developed to support the activities, met three times during the first year of EEA-P implementation. These meetings focused on the sub-grant solicitation process and contributed to the selection and refinement of effective workforce development projects, but were unable to contribute to the development of robust partnerships to sustain and scale EEA-P program activities. Keeping Steering Committee members engaged in the program outside of steering committee members and then after the sub-grantee selection process was a major challenge for EEA-P. Cost Effectiveness: The overall cost effectiveness of the EEA Pakistan country program has been assessed on two levels. The first level is related to country program subgrants, which include both seed grants administered by USAID and cash and in-kind leverage contributions contributed by other partners to support EEA country programs. The second level more broadly includes costs classified as alliance building and program support costs. This second level includes costs outside of those directly incurred by project implementers. They accordingly include costs related to the functioning of the EEA National Secretariat RSPN, local consultants in Pakistan, as well as IYF direct and indirect expenditures attributable to the Pakistan program. Each of these cost categories are discussed in more detail below. Subgrant Project-level Costs: As noted above, subgrant project-level costs include seed funds and leverage contributions invested toward individual projects which typically covered curriculum design or improvement, training, job placement costs, and project specific costs of NGO subgrantees. The amount of seed funding support from USAID toward individual projects is a useful metric to assess cost effectiveness given that such Page | 15 funds are directly focused on implementation of projects, are directly leveraged by investments from other parties, and are best positioned for increased efficiency when additional beneficiaries are added to successful pilot programs. In Pakistan, the USAID support of approximately $586,248 in seed funds that were coupled with leverage investments of $859,5262 reached a total of 1,406 beneficiaries. If all project-level costs are taken in account, this brings the cost to $1,028 per beneficiary, of which $417 was borne directly by USAID and $611 was shouldered by the alliance partners. Figure 6: Project-level Costs Subgrant Name Number of Beneficiaries USAID Funds Leverage Total Project Budget Cost Per Beneficiary Aga Khan Foundation 363 $ 200,000 $ 201,872 $ 401,872 $ 1,107 ASK Development 301 $ 141,531 $ 249,217 $ 390,748 $ 1,298 Sahara Vocational Training 250 $ 10,000 $ 32,207 $ 42,207 $ 169 Sabawon CNG Training 312 $ 81,520 $ 116,230 $ 197,750 $ 634 RSPN Life Skills Training 180 $ 153,197 $ - $ 153,197 $ 851 PVTC IT Training NA $ - $ 260,000 $ 260,000 NA TOTAL 1,406 $ 586,248 $ 859,526 $ 1,445,774 $ 1,028 Alliance Building and Program Support Costs: A more expansive examination of costs by necessity also includes those alliance building and program support costs related to the EEA-Pakistan program. These costs supported significant, collective efforts by IYF working together with RSPN as National Secretariat, and local program consultants. Their efforts focused on a number of items related to program activities, including managing the design and implementation of alliance based interventions; providing international technical expertise on successful employability approaches; assisting in leveraging of resources; providing capacity building support to implementing partners in skills training, job placement and post training support to entrepreneurs; and providing financial oversight, compliance, and direct monitoring and evaluation support. Upon examination of IYF accounting records, these costs taken together with indirect costs attributable to the Pakistan program, totaled $686,162. This USAID-borne cost adds approximately $488 on a per beneficiary basis. Figure 7: Alliance Building and Program Support Costs Subgrant Name Number of Beneficiaries Alliance Building & Program Support Costs Cost Per Beneficiary Aga Khan Foundation 363 $ 114,360 $ 315 ASK Development 301 $ 114,360 $ 380 Sahara Vocational Training 250 $ 114,360 $ 457 Sabawon CNG Training 312 $ 114,360 $ 367 RSPN Life Skills Training 180 $ 114,360 $ 635 PVTC IT Training NA $ 114,360 NA TOTAL 1,406 $ 686,162 $ 488 Taken together, these overall costs of both seed funds and alliance building program support costs total approximately $1,516 per beneficiary, and appear roughly equivalent compared to other good practice country projects of this duration and that include internships, on the job training and international technical assistance. Of the cost of $1,516 per beneficiary, $905 (60%) was borne by USAID and $611 (40%) was contributed by alliance partners. It is noted that this analysis does not take into account PVTC beneficiaries who benefitted from improved curricula supported by EEA. When assessing costs of the EEA-P program, it is important to note that we have deliberately not included beneficiaries from the largest training program. This program, implemented by the Punjab Vocational Training Center, upgraded the Microsoft Unlimited Potential IT curriculum and supported the training of 10,000 beneficiaries through PVTC’s extensive network of Vocational Training Centers across the Punjab region of Pakistan. Based on the tracking information PVTC provided, the target of 10,000 youth was reached. However, 2 This leverage figure does not include RSPN’s contribution of $3,217. Page | 16 given the inability of PVTC to fully track these graduates and performance under the EEA monitoring and evaluation system, we have chosen conservatively not to include such numbers in our overall calculations. Should these 10,000 beneficiaries be counted toward the overall totals for the purpose of cost-effectiveness, the overall cost per beneficiary would be significantly reduced to $187 (60% by USAID and 40% by the alliance), including the project level costs of $127 and alliance building and program support costs of $60. Finally, and perhaps most significantly, a full assessment of cost efficiency would also require a more thorough assessment of the effectiveness of alliance building activities on long term sustainability and scalability of proven practice programs. As such, the PVTC model of programming for example demonstrates the significant increase in beneficiaries and benefits possible through an alliance building approach, particularly with government or quasi-government agencies as partners. Should the new efforts to substantially expand the EEA model be successful, the cost effectiveness of initial EEA investments could be dramatically expanded. While the longitudinal study required for this type of evaluation goes beyond the scope and resources available for this study, such factors should be addressed more directly in similar employability projects in the future. Indicator 1.3: Number of private sector, government and civil society partnerships formed through the program to jointly offer employment and business development training activities in targeted areas The program formed partnerships with nineteen partners, including AKF, ASK Development, Sahara Welfare, SABAWON, and NRSP-IRM and their strategic partners to jointly offer employment and business development training activities in Lahore, Karachi, Gilgit, Peshawar, Sialkot, Faisalabad, Muzaffarabad, Rawalpindi/Islamabad. Figure 8 provides more details on EEA-P partners and their contributions. For example, ASK Development partnered with community-based organizations (CBOs) such as DOVE, SUKHI, and Bedari. These CBOs helped ASK Development match the technical training programs, which focused on the service industry, with the needs and opportunities in the communities and also reached into the community to identify training participants. ASK Development and these CBOs then approached a variety of businesses to provide internships and consider job placement. As a result, more than twenty businesses, including banks, hotels, mobile phone companies, and restaurants, provided on-site training and/or internships to youth participants, and several of these same companies subsequently hired the participants after they completed the training program. Figure 8: Typology of EEA-P Partners and Their Contributions Partner Name Partner Role and Contributions Corporations DHL Pakistan Steering Committee member Microsoft Providing cash and in-kind assistance / Steering Committee member Engro Chemicals Pakistan Limited Steering Committee member and Donor Serena Hotels / Tourism Promotion Services Resource Partner providing training venues and other support Chef’s Association of Pakistan Resource Partner during training curriculum development CNG Association of Pakistan Donor and resource partner Thardeep Rural Development Program Steering Committee member Government National Vocational and Technical Education Commission (NAVTEC) Providing cash contribution for continuation of training activities Hydrocarbon Development Institute of Pakistan (HDIP) Resource Partner during training curriculum development College of Tourism & Hotel Management (COTHM) Resource Partner Oil & Gas Regulatory Authority Resource Partner during training curriculum development Federation of the Chambers of Commerce Pakistan Potential partner in creating linkages with private sector partners, particularly on job placement issues Pakistan Institute of Tourism & Hotel Management (PITHM) Resource Partner Pak Austrian Institute of Tourism & Hotel Management (PAITHOM) Resource Partner Non-governmental Organizations Rural Support Programmes Network National Secretariat Punjab Vocational Training Council Steering Committee member; project partner Society for Upgradation of Knowledge, Health, and Infrastructure (SUKHI) Resource Partner – mobilizing youth and helping place training participants into internships and jobs Civil Junction Resource Partner Page | 17 Aga Khan Foundation Implementing Partner ASK Development Implementing Partner Sahara Welfare Society Implementing Partner SABAWON Implementing Partner Institute for Rural Management Training provider for Life Skills project Pakistan Centre for Philanthropy Steering Committee member Develop Our Valuable Earth (DOVE) Resource Partner – mobilizing youth and helping place training participants into internships and jobs Innovation Alliance Training provider for PVTC project Rotary International (Rawalpindi Club) Steering Committee member Ali Institute for Education Technical Resource Partner for Sahara Arts & Crafts teacher training Indicator 1.4: New partnerships, mechanisms and approaches created as a ripple effect of working with EEA programs to support youth employability USAID/Pakistan developed a US$80 million Empower Pakistan: Jobs project based partly on the framework of the existing program. In addition, NAVTEC agreed to provide Rs.6.5million (approximately $100,000) to ASK Development to train 540 youth in the services sector related skills provided under EEA-P. Indicator 1.5: Number of non-target institutions that request to adopt EEA promoted programs, models and tools as part of their activities A majority of the firms bidding on USAID/Pakistan’s US$80 million Empower Pakistan: Jobs project requested IYF and RSPN to partner with them and to adopt the EEA-Pakistan promoted projects, models and tools as part of their bids. In total, four out of six major bidders requested partnerships from IYF and RSPN. Page | 18 Intermediate Result II: Improved Access to and Relevance of Education and Employability Training Summary of Intermediate Result Two Findings The study shows that EEA-P is successful in not only mobilizing local resources but providing quality employability skills training to marginalized youth focusing on specific industries including compressed natural gas, hospitality and services. The program has trained a total of 1,406 youth (vs. 1,430 targeted) of which 41% are women, and 98.5% completed the training. Additionally, more than 10,000 trainees across one hundred Vocational Training Institutes throughout Punjab benefitted from the basic IT training that was improved through this program. The quality and relevance of training was rated high by graduates and employers. For instance, almost all of the graduates surveyed report to have gained the skills they need to find employment. The study also finds substantial improvements in life skills particularly in areas of self-confidence and conflict resolution. Indicator 2.1: Number of young men and women participating in EEA projects The program reached 1,406 men and women out of the target of 1,430. A synopsis with gender breakdown is presented in Figure 9 below: Figure 9: Number of Youth Participated by Project Target Actual Female Male Percentage Total 1,430 1,406 583 810 97% Aga Khan Foundation-Hospitality Training 400 363 112 251 91% ASK Development-Service Industry 300 301 124 177 100.3% Sahara Welfare-Arts and Crafts Centre 250 250 250 0 100% SABAWON- CNG Safety and Service 300 312 0 312 104% RSPN – Passport to Success 180 180 97 83 100% Additionally, more than 10,000 trainees across 100 Vocational Training Institutes throughout Punjab benefitted from the basic IT training that was improved through this program. The improved curriculum and instructor training is expected to continue to benefit 10,000 trainees annually. Indicator 2.2: Number and percentage of young men and women completing EEA supported projects The global EEA program required EEA-P to ensure that 90% of the program trainees complete the EEA supported trainings. The program planned to train 1,287 men and women, but it exceeded this target by training 1,393 youth (810 men and 583 women). The details are given in Figure 10 below: Figure 10: Number of youth Completed by Project Target Actual Male Female Percentage Overachieved Dropout Total 1,287 1,386 803 583 7.7% 20 AKF-Hospitality Training 360 363 251 112 1% 0 ASK -Service Industry 270 299 175 124 11.1% 2 Sahara - Crafts Centre 235 250 0 250 10.6% 0 SABAWON- CNG Safety 270 294 294 0 10.9% 18 RSPN – Passport to Success 162 180 83 97 11.1% 0 Indicator 2.3: Number of teachers, trainers, facilitators and counselors trained under EEA supported programs The project trained a total of 51 trainers against a target of 54 (94% of the target). The trainers and facilitators were primarily chosen from the industry and from various training institutes working with the project partners. The trainers not only imparted the required technical skills to youth but also worked on their life skills to build their self confidence in some instances like the Hospitality and the Services sector projects. Each project had its own mechanism of identifying trainers and facilitators. AKF was working with various hospitality institutes including COTHM, PITHM, and PAITHOM, who provided their regular faculty to teach these courses. These Page | 19 No, there has been no change. 0% Yes, there has been some improvement. 33% Yes, there has been good improvement. 51% Yes, there has been a huge improvement. 16% No comment. 0% Did  your communication skillsimprove as a result of training? No, there has been no change. 0% Yes, there has been some improvement. 43% Yes, there has been a good improvement. 42% Yes, there has been a huge improvement. 15% No comment 0% Did your ability to resolve conflict improve as a result of training? faculty members were well aware of the course contents thus the project did not build their capacity in any new skills. Sabawon worked with All Pakistan CNG Association to identify four trainers from their industry partners/counterparts: Hydrocarbon Development Institute of Pakistan, and Oil and Gas Regulatory Authority. On the other hand, ASK Development and Sahara Welfare Society already had their own trainers. Figure 11: Number of Teachers and Trainers Trained by Project Target Actual Percentage achieved Total 54 51 94% ASK -Service Industry 20 20 100% Sahara -Arts and Crafts Centre 6 4 67% SABAWON- CNG Safety 8 4 50% RSPN – Passport to Success 20 23 115% Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in EEA supported projects Survey results show high percentages of youth reporting improved capabilities through participation in EEA-P supported projects. • Ninety-nine percent of the respondents felt that they gained technical knowledge and skills through the training provided by the program. • All respondents felt that there communications and conflict resolution skills have improved. • Ninety-eight percent felt that their ability to listen to others improved. • Ninety-seven percent felt that there self-confidence and sense of initiative has improved. • Seventy-five percent felt that their relationship with family and friends have improved after completing the training. Figure 12: Improvements in Communication Skills Figure 13: Conflict Resolution Skills Page | 20 Indicator 2.5: Level of satisfaction of young trainees with the quality and relevance of the training, counseling and services received through EEA-P program Overall, 73% respondents felt that they have gained needed knowledge and skills to start a new business and eighty percent felt empowered as a young entrepreneur because of the skills gained. However, only 34% of the trainees actually got some entrepreneurship-related training through the projects implemented by AKF and Sahara Welfare Society. It should be noted that the figures of 73% and 83% was solely based on the trainee perception and understanding of the survey questions. Although most of the projects did not provide any entrepreneurship related training, many trainees felt they can undertake a business with their newly gained confidence and skills. This demonstrates that if youth are provided with an opportunity for demand driven training, their own creative thinking process comes into play and they are able to think of new ideas of earning income including entrepreneurship. Indicator 2.6: Number and percentage of youth who completed EEA supported programs who report that they have referred or will refer this program to other youth Seventy-three percent of the respondents have referred youth to this program and 93% would refer youth to this program as depicted in the graphs below: These implicit endorsements speak to both the quality and the relevance of the training programs. The participants were satisfied with the training programs and have seen their new skills translate into new employment opportunities, and thus would overwhelmingly recommend that others participate in EEA-P training programs. Page | 21 Intermediate Result III: Improved Prospects for Employment and Successful Entrepreneurship Summary of Intermediate Result Three Findings The study found that EEA-P has been able to use its alliance networks to place youth in jobs effectively within six months after graduation. EEA-P has placed 551 youth within six months upon graduation. Twelve graduates have decided to establish small businesses with new the skills acquired. The study finds that income levels of most youth who were working previously have increased as a result of participation in the training. In terms of employer satisfaction, 95% of employers surveyed thought that EEA graduates were better than other graduates from similar age groups, illustrating the effectiveness and relevance of EEA-P skills training provided. Additionally, all employers interviewed thought that EEA-P provided qualified graduates for the business sector, thereby closing the gap between employers’ needs and employees’ qualifications. Indicator 3.1: Number and percentage of trainees who gain employment within six months of completing program A total of 551 youth secured jobs within six-months after completing the training which represented 63% against the original target of 875 youth. The reason for a lower achievement rate was primarily because the Sahara Art Craft Center could not facilitate trainees in obtaining jobs. The Sahara Center is located in a remote and culturally traditional area of Pakistan, where job opportunities for women outside of the home are rare. Both geography and social norms prevented follow-up, but we heard anecdotally that some women have started home businesses in sewing and embroidery. Otherwise, the employment rate for men and women was roughly equivalent. A somewhat unexpected result came from the hospitality project, where women found greater than expected success due to their interest and willingness to work in commercial kitchens and restaurants’ desire to hire women, who are seen as well suited for the work because of their discipline and cleanliness. Indicator 3.2: Number and percentage of trainees who continue education, obtain an internship or engage in further professional training within six months of completing program Ninety-five percent of the respondents showed interest in continuing education and professional training at some point in their career, and 52% of the respondents were able to be placed as interns or apprentices as a result of the training they received from EEA-P. Indicator 3.3: Number and percentage of EEA trainees starting their own businesses within six months of completing training Overall, eighty percent of the respondents felt empowered to start a business as a result of the training, and fourteen respondents have actually started one. These were trainees trained under the hospitality project. The Sahara Art & Crafts Centers was the only project that had some entrepreneurial thrust in it. It is important to mention that none of the projects purely focused on creating entrepreneurs, rather they focused majorly on technical skills that could facilitate employment. Thus these fourteen respondents that actually started their own businesses initiated this effort solely with their own initiative and resources. The program did not have any provision to help them start a business. Moreover, for anyone wanting to start a business, accessing finance is also a hindrance. The project did not dedicate resources to connect the youth with various financing institutions as it was not planned or designed to provide this connection. Indicator 3.4: Number and percentage of youth with satisfactory internships or apprenticeships Overall, 52% of the total participants trained have been inducted into either an internship program or as an apprentice. The projects that facilitated internship included the ones implemented by AKF, ASK Development, and Sabawon. Figure 14 shows the level of satisfaction with internships among the interviewed trainees: Figure 14: Youth Satisfaction with Internships and Apprenticeships Program Satisfaction ranking Excellent Good Fair Poor N/A Overall internship experience 45.7% 34.3% 11.4% 0.0% 8.6% Relevance of the assignment to training 35.3% 47.1% 8.8% 0.0% 8.8% Technical skills learned 40.0% 31.4% 14.3% 5.7% 8.6% Coaching received during internship 48.5% 30.3% 12.1% 0.0% 9.1% Page | 22 Thus, a majority of trainees were very satisfied with the internship opportunities that the projects offered. These internship opportunities were secured by implementing partners through strong outreach to the private sector. The CNG project, for example, partnered with the All Pakistan CNG Association whose membership includes more than 300 CNG stations. The hospitality project engaged the private sector early in the process by holding a series of strategic roundtable discussions. These discussions not only helped AKF develop a strong curriculum, but also build momentum among stakeholders, including small hotels and restaurants, which were willing to provide internships to training participants. Indicator 3.5: Number and percentage of placed youth who declare having a satisfying and quality job The study finds that ninety percent of the respondents felt satisfied with the quality and remuneration of jobs they were able to obtain as a result of the training. More than 75% of the respondents found that their training was relevant to their jobs. A break-down of the responses is provided in the table below: Figure 15: Job Satisfaction of Youth Are you happy with the current job, level of job and other type of compensation provided? Indicators Very Poor Poor Fair Very Satisfying Overall work environment. 0.0% 9.8% 52.9% 37.3% Relevance of the job in the field of study 2.0% 8.2% 38.8% 51.0% Support from supervisor and or peers 2.2% 8.9% 48.9% 40.0% Possibilities for professional growth. 2.3% 11.6% 51.2% 34.9% Salary 2.1% 23.4% 40.4% 34.0% Other Compensation 9.5% 14.3% 42.9% 33.3% Indicator 3.6: Net salary increase of youth employed through the EEA programs (within six months after placement) A synopsis of the salary before and after the training is provided in the table below. It is worth noticing that percentage of youth with less than $50 and $50-$100 monthly salary reduced, while the youth earning a monthly salary between $100-$300 and $300-$500 increased. Therefore, trainee salary levels have increased from a lower level to a higher level. This can be attributed to the relevance of the skills imparted to the ones required by the industry, and in general willingness of employers to pay the program trainee higher than other employees. The figure below compares salary levels of EEA-P graduates before and after the intervention. Figure 16: Job Income Comparison Income Level in US$ Before Intervention After Intervention Less than 50 25.90% 24.10% 50-100 44.40% 37.90% 100-300 22.20% 27.60% 300-500 3.70% 6.90% 500-800 3.70% 3.40% Over 800 0.00% 0.00% Figure 17: Income Level Comparison (Before and After the Intervention) Page | 23 Indicator 3.7: Level of employers’ satisfaction with the soft skills and/or technical skills of EEA trained youth Interviews with employers revealed that the performance of the EEA graduate is relatively better as compared to the non-EEA graduates. Several recurring trends can also be drawn - although these were mentioned in response to suggestions for improvement of the EEA-P program, they can be applied generally to any technical vocational program for youth: • Training duration should be adequate to cover material at a reasonable pace • Training should include practical application of skills being taught to reinforce lessons • Training schedule should be flexible in order to provide access to a greater number of youth • Training programs should include modules on improving English language communication Employers also provided important feedback on the EEA-P graduates they have hired and observed in the workplace. These quotes demonstrate the positive effect EEA-P training programs had on critical employability skills. Direct quotes by Employers about the EEA graduates they employed: • “The confidence level, attitude and discipline were good among the program trainees” • “Trainees have gained exceptional knowledge and education through the program” • “The trainees have increased motivation level” • “The female trainees we have hired are dedicated and regular” • “The trainees have good management, communication and data handling skills” • “The trainees require less effort to understand the assigned tasks” • “The trainees are well aware of the sector they are trained in” Further data from the twenty employers that participated in the EEA-Pakistan evaluation demonstrate their support for EEA. How did you know about EEA program? • Fifty-six percent said they were approached by EEA project partners • Twelve percent found out about the program from media (brochures, newspapers and magazines) • Nineteen percent reported that somebody/some company refereed EEA graduates Performance ratings • Fifty-five percent of employers rated overall performance of graduates as good; 40% as excellent; and 5% as fair • Technical skills: Fair – 20%; Good – 40%; Excellent – 35% • Life skills: Fair – 30%; Good – 40%; Excellent – 25% Comparing EEA graduates with other graduates • Seventy-four percent of employers said EEA graduates are better than other graduates and 21% as the best; 5.3% as the same • Eighty-eight percent said EEA graduates met their expectations or needs Would you consider offering jobs, internships or apprenticeships to EEA graduates in the future? • All employers responded Yes In your opinion, do you think EEA training is closing the gap between employers’ need and the employees’ qualifications? (i.e., providing qualified candidates to the business sector meeting the market needs) • All employers responded Yes What do you know about EEA? • “They are providing quality services for producing quality staff for organizations” • “They provide capable and skilled employees” Page | 24 In your opinion, do you think EEA training is closing the gap between employers’ needs and the employees’ qualifications? (i.e. providing qualified candidates to the business sector meeting the market needs)? • All employers that responded to this question agreed (nineteen respondents). They all agreed to offer internships and jobs to EEA graduates in the future as well. Explain whether your EEA experience has any effect on your approach their human resources management approach in recruiting youth for internships or jobs. • Thirty-eight percent of responding employers stated that the company plans to become more involved in alliance programs focusing on youth employability. • Nineteen percent of employers stated that the company is more open to the idea of providing internships and mentorship support to youth in general. • Thirteen percent of responding employers stated that the company is likely to recruit youth through EEA programs and other similar programs. Page | 25 Intermediate Result IV: Positive Indirect Effect on Families and Communities of Youth Summary of Intermediate Result Four Findings The study also finds positive if indirect effects of the program on families youth served. Over eighty percent of youth are sharing their incomes with their families for health, education and household expenses. It also seeks to determine how many youth feel like they now have ability to improve financial and social conditions of their families. Seventy percent of youth surveyed reported that their financial support helped improve the financial and social situations of their families. Indicator 4.1: Number and percentage of employed youth financially helping their families for household expenses, health and education The study revealed that 82% of the respondents are helping their families in terms of household expenses, health and education to a certain degree. Eighty-two percent of the respondents reported that their support helped improve their families’ financial and social standing, and 18% reported no change. Additionally, 34% of the respondents said they noticed major improvements in their relationships with family and friends after completing the program, 41% reported some improvements, and 25% reported no change. Indicator 4.2: Number and percentage of youth who have reported improved financial and social conditions of their families The study sought to determine not only how many youth are helping their families but how many feel like they now have ability to improve financial and social conditions of their families. The results are presented below: • Seventy-five percent of graduates stated that that their financial support (with the income they shared) helped improve their family’s financial and social situation. • Eighteen percent of graduates reported that their support somewhat improved the situation. • Eleven percent of youth reported no change. Page | 26 Section IV: Challenges, Lessons Learned and Recommendations Effectiveness, Relevance and Sustainability The EEA-P final evaluation has led to several conclusions and recommendations presented below. This section presents the report’s main conclusions and recommendations on the EEA-P program’s effectiveness, relevance and sustainability. Main conclusions and recommendations are as follows: • Capacity of implementing partners has been enhanced through the program. • Partners are willing to scale the programs, but funding sources have not been identified. • Recommendations include: o Longer training duration o More training content on English language and computer skills o More practical training opportunities o Improved job placement services. Effectiveness: The effectiveness of any intervention is gauged through measuring its performance against the intended benefits. The EEA-P was well designed and addressed the evolving needs of the beneficiaries. It remained responsive to the needs of youth being trained, the alliance partners, and employers. The training programs were developed based on needs assessment of the employers and historical context and experience of the implementing partners/training agencies. Both the CNG and Hospitality project conducted extensive market research prior to developing training manuals. And Both ASK Development and AKF revised their training plans during implementation to respond to new realities: ASK Development identified a high-demand training program and location mid-way through implementation and quickly made the necessary adjustments to deliver the training program; AKF originally planned to provide hospitality training in Swat, but quickly realized that the tourism market there was rapidly declining, so they canceled that training workshop and added more participants to locations with more likelihood for job placement. Good planning and swift responses to changing environments and market needs were essential in the effectiveness of the training programs and ability to place participants into jobs. And employers rated EEA graduates higher as compared to the non-EEA graduates, which highlights the effectiveness of EEA-P. The program had various effects on the stakeholders involved in its implementation, primarily the youth participants, their families, and their employers. Over the life of the program, it trained 1,386 youth of which 508 (37%) -- 411 male, 97 female -- secured employment. These youth are contributing significantly to their families and the local economy. Approximately 76% of the interviewed youth declared that their family’s financial and social conditions improved after the completing the training program, and about 96% expressed that they would be interested in pursuing further studies or training after completing the program. The program led to increased profits for various businesses that employed the EEA-P trained youth. Youth also reported increased confidence as a result of the training, which may not be directly attributable to results, but has apparently encouraged twelve entrepreneurs to start their own businesses. Recommendation: One target that was not met in EEA-P was the percentage of female participants. Effectiveness of the program in terms of gender inclusiveness can be increased by training females as Master Trainers which was not specifically targeted in the program. It can also be enhanced by creating innovative avenues for young women to participate in trainings which are much more accessible to them. Relevance: As suggested in USAID’s RFA, which is quoted in the Problem Statement section above, Pakistan’s current education system does a poor job imparting skills which are required to get a decent employment or are irrelevant to the market needs. In response to this problem, the EEA-P program designed demand-driven trainings in consultation with industries that were relevant to the kinds of employees needed by these industries. EEA-P training programs also featured guest speakers from the private sector, field visits to potential employers, and seminars attended by all stakeholders—participants, implementers, and potential employers— which served to raise awareness and provide young people an opportunity to test their new communications skills. By enrolling in the EEA-P designed projects, youth acquired skills that were valued and appreciated by the employers as well as helped them in getting employed. Recommendation: The relevance of the program could have been further enhanced by increasing the number of exposure visits and the hands-on training. This recommendation was identified by the youth in the focus group Page | 27 discussions and feedback forms. Moreover, internship periods should be increased under the training projects to further increase the relevance of the training. Sustainability: For the most part, the training programs developed under EEA-P lack financial independence due to the inability to generate revenue through fee-based training and a lack of external funding to support more training workshops. RSPN is not continuing to build alliances to provide such trainings due to absence of funding to run the secretariat and provide grant funds for additional projects. The trainings imparted by various implementation partners were subsidized through the project funding and the trainees were not charged the cost of service delivery based on actual. The reason for the subsidy was the program’s target: unemployed youth who are not able to pay fees to support the cost of the training programs. Nevertheless, EEA-P made a strong case to civil society and private sector stakeholders that youth employability is a critically important issue that can be addressed with innovative approaches and moderate levels of funding. It is too soon at the writing of this report to determine whether the strategy and approach of EEA will sustain in Pakistan, but NAVTEC’s funding of a subsequent round of training by ASK Development and USAID/Pakistan’s endorsement of EEA as a successful pilot worthy of expansion indicates that it will be sustained in some manner. Recommendation: The long-term effectiveness program could have been even greater if the sub-grant projects would have been replicated and scaled up immediately following the pilot training phase supported with EEA funding. Only the ASK Development project received additional external funding to continue offering training programs. Nevertheless the program approach had a significant effect in terms of adoption by project partners and donors like USAID/Pakistan replicating the approach in its Empowering Pakistan: Jobs program. Also, the program should have made more efforts to engage in policy level discussions with the Government of Pakistan to initiate the EEA approach at its training institutes. It could have also worked with the private sector to offer these trainings on a profit-oriented basis. Finally, corporate social responsibility wings of various private sector concerns could have been approached to generate funding for scholarships for unemployed youth. Conclusion The challenge of workforce development is to ensure that the needs of both unemployed youth and industry are met. For employers, the workforce system must ensure that there is an available labor pool of sufficient size and skill to support continued economic vitality and efforts to improve productivity. For workers the system should ensure that individuals have opportunities to acquire the skills necessary to gain and maintain decent employment at a living wage. The USAID funded EEA-P program did not attempt to address the policy environment issues discussed above as this was not within its mandate. However, the program was able to introduce a model that can be replicated at the national level to offer training services that are demand driven. This would entail continued multi-sector collaboration to identify and address the employability needs of young people and industry and increased involvement of national public and private sector vocational training providers. EEA-P was able to demonstrate its positive effect on the lives of the youth trained by facilitating new employment opportunities for unemployed youth, better opportunities and increased salaries for those who were already employed, confidence and personality building through provision of life skills, and most of all a sense of strength to do something meaningful in life. The EEA-P sub-grant projects, however, were not able to demonstrate immediate replication, which indicates that further work needs to be done to improve the program design to ensure sustainability. The program’s approach nevertheless was well appreciated by the partners, employers, youth, and donors like USAID Pakistan. The alliance model introduced through this program can nurture future youth employability efforts in Pakistan. ©2010 International Youth Foundation. All rights reserved. No portion of this report may be reproduced in any form without the written permission of IYF. This project was made possible by the generous support of the American people through the United States Agency for International Development (USAID) under grant agreement no. RAN-A-00-04-00043-00 This report was funded in part under grant agreement no. EPP-A-00-08-00006-00. The International Youth Foundation (IYF) invests in the extraordinary potential of young people. Founded in 1990, IYF builds and maintains a worldwide community of businesses, governments, and civil-society organizations committed to empowering youth to be healthy, productive, and engaged citizens. IYF programs are catalysts of change that help young people obtain a quality education, gain employability skills, make healthy choices, and improve their communities. To learn more, visit www.iyfnet.org ANNEX G: EEA Philippines Evaluation Report CONTENTS Executive Summary Section I: Introduction Section II: EEA-Philippines Results Framework Section III: Findings and Analytical Review of Program Achievements Section IV: Challenges, Lessons Learned and Recommendations Conclusion FIGURES Figure 1: Classification of Resources Provided by EEA Alliance Members Figure 2: Sustainability Status by Project Figure 3: Project-level Costs Figure 4: Alliance Building and Program Support Costs Figure 5: Cost per Beneficiary Figure 6: Out-of-School Youth Statistics in Mindanao Figure 7: Number of Youth Participating in EEA projects Figure 8: Number and Percentage of Young Men and Women Completing EEA Projects Figure 9: Number of Teachers, Trainers, Facilitators and Counselors Trained Figure 10: Youth Satisfaction with EEA Trainers and Counselors Figure 11: Percentage of Young Men and Women Demonstrating Improved Capabilities Figure 12: Youth Satisfaction with the EEA Training Figure 13: Life Skills Acquisition Figure 14: Youth’s Vision for the Future Figure 15: Number of Graduates Placed in Jobs Figure 16: Job Placement Rates of EEA Projects Figure 17: Comparison of Income Levels Figure 18: On-the-job Training Experience Figure 19: Job Satisfaction ANNEX Annex A: Typology of EEA Partners and Leverage Contributions Acronyms ARMM Autonomous Region of Muslim Mindanao CSR Corporate Social Responsibility DCCCI Davao City Chamber of Commerce Inc Foundation DOLE Department of Labor and Employment DSWD Department of Social Welfare and Development EEA Education and Employment Alliance EU European Union FLEMMS Functional Literacy, Education and Mass Media Survey IYF International Youth Foundation LGU Local Government Unit MIAP Metal Industries Association of the Philippines NGO Non-government Organization NSCB National Statistical Coordination Board OJT On-the-job Training OSY Out-of-school Youth STI Systems Technology Institute TESDA Technical Education and Skills Development Authority USAID United States Agency for International Development YPS Youth Productivity Services Executive Summary The Philippine Government estimates that there are roughly 12 million out-of-school youth in the Philippines representing fifty percent of the total youth population.1 This is a sector rarely assisted by government programs, yet often cited for being responsible for crime, drug use and other negative behavior. In response to this challenge, the Education and Employment Alliance (EEA) Philippines program was launched in February 2006 to equip out-of-school youth in Mindanao with the necessary employability skills and help them become productive workers and entrepreneurs. In partnership with leading non-governmental organizations (NGOs), EEA has provided integrated technical education training to 3,036 out-of-school youth (of which 2,669 graduated) over its three-year project period. The summative evaluation was conducted to determine the overall value of the public-private alliance approach used, and the effect of employability skills projects on youth and their families. Based on the analysis of program evaluation data and interviews with participants, employers and implementing partners, this study shows that the program was successful in providing demand-driven skills training and placing youth in jobs. A total of 1,961 youth or 73% of graduates have become either employed or self-employed. Nine graduates are continuing their education. EEA’s job placement success exceeds national benchmarks and is attributed to the ability of partners to help fully prepare graduates for the job market by providing both technical and soft skills and equipping them with government certificates, as well as the industrial knowledge of partners, their active liaison with the target industries, networking with employers to find job opportunities, and using corporate social responsibility as an entry strategy to access jobs for the youth. The study also finds that EEA was effective in forging public-private alliances to mobilize community resources, provide relevant skills training and placing youth in jobs. More than 110 institutions, agencies and private corporations partnered with the alliance on both national and local levels in support of youth employability. The evaluation discovered that using the alliance approach was regularly regarded as the key reason why partners felt that their projects succeeded. In this regard, the alliance based approach to programming appears to foster creativity, sharpen management focus, and help promote cost effectiveness. Nonetheless, building alliances around youth employability programs does pose challenges, and partners also acknowledged the difficulty of building an alliance, in convening meetings, troubleshooting among multiple partners, and engaging alliance members in resolving operational issues of the program. A concerted focus on alliance building also brought tangible benefits to all donors to programs, with the overall program receiving a total of $1,802,681 in leverage contributions of which $778,723 represented cash contributions and $1,023,958 included in-kind contributions. This was against seed funding provided by USAID to the projects of approximately $600,000, and the program achieved a leverage ratio of 2.5:1. From a sustainability perspective, the multiplicity of partners working together seems to have provided important avenues for building on successful programs and promoting long term sustainability of efforts. In this regard, in addition to the six projects supported by EEA directly, at program end a seventh program was initiated, which is a spin off or replication of the EEA approach that has been adopted by the Philippine government. In this effort, known as the Youth Productivity Services (YPS) program, the Philippine government is seeking to utilize alliance building methods in nine municipalities in the province of Misamis Oriental to encourage local ownership over initiatives and training that is directly relevant to local needs. As such, while early in implementation, it appears that through this avenue, significant numbers of out-of-school youth can be added to EEA’s efforts to promote access to relevant technical skills training and find jobs. In other areas, the study shows that youth were largely satisfied with the quality and relevance of training provided and rated their training as either “good” or “excellent.” However, they were not as satisfied with the quality of internships received, citing the need for stronger and more structured on-the-job training programs. The employers who provided on-the-job training to the youth or employed them were similarly positive in their evaluation of the youth. Suggestions for improvement include: providing regular follow-up support to new workers on the job, more on-the-job training by companies, training in advanced skills and further basic knowledge such as reading engineering plans or learning specialized welding skills. Parents or relatives of the youth were also seeing changes in the attitudes and behavior of the youth, which have set examples for other siblings in a family and for out-of-school youth in the community. The study finds that a majority of youth are remitting their wages to their families. As a result, more than half of the youth felt, their finding a job has improved the lives of their families. EEA graduates who were out- 1 National Youth Commission, Office of the President, 2003 (projected) of-school youth now feel confident, positive about their futures, and know they can keep decent work utilizing their newly acquired skills and government trade licenses. Additionally, companies that have hired EEA graduates find that trained out-of-school youth can perform as well as high school graduates if not better, thereby helping them shift their views on working with disadvantaged youth. In conclusion, this study finds that the EEA Philippines program, after three years of implementation, succeeded in meeting all of its intermediate objectives of building alliances, training youth and opening access to on-the-job training, apprenticeship and employment for youth. It engaged families of youth and local government officials in the program which has helped sustain program results. On the last intermediate objective of family engagement, this became a natural consequence of targeting out-of-school youth as their families had to be involved actively in the program for it to succeed. While these results are quite positive, it is also important to note alliance building did prove challenging in some important respects. In particular, partners argued that the time of the program was too short to build an alliance, and implementation was sometimes shortened by longer timeframes for project design. Additionally and more generally, issues that needed further development include: creating more effective ways of linking skills and jobs; addressing specific challenges of working with at-risk youth; developing better programs for apprenticeship and on the job training; and better leveraging lessons learned to launch a program of policy advocacy and reform that could make youth employability a critical issue for national and regional governments. Section I: Introduction The Education and Employment Alliance (EEA) program in the Philippines was implemented by the International Youth Foundation (IYF) and Consuelo Foundation, Inc. (CFI) to help at risk, out-of-school youth in Mindanao gain skills to access the job market or establish livelihoods. The program, implemented from late-2005 to mid-2009 aimed to use an alliance-building approach, deliberately seeking to leverage and combine the skills and resources of partners from the public, private and civil society sectors, to ensure program results and sustainability. While public private partnerships have been common features in Philippine development projects, and training and jobs are indicated for most vocational technical training programs, these elements had not been previously combined to ensure successful job training programs to a particularly vulnerable group, out-of-school youth in Mindanao. Generous funding support from the United States Agency for International Development (USAID) made possible this effort by EEA Philippines to explore innovative ways to combine these elements to effectively train at-risk youth and help them find work. When EEA was launched in late-2005, only a few private and government agencies were addressing the problems of out-of-school youth. There was no formal measurement of the extent of the problem of idle youth who dropped out of formal schooling due to poverty, conflict and family problems. However, being out-of-school was recognized as one reason why young men were joining the armed conflict in Mindanao. To begin the program, a Mindanao Youth Labor Study was conducted that revealed the mismatch between the entry level skills required by industries for employment and what graduates of traditional Technical/Vocational Education Training (TVET) courses offered. It scanned opportunities where EEA could build on and design six innovative programs for integrated employability skills training. This evaluation study sought to capture the program learnings generated by these innovative pilot projects which used the alliance approach to pursue employability among at-risk youth in Mindanao. A. Evaluation objectives, design scope and methodology The final evaluation of EEA Philippine program has been conducted as part of IYF’s EEA global evaluation. The goal of the evaluation is to assess: 1) the value and utility of the EEA alliance approach in terms of sustainability and scalability, and 2) the effect of the EEA alliance projects on direct beneficiaries. The first evaluation objective is to assess if and to what extent the program is successful in attaining four key program results: sustained alliances for youth education and employment; improved access to relevant training leading to jobs for youth; improved prospects for employment and entrepreneurship; and affecting positive, indirect effect on their families and communities. A second objective is to understand how the alliances, implementing partners and the secretariat performed on the following key alliance results: 1. The effectiveness of the EEA alliance approach in building partnerships, and promoting sustainable and scalable youth employability programs 2. In-kind and cash resources leveraged for the program through private sector and government partners 3. Number of private sector, government and civil society partnerships formed through the program to jointly offer employment and business development training activities in targeted areas 4. New partnerships, mechanisms and approaches created as a rippling effect of working with EEA programs to support youth employability The evaluation of the EEA Philippines program was undertaken by Ms. Marcia Miranda, IYF Consultant, between November 2008 and June 2009 using both qualitative and quantitative methods to assess the immediate and mid-term results of EEA Philippines’ subgrant projects. Field visits were undertaken from November to December 2008 in all project sites. Ms. Miranda worked closely with project implementing partners who organized focus groups, made appointments with partners, and traveled to administer survey questionnaires in the difficult-to-access areas of Tawi-Tawi and Manguindanao. The evaluation capped the intensive monitoring of projects under the EEA global monitoring and evaluation framework that had been put in place early in the program. The monitoring and evaluation process was managed by CFI as the lead implementing partner and Secretariat. Throughout the program, in addition to quarterly reports from implementing partners, and field monitoring visits by CFI, EEA implementing partners were convened for special training workshops to share project learnings that were surfacing as a result of regular monitoring and evaluation of progress toward program outcomes. For the purpose of this evaluation, a total of 169 respondents from six different stakeholder groups were asked to provide insights into how the project was implemented. These were: (a) the alliance partners which included members of the national alliance, steering committee and the community; (b) employers who hired youth or helped by accommodating youth internships; (c) direct youth beneficiaries; (d) trainers, (e) friends and family members of youth; and (f) implementing partners including the Secretariat. Survey instruments were developed to capture their insights and administered in the project areas. The global instruments adapted were translated into Visayan and Tagalog. In addition, focus group discussions of youth and alliance members, and key informant interviews were done to deepen understanding of their responses. A purposive sample of 110 youth, chosen from those who had completed their training and had gained employment, participated in the survey and focus groups. The other survey respondents included nine employers who employed youth of two projects, 19 alliance members, 13 trainers, 12 implementing partners, and six friends and family members from one remote barangay in Maguindanao and the National Secretariat. The findings of an evaluation workshop conducted by the Davao City Chamber were also incorporated in the study. B. Evaluation Limitations and Challenges The study was limited in terms of being able to comprehensively cover all the locations where community based training activities were implemented. During the field work, parts of Maguindanao were inaccessible due to renewed conflict and military operations. The project site in Sitangkai could not be accessed by the lead evaluator because of security concerns. For both of these areas, implementing partners administered the survey questionnaires after receiving guidance from the Consultant. A second limitation was the inability to randomly survey youth who had completed the training. The sampling of the youth respondents were purposive for only youth that had completed the program, meaning they had finished the training and been employed. There were limitations in this aspect since not all youth could leave their workplaces to participate in the survey. The respondents were not randomly sampled and were only those who were available during the scheduled field visit. To maximize involvement of the youth, the surveys and focus groups for the youth were administered after work in the evening or on weekends. Additionally, as discussed in detail in the overall global evaluation, this study is inherently limited by the lack of a fully external evaluation of program outcomes. As noted in the global introduction, given both resource constraints and the desire to leverage the broad experience of program participants to fully compare and contrast approaches used in the different countries for governance, program design, leveraging of resources and other elements, a strategic decision was made to utilize participants in the overall program to support this evaluation. In the interest of promoting learning and improvement, the EEA teams have done their best to express challenges and problems within programs and the alliance structure, grounded in the reality of their experience in the program; however, the study does have this inherent limitation, which is important to disclose fully. C. Project Summary Over its three-year project period, a multi-leveled alliance building approach was developed to design and implement six integrated workforce development training projects to help out-of-school and at-risk youth in Mindanao. More directly, the goal of EEA Philippines has been to develop community-based partnerships and alliances that pilot, expand and sustain innovative education and employment programs for at-risk and out-of-school youth. In reaching this overall goal, the program has sought to meet the following program objectives: 1. To develop a sustainable alliance structure and pilot innovative partnership activities that expand relevant education and employment opportunities among disadvantaged youth, and 2. To develop and test models of sustainable public-private alliances, and those that promote best practices for the use of technology to improve the quality of education at all levels, both in and out of school. As it constructed alliance based programs, EEA Philippines created six workforce development programs that were ultimately targeted to reach a total of 2,958 at-risk and out-of-school youth in Mindanao, a region where development has been skewed by war and poverty. In doing so, it also attempted to create concrete manifestations of the value of public-private alliances, by leveraging the initial USAID seed funds of $617,968 at a ratio of 3:1 through collaboration between global, national and community alliances. Problem Statement Mindanao is the second largest island in the Philippines with a population of 18 million. It is a land of abundance and opportunity, but untapped because of years of war and conflict. The Mindanao war has been described as a five-century struggle that is the second oldest on earth.2 The impact of this strife is seen in the high poverty incidence in Mindanao. In 2003, almost 23.8 million people lived below the Philippines' poverty threshold of slightly over US$ 100 a month. This represents 24% of Philippine families and 30% of the entire population. In addition, the country suffers from one of the highest levels of income inequality in Asia, with the poorest 20 percent of the population accounting for only five percent of total income or consumption. Fourteen of the poorest Philippine provinces are in the island of Mindanao. The poverty in Mindanao is sharply felt in areas which have been affected by conflict. In these areas, education is seriously affected. In some conflict areas, the ratio of students to classrooms is at 83 to 1. Correspondingly, while the Philippines boasts of a literacy rate of 93.7% for males and 94% for females, in Mindanao, literacy rates for Central Mindanao were 85.9% for males and 88.89% for females while in ARMM, it dropped to 71% for males and 69.4% for females.3 The dropout rate in Mindanao is also the highest in the Philippines at 12%, compared with 6.1% for the island of Luzon and 7.3% for the Visayan Islands. Some conflict affected parts of Mindanao have dropout rates of 23%. The percentage of out-of-school youth (OSY) is also the highest in Mindanao at 23.1% compared to the national average of 14%. While out of school, Mindanao youth find that they are unable to find jobs, which is the main reason cited for dropping out. A USAID study estimates that on average, youth unemployment in post-conflict areas is approximately six to seven times higher than the national average. In the ARMM provinces, youth unemployment is 42%, compared to the overall national adult average of six percent and youth national average of 16%.4 In an effort to overcome the challenges of poverty, high youth unemployment and prolonged conflict, the Education and Employment Alliance program in the Philippines sought to provide the necessary training to out-of-school youth (OSY) to find jobs and prepare them to succeed in a knowledge-based economy as responsible, productive adults. The training was matched to meet the job needs in Mindanao such as agriculture, fisheries, services and self-employment. Programs also provided employability and life skills, such as literacy and numeracy, inter-personal communication and conflict management to build the intangible skills so important in the workplace and in life. Particularly in conflict and post-conflict areas, EEA programs were designed to be sensitive to the needs of OSY who often faced significant barriers to employment. Employers were often reluctant to hire out-of-school youth, particularly those who did not have a high school diploma and had minimal or no prior work experience.5 The problem of OSYs has been connected to the growth in terrorist activities as youth had the propensity to join in the conflicts that surrounded them. Youth spoke of the poverty that forced them to drop out of school, leaving them with few or no marketable skills and vulnerable to recruitment by armed militias, kidnap by gangs and drug traffickers. With low self-esteem, many youth could not envision a future. More positively, however, the youth surveyed placed education at the top of their list of priorities, citing it as a viable solution to their problems and the passport to a better future.6 Education was important to youth in conflict areas. They said that education introduced a daily routine and a sense of purpose and order in their lives. It helped them earn respect and prestige in their communities. Older youth were keen on non￾formal education and skills training that would help them get remunerative employment. Irrespective of whether they chose formal or job-linked training, the youth saw access to education as one of the first steps in resuming control of their strife-affected lives.7 The Philippine Experience in Assisting the OSYs through Alliances Among the many development programs in the Philippines, there are only a few that focus on youth employability, despite the fact that about forty percent of the Filipino youth drop out of school before they acquire their high school diploma. A majority of youth programs have focused on in-school youth, and 2 Salvatore Campo and Mary Judd. The Mindanao Conflict in the Philippines: Roots, Costs and Potential Peace Dividend. Social Development Papers. World Bank. Feb 2005. http://quickplace.emindanao.org.ph:9080/StaticProjectPage/pdf/Roots.pdf 3 FLEMMS Survey. National Statistics Office. 2003 4 USAID/Asia and Near East Bureau. Jobs for the 21st Century: Synthesis Paper. June 2007 5 USAID/Asia and Near East Bureau, USAID/Philippines. Jobs for the 21st Century: Philippines ARMM Assessment. December 2005. 6 Rajendran, Shobhana. The Impact of Armed Conflict on Male Youth in Mindanao. World Bank. July 2006 7 Ibid. initiatives to assist in the development of the OSY have come from only a few private institutions. A study conducted in 1999 by Consuelo Foundation, funded by International Youth Foundation and the World Bank, showed that in addition to the need for more organizations to train OSY for employability and or provide them with alternative learning system, there was also the need for a more organized participation of the private sector, especially the business community. It was noted that most business entities limited the operations of their corporate social responsibility projects to areas where they operated and/or those that can be directly linked to their businesses. With the prevalence of poverty and the resulting high incidence of out-of-school youth in Mindanao, the EEA program became an urgent need. Analytical Review of EEA Philippines Project Achievements As of 31 March 2009, through six projects, a total of 3,036 at-risk youth (vs. 2,958 targeted) were trained in technical and entrepreneurship skills, of which 2,669 youth graduated. Of these graduates, 1,794 youth found employment and 167 established micro businesses. When compared against program targets, the results are positive. From these graduates, 1,794 or 67% of graduates found jobs and six percent of graduates set up small businesses within six months. The job placement rate surpasses national benchmarks set by TESDA, the government technical training agency. The innovation of EEA was to bring technical training to remote and difficult to reach communities and enroll at-risk and out-of-school youth. To do this, NGO partners adapted government approved curriculum to be community-based and mobile, trained new teachers and operated with flexibility to reach remote islands of Sitangkai, or inland to the hills of indigenous tribes people and conflict affected communities or in municipal spaces. A total of 103 trainers were identified from the community and further trained or certified with the government TESDA8. The projects were implemented by six implementing partners, all non-profit organizations, of which two were industry associations. The alliances that operated during the project were multi-tiered. At the top tier was a national alliance and steering committee based in Manila. This alliance was supported by the Consuelo Foundation as National Secretariat, with the responsibility to engage the national alliance in supporting and approving the design of projects, leveraging resources, and assisting in sustaining and taking successful projects to scale. As projects were designed and implemented, they were overseen by the Consuelo Foundation – however, each looked to also create within the partnership, a community based alliance that would support the program technically and financially, as well as employ youth. In the end, more than 110 institutions, agencies and private corporations were tapped to help the projects from early 2006 to 2008. A measure of its success was the resources that were leveraged as of March 2009. Against USAID seed funds of $617,968, the project leveraged a total of $778,723 in cash and $1,023,958 in in-kind contributions. There were seven EEA projects at program end. Six of which were funded by EEA IYF and one was a spin off or replication of the EEA approach with joint support by the national and local governments. Summaries of these seven projects and their achievements are presented below. (1) Civil Trades Training for Ex-Combatants in Mindanao Phase I and II (April 2006-June 2008): This project has provided vocational skills and job opportunities to ex-combatants and disadvantaged youth from communities displaced by war. It was implemented in two phases and provided 735 OSY in Mindanao and Luzon training in construction skills. A total of 724 youth completed the training of which 426 have been placed in jobs. Of 426 youth who secured jobs, 75 youth have been employed in the construction of school buildings which is funded by the Petron Foundation under EQuALLS and ten youth are working in Chevron-sponsored elementary schools. The project was implemented as a partnership between Habitat for Humanity and the EEA Implementing Partner, Alterplan. (2) Enhancing the Capacity of Davao Out-of-School Youth in Agriculture Production (August 2006- May 2008): The project has trained 180 OSY in food production and processing, fruit production and organic vegetable farming. The program was supported by an alliance of academic institutions, local governments and their line agencies led by the Davao City Chamber of Commerce and Industry. The project ended in May 2008; however, the Davao Chamber of Commerce is continuing the program as a corporate social responsibility (CSR) initiative, albeit on a limited scale, responding to city communities that wanted to improve skills of their youth. Since EEA funding ended, the project has trained 100 additional youth. (3) Integrated Agri-Fisheries Entrepreneurship Training for Rural Youth in Mindanao Phase I and II (August 2006-June 2008): The project has focused on youth entrepreneurship among out-of-school youth 8 TESDA or the Technical Education and Skills Development Authority is a national government agency under the Department of Labor that formulates manpower and skills plans, sets appropriate skills standards and tests, coordinates and monitors manpower policies and programs, and provides policy directions and guidelines for resource allocation for the TVET institutions in both the private and public sectors. http://www.tesda.gov.ph in remote communities, often conflict-affected. Entrepreneurship training and business mentoring was provided to 359 marginalized OSY – the majority of whom were women. Phase II of the program dedicated funds to finalize and develop a manual for the innovative entrepreneurship curriculum. Second, 150 young entrepreneurs received additional mentoring from the business sector. Third, the curriculum was endorsed by the Technical Education and Skills Development Authority (TESDA), an important step in promoting the widespread and long-term use. Start-up capital was secured for 33% of trainees. (4) Integrated Small Engine Repair in ARMM: Marcellin KAPATID Program (August 2006-October 2008): The project has focused on helping urban, poor OSY and youth in conflict. A total of 375 disadvantaged youth were trained, initially in small engine repair then in basic welding and 225 youth have been placed in jobs. The trainings were community-based and in three topic areas: basic welding and small engine and automotive repair; math, reading, English; and life skills. With $56,000 in financial support from Chevron Philippines, the project has established three training sites or “service centers” where trainees are employed as center mechanics. The project will use other sources of funding (including Chevron funds) to continue community-based training activities in other barangays in General Santos and Saranggani province in FY2009. Marcellin is recognized by local barangays as an effective provider of community-based technical training and intends to use this as leverage in finding avenues to continue providing employability training for youth. (5) Empowering Youth to Modernize the Seaweed Industry in Tawi-Tawi (June 2007-August 2008): This project has trained 1,042 disadvantaged youth to become skilled and active participants in Tawi￾Tawi’s seaweed industry. The goal of the one-year project was to increase the productivity of seaweed farms to compete in the growing global seaweed market. It targeted Muslim youth who lived in remote water villages with limited access to basic education. The project also adapted the curriculum for selected EQuALLS program sites in Tawi-Tawi and trained 100 youth in seaweed production last quarter. Local partners also implemented the recently-approved project funded by a German Foundation, Kindernothilfe, to continue the training. The seaweed associations, of which program graduates are part of, continue to receive technical and financial assistance from the Philippine Development Assistance Program under a larger enterprise program for the seaweed industry of Sitangkai, Tawi-Tawi. (6) Metal Works Industry Association of the Philippines (MIAP) Plus Phase II (April 2006- June 2009): This project built on the successes of phase I where 120 out-of-school youth in Misamis Oriental were successfully equipped with critical academic, life and technical skills required for employment. Phase II trained over out-of-school youth in welding, academic subjects and life skills. MIAP is recognized for its demonstrated ability to provide basic skills training and job placement by offering advanced welding training, life skills and intensive academic training through the Department of Education’s Alternative Learning System (ALS). In terms of sustainability beyond EEA, MIAP is committed to mobilizing local resources to continue to provide technical training to youth and prepare them to work in the local metal works. In addition, MIAP plans to institutionalize its training interventions through its Metal Village project within the Phividec Industrial Zone in the province. (7) Youth Productivity Services (YPS) Program (October 2008-Present): The Youth Productivity Services program is modeled on EEA and launched as a pilot of the national government following advocacy by EEA and IYF. The success of the YPS is seen to sustain the work of EEA Philippines to help out-of-school youth access training and employment. The scheme will be tested in nine municipalities in Misamis Oriental in partnership with the EEA Partner, MIAP. It will support nine municipalities in Mindanao to develop their own community supported employability programs for youth through alliance building training and government capacity building workshops. The target for the pilot which will run for two years until 2010 is to train 450 out-of-school youth in various technical, life and employability skills. The government has allocated $151,600 to fund the program across nine municipalities. Different from the EEA projects, the training areas are varied, such as welding, masonry, electrical installation, plumbing, physical therapy, and culinary arts. Labor market assessments were completed and five municipalities have initiated welding and culinary arts trainings. Assuming success of the provincial pilots, they will be used to encourage other local governments to adopt the approach. The following section presents the results framework that EEA Philippines built upon in order to achieve three immediate results: 1. Alliance building and leveraging of resources 2. Improved access to and relevance of education and employability training 3. Improved prospects for employment and successful entrepreneurship The fourth intermediate result was added as part of the summative evaluation to determine whether EEA had any indirect effect on youth’s families and communities, and if so, to what extent. Section II: EEA-Philippines Results Framework Philippine Education and Employment Alliance Goal: Improving the quality and relevance of education to better prepare young people (ages 15-25) for jobs and self-employment Intermediate Result 1: Alliance building and leveraging resources Intermediate Result 2: Improved Access to and Relevance of Education and Employability Training Intermediate Result 4: Positive Indirect Effect on Families and Communities Intermediate Result 3: Improved Prospects for Employment and Successful Entrepreneurship Indicator 1.1: In kind and cash resources leveraged for the project through private sector, government, civil society and other partners Indicator 1.2: Number of private sector, government and civil society partnerships formed through the project to jointly offer employment and business development training activities in targeted areas Indicator 1.3: The effectiveness of the EEA Alliance Approach in building partnerships, and promoting sustainable and scalable youth employability programs Indicator 1.4: New partnerships, mechanisms and approaches created as a rippling effect of working with EEA programs to support youth employability Indicator 1.5: Number of non￾target institutions that request to adopt EEA promoted programs, models and tools as part of their activities Indicator 2.1: Number of young men and women participating in EEA programs Indicator 2.2: Number and percentage of young men and women completing EEA supported programs Indicator 2.3: Number of teachers, trainers, facilitators and counselors trained under EEA supported programs Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in EEA supported programs Indicator 2.5: Level of satisfaction of young trainees with the quality and relevance of the training, counseling and services received through EEA programs Indicator 2.6: Level of employers’ satisfaction with the soft skills and/or technical skills of EEA trained youth Indicator 2.7: Number and percentage of graduates who report that the y will refer EEA to other youth Indicator 3.1: Number and percentage of trainees who get a job within six months of completing the program Indicator 3.2: Number and percentage of trainees who continue education, obtain an internship or engage in further professional training within six months of completing the program Indicator 3.3: Number and percentage of EEA trainees starting their own businesses within six months of completing the training Indicator 3.4: Number and percentage of youth with satisfactory internships or apprenticeships Indicator 3.5: Number and percentage of placed youth who declare having a satisfying and quality job Indicator 4.1: Number and percentage of employed youth financially helping their families with household, health and education expenses Indicator 4.2: Number and percentage of youth who have reported improved financial and social conditions of their families as a result of participation in the EEA program Section III: Findings and Analytical Review of Program Achievements Intermediate Result I: Alliance Building and Leveraging Resources Summary of Intermediate Result One Findings In summary, the evaluation has found under Intermediate Result I that the use of the alliance approach has been an important determinant in the effective and efficient implementation of EEA Philippines employability projects and achieving optimal project results. A significant benefit of the alliance approach has been in the mobilization of resources, both in-kind and cash, which enabled EEA to deepen program effect. The Philippine program mobilized $2.50 for every $1 in seed grants. However, it is also important to note that the short project time span affected the ability to build up strong commitment among partners whose mandates were not devoted to helping disadvantaged youth. Nonetheless, EEA Philippines has shown that once models are efficiently operating on the ground, it becomes easier to advocate for more support from partners across sectors, paricularly as demonstrated by the National Government’s Department of Social Welfare Development adopting the EEA program, which is now known as the Youth Productivity Services (YPS) program. Introduction On 2 February 2006, the Consuelo Foundation, one of the leading non-government organizations in the Philippines and a key partner of the International Youth Foundation, convened twelve private and public institutions to be part of the National Steering Committee for EEA. The national committee was composed of: • Two representatives from the Government: the Department of Education – Bureau of Alternative Learning Systems, and Technical Education and Skills Development Authority or TESDA • Four representatives from the private sector: Caltex Philippines, Pilipinas Shell Foundation, Globe Telecom and the Makati Business Club • Four representatives from local NGOs: Consuelo Foundation, Don Bosco North Province, Philippine Business for Social Progress and Notre Dame Foundation for Charitable Activities from Mindanao • Two representatives from international groups: IYF and SEAMEO-INNOTECH All of these agencies had been involved in training youth but not have significant experience in working with out-of-school youth. Through a series of meetings, the Alliance established the roles of members and envisioned for the National Steering Committee to have three major roles: programmatic approval, resource mobilization and operational support by participating in quarterly meetings and in advocacy and networking events. The organizational structure was also established with a chairperson and two sub￾committees: 1) Program Review Committee to review proposals and provide technical inputs before they are fully approved by the Steering Committee; and 2) Resource Mobilization Committee to review and strengthen the EEA leverage strategy. The Committee reviewed and agreed that a request for proposals should be circulated to potential EEA implementing partners. They also affirmed the focus on helping out￾of-school youth, ages 15-24, in the island of Mindanao, and approved the project selection criteria. The National Committee also defined “innovativeness,” a key feature of EEA projects, as projects that are new to a target site, with new components and taking an approach to address a specific need. After its establishment in February 2006, the national alliance convened a total of eight meetings. It took three program meetings and four Steering Committee meetings before projects were fully developed and approved for implementation. Its last meeting was in March 2008, after which the projects and Secretariat fund support drew to an end and no further meetings were convened. The national alliance members assessed that the alliance approach was able to help reach the target beneficiaries and helped develop project management capacities of local implementing partners. National alliance members were experienced in working with public or private institutions and their experience with EEA affirmed their own experiences that the synergy of partnership and pooling of resources substantially enhances program delivery and potential. Similarly, the six EEA local implementing partners said that the EEA alliance approach had allowed them to amplify the potential of the project in terms of numbers, visibility, and potential for sustainability. Four of the six partners said that the use of the alliance approach proved “much better” or “better” when compared against traditional development methods. Specifically, the partners said it increased the responsiveness of partner agencies or corporations to share resources, participate in meetings, and find common solutions to problems. The Alliance approach was felt to have brought about better project results, if not “almost assured it.” But partners commented that the approach worked better, if all partners shared strong commitment or belief in the program objectives rather than token participation. By program end, the implementing partners had changed their attitudes towards working with at-risk youth, on youth employability and towards using the alliance approach. Their responses are varied but show common themes. Most of the implementing partners have come away from the project with a new attitude towards working with OSY and to working on development using the alliance approach. When the program started, not all of the EEA NGO partners had youth as their prime focus. After their involvement, they have made institutional decisions to sustain their work with youth. They realized that the youth sector, particularly OSY, were critical to community development in Mindanao, to help build up a skilled labor force that could perform positively at work. By project end, the partner NGOs had changed their perception of at-risk youth, understanding how they could become productive and self-directing instead of vulnerable to lawlessness and recruitment into terrorist bands. Indicator 1.1: In kind and cash resources leveraged for the EEA program through private sector, government, civil society and other partners As of June 2009, EEA partners generated $1,802,681 in cash and in-kind support to leverage the USAID grant of $617,968 for seed funds, resulting in a leverage ratio of $2.50 for every $1 dollar invested in out￾of-school youth. This total leverage amount includes contributions from the Secretariat which similarly generated cash and in-kind leverage. The cash leveraged, totaled $778,723 or 43% of the total, while the total in-kind resource mobilized totaled $1,023,958, representing 57% of the total leverage. More than 100 community partners participated in different projects. As the projects were implemented and the unplanned expenses cropped up, one significant benefit of partnership was the willingness of partners to help mobilize additional resources. They also appeared increasingly willing to expand project operations and scope to help more youth as success of activities became more apparent. The in-kind donations provided were varied and included classrooms for the lectures, space for training and provision of food during the graduations, building a training center, access to jobs by marketing the youth to members of their associations, use of partner vehicles, bunk houses for the participants, lecturers on specific topics such as Caltex lecturers on lubricants, or facilitating the clearance of trainees so they could find jobs. Personnel time was the resource most frequently provided by alliance members, followed by sharing their social capital through linkages, networks and funds – see Figure 1 below. Alliance members participated in community alliance meetings, attended networking events and worked directly with the youth from their communities. See Annex A for the typology of partners and leverage contributions. Figure 1: Classification of Resources Provided by EEA Alliance Members Source: EEA Philippines Evaluation, November to December 2008, Mindanao Indicator 1.2: Number of private sector, government and civil society partnerships formed through the EEA program to jointly offer employment and business development training activities in targeted areas A total of 110 partnerships were formed as part of the process of implementing the project. Partnerships were formed at the community level and among businesses, government, NGOs and included parents of youth. As at risk and out-of-school youth, their needs went beyond the technical training courses. The partnerships primarily involved the sharing of resources, both in-kind or cash. These resources included sharing personnel time to coach youth that needed to take equivalency tests for elementary or high Resources Provided by Alliance Members Response Frequency Response Count Funds 31.6% 6 Personnel Time 84.2% 16 Trainers 31.6% 6 Goods & Materials 36.8% 7 Curriculum Design 26.3% 5 Job Opportunities 26.3% 5 Linkages & Networks 52.6% 10 school, or teaching parents how to raise the additional funds for non-tuition expenses and donating tool kits to start youth onto a career as auto mechanics. The bulk of resources shared were the corporate time to mentor, apprentice and train youth on the job. Partnerships were linked to the community-based alliances which were a unique feature of EEA Philippines. Indicator 1.3: The effectiveness of the EEA alliance approach in building partnerships, and promoting sustainable and scalable youth employability programs9 The Philippine program was found to be effective in implementing the alliance approach and benefiting from the increased resources they were able to mobilize and collective spirit that helped in project implementation. Specifically, the partners through the evaluation survey and key informant interviews cited reasons why the alliance approach helped the EEA in terms of building partnerships, gaining scale and initiating sustainable operations. Building Partnerships: By using the alliance approach in implementing the project, NGO partners learned to work with sectors outside their traditional network, and in the process, realized they could scale up their program outreach and potential. In their communities, they gained experience, expertise and recognition for their success in working with the youth. They have been contacted to continue what was started under EEA, although primarily by local governments, whose mandate is to provide employment opportunities for unemployed has become more critical in light of the economic downturn. For example, Marcellin is working with a new Barangay and training their youth in welding. The MIAP has begun to work directly with the provincial government to train OSY using the EEA approach. The University partner of the Davao Chamber said that their community outreach methodologies and strategies have been enriched: “We have used the EEA approach and curriculum in implementing a skills training for OSY in Sta Cruz, Davao del Sur with the LGU of Sta Cruz. We have become advocates following the results of the Mindanao Employability study and deepened with our actual experience in the EEA project. It opened partnerships with industry partners, paving the way to employability, which is one area we barely explored in the past.” Promoting Scalability: Using the alliance approach was cited by partners as a key reason for project success. The partners said that the approach fostered creativity, sharpened management focus and lighter work. However, they also said that it was difficult to convene meetings and engage in sufficient consultation with their alliance members. In this respect, partners cited a lack of dedicated staff for alliance work as one reason for this as well as insufficient outreach and communication to members. All the partners acknowledged the importance of a mind shift from managing a project alone to managing and achieving results with the help of other stakeholders or alliance members. Implementing partners commented: “Self dependence in implementing development projects is self defeating. You can’t depend only on yourself in any project. A big factor in the realization of the outputs and outcome results of the project has been the partnerships, alliance. It has enhanced and broadened our understanding, framework, and approach to OSY development.” “Resources are scarce and not one organization or entity can bring about local economic development. Needs and challenges are huge that no one organization can solve these alone. Rural enterprise and industry development is basically a multi-stakeholder approach.” “Things can be done better in conjunction with other stakeholder partners. Projects implemented by stand alone proponents are most likely to be less successful than those that are conducted 9 EEA Philippines defined community based alliances as a group of institutions who were committed to working together and sharing resources to find solutions to community challenges. Community-based alliances (CBAs) were accessible and invited international, national and local decision makers to participate in the development process. They were seen to exist in different forms with different development objectives. Some are formal and based on memorandums of agreement, while others are informal and largely based on verbal commitments. A common characteristic of CBAs was their ability to integrate programs into the community by responding to local needs, generating local buy-in and sustaining programs. EEA Philippines also designed CBAs to be able to (1) Respond to a demand in or around the community for training, employment or economic development.; (2) Bring together capable, respected and committed leaders from all sectors who work together to achieve a common goal; (3) Raise awareness about programs and mobilize resources to sustain activities; (4) Actively involve local government and community leaders to foster local ownership. using alliances as strategies. Also, peace building in Mindanao is premised on eradicating poverty, marginalization and injustice, issue that can only be resolved by a united populace and a partnership, even social contract between private sector, government and civil society.” “Working with an Alliance allows us to be creative and innovative through resource-sharing and manpower pooling using competitive advantages. Tasking and complementation can result in better, more satisfying results. Almost all aspects in managing the project are distributed and this makes the project work, lighter.” Through these surveys, it appears that the alliance approach has changed the way the NGO grantees implemented projects and mobilized resources. Instead of relying on only one institution, the partners found they could count on an alliance of institutions and individuals. Partners said that they have found that time required was cut down in accomplishing activities, in finding resources or identifying advisors to solve problems during project implementation. Among other items cited by survey participants, the alliance was useful in helping youth who were problematic, finding opportunities for jobs, checking that a curriculum was relevant to industry needs and mentoring youth during their first month on the job. By program conclusion, the implementing partners said that they approached EEA problems with the knowledge that solutions could be found through collection thinking and action. One partner tagged a motto for their project - "tulong tulong vs kami kami” (working together vs. going it alone). Sustainability: After USAID funding was completed in March 2009, five projects were still in operation and expected to continue their EEA projects. Their sustainability was supported by having completed a successful project, gained a positive reputation in implementing youth projects, built relations with members of their community-based alliances, and expanded networks. But the sustainability of the project was mainly due to the NGO leaders, who had became convinced in the course of the EEA project, in the relevance of youth employability and alliance approaches. They submitted proposals to donors to expand, replicate or restart their EEA projects; changed their institutional missions; and continued to promote their new capacity to help others implement EEA projects. For example, MIAP plans to include a training center for OSY in the Metal Village Complex they were planning. Alterplan also made changes in their NGO mission to open them up to continuing training of youth in construction skills. Marcellin is being seen by the General Santos City as their resource for youth employability for at-risk youth. All implementing partners said that the one to two year project time period was short and did not provide the time needed to build a strong and committed alliance. They said their initial efforts had been focused on making the projects work. With a working and successful model, they said it was easier to multiply efforts. Marcellin Foundation found this to be true as barangays approached them to implement similar EEA projects. NGOs said that one problem in expanding their programs was the focus on at-risk youth, which appeals more to social welfare institutions. While such sustainability is an important aspect of the program, long term sustainability is promoted thorough government buy-in and expansion, as with the nascent effort through the YPS as a pilot project. Figure 2 summarizes the sustainability status of EEA subgrant projects: Figure 2: Sustainability Status by Project Project Program Summary Partners Sustainability Status Civil Trades Training for Ex￾Combatants To equip 900 out-of-school youth of ex-combatants in skilled construction trades such as carpentry, plumbing and masonry and life skills, 754 of these trainees to be placed in jobs ALTERPLAN, Habitat for Humanity￾Philippines, UNDP Act for Peace, local employers Project continues – because of the need for construction workers and a core project of Alterplan MIAP Plus To train 240 out-of-school youth with academic, life and technical skills required for employment in Metalworks and welding jobs in Metalworks Industry Association member companies, and to place 194 of them in jobs Metalworks Industry Association of Philippines MIAP, Cagayan de Oro Chamber of Commerce, City Government of Cagayan de Oro Project being expanded and launched as Youth Productivity Services (YPS) project reaching an estimated 450 youth in nine municipalities of Misamis Oriental and also a core activity of MIAP Project Program Summary Partners Sustainability Status Integrated Small Engine Repair in ARMM: KAPATID Program To provide 130 rehabilitated juvenile offenders with basic welding, small engine and automotive repair training at 3 training “service” centers, with 94 trainees expected to secure jobs after training Marcellin Foundation, Chevron, TESDA, ARMM Chamber of Commerce Project being expanded to communities and operating in General Santos City with Chevron funds Integrated Agri-Fisheries Entrepreneur￾ship Training for Rural Youth To provide 150 marginalized out￾of-school youth, the majority of them women, with technical and entrepreneurship training in agri-fisheries production and, in turn, to create 50 social enterprises and 300 new jobs Kasanyangan Foundation, Local Government Units, TESDA, Zamboanga Colleges Project stopped due to lack of funds for the next phase Enhancing the Capacity of OSY in Agriculture Production - Mag-Agri To equip 180 out-of-school youth with demand-driven skills in food production and processing, fruit production, and organic vegetable production, with 113 to gain employment Davao City Chamber of Commerce and Industry, Mango Industry, University of Southeastern Philippines Project continues as Davao Chamber CSR program and in the University of Southern Philippines community outreach program Empowering Youth to Modernize the Seaweed Industry in Tawi-Tawi To support integrated technical training education projects for 1,040 in- and out-of-school youth in seaweed production, with 839 to find jobs Philippines Business for Social Progress, the Philippines Development Assistance Program, Mindanao State University Project continues with new donor funds and loan program for seaweed farmers from Development Bank of the Philippines Source: EEA quarterly programmatic reports. Cost Effectiveness: Under EEA’s global framework, which included a series of both country and global level activities, the overall cost effectiveness of country program’s has been assessed at two levels. The first level of such costs are directly related to country program subgrants, which include both seed grants administered by USAID and leveraged resources contributed by other partners to support EEA country programs. The second level of costs includes Alliance building and program support costs, which include costs related to the functioning of the EEA National Secretariat Consuelo Foundation, local consultants or/country advisors in the Philippines, as well as IYF direct and indirect expenditures attributable to the Philippines program. Each of these cost categories are discussed in more detail below: Subgrant Project-level Costs: As noted above, subgrant project-level costs include seed funds and leverage contributions invested toward individual projects which typically covered curriculum design or improvement, training, job placement, enterprise development costs, and project specific costs of NGO subgrantees. Projects typically lasted between two and eight months, inclusive of internships and on-the￾job training efforts. The amount of seed funding support from USAID toward individual projects is a useful metric to assess cost effectiveness given that such funds are directly focused on implementation of projects, are directly leveraged by investments from other parties, and are best positioned for increased efficiency when additional beneficiaries are added to successful pilot programs. In the Philippines, the USAID support of approximately $617,968 in seed funds that were coupled with leverage investments of $1,802,681 reached a total of 3,036 beneficiaries. If all project-level costs are taken in account, this brings the cost to $797 per beneficiary, of which $203 was borne directly by USAID and $594 was shouldered by the alliance partners. Figure 3: Project-level Costs Subgrant Project Name Number of Beneficiaries USAID Funds Leverage Total Project Budget Cost Per Beneficiary ALTERPLAN 735 $ 220,426 $ 880,032 $ 1,100,458 $ 3,032 MIAP PLUS 345 $ 82,426 $ 183,351 $ 265,777 $ 770 Marcellin 375 $ 60,135 $ 202,328 $ 262,463 $ 700 Agri-fisheries 359 $ 70,094 $ 120,071 $ 190,165 $ 530 Mag-Agri 180 $ 109,893 $ 175,638 $ 285,531 $ 1,586 Tawi-Tawi Seaweed Project 1,042 $ 74,994 $ 241,261 $ 316,255 $ 304 TOTAL 3,036 $ 617,968 $ 1,802,681 $ 2,420,649 $ 797 Source: EEA Accounting Records, IYF, 2009. With respect to individual project costs, it was found that the cost of some technical training interventions were also relatively high due to the expense of consumable materials (welding rods in the case of welding and construction supplies for masons) and a focus on remote areas which increased travel, logistics and other costs. In particular, training costs often included allowances for youth to travel to the training center which were often far from where the youth live. Alliance Building and Program Support Costs: A more expansive examination of costs by necessity also includes those alliance building and program support costs related to the EEA Philippines program. These costs supported significant, collective efforts by IYF, Consuelo Foundation as National Secretariat, and program consultants to: manage the design and implementation of alliance based interventions; provide international technical expertise on successful employability approaches; assist in leveraging of resources; provide capacity building support to implementing partners in skills training, job placement and post training support to entrepreneurs; provide financial oversight, compliance, and direct monitoring and evaluation support. Upon examination of IYF accounting records, these costs taken together with indirect costs attributable to the Philippines program, totaled $640,657. This USAID-borne cost adds approximately $211 on a per beneficiary basis as presented in the figure below. Figure 4: Alliance Building and Program Support Costs Subgrant Project Name Number of Beneficiaries Alliance Building and Program Support Costs Cost per Beneficiary ALTERPLAN 735 $ 106,776 $ 145 MIAP PLUS 345 $ 106,776 $ 309 Marcellin 375 $ 106,776 $ 285 Agri-fisheries 359 $ 106,776 $ 297 Mag-Agri 180 $ 106,776 $ 593 Tawi-Tawi Seaweed 1,042 $ 106,776 $ 102 TOTAL 3,036 $ 640,657 $ 211 Source: EEA Accounting Records, IYF, 2009. Taken together, these overall costs of both seed funds and alliance building program support costs total approximately $1,008 per beneficiary, and appear roughly equivalent compared to other good practice country projects of this duration and that include internships, on the job training and international technical assistance. Of the cost of $1,008 per beneficiary, $414 was borne by USAID and $594 was contributed by alliance partners as calculated below. Figure 5: Cost per Beneficiary Cost per beneficiary USAID Alliance Contributions Total Cost Per Beneficiary Project-level costs $ 203 $ 594 $ 797 Alliance building and program support costs $ 211 $ - $ 211 Total $ 414 $ 594 $ 1,008 Source: EEA Accounting Records, IYF, 2009. Finally, and perhaps most significantly, a full assessment of cost efficiency would also require a more thorough assessment of the effect of alliance building activities on long term sustainability and scalability of proven practice programs. For example, should the new efforts to substantially expand the Youth Productivity Services Project in Mindanao be successful, the cost effectiveness of initial EEA investments could be dramatically expanded. While the longitudinal study required for this type of evaluation goes beyond the scope and resources available for this study, such factors should be addressed more directly in similar employability projects in the future. Indicator 1.4: New partnerships, mechanisms and approaches created as a rippling effect of working with EEA to support youth employability The primary mechanism developed as a result of EEA was the Youth Productivity Services program by the national government. By the second year, the first year project results became available and local governments took notice and invited implementing partners to extend their projects to their barangays. The EEA Secretariat took this opportunity and presented the program to the national government’s Department of Social Welfare and Development (DSWD) and suggested that the EEA program framework became adopted as a national program and pilot tested in the province of Misamis Oriental. The results of the pilot have been positive as local government budgets have been increased for youth employability training, training facilities have been established, tools developed for a systematic identification of OSY, local labour demand and opportunities for employment have been done and government social workers are now trained to train in employability and life skills. A total of nine of the 22 municipalities in the province of Misamis Oriental have voluntarily signed up for the program, committing their own funds as counterpart to provincial and national counterpart funds. A total of Php 4 million pesos or US$ 89,000 ($1: Php 45) has been committed to the pilot. The DSWD is not the national agency tasked with employment, but they are the national agency with a mandated focus on youth, especially OSY. In adapting the EEA program, DSWD placed a greater emphasis on the “life skills” and community nurturing, which was more their mandate than placing an emphasis on jobs and technical skills. If the program pilot is successful, the Youth Productivity Services will be integrated in the Comprehensive Program for Out-of-School Youth (enhanced Unlad Kabataan Program) as one of its major components. This will then be advocated for institutionalization in local government units nationwide. The same initiative was done by other local government units. Barangays in General Santos City and Davao City, where youth unemployment was a community concern, asked implementing partners to implement EEA type projects in their areas. Indicator 1.5: Number of non-target institutions that request to adopt EEA promoted projects, models and tools as part of their activities A finding of note in the evaluation was that all implementing NGO partners decided to adopt and make youth productivity a part of their programming priorities. Participating schools similarly adopted and began offering the curriculum that had been developed by EEA approach in their community outreach programs. For example, Systems Technology Institute (STI) College,10 the partner for the youth entrepreneurship project in Zamboanga City, continued to offer the entrepreneurship curriculum to other youth enrolled in their school. The social welfare officers of Misamis Oriental received training in Skills for Life and Employability, and can now provide life skills training to their constituents. The key beneficiaries of the technology of the EEA approach were the implementing partners themselves. The partners were provided with training workshops on alliance building and Skills for Life and Employability as well as field visits to different projects to learn about e-learning and trade tests. 10 STI College has branches throughout the Philippine and offers information-technology based short term courses. Intermediate Result II: Improved Access to and Relevance of Education and Employability Training Summary of Intermediate Result Two Findings A majority of EEA graduates have benefited from the integrated training and job program through new skills, changed attitudes and a vision for the future. Of the 3,036 youth who joined the program, 2,669 youth or 88% of participants completed the courses and on-the–job training with industry or on their micro enterprises. Drop outs were few and primarily motivated by the need to work immediately, following pressure from their parents. When possible, all EEA projects had mentored and financially supported their graduates to attain government trade certificates to increase their chances of job placement. EEA graduates from three out of four projects attained high marks, from 100% to 98% on the trade tests. Ninety-two percent of these youth attributed their performance to the good to excellent training they received. Sixty to seventy percent of the youth also assessed that their life skills had improved in communicating, resolving conflict, listening, taking the initiative or being responsible for work and in their vision for the future. Seventy of the youth who participated in the evaluation said they would refer EEA course to their peers, being very satisfied with its effect on their lives. Introduction EEA Philippines focused on serving out-of-school youth in Mindanao. In 2003, it was estimated that OSYs were 67% of the total youth population in the island, aged 16 to 24 years (FLEMMS 2003). Estimated at 2.2 million, these were youth who could not, or chose not, to complete school or had completed some level of school and were engaged in other activities. They often cited reasons of family poverty and armed conflict as reasons why they did not seek higher education. Unemployment of youth was related to the mismatch between their skills and what the industry was demanding. This was found by the 2005 EEA Mindanao Youth Study which suggested areas where there were opportunities for jobs in Mindanao. Figure 6: Out-of-School Youth Statistics in Mindanao Total Mindanao Youth Total OSY Percentage of OSY Aged 12 to 15 2,014,000 310,678 15% Aged 16 to 24 3,323,000 2,219,578 67% Source: FLEMMS, National Statistics Office, 2003. Indicator 2.1: Number of young men and women participating in EEA programs The EEA program was mandated to help at-risk and out-of-school youth. At risk youth included youth in conflict with the law, youth ex-combatants and urban poor youth. Targeting this sector, NGO partners narrowed down their recruitment process by going to communities that were poor, remote or in need of help. For example, the Alterplan-Habitat alliance primarily worked with Peace and Development communities, camps of ex-combatants of the Mindanao National Liberation Front (MNLF). Partners began by explaining the program to the local government officials and in the process, looped in the barangay into the alliance approach, tasking them to help in recruitment, allocate space for training, take care of food and transportation and monitor youth who were absent or discouraged to continue their courses. Local Government Units (LGUs) were receptive to the proposal to help their disadvantaged youth. While political patronage became an issue in some of the recruitment, these were managed by good selection techniques and criteria. Figure 7: Number of Youth Participating in EEA projects Projects 2006 2007 2008 2009 Total Target Achieved Target Achieved Target Achieved Target Achieved Target Achieved Total 540 400 1,050 1,483 1,218 1,058 150 95 2,958 3,036 MIAP Plus Technical Skills Training 100 75 50 75 120 100 150 95 420 345 Civil Trades Training for ex￾combatants 330 215 150 115 450 405 - - 930 735 DAVAO: Rural Agri Skills Training 60 60 120 109 - 11 - - 180 180 KFI Youth Entrepreneurship - - 150 359 - - - - 150 359 Marcellin KAPATID Project 50 50 80 83 108 242 - - 238 375 PDAP/PBSP Seaweeds Training - - 500 742 540 300 - - 1,040 1,042 Source: Benefits Tracking, EEA Philippines Secretariat, Consuelo Foundation, March 2009. Selection in the entrepreneurship project supported by EEA resulted in significant attrition, however, as this project implemented a self-awareness training that filtered in only those youth who believed they could be entrepreneurs and self-employed. From a cohort of 325 youth who attended a “Discovery Weekend,” only 206 remained to take the formal business courses and only 163 graduated with business plans for small businesses. This high attrition rate in the entrepreneurship trainings were different from the technical skills training courses where 88% of youth who began the trainings, graduated. In the end, in this project, the implementing partner was unable to mobilize significant support to mentor the graduates or activate a strong alliance with a microfinance organization to provide youth with micro loans. Indicator 2.2: Number and percentage of young men and women completing EEA supported programs Of the 3,036 youth who began different training programs, 2,669 youth or 88% completed the different courses, which ranged from two to eight months, inclusive of internships and on-the-job training. Twelve percent of participants dropped out because of employment opportunities that became available, and when requested by their parents to leave. A few youth left the program to enroll in formal schooling. Figure 8: Number and Percentage of Young Men and Women Completing EEA Programs Projects 2006 2007 2008 2009 Total Target Actual Target Actual Target Actual Target Actual Target Actual Total 330 215 413 489 1,824 1,900 135 65 2,702 2,669 MIAP Plus Technical Skills Training - - 68 75 175 131 135 65 378 271 Civil Trades Training for ex-combatants 330 215 - 105 540 404 - - 870 724 DAVAO: Rural Agri Skills Training - - 162 117 - 46 - - 162 163 KFI Youth Entrepreneurship - - 135 153 - - - - 135 153 Marcellin KAPATID Project - - 48 39 69 298 - - 117 397 PDAP/PBSP Seaweeds Training - - - - 1,040 1,021 - - 1,040 1,021 Source: Benefits Tracking, EEA Philippines Secretariat, Consuelo Foundation, March 2009. Indicator 2.3: Number of teachers, trainers, facilitators and counselors trained under EEA supported programs Most EEA projects trained their own teachers since courses had been adopted, revised or changed to meet the needs of industry and the special needs of their students, most of whom had limited literacy or formal education. Only Davao City Chamber which partnered with a State University had academically qualified trainers. For Partners who chose to bring the trainings to communities, it became necessary to work with trainers, who were often NGO staff or community workers. As the projects expanded, trainers were also found among graduates or in the case of the seaweed farming project, peer trainers became a practical strategy for remote areas. Figure 9: Number of Teachers, Trainers, Facilitators and Counselors Trained Projects 2006 2007 2008 Total Target Achieved Target Achieved Target Achieved Target Achieved Total 86 68 25 11 - 24 111 103 MIAP Plus Technical Skills Training 12 12 2 - - - 14 12 ALTERPLAN Civil Trades Training 21 21 20 7 - 24 41 52 DAVAO: Rural Agri Skills Training 18 12 - - - - 18 12 KFI Youth Entrepreneurship 18 7 - 1 - - 18 8 Marcellin KAPATID Project 17 16 - - - - 17 16 PDAP/PBSP Seaweeds Training - - 3 3 - - 3 3 Source: Benefits Tracking, EEA Philippines Secretariat, Consuelo Foundation, March 2009. The strategy of using community trainers, however, did not necessarily result in good trainers. Youth evaluated their trainers using a four-point rating scale of from very poor to excellent, and gave higher marks to their trainers for their role as counselors and for their ability to provide mentorship and career counseling, and lower marks for their performance as technical trainers. Figure 10: Youth Satisfaction with EEA Trainers and Counselors Please rate program trainers/counselors/facilitators in the following areas. 3.18 3.20 3.22 3.24 3.26 3.28 3.30 3.32 3.34 Overall performance Technical knowledge (if applicable) Counseling capacities (if applicable) Capacity to provide mentorship and career counseling Capacity to keep you motivated and engaged Source: EEA Philippines youth beneficiary survey, IYF, 2009. Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in EEA supported programs Four of the six EEA projects sought trade certification from TESDA for their graduates as an integral element of their training programs. A total of 1,208 graduates or 45% of all graduates who were eligible to take the tests, received substantial mentoring support and financial support from EEA to take and pass trade tests as welders, masons and construction workers, food processors and fruit technicians. Trade certification was viewed as an indicator that the newly acquired skills met the competencies that were needed for a job. With a trade certificate, the graduates could find employment more readily, including overseas work. Results from three of the projects were high, from 98% to 100% passed TESDA tests. Alterplan explained that their high pass marks was helped by developing a group of national trade certifiers to supplement government training in areas where there were only one or no certified trainers. On the other hand, Marcellin graduates barely passed their trade certificate exams and this was attributed to the limited literacy of their trainees, many of whom had barely finished elementary school. Figure 11: Percentage of Young Men and Women Demonstrating Improved Capabilities Projects 2006 2007 2008 Target Actual Target Actual Target Actual MIAP Plus 80% 100% 80% - 80% - ALTERPLAN Civil Trades Training 75% 98% 75% 100% 75% - DAVAO: Rural Agri Skills Training 33% - 33% 98% 33% 100% Marcellin KAPATID Project 70% - 70% 27% 70% 20% PDAP/PBSP Seaweeds Training 0% - 0% - 85% - Source: EEA Monitoring Chart, IYF, 2009. Indicator 2.5: Level of satisfaction of young trainees with the quality and relevance of the training, counseling and services received through EEA programs Most of the youth were mobilized to join the program by their local government (61%) while others (29%) heard of the program from a friend or family members. Survey respondents recounted how waiting lines for the second and third batches of EEA training programs became longer and longer as the first batch graduates reported to their LGUs and families that they had successfully found jobs. Overall, 92% of the youth said that the training was either “good” (48%) or “excellent” (44%) using a 5-point rating scale. Specifically, they rated the relevance of the content to the needs of the job as good (69%) to excellent (32%). While there were similar rankings for assessments on the interactive nature of the course, life skills and technical skills learned, more youth assessed the life skills acquired as excellent (33%) or good (62%). Figure 12: Youth Satisfaction with the EEA Training Source: EEA Youth Beneficiary Survey, Mindanao. November to December 2008 Through surveys and focus groups, the majority of EEA youth said that they had experienced personal change as EEA trainees. Specifically, ƒ In terms of communication skills, before the training, 70% of the youth rated their communication skills as “fair” to “very poor” while after the training, 64% said they had improved these skills. 32% said there was some improvement or no change. ƒ At baseline, the 57% of the youth rated their listening skills as poor to fair while 43% rated their prior listening skills as “good” to “excellent.” After the training, 33% reported some improvement while 66% said there has been a “good” to “huge” improvement in these skills. ƒ While 22% of the youth said their self confidence was “very weak” to “weak” before the training, after the training, 71% said there has been a “good” to “huge” improvement in their self confidence – see Figure 13 below. ƒ Sixty-three percent of youth reported their sense of initiative to be “very poor” to “fair” among and after the training, seventy percent saw a good to huge improvement in their sense of initiative. ƒ While 70% of the youth assessed their sense of responsibility as “very poor” to “fair” before the training, after the training, 76% said they has been a good or huge improvement. ƒ Resolving conflicts was assessed as “fair” to “poor” among 70% of the youth, and after the training, 59% reported to have experienced “good” to “huge” improvements. ƒ The change is also noticed in their relationship with family and friends where 44 % assessed that there has been a “huge” improvement and 42% reported “some” improvement. Please rank your satisfaction with the program: 3.00 3.05 3.10 3.15 3.20 3.25 3.30 3.35 3.40 Overall content of the training program Quality of training materials distributed Relevance of content and training materials to employment needs Interactive/practical exercises and activities (if any) Life Skills and attitudes learned from training (if any) Technical skills learned from training (if any) Figure 13: Life Skills Acquisition Graduates were asked to rate the relevance of their training to employment opportunities, and 96% of them rated the training either “good” or “excellent.” Before the training, 5% of the youth said they had no vision of their future, and 35% knew vaguely what they wanted to do, while 41% knew what they wanted but did not think they could attain it. After the EEA program, eighty percent said their vision of their future had changed while 17% felt there was “somewhat” of a change and three percent experienced no change. After the training, 52% of the youth envisioned continuing to study and work while 23% wanted to study exclusively – see Figure 14. Figure 14: Youth’s Vision for the Future In Maguindanao, where Alterplan targeted ex-combatants from camps of the Moro National Liberation Army (MNLF), parents and relatives of the youth graduates said all of their sons or brothers had changed. “There has been a big change. Youth stopped hanging out with their gangs.” One father said his son seemed to have “become kinder because you could not speak to him before. He was hard-headed.” Other relatives commented: ƒ “In the past, he just slept and ate. Now he is responsible.” ƒ “He became nicer and now calls me often and asks about his siblings. He now helps out with family expenses.” ƒ “His rough behavior smoothened out after the training.” ƒ “He now listens to my advice and he encourages his brothers and sister to enroll in trainings if there are more to be offered.” ƒ “Now, when you ask for something, he is generous and helpful.” Indicator 2.6: Level of employers’ satisfaction with the soft skills and/or technical skills of EEA graduates Employers were asked to rate the quality of and satisfaction with the soft and technical skills of EEA graduates, and also provide suggestions for improvement. Their attitudes were also assessed during an evaluation workshop hosted by the Davao City Chamber. Profile of Participating Employers: There were three kinds of employment that was accessed by the youth. The first was formal employment in companies and small enterprises such as steel and machine shops, auto repair, ship building and construction companies. The second was self-employment as youth entrepreneurs. The third was employment in family farms as agri youth entrepreneurs. The employers in two projects were partners of the implementing NGO from the project’s onset. These were Habitat for Humanity which was co-partner of Alterplan and Sitangkai Seaweeds, which was based in Tawi-Tawi, and a buyer of seaweeds and co-partner with PDAP. Both enterprises had a significant interest in the outcome of the youth training programs since the quality of training outcomes had a direct consequence on their core business, whether by way of the quality of homes constructed or the quality of seaweed grown and harvested. Two employer groups were members of EEA partners that were industry or business associations. These were employer-members of the Metal Industry Association of the Philippines/Cagayan de Oro chapter and the Davao City Chamber of Commerce. The employers took in the youth as on the job trainees and hired those who excelled during their on-the-job training. Assessing Performance of Masons, Welders and Small Engine Repair Workers: A sample of nine employers in Cagayan de Oro and General Santos, who hired youth with welding or small engine repair or masonry skills were asked to assess the EEA youth they hired. In summary, all of the employers said they would continue to offer on-the-job training to EEA graduates of welding and small engine repair. All similarly felt that EEA is closing the gap in providing skilled workers since many were previously experiencing fast turnover of staff who left for overseas work. They assessed the youth as “good” to “excellent.” Similarly, these employers assessed the soft skills or attitudes of the youth as “good” to “excellent.” However, despite self assessments on life skills by young people, employers commented that the youth needed to improve their employability skills: coming to work on time, following supervisors’ instructions, maintaining their openness to taking on and learning on the job and more exposure to work sites. One employer observed that they had difficulty integrating the youth into the workforce of older workers who had no advantage of formal training and another saw a pronounced air of bravado or confidence (“yabang”) among the new workers who felt they already knew enough after the training and trade tests. Suggestions by employers to improve life skills training included: conducting regular follow up since skills cannot be learned overnight, more training by companies, training in advanced skills and further basic knowledge such as reading engineering plans. One suggestion is to set in place a “buddy-buddy system where an older worker can mentor a younger worker on the job. But compared with other employees, most of whom have completed formal education, they said that the EEA youth are the same, if not better than similar workers. Overall, did the EEA youth meet employers’ expectations? About sixty-seven percent of the employers said “yes” while a third felt that the youth needed more training. Of interest and perhaps useful for further examination, negative responses were often from larger companies where automation and more complex skills were needed for the work. Assessing Fruit Technicians and Food Processing Workers: The positive and negative attributes of new EEA trained workers were assessed in a focus group among employers as part of the terminal review of the Davao project. Employers commented that the EEA youth were seen as “moldable, easy to teach and would closely follow instructions” as opposed to “lacking leadership and initiative” or bringing to work the emotional problems of their family. The employers commented that the program strengthened the leadership, values and attitudes of the youth through its life skills training. They admitted that they were initially concerned over hiring OSY, but the EEA experience has influenced them to shift their views. Indicator 2.7: Number and percent of graduates who report that they will refer EEA to other youth In order to determine the level of satisfaction of youth with YEEI projects, the assessment also asked whether graduates would refer other youth to EEA Philippines projects and received responses from 95 youth. Eighty-four youth (88% of respondents) are willing to refer the training to other youth. Sixty-six youth surveyed said that they have in fact referred this program to others. This also reinforces the earlier conclusion that a majority of participants has had positive experiences with the program and are satisfied with the quality and relevance of training they received. The program, youth said, gave them a source of pride. Focus group discussions and interviews with the youth show that the program has allowed them to have stature in their communities opposed to when they were bystanders at corner stores. Intermediate Result III: Improved Prospects for Employment and Entrepreneurship Summary of Intermediate Result Three Findings The study shows that the program was successful in providing demand-driven skills training and placing youth in jobs. A total of 1,961 youth or 73% of graduates found jobs or successfully launched small businesses with support from EEA by program end. Specifically, 67% of graduates found jobs and 6% established small businesses. Nine graduates are continuing education. Jobs are typically entry level jobs in industry as welders, construction workers, auto mechanics, fruit technicians and food processing workers. Among entrepreneurs, many are agri-entrepreneurs, mostly as seaweed farmers in remote islands. The industry employers assessed that EEA graduates possessed skills on par with other workers that applied to them. However, youth said they were not as satisfied with the quality of internships they received, citing the need for stronger and more structured on-the-job training programs. Techniques used to secure jobs included active liaison with industry, finding the right job for the skills of the youth worker, fair wages, expanding networking for jobs, and using corporate social responsibility as an entry strategy for jobs. Introduction Finding jobs for the newly trained OSY was a main challenge for the NGO partners. The EEA program was different since the project did not end with the graduation of the youth from the training course, but with access to a job. Partner NGOs which were industry associations or chambers of commerce had easier access to their corporate members but had to persuade corporate members to accept trade certificates in lieu of high school diplomas. In the case of the entrepreneurship project, on-the-job training was replaced by an enterprise mentoring component where mentors visited and provided advice to youth entrepreneur students. Indicator 3.1: Number and percentage of trainees who got a job within six months of completing the EEA program Of the targeted 2,308 youth who graduated from the program, 1,794 or 77% of graduates found employment. However, when based on the total number of 2,669 youth who graduated, the ratio is lower at 67% but still higher than the acceptable job placement rate of TESDA of 60%. The highest placement rate of all graduates is of MIAP Plus graduates at 90%, thus nine out of ten graduates found jobs. Alterplan and the Davao Chamber projects had job placement rates within similar ranges of from 55% to 58% for youth trained in masonry and food processing. For seventy percent of youth surveyed, it took them between one to three months to find employment. It took four to six months to find jobs for another 18% and over six months for 12% of youth surveyed. Figure 15: Number of Graduates Placed in Jobs Number of trainees who get a job within six months of completing program 2006 2007 2008 2009 Total Target Actual Target Actual Target Actual Target Actual Target Actual Total 215 43 562 303 1,436 1,356 95 46 2,308 1,794 MIAP Plus Technical Skills Training - - 75 20 119 163 95 21 289 244 85ALTERPLAN Civil Trades Training 215 43 239 168 300 215 - - 754 426 DAVAO: Rural Agri Skills Training - - 75 44 38 50 - - 113 94 KFI Youth Entrepreneurship - - 135 58 - 27 - - 135 85 Marcellin KAPATID Project - - 38 13 140 187 - 25 178 231 PDAP/PBSP Seaweeds Training - - - - 839 714 - - 839 714 Source: Benefits Tracking, EEA Philippines Secretariat, Consuelo Foundation, June 2009. The lowest job placement rate was for the youth entrepreneurship training course. The partner, Ploughshares for the KFI Youth Entrepreneurship Project, commented that one of its weaknesses was not being able to link with microfinance institutions to open up access to credit. Those youth which established small businesses were able to do so from a small loan facility, which has been accessed for only one of the three project sites or with family loans. Figure 16: Job Placement Rates of EEA Projects MIAP Alterplan Davao Marcellin KFI Entrepreneurs PDAP Seaweed Farmers TOTAL Youth graduated 271 724 163 337 153 1,021 2,669 Employment target 289 754 113 178 135 839 2,308 Youth with jobs 244 426 94 231 85* 714* 1,794 Job placement based on original targets 84% 56% 83% 130% 53% 85% 86% Job placement rate of EEA graduates 90% 59% 58% 69% 56% 70% 67% *Self-employed or farm workers. What Worked in Finding Jobs for Youth Active Liaison with Industry: Active industry outreach was a success factor for high placement rates. This was made possible with the hiring of industry liaison officers. For example, Marcellin’s industry liaison officer fielded 15 calls to a construction company before it was able to place graduates in jobs to for the construction of the huge Robinson mall in General Santos City. Having good contracts within industry was less of an issue for the industry association partners which understood the needs of their business members more than NGO partners. Irrespective, the Davao Chamber project found that it still needed to actively market the young graduates, using strategies such as placement of ads in local papers, participation in trade fairs where the youth manned an EEA booth, personal calls and formal presentations to member companies. Finding the Right Job: Almost all the youth said they gained the technical knowledge and skills needed to get jobs (98%), along with life skills (95%). But Partners understood that it was important to match the youth to the right job. Most of the EEA trainees were out-of-school youth, from poor households with limited education and exposure to urban living. When placed in a large, formal company, some youth found it difficult to perform well. Marcellin observed this when it placed a youth in a large company with sophisticated automation for canning. Without the needed skills, the youth were assigned to clear metal scraps. When moved to a medium sized engine rebuilding firm, they however found they could use their skills, get promoted, recruit other classmates and gain better compensation. Fair Wages: Finding the right job with the correct and fair compensation was similarly an issue in placing EEA youth, once the most disadvantaged in communities. Alterplan reflects that: “Starting with zero￾knowledge training participants, the project team found it very challenging to market the graduates to employers outside the Habitat system. Even with TESDA certificates of competence in given trades, many of the training graduates would have been at a loss if forced to work outside Habitat immediately after completing the course and the trade tests. This was the assessment of the project team and Habitat managers who guided the graduates through their first months of employment. Additionally, at the entry level of workers in a typical private-sector construction project, the prevailing compensation modes are sub-contracting and piecework. Without the consideration given to them by Habitat, the new graduates’ inexperience and (very probable) low work output would have made them very poorly-compensated for quite a while in a typical workplace.”11 In evaluating their participation in EEA, the youth said that after the EEA training, their income levels have increased. Before the training, 50% of the youth said they earned less than $50 a month and 48% said they earned from $51 to $100 a month. After the training, only 27% said they were earning less than $50 a month while the majority at 70% said they were earning form $55 to $100 a month with three percent earning from $150 to $3000 a month – see Figure 17. 11 Sarah Redoblado, Executive Director, Alterplan. Figure 17: Comparison of Income Levels Before and After the EEA Training Networking for Youth Jobs: In finding jobs for the youth, the EEA partners relied on their network of contacts. Partners reached out to Chambers of Commerce, and tapped into industry needs. For example, the Marcellin Foundation has offered a solution to the problem of fast turn-over of trained personnel to overseas job markets. They have offered and entered into cooperative arrangements to supply skilled manpower to industry groups which were part of the network of the Marcellin Executive Director. Members of the network include the Kimball Group, Raffles Engineering, Vercide Engineering, and Marfenio Tan’s Group. Corporate Social Responsibility: Placement was not always the result of vigorous marketing - it was often because a strong appeal was made to the corporate social responsibility of an employer. But CSR only worked to open the door to a potential employee. Once the opportunity was there, the youth were assessed using the same criteria as other applicants. The CSR strategy was only effective during a first placement, a second and third batch of youth were looked at without the lens of helping communities and more on whether they could deliver on the job. Indicator 3.2: Number and percentage of trainees who continue education, obtain an internship or engage in further professional training within six months of completing the program All of the EEA youth underwent internships as part of the training course. It is important to note that their main concern was to find a job. Only a few youth said they wanted to continue to study. Based on EEA monitoring, only nine youth out of a target of 309, decided to continue and seek further training and education. EEA Partners said that many youth had to prioritize finding work since they came from poor families. But survey results showed that 68% of youth wanted to continue their education and take further professional training and only a third of the youth or 33% said they would not. Indicator 3.3: Number and percentage of EEA trainees starting their own businesses within six months of completing the training Only one of the six EEA projects was designed to focus on youth entrepreneurship, but by the program end, entrepreneurship had been introduced in all of EEA projects resulting in a total of 167 micro businesses. The three projects that produced entrepreneurs are the Ploughshares, Tawi-Tawi and Davao City Chamber projects. At the beginning, the Ploughshares Agri Fisheries Training for OSY project was designed and implemented to promote youth entrepreneurship in two areas: Zamboanga City and Zamboanga Sibugay, and in Davao del Norte. A total of 325 youth joined the initial training events which were two-day Discovery Weekends or youth self-awareness camp that served to orient the youth on entrepreneurship and the traits that were needed, and which filtered in those who felt they could become entrepreneurs. A total of six Discovery Weekends were held in various sites in partnership with local governments. From the 325, 200 youth continued on to take the formal business training courses. At the end, the project resulted in 85 youth entrepreneurs who set up and operated small businesses A total of 714 agri-entrepreneurs were encouraged by the training in the Tawi Tawi Seaweed Agri Entrepreneur Farmers project. The high number was justified as in Sitangkai, Tawi tawi, a remote island in Southern Philippines; all the graduates could only become agri-entrepreneurs. The implementing partner explained the project effectiveness by referring to the example of one graduate, Mr. Ummik Sabung. Mr. Sabung was able to successfully harvest a total of Php 100,000 from an investment of Php 1,000 capital in seaweed farming. Ummik, was called the silent one in the group and did not always standout, carefully used his added knowledge on seaweed production to buy two small motorized boats, a television set, and has now become the primary bread winner of his family. He said that before the project, he was hardly able to make any good harvests because diseases like “ice-ice” spoiled them. When the EEA-SPE3 project was offered, it was able to contribute to the improvement of his seaweed farming. With the new learnings from SPE3, he has been able to control, the occurrence of these diseases. The result is an additional 100 lines of produce seaweeds per cropping. As the projects progressed, some partners realized that entrepreneurship training was an important skill that the youth needed to have to sustain their gains. This was because available jobs were often seasonal, contractual or short term. For example in Davao City, where youth were trained as fruit technicians or food processors, jobs in the canneries were only available during the seasons of fruit harvests. Davao City Chamber decided to “add￾on” entrepreneurship by introducing a business planning workshop and creating links with microfinance institutions in Davao City. Participation in the workshop was voluntary and was offered to trainees who had completed their courses and expressed interest to pursue entrepreneurial activity after the program. A total of 76 OSY from Food Processing and Fruit Production training successfully finished the five-day basic entrepreneurial seminar/workshop at the MIC Cursillo Retreat House on separate dates. From these graduates, seven youth successfully accessed loans from the City Social Services and Development Office for some funding or seed capitalization assistance to the seven OSY from Food Processing. Originally, ten youth applied for seed capital assistance, but three backed out because of the repayment responsibility. Because of the project, the Davao City Chamber was encouraged to approach and enter into a Memorandum of Understanding with the Development Bank of the Philippines (DBP) for a Microfinance program for SMEs. The Project Team will arrange with the program head for possible inclusion of the OSY graduates in the microfinancing activity. DCCCII also partnered with the Department of Labor and Employment XI in implementing its Grow-Micro Program. This program was designed to provide different kinds of assistance to an informal sector to formalize micro businesses. Out-of-school youth were qualified to secure funding support from this program. Youth respondents in the evaluation survey said they are satisfied with the entrepreneurship training and counseling they received. Of the 53 who received the training, forty youth rated it “good” while twelve rated the training as “excellent.” Additionally, seventy-four percent of these youth started a small business with under $500 while 26% invested between $500 and $1,000. When asked about the performance of their business, 85% said they have been able to cover all business expenses with their profit and 65% said their businesses are making profits. Was the EEA project a reason for their being able to set up businesses? Sixty-five percent of the youth said “yes.” One youth explains why: “Because of our training, I have learned a lot, not only how to start a business but how to change myself to prepare for the future.” Indicator 3.4: Number and percentage of youth with satisfactory internships or apprenticeships The on-the-job training (OJT) experience was highly appreciated by the youth. While a third of the youth said there was nothing they did not like in the OJT program, there were a number of youth who were critical about the weak quality of the mentoring they received during OJTs or apprenticeships. This is because of their disappointment that they have not been hired after the OJTs and the lack of time during the training to practice the skills, such as welding. The youth suggested that the OJT/apprenticeship program be improved with more time for the training, better OJT trainers, being placed in larger companies versus internships in small businesses, and the addition of more courses. The youth also pointed to the need to be listened to by the trainers, to their needs and problems and not be ignored. They asked that the quality of the trainings be improved since there were times when the lecturer could not be understood by some of the trainees. They added that they needed more time to learn and to practice the new skills. Most of the youth respondents completed their internships (94%) with only six percent that said they did not have OJTs. Of these, approximately twenty percent of the respondents had issues with the OJTs, assessing these as “fair” to “poor” using a five-point rating scale of very poor - excellent. However, 76% of respondents assessed their OJT experience as either “good” to “excellent” – see Figure 18. During the OJT, they were able to deepen their learning of technical skills with only a few citing employability skills such as coming to work on time or determination to work. Most of the jobs have been offered as a result of youth undergoing on-the-job training. This is the case for the Alterplan-Habitat construction workers where 426 of the 724 trained youth have been hired in Mindanao and other construction sites. Figure 18: On-the-job Training Experience Indicator 3.5: Number and percentage of placed youth who declare having a satisfying and quality job Most youth said they found their jobs “fair” to “very satisfying” while around twenty percent had issues with their current jobs. Some of the issues were: some graduates are viewed suspiciously by older workers, who had no formal training; work was not continuous but seasonal; their work did not match their field of study; and some were being made to perform tasks such as cleaning and sweeping factory floors. Graduates who are ex-combatants, who had not completed high school, or even elementary school, had problems in learning more advanced technical skills. Even when given access to jobs, many of the youth found it difficult to take on jobs that were far from their homes, as transportation and housing were expensive. As one remedy, one partner, Alterplan had set up a special transport fund for graduates to travel to Manila-based work. Barangays also offered help in getting official documents to help with job search or getting government trade certificates. Figure 19: Job Satisfaction Intermediate Result IV: Positive Indirect Effect on Families and Communities Summary of Intermediate Result Four Findings The study found that a majority of youth were remitting their wages to their families. As a result, more than half of the youth felt, finding a job improved the lives of their families. The families, on the other hand, said that they had observed significant changes for the better in the youth after the training course. This had in turn influenced and motivated other siblings in their family. The evaluation study triangulated the results of the program by interviewing families and asking youth to assess how they related to their families, post EEA. The findings were positive. Indicator 4.1: Number and percentage of employed youth financially helping their families for household, health and education expenses Most of the earnings of the youth were used to help their parents and siblings (88%), spent for personal uses (63%) and saved for the future (62%). A majority of the respondents said youth helped with family expenses and sent money home. Youth estimated that some would send back as much as thirty percent of their salaries to their families. Would they return to their homes? The youth said they preferred not to, since being away from Mindanao with a productive job was an appealing alternative to family and community pressure to take up arms and join the struggle. Indicator 4.2: Number and percentage of youth who have reported improved financial and social conditions of their families The study sought to determine whether youth felt they had gained the ability to improve financial and social conditions of their families. • Sixty percent of graduates stated that that their financial support (with the income they shared) had helped improve their family’s financial and social situation. • Thirty-six percent claimed that their support had somewhat improved the situation. • Four percent said there had been no change. Section IV: Challenges, Lessons Learned and Recommendations The EEA Philippine program, after three years of implementation, succeeded in meeting all of its intermediate objectives of building an alliance, training youth and opening access to on the job training, apprenticeship and employment for youth. It also engaged youth, families and local government officials in the program. On the last intermediate objective of family engagement, this became a natural consequence of targeting out-of-school youth as their families had to be involved actively for project success. While the evaluation was generally positive, however, some partners felt that the time of the program was too short and as such did not fully support building an alliance. As EEA Philippines reflected on lessons learned, the issues that surfaced were primarily on sustaining the alliance, creating more effective ways of linking skills and jobs, and overcoming challenges of working with at-risk youth. 1. Sustaining the Alliance The EEA Philippines alliance structure was a two-tiered mechanism at the national level, with a national group of respected leaders and at the project level through community-based alliances of institutions and individuals who had decided to work together to help out-of-school youth. The alliance structure was both formal and informal. Memoranda of Agreements were signed among members and the implementing partners at the community level. While easy to do, these formal commitments were not the impetus for real alliance building. Rather, what appears to have prevailed were informal commitments that were forged among communities and friends and nurtured by the EEA alliance building experience at local community levels. These commitments were “social capital” that could be directed to youth employability. All partners said they would use these relationships in future endeavors. Financial support for alliance work was not easily found. Specifically, Sarah Redoblado, Executive Director, Alterplan concluded that sustaining funding for alliance work still remains to be seen. All implementing NGO partners had developed strategies to secure funding after project completions. Plans had been initiated by the Secretariat through workshops for resource mobilization and sustainability to help implementing NGOs. For one, Alterplan, which was able to significantly leverage support during project implementation, has been the most active in creatively seeking ways to sustain their work. Some of their strategies are to seek support from government agencies such as for TESDA vouchers or scholarships and to tap local government development funds. Another option has been to seek subsidized or full payment of tuition by training participants or their sponsors. Alterplan sees the possibility of getting support from construction levies but realized that large construction companies have limited interest in helping out-of-school youth. In the future, new project programming shall benefit from a strong and closer relationship with industry, or having more industry partners. For example, Alterplan finds that EEA has made it possible for them to talk to Chevron Philippines for a school building project. From the implementing partners, most of the national alliance steering committee members said that the alliance will not be sustained at the national level. It has proved to be challenging to convene the EEA Philippines alliance that was composed of officers from different national offices of the government and non-government and business organizations. Furthermore, participation in, or assistance from their respective organizations to the EEA project and its implementing partners were always referred to their local offices. In view of this experience, Consuelo Foundation deemed it more effective to pursue local alliance building in each project site to generate additional resources for education and employment projects for the out-of-school youth. But at the local level, national committee members believe that the alliance made a difference and can be sustained. EEA Chairperson Ray Dean Salvosa commented as follows: “Members of local alliances tend to identify with the project because of its implementation in their own communities. They are also in better positions to identify local needs and resources related to education and employment. It is also easier to convene them for meetings because of proximity and less busy schedules. These meetings are very important venues for their participation in project assessment, planning, strategizing and resource generation.”12 The Secretariat affirms that a lot depends on local people but the national support is important. In the case of the Philippines, the national alliance provided direction and basic policies, which allowed for a 12 Ray Dean Salvosa, Executive Director, Consuelo Foundation and Chair of the EEA Steering Committee. high level of transparency. But, unless an alliance member was already working with the chosen beneficiaries, their interest was short lived. Specifically, the corporations supported the work when they were able to tie their business agenda to the project. Chevron found this to be possible since they have presence in a project site, General Santos City, and the EEA projects are similar to what they have previously funded to promote youth employability. Whether national or local, alliances work better because members have a direct stake in the outcomes of the project. The Secretariat reflects that it is natural for members of an alliance to have ulterior purposes to join and to stay on. The strength of an alliance is also based on who the lead institutions or who the lead alliance conveners are. The Secretariat again points to the importance of having institutions that can outlive projects. The Philippine experience shows that small, start up institutions are not good anchors when an alliance is being built up. For example, the Zamboanga NGO, Ploughshares was primarily interested in pilot testing the approach while other NGO partners have added the EEA project to their overall work program. The Secretariat also points to the importance of choosing right leaders for an alliance. It is not only a strong institution but a strong individual leader that is a prerequisite for program success. The EEA experience shows that leadership in an alliance is exercised at two levels: the institutional level and the individual level. Institutional membership provides some stability, since it protects from internal institutional changes such as staff turnover. However, the individual that represents the institution influences the degree to which an institution participates in an alliance. The representative promotes the alliance within the institution and ensures the goals of both remain complimentary over the life of the project. Executive Directors of EEA Partners have advocated within their organizations to maintain their focus on youth employability. Of the six NGOs, five have made youth employability as part of their new work. While implementing partners plan to sustain the EEA approach by integrating it into their NGO missions and programs, the primary strategy to sustain the results of the initiative has been to advocate with the government to adopt EEA. By late 2008, the YPS program was taken up and funded by the Department of Social Welfare and Community Development in Misamis Oriental. The EEA Secretariat and local implementing partners agree that local government have an advantage in advocating with new or local industries to hire OSY workers. LGUs are often able to negotiate with these industries to fill a percentage of the jobs with local hires. The value added of the EEA approach was able to provide a proven package of technical and life skills to LGUS so their youth candidates can compete effectively with more seasoned workers. By March 2009, five pilot municipalities had begun training OSYs to be productive youth workers, allocating their own funds, and using industry curriculum with their own EEA-trained life skills trainers. It is foreseen that in areas where the EEA program is implemented, local governments will easily respond to a national directive to implement YPS projects. The Youth Productivity Service Program of the DSWD thus provides a promising avenue to sustain what EEA has started to transform OSY as productive young workers in their communities. It also has the potential to make interventions supported by EEA increasingly cost effective, should the initial investment pay back significantly in terms of broad uptake from government and increasingly larger pools of beneficiaries accessed by government. 2. Linking Skills and Jobs In spite of strong placement rates, by project end, the NGO partners continued to emphasize their limitations in working with industry to access jobs for the youth, an area in which they felt they did not have the requisite skills or experience In retrospect, they suggest that one solution would be to make industry more visible in the program from its inception. To do this, NGOs will need to strengthen their skills working with industry, such as learning how to interpret industry forecasts or trends and how to recruit and place youth in jobs. But in working with industry, EEA partners expressed concerns that the target beneficiaries of at-risk youth have made it more difficult to meet industry demands. Formal employment requires high school diplomas and discipline in attendance and on the job. Alterplan commented that the level of maturity of youth was something to worry about, particularly when they are asked to cope in less than familiar locations or travel far from home. Some partners also suggested working with academic institutions in developing curriculum and providing skills. The Davao City Chamber partnership with a university provided high quality curriculum and teaching as well as an opportunity for the youth to return to school through a ladderized curriculum. The Marcellin Foundation, with its links to the Notre Dame School system has been able to oversee the learning activities tapping academic trainers. However the other EEA partners had to train trainers as they trained the youth, with varying levels of success. Youth noted this in their evaluation, commenting that the trainers were better counselors than teachers of technical skills. An area for improvement in future projects is to better manage the apprenticeships with formal industry. These have been implemented as less than formal programs and perceived by the youth as weak. There were also instances where apprenticeships were welcomed by formal employers because they provided an opportunity for cheap labor. The NGOs quickly became aware of this and held back on job placement, making sure that fair wages were provided to the youth. A few employers wondered why youth did not jump at the chance for any job while in contrast, some youth felt their new skills entitled them to better opportunities. How long after job placement should an NGO help out? This issue became evident as youth experienced problems in their new jobs. For youth who were not used to living in urban areas, the complaints were that they did not have the skills to interact with more experienced older workers. Some NGOs see the need to encourage the formation of worker cooperatives and employment agencies to handle the social displacement that was sometimes experienced by young workers. These employment organizations are also important to manage the seasonal work that many youth were encountering in the construction, food processing and steel industries. The seasonality of work is also because the skills of the youth are basic. 3. Working with At-Risk Youth EEA is mandated to work with at-risk youth in conflict areas and this was not a priority sector for Philippine government agencies. Only the DSWD and the Department of Labor were focusing on OSY issues at the time. In 2006, they were not known or tracked, despite some estimates that for every one youth in school, there was one who was out-of-school or that the bulk of the armed groups in Mindanao were aged 15 to 25 years old and out of school. Advocacy from the Education for All initiative pushed the agenda to make this sector, visible. But still, unless the community finds these youth, they remain invisible and untracked. The cost of working with out-of-school youth is higher than with in-school youth. The education required of these youth is more than just technical. It requires life skills and in some instances, reintegration skills. The participation of the community and the youth parents are critical. And the financial support includes not just tuition for the skills training classes, but support for transportation, supplies and food. Youth shared that they had sometimes not attended classes prior to this training because of hunger. Conclusion In sum, based on extensive interview and surveys with program participants, implementing partners, employers, and other partners this study finds that for the 2,669 who graduated from EEA technical courses, the program effect stays on with them and their families. The youth who graduated know they can find jobs with their new skills and government trade licenses. Youth families also benefited since they now have role models for other family members. Parents have said they now realize that their sons can do more than farming and fishing and that other children are seeking to enroll in similar programs. The companies who hired OSY, have gained new views on working with disadvantaged youth, seeing that they can compare with schooled youth in doing the job. And NGOs have seen the importance of youth programming in their development work. NGOs have also seen the efficacy and advantages of the alliance approach as critical to the project successes. The study also suggests that youth employability projects need to be integrated with the broader clientele they serve – youth, employers and communities – and can benefit from an alliance approach in developing integrated systems. However, working with the alliance approach means NGOs need to have new skills to enable them to relate with different sectors with different agendas. NGOs will need skills as integrators, with the ability to translate and link different agendas. Positively, the EEA Philippines program showcases how the EEA approach can be adopted and scaled up, with the initial program serving as a testing ground for initiatives that could gather the necessary attention and resources in this pursuit. Cost also appears to be fairly similar to other initiatives. It appears that partnerships can continue to promote successful approaches through their own networks. While the three-year program succeeded and in some instances, surpassed its target and objectives, there are new areas that need to be addressed to become more effective in helping youth gain quality employment. One area is how to effectively interface with the business sector and develop relevant training and youth apprenticeship or OJT programs. Another issue that was not fully addressed by the EEA initiative is how to step up the skills of at-risk youth considering their low educational levels so they can access more stable jobs and how to support the new worker on the job, to include worker associations for job placement and job benefits. A further area which needs to be explored, and innovations should be devised, is in the area of youth entrepreneurship for urban based workers, family enterprises and rural agricultural youth entrepreneurs, particularly for those at-risk. Finally, a lasting effect of EEA Philippines is the adoption by national government of the EEA approach in the YPS program being piloted in Misamis Oriental. Analysis of the full impact of this expansion of EEA models is not possible at this early juncture, but the possibilities and potential of successful implementation of YPS would suggest significant value to the alliance building approach engendered by the EEA program. As such initiatives develop; however, there will also be a need for policy reform and advocacy to make youth employability a continuing and critical concern for national and regional governments. ANNEX A: Typology of EEA Partners and Leverage Contributions Partner Name Partner Role and Contributions Non Government Organizations ALTERPLAN Implementing partner Asia Pacific Prosperity Co-implementing partner Fisheries Improved for Sustainable Harvest (FISH), Bongao, Tawi-Tawi Educational field visit for the youth Civil Society Network on Education Partnership in developing Entrepreneurship Courses for out-of￾school (OSY) that can be mainstreamed with the Bureau of Alternative Learning Systems Of the Department of Education Davao City Chamber (DCCCI) Implementing partner and project-level Secretariat Habitat for Humanity Philippines (HFHP) – Philippine National Office (PNO) and Habitat Building Resource Center (HBRC) Secured Habitat project sites as venue for OJT component of skills training; Upgraded the quality of Habitat workers; Linked training graduates to employment in Habitat projects; Coordinated skills training with day-to-day project management (in HBRC-managed projects) Habitat for Humanity Chapters, General Santos City (5 chapters) Integrate skills training with day-to-day Habitat project management; opened access to LGUs Kasanyangan Foundation Inc. Assistance in identifying and mobilizing OSY for the training KFI Center for Community Development Foundation Inc. Assistance in identifying and mobilizing OSY for the training Marcellin Foundation Implementing partner Metal Working Association of the Philippines/ Cagayan de Oro Chapter Implementing partner and project-level Alliance Secretariat Mindanao Business Council Provided access to jobs Mindanao Center for Research & Development Coop Assistance with micro-enterprises Philippine Development Assistance Program Implementing partner Ploughshares Implementing partner Southeast Asian Fisheries Development Center (SEAFDEC), Tigbauan, Iloilo Shared technology on seaweed farming Tawi-Tawi Family Life Foundation (TFLF), Bongao, Tawi-Tawi Arranged educational field visits for the youth Private Sector Asia-PRO Cooperative Manpower services and placement of graduates Ateneo de Zamboanga University Part of the implementation team and provided a resource person during training Bloomingdale Store Part of the implementation team, provided a resource person for the training, and provided space for the retail store for OSY products Cagayan de Oro Steel Center, Inc., Tagoloan, Misamis Oriental Provided on-the-job training (OJT) and employment opportunities Edisons Machinery Works, Corp., Cagayan de Oro City Provided OJT and employment opportunities Chevron Philippines Funds for training and materials Davao Region Mango Contractors Association Apprenticeships and access to companies for OJT of youth Durian Industry Council Apprenticeships and access to companies for OJT of youth EEI Construction Corp., Villanueva, Misamis Oriental OJT and employment opportunities Partner Name Partner Role and Contributions Fitweld Limited, Inc., Cagayan de Oro City OJT and employment opportunities Food Processors Association of Davao Apprenticeships, access to companies for OJT of youth Goldtown Industrial Sales Corp., Cagayan de Oro City OJT and employment opportunities Hanjin Heavy Equipment Corp., Villanueva, Misamis Oriental OJT and employment opportunities JENARAs Manpower Services Manpower services and placement of youth graduates Lapanday Agricultural Development Corporation Apprenticeships, access to companies for OJT of youth MENZI Farms OJT of youth Mindanao Fruit Council Apprenticeships, access to companies for OJT of youth Mindanao Oriental Builders, Inc., Villanueva, Misamis Oriental OJT and employment opportunities Orophil Industries, Cagayan de Oro City OJT and employment opportunities Philippine Green Farm Employment and training in the Banana Sector Phividec Industrial Authority Member of Alliance and access to OJT opportunities Private Individuals/Professionals Design of Youth Entrepreneurship Curriculum and training of youth in remote villages Sikulan Seaweeds Farmers’ Association, Tawi-Tawi Mobilization and monitoring of participating youth Sipangkot Seaweeds Farmers’ Association Mobilization and monitoring of participating youth Sitangkai Seaweed LMC Corporation Additional market for young people’s seaweed harvest and a potential source of employment for youth graduates for quality control of seaweeds Sanga-Sanga, Bongao, Tawi-Tawi; Kasangyangan Nursery Seaweed Enterprise (KNSE), Tigtabon Island, Zamboanga City Educational field visit for the youth SITEXPORTS Seaweed market for the youth’s raw dried seaweeds produce Southern Mindanao Mango Industry Development Council Apprenticeships, access to companies for OJT of youth STI School in Zamboanga City Co implementer of the modules, provider of classroom, training for the STI teachers on doing OSY training SUMIFRU Corporation Apprenticeships, access to companies for OJT of youth TADECO Apprenticeships, access to companies for OJT of youth Tinambak Lugus Seaweeds Farmers’ Association Mobilization and monitoring of participating youth Vegetable Industry Council of Mindanao Apprenticeships, access to companies for OJT of youth Vercide Engineering Services, Inc., Tagoloan, Misamis Oriental OJT and employment opportunities VM Paras, Tagoloan, Misamis Oriental OJT and employment opportunities Government: National & Local Government Units (LGU) Bureau of Alternative Education Mentoring and follow up of youth for equivalency for High School City Government of General Santos City Policy support for Marcellin program City Social Services Development Office, Davao City Microfinance loans and access to the city government services Davao City Barangays (six) Community support in recruitment and monitoring of Youth Partner Name Partner Role and Contributions Department of Agriculture, Davao City Development of curriculum Department of Agriculture Region 9 Technical assistance in agricultural training Department of Education through their BALS representative for the Municipality, Misamis Oriental Mobilizer of OSY, Assistance in Implementation of Program Department of Education (DepEd), Sitangkai District Functional education curriculum Department of Labor and Employment, Davao City Participation in the Youth Labor Study for the City Government Department of Social Welfare & Comm Dev, National Office Funds for Youth Productivity Service Model Department of Science and Technology Development of curriculum Department of Trade, Davao Technical training and seed capital for livelihood loans ILO/City Government Project, Davao City Youth entrepreneurship training LGU of Davao del Norte, Provincial Department of Agriculture Technical assistance in Hog raising and other agricultural training; provider of facilities for training based on government fees Local Government Units of Misamis Oriental participating in YPS program Development of operating plans for the Youth Productivity Services project Local Government of Davao Norte Technical and market access assistance to OSY enterprise products Mindanao State University - TCTO Technical support for seaweed tissue culture that will help propagate good and productive seaweed seedlings Municipality of Opol, Misamis Oriental Funds, materials and training venue for training of youth under YPS model Municipal Departments of Social Welfare and Development, Zamboanga Sibugay and Zamboanga City Provision for technical assistance; mobilizing out-of-school youth and co-building enterprises for the youth Municipal Local Government Units of Sitangkai and Sibutu; Barangay Abinsanga (Sipangkot), Tawi-tawi Development of a seaweed industry master plan Local Government of Tagoloan Additional resources and venue for the training of youth Local Government of Villanueva Additional resources and venue for the training of youth Philippine Rice Research Institute Technical Assistance in securing copyright Provincial Department of Labor and Employment Office Technical Assistance in Building Rural Organizations for OSY and provision of funding social enterprises Provincial Government of Misamis Oriental Funds for Youth Productivity Service model and mobilization Region 9 Department of Labor and Employment Technical assistance in building rural organizations for OSY and provision of funding social enterprises Regional Department of Agriculture in Davao Provinces Technical assistance in agricultural training Technical Education and Skills Development Authority (TESDA) in Zamboanga, Davao del Norte, Davao City and Zamboanga Sibugay Doing the OSY training, provider of funds for OSY course TESDA, Misamis Oriental Review and approval of adopted curriculum TESDA & Bureau of Fisheries & Aquatic Resources, Tawi-Tawi Accreditation of youth trainers and recognition of trainings initiated by PDAP; short listing of the youth graduates for futures TESDA training Partner Name Partner Role and Contributions TESDA Kor-phil (Korea - Philippines Facility), Davao City Training of youth University of Southern Philippines, Davao City Co-implementing partner WNAS (Wangan National Agricultural School) in Davao City Training of youth Zamboanga City Department of Social welfare and the Department of Agriculture Mobilization of OSY; Assistance in program implementation Zamboanga Sibuguey Provincial Local Government Unit through the offices Governor George Hoffer and the Provincial Cooperative Development Office Provision for technical assistance, funding mobilizing OSY and co-building enterprises for the youth International Organizations Act for Peace Programme and World Food Programme Enhanced access to target beneficiaries in Mindanao; Enhanced access to LGUs with Peace and Development Communities; Resources for additional support and services to beneficiaries Building and Woodworkers Int’l (BWI) and National Union of Building and Construction Workers (NUBCW) Linked training graduates to industry formation and potential benefits of membership in workers organization; Additional resources for training and certification of trainers-assessors; Linkages to industry association International Labor Organization (ILO) Enhanced access to target beneficiaries and LGUs in Guimaras; Resources for training of additional beneficiaries in distressed economic conditions ©2010 International Youth Foundation. All rights reserved. No portion of this report may be reproduced in any form without the written permission of IYF. This project was made possible by the generous support of the American people through the United States Agency for International Development (USAID) under grant agreement no. RAN-A-00-04-00043-00 This report was funded in part under grant agreement no. EPP-A-00-08-00006-00. The International Youth Foundation (IYF) invests in the extraordinary potential of young people. Founded in 1990, IYF builds and maintains a worldwide community of businesses, governments, and civil-society organizations committed to empowering youth to be healthy, productive, and engaged citizens. IYF programs are catalysts of change that help young people obtain a quality education, gain employability skills, make healthy choices, and improve their communities. To learn more, visit www.iyfnet.org ANNEX H: EEA India (QUEST) Evaluation Reports QUEST Alliance Evaluation of the Computer-Assisted Learning Programme Dr. Mythili. R R.V. Educational Consortium Rashtreeya Sikshana Samithi Trust Bangalore -11 January, 2009 Field Assistance Ms. S.K. Prabha Ms. Brinda Rao Ms. Akila.R Ms. Suma Ms. Gayathri Data Entry & Typing Assistance Ms. S.Annapurna Ms. Varahi Ms. Mamatha Content List of Abbreviations List of Figures 1. Introduction 1.1 An overview of the programme 1.2 Design of the present study - Objectives - Methodology - Tools 2. Implementation processes of the programme 2.1 Resource provision (Physical and Human) 2.2 Capacity building 2.3 Community mobilization 2.4 Monitoring and Evaluation 2.5 Challenges and problems faced 3. Supporting and Detrimental Factors 3.1 Supportive factors 3.2 Collateral Benefits 3.3 Detrimental factors 3.4 Case Study 4. Perception of stakeholders 4.1 Teachers 4.2 Students 4.3 Parents, SDMC members and Community members 5. Implications of the programme 5.1 Classroom integration 5.2 Gains for students 5.3 Changes for teachers 6. Conclusions 6.1 Resources 6.2 Training 6.3 Follow-up and feedback 6.4 Sustainability (support from Community, Panchyat and Government) 6.5 Classroom environment 6.6 Pedagogic approach References Appendices List of Abbreviations APF Azim Premji Foundation B.Ed Bachelor of Education BRC Block Resource Centre BRP Block Resource Person C.Ped Certificate course in Physical Education CAL Computer Assisted Learning CRP Cluster Resource Person D.Ed Diploma in Education DIET District Institute of Education and Training HM Head Mistress/Master KSQAO Karnataka School Quality Assessment Organisation M.Ed Master of Education NCERT National Council of Education Research & Training NCF National Curriculum Framework PUC Pre University Certificate QUEST Quality Education and Skills training SC Schedule Caste SDMC School Development & Management Committee SSLC Secondary School Leaving Certificate ST Schedule Tribe TLM Teaching Learning Material UNESCO United Nations Educational, Scientific and Cultural Organization List of Figures Fig 1.1 Percentage of Male and Female teachers Fig 1.2 Academic qualification of teachers Fig 1.3 Professional qualification of teachers Fig 1.4 Percentage of support teachers with D.Ed Fig 2.1 Percentage of computers working – Phase-1 schools Fig 2.2 Percentage of computers working – Phase-2 schools Fig 2.3 Number of support teachers trained by APF Fig 2.4 Percentage of teachers who have received training from APF Fig 5.1 Comparison of Average enrolment of students in Phase-1 Schools Fig 5.2 Comparison of Average enrolment of students in Phase-2 Schools Fig 5.3: Comparison of average KSQAO scores of students in Phase-1 schools Fig 5.4 Comparison of average KSQAO scores of students in Phase-2 schools Chapter I: Introduction The Quality Education and Skills training (QUEST) Alliance supports initiatives that help improve the quality and relevance of basic education and skills training for vulnerable and marginalized children and youth through advocacy and provision of funding support. The Computer Assisted Learning (CAL) programme in Bangalore, which was implemented by Azim Premji Foundation (APF) with support from the QUEST Alliance, is the focus of this evaluation study. 1.1 Overview of the programme APF has extensive experience in running CAL programme having initiated 225 computers assisted learning centres all over Karnataka. Learnings from APF’s previous initiatives have been integrated into this current programme being evaluated. This programme was initially planned for implementation in twelve Government higher primary schools at Bangalore. Subsequently, it was extended to eight more schools as additional resources were mobilized from local communities. The duration of the programme was two years. The schools were short listed with the help of block officials on the basis of the following criteria: • School strength of more than 250 • Availability of an extra room • Presence of legal electricity power connection • Interest envisaged by teachers • Contribution from the community over the last three years The head teachers of these 23 short listed schools were then oriented by APF personnel about inviting community for a meeting and the arrangements that needed to be made for the meeting. The head teachers were also informed of the process of school selection. Subsequently, twelve schools were finalized on the basis of the interest and support lent by the community members. The programme was extended to another eight schools after one year of implementation. Each school was provided with six computers. In addition one computer and printer were supplied for the exclusive use of teachers, at their behest. Another hall mark of the programme was the appointment of a support teacher in each of these schools. A fairly rigorous process was employed in the selection and training of these support teachers. A coordinator was appointed exclusively for the monitoring, feedback and follow-up of this programme. The coordinator was then supported by three personnel from APF. The time table for using computers was prepared by the head teachers and teachers of individual schools. It was planned that on an average, each child was to get two periods at the computer every week. This CAL programme differs from the previous CAL programmes implemented by APF on the following aspects: • Role of support teacher has been perceived differently from that of Young India Fellow of previous programmes. • Support teachers’ basic qualification was pre-university certification, preferably with a Diploma in Education (D.Ed). • Training for support teachers was more rigorous. • More tried and tested CDs were provided to schools. • Grouping of children was done more carefully. • There was a greater emphasis on documentation. • Monitoring was more intensive. • Teachers were encouraged to view CDs with students. • Monitoring software has been installed on an experimental basis in three schools. • CAL model for low electricity consumption is being piloted in eight schools. 1.2 Design of the study The present study was conceptualized as a process-based evaluation. The objectives of the study are as follows: 1. To capture the implementation processes of the programme 2. To identify the challenges and problems faced while implementing the programme 3. To document best practices 4. To ascertain the perception of stakeholders (students, teachers, parents and community members) about the effectiveness of the programme 5. To study the implications of the programme in terms of: children’s attendance; children’s learning levels; retention of students; classroom practices; learning gains of students; and changes for teachers. Apart from interacting with APF personnel involved in the programme and studying relevant documents, a field survey of all the twenty schools was administered. Out of these twenty schools, in twelve where the programme was launched in 2006, the programme ended in June, 2008. We refer to these as Phase-1 schools. In the remaining eight schools (Phase-2) the programme was launched in 2007 and continued until 2009. A team of six investigators visited the schools to collect data. A workshop was conducted to orient the investigators about the nature of the CAL programme: objectives of the evaluation; conducting interviews and focus group discussions; observing classrooms and the kind of information to garner. It took twelve weeks (September to November, 2008) to complete the field work. The delay was primarily due to the examinations followed by a month long vacation for schools. Sample While general information was collected from all twenty schools, the following sampling procedure was used to get data from stakeholders: Teachers: The mandate was to interview two teachers in each school. However, in most schools all teachers showed inclinations to interact with the study team. Hence, focus group discussions were held. The following figures give the details of 62 teachers who formed our sample. Fig 1.1: Percentage of Male and Female teachers 80.6 19.4 Female Male Fig 1.2: Academic qualification of teachers 32.1 42.9 25 0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0 45.0 SSLC PUC Degree % of Teachers Fig 1.3 Professional qualification of teachers 92.3 5.8 0.0 1.9 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0 TCH B.Ed M.Ed C.Ped % of Teachers Support teacher: In schools where services of support teachers were continued, we interviewed these support teachers. Figure 1.4 presents the percentage of support teachers with D.Ed. Fig 1.4: Percentage of Support teachers with D.Ed 27% 73% With D.Ed Without D.Ed Students: Ten students of class 5 and ten students of class 7 were randomly selected, ensuring proportional representation of girls and boys. Discussions with class 5 and class 7 students were held separately. Parents or School Development and Management Committee (SDMC) members: In each school, we interviewed the SDMC President where available. In five of the schools where we could not meet the President, we ensured that atleast one SDMC member was interviewed. In addition we interacted with four to eight parents, whose children study in the school. Community members: We spoke to a few members of the community living in the vicinity of the school. In addition to interacting with stakeholders, one computer class and one regular classroom were observed to capture integration process, in any. A Classroom in progress Tools Prior to the preparation of tools, this researcher visited two schools where the programme was on￾going. This helped gain a perspective for tool development. A set of nine forms were developed to aid data collection. Form-1 captured basic school details. Form-2 provided guidelines for classroom observation. Guidelines for observing a computer class were given in Form-3. Forms 4 to 8 had guidelines for interviewing or conducting focus group discussions with regular teachers; support teachers; students; parents/SDMC members; and community members respectively. Form-9 had leading questions for discussions with APF personnel. These forms are presented as annexes of the report. Data The following table maps the data required together with the objectives. Table 1: Nature of data collected and source of data Objectives Data collected Data Source 1. To capture the implementation processes of the programme Primary data from organization / individuals implementing the programme • All available documents • Interactions with people involved 2. To identify the challenges and problems faced while implementing the programme Primary data from school teachers, support teachers and programme implementers Interviews and focus group discussions with concerned personnel 3. To document best practices Primary data from school teachers and support teachers; schools • Interviews • Classroom observation Objectives Data collected Data Source 4. To ascertain the perception of stakeholders (students, teachers, parents, community members) about the effectiveness of the programme Primary data from stake holders • Focus group discussions with stake holders 5. To study the implications of the programme in terms of: Children’s attendance, Children’s learning levels, Retention of students, Classroom practices Secondary data pertaining to students; primary data from classrooms • Attendance registers before and after implementation of the programme • Students’ progress report/KSQAO scores (where available) • Classroom observation Data Analysis Since much of the data were qualitative, descriptive analyses were used. Some quantitative data we collected were entered in Excel spreadsheet and subjected to frequency analysis. Limitations of the study The study involves mainly process indicators alone. Input indicators have been adequately captured by the implementation agency. We assessed a few of them during field visit. Output indicators have not been adequately captured for lack of baseline data. Hence, assessing the impact of the programme in quantifiable terms is not meaningful. The next chapter discusses the implementation processes of the programme. CHAPTER II: Implementation Processes of the Programme One of the primary objectives of this evaluation was to capture the processes that went into the implementation of the programme in terms of providing resources, building the capacity of concerned people, monitoring and feedback mechanisms. Data for this study were obtained from the internal reports and other pertinent documents prepared by APF, as well as through interactions with persons involved in the implementation of the programme. These were then substantiated during field visits. 2.1 Resource Provision The following physical resources have been provided in each school: ¾ Six computers: The rationale for providing with six computers was that 24 children could be accommodated at a time in the computer class with a maximum of four children per computer. A typical computer class ¾ All computers have speakers; headphones were also provided for each child. ¾ Around 75 multimedia, interactive CDs created by APF have been given. ¾ The computer room has been equipped with basic furniture. ¾ A printer and an additional computer have been provided for the exclusive use of teachers. ¾ In the eight phase-2 schools, a multi-user computing approach is being piloted. This technology provides each end-user with his/her own monitor, keyboard, mouse and speakers that connect to small access terminals. These access terminals in turn are connected to the host computer. The vendor has also supplied each access terminal with the core terminal services software. This technology promises to deliver high performance at a low cost and is compatible with standard PC applications. What we found during field visit Fig 2.1: Percentage of computers working – Phase-1 schools 2 3 3 4 None of the computers w orking < 50% of computers w orking 50-80% of computers w orking >80% of computers are w orking • In two of the twelve Phase-1 schools, none of the computers are working. • In three more schools less than half of the computers provided are working. • In the remaining schools, either all computers are working (in four schools) or only one computer is under repair. Fig 2.2: Percentage of computers working – Phase-2 schools 3 3 2 < 50% of computers working 50-80% of computers working >80% of computers are working • Among Phase-2 schools, all computers were working in only two schools. In the remaining schools, only two or three of the six computers were working. • Children tend to misuse headphones – biting on the wire or pulling it. In only a few schools, were the headphones functional. • Each school was equipped with around 50-60 CDs. • Printer and computer meant for teachers remained unopened, in all but three schools. • The teachers have not been able to discern any difference between a standard PC and the new cost effective technology supplied. Man Power Resources • One support teacher was appointed for each school. • 50% of the support teacher’s salary of Rs.2000/- was borne by the programme, for the two years it was in effect. • A coordinator was appointed to oversee the programme in all twenty schools. In addition, three coordinators from APF were also involved in the close monitoring of the programme. What we observed • Support teachers are there in all of Phase-2 schools. • Services of the support teachers continued in three of the twelve phase-1 schools where the programme has come to an end. • The programme coordinator had left the organisation soon after the programme ended in phase-1 schools. Coordinators from APF are monitoring phase-2 schools. 2.2 Capacity Building Support Teachers As mentioned earlier, a fairly rigorous process was employed in selecting and training the support teachers. SDMC members/head teachers were asked to identify a candidate who has completed PUC, preferably with D.Ed qualifications, else with an inclination for teaching. The shortlisted candidates were observed during group discussions they were asked to participate in. This was followed by an interview. Two candidates were selected for each school, with the understanding that they would be appointed, after their training. A five-day residential training programme was conducted for over 20 candidates. The training was intended to meet the participants with basic computer skills and to orient them to pedagogic skills. Accordingly, the content of the training programme was: - Overview of learning theories - Introduction to how children learn; learning styles of children - Basic computer skills - Use of CDs and selection of CDs - How to group children - Hands-on experience for working with children - Working with teachers, head teachers, parents and government officials - Developing positive thinking - Improving communication skills The modality of this training programme was: - Observation - Hands-on with children - Self-reflection - Discussions After the training, twelve were appointed as a support teacher for each school. Follow-up of this training was done at these schools. What we found Fig 2.3: Number of support teachers trained by APF 12 8 3 8 2 6 0 2 4 6 8 10 12 14 Phase 1 Phase 2 Total No.of Schools No.of Support Teachers No.of Support teachers trained by APF In all eight phase-2 schools, support teachers are there. Two of these eight teachers have not received training from APF. In three of the twelve phase-1 schools, support teachers are continuing. Both these teachers have been trained by APF. We gathered that attrition had been rather high, with support teachers leaving, once they get government jobs etc. However, all the support teachers opined that the training provided by APF was useful. They were especially appreciative of the soft skills component of the training. Regular Teachers Training for the regular school teachers was given at their respective schools by the coordinators. The duration of the programme was three days. The objectives of this training were to: • Give an overview of the programme • Teach teachers to switch the computers on and off • Help teachers view the CD content Discussions were held on integration and ideas/issues for helping support teachers. What we observed Of the 62 teachers from 20 schools we had interviewed, 28 said they had received training from APF personnel. Fig 2.4 Percentage of teachers who have received training from APF Not Trained, 54.84 Trained, 45.16 Among these 28, ten have received training through Government initiatives as well. These have been conducted by DIET/BRC/CMC. In one school, two teachers had undergone 40 days training by Intel. Most of the teachers do not seem to have taken the training provided by APF seriously. Their typical response was, “APF has not given us any training.” When prodded about it they replied, “Oh! We were taught how to turn a computer on and off.” As to the CDs except in seven schools, teachers in other schools have seen only one or two CDs. We also came across six or seven teachers (most of them were male) who had not seen a single CD. Those teachers who have seen the CDs find value in them. This is discussed in section 4.2. Most of these teachers said the support teacher had helped them handle the computer, when they faced difficulties. In four schools, a teacher has taken the initiative in entering the time table, marks list and students’ details. In one of these four schools, the support teacher types out the question papers for subject teachers. In another school, a teacher who is learning to type in Kannada said she saves her notes of lessons in the computer. 2.3 Community Mobilization Community buy-in for the programme has been its mainstay. One of the criteria for selection of schools was the level of participation of community members in the meetings organized at selected schools. The head teachers were requested to invite parents, SDMC members, Panchyat members, local leaders and donors for the meeting. During the meeting, APF personnel taught two-three children of the school to handle a mouse and displayed the CD content on big screen to create awareness among the community members. To ensure sustainability of the programme, community members were asked to bear fifty percent of the salary of the support teacher and electricity bill. They were also informed that the local community has to garner funds to run the programme independently, once external funding ceases. Local community members had seeded money at the time of commencement of the programme. What we found During our interaction with parents and SDMC members, we found that they were rather proud that the school their children are studying in equipped with computers. A couple of them said that during the initial days soon after computers were installed they would stand outside the computer class just to watch their wards handling the computer. The SDMC President in most schools has been taking keen interest in trying to mobilize funds to continue the programme. They informed us that they were able to pool in resources from the local community for paying electricity bill and attending to minor repairs. In the three phase-1 schools where support teacher is continuing, individual donors are contributing towards her salary. But other than parents or relatives of children studying in the school, there was little awareness about the presence of computers in the school among other community members. 2.4 Monitoring and Feedback APF had appointed a coordinator exclusively to monitor the twenty schools. In addition, there were three other APF personnel who were monitoring the programme closely. This had helped attend to problems without delay. Periodic feedback given to support teacher helped ensure the programme remained on track. A head teachers’ meeting was arranged by APF to provide them with an overview of the CAL programme. The aims of the programme and the direction of learning envisaged through the programme were discussed with the head teachers. They, in turn, came up with practical problems they face, in terms of sustenance of support teachers, shortage of regular teachers, and management of mid-day meals. What we found All the teachers of phase-1 schools were unanimous that the close monitoring resulted in attending to repairs in computers with little time lag. Only in one school (Bidaluru) was there a complaint about none of the computers working even during the project period. This was again due to faulty electrical wiring. The coordinator had interacted only with the support teacher and not with the regular teachers. The support teachers have found the interactions with the coordinators very useful. They said it helped them gain confidence and remain motivated. 2.5: Challenges and Problems faced We captured the challenges and problems faced during the implementation of the programme, through interactions with APF Personnel, head teachers, regular teachers, support teachers and parents. The following were the recurrent issues that emerged: • Difficulty in getting qualified support teachers, as the salary offered was not lucrative enough; • High levels of attrition among support teachers, especially those with D.Ed qualification; • Erratic power supply; • High electricity bill (in phase-1 schools); • Paying salary for support teachers after the programme ended; • Attending to technical problems schools said they find it difficult to find manpower and funds to attend to the repairs. APF has provided schools with contact details of computer technicians. According to the head teachers, technicians refuse to attend to their problems saying that their annual maintenance contract has ended or that the commute is too difficult; • Maintenance of head phones is a challenge. Children tend to handle them carelessly. In the absence of headphones, the noise level is very high; • There are only a few CDs available. In schools where children are not being taught computer skills, they complain of getting bored seeing the same CDs after a year; • Grouping of children based on their academic performance wherein one or two children who perform well academically are grouped with two or three who do not - which appears to be detrimental to the latter; • During our field visit, we found children using CDs much below their cognitive level. For example, class 5 students were viewing ‘Akshara Bandi’ meant for six to seven years old; and • Purpose of using CD is unclear to both regular teachers and support teachers. They are using the CDs for reinforcement, if at all. The next chapter documents some of the best practices we observed. Chapter 3: Supporting and Detrimental Factors In any education-related programme given a similar set of inputs, the outputs are varied. It is therefore of interest to look into the factors that help realize the objectives of a programme and those that are detrimental. While no doubt, some of these factors are context specific, generic lessons can be drawn from them. 3.1 Supportive Factors • Enthusiasm and interest shown by children, in general. Everywhere we found the level of confidence shown by most children in handling computer was enormous. • Where two or more regular teachers of a school have involved themselves, programme has continued effectively after end of project period. • English learning in terms better pronunciation and sentence formation has improved. In a couple of schools, children themselves have recognized this. • CDs pertaining to mathematics and science concepts have helped improve understanding in a few children and in one school. • Children have used computers during holidays. Both teachers and parents were appreciative of this opportunity, as otherwise the children would have whiled away their time. • Children have enjoyed animations in the CDs. • CAL has helped initiate small group discussions among children. • Role of external agency was effective. Monitoring and feedback was helpful. • APF trained support teachers have been far more effective than support teachers without training. • As long as the project was in effect, computer classes were being held regularly. We were also able to document instances wherein the programme yielded benefits that went beyond the intended objectives. These have been enumerated in the following section. 3.2 Collateral Benefits Since these are school specific instances, the name of the school is mentioned in brackets • Children won a quiz competition at the taluk level for the first time and the school attributes this success to CAL progamme (Ajjanahalli); • The Block Education Officer has shown interest in the programme and asks for frequent updates (Kachuvanahalli); • Increased involvement of parents (Ramagondanahalli); • Teachers utilise activities/ideas from CDs in their classroom teaching (Kachuvanahalli, Bashattihalli); • A teacher who is a resource person for block level training has got many ideas for conducting his training programmes, from the CDs (Vishwanathapura); • A private school had to close due to decreased enrolment after CAL programme started in the Government school (Bidaluru) [While this may not be direct benefit, parents and teachers consider it a great pride that they have been able to wean back children from the more ‘privileged’ private school]; • CAL is being used for remedial teaching (Kaikondahalli); and • One group of class 7 students have visited a browsing centre to get information for their social science project (Vishwanathapura). 3.3 Detrimental Factors Based on our observations and interactions we culled out the following factors that are coming in the way of effective continuation of the programme: • Difficulty in maintaining the hardware - computers, UPS and especially the headphone. The difficulty stems both from raising money to attend to the repairs and getting technicians; • Low salary offered for support teacher has led to high levels of attrition; • Little buy-in from teachers and department officials; • CD meant for single user is being used by four children; and • Children complained of boredom after a year, as they were being made to see the same CDs again the following year (Around 50 CD titles have been provided to each school). These factors were common to most schools. The presence of an external agency-APF had helped mitigate these factors as long as the project was in force. However, as enumerated in the previous section, some of the phase-1 schools have been able to overcome them – where teachers have taken interest (two schools) or the HM/SDMC President has been able to garner funds for continuing the support teacher. To gain a better perspective of these factors in context, we have included two contrasting case studies in the next section. 3.4 Case Studies Both of these schools are phase-1 schools. Kachuvanahalli Kachuvanahall, is a small village tucked away in the paddy field about 20 kms away from the Kanakapura-Mysore road. We went to Kachuvanahalli unannounced. The school has a well maintained garden. The corridors were clean. Children were sitting in small groups revising their lessons for the upcoming semester I examination. The school has 4 male teachers and a headmaster. The teachers came across as a highly motivated group of individuals working together well. Computer room: Computer room was well maintained. At the time of our visit, class 7 students were being shown the CD “Shakthiyodane Sarasa”. The children were sitting on the floor and watching the CD on one monitor. Upon being asked to see children use the computer themselves, children were made to work in groups of three and were made to watch CDs of different subjects. The students were quite well versed in handling computers. Subject teachers: Teachers have made use of the computers to teach subjects like English, Mathematics and Science. They opined that the computers helped them teach children going beyond their syllabus. For example, speaking in English fluently; teachers have also been giving students the English equivalent terminologies in mathematics and science which they picked up from the CDs. The teachers also felt that the students get an opportunity, thanks to CAL to discuss in small groups about a problem/question which other wise they would not have got. Teachers are adept in handling computers and have been using computers to prepare question paper, for maintaining records as well as invitation cards for school function. They appreciated the APF staff for periodic monitoring and feedback. The overall impression was that even after the support teacher had stopped coming to Kachuvanahalli the teachers have continued good work. They appear to be making a conscious effort to link what is being taught in classroom with the available CDs. With more training, hand holding and guidance; integration could be more successful and children would reap the benefits. Students: They were excited to talk about what they learned from computers. The children on their own said that pronunciation of English in CDs sounded better than their teachers and they learned how to form sentences in English from CDs. Children have also learned MS Paint and were able to use word-pad for typing. They seemed to have liked and learned a lot from Ms.Shylaja, an APF trained support teacher. The students had come to school during the summer holidays from 10.00 am to 2.00 pm to play on the computer. Community members: Kachuvanahalli is a typical village with a small farming community. Talking to the SDMC president, it was evident that most land owners live in the city while only landless labourers and with farmers with very small land holdings live in the village. Hence collecting money to pay Rs.500/- for electricity charges was an uphill talk for them. Even though both the teachers and the SDMC President felt a support teacher was necessary, they did not think they would be able to collect enough money to pay. The parents felt that CAL programme had helped their children compete with those from private schools. Madabal This school is a little away from the main road. The school is to celebrate its centenary year. The HM was not informed about our visit. The school has seven female teachers and two male teachers (including the HM). Computer room: It was very untidy. The children were asked to dust it as we entered it. A television set was placed in the middle of the room. One computer and printer supplied for the teachers’ use were not even opened from their cover. As long as the support teacher was there the computer classes were regular. The UPS was making shrilling noise which made it difficult to be in that room. But the children were not bothered with the noise and they were fully involved in watching their CDs. The HM told us that this is how it has been for the past few months. The support teacher had taught the students to operate the computers and they were doing so independently. The students have learned drawing and painting from their support teacher. They looked very confident in handling the computers. Subject teachers: Only two subject teachers have watched the CDs that too, one or two. They were unable to recall the names of these CDs. They said group activity has increased among children as they helped each other. They appeared disinterested in guiding their students to computer room as they feel that their workload is already too much. They also said that because of computers, Edusat and radio programme it is difficult for them to finish their syllabus. They wanted a support teacher who can take care of the computer room and handle computer classes. Teachers do know how to switch on/off a computer. They say they are ready to learn more but are not ready to undertake computer classes for the children on regular basis. One of the teachers said the CDs are helping the slow learners. Students: They enjoy the computer classes but they said the frequency of these classes had reduced drastically after the support teacher left the school. They enjoy the Maths CD as it has games and makes it easy for them. One of the students whom we met had pronunciation problems and we were told that with the introduction of computers, he has learned to speak more clearly. Community members: Community members other than parents are unaware of this computer programme. The parents are happy to see their children learning from computer and CDs. Their only concern was that they do not have funds to continue their APF trained teacher. The next chapter deals with how teachers, children, their parents and other community members perceive the programme. Chapter 4: Perception of Stakeholders In order to ascertain their perception about the utility of the programme and its sustenance, we interviewed teachers, talked to class 5 and class 7 children separately, and held focus group discussions with parents/SDMC members/community members 4.1 Teachers The role of teachers is rather crucial in this programme. We therefore spent considerable time in getting their views. However, most of their comments were general, interspersed with a list of requirements. Since most of them were recurrent, we have included both in this section. Also, there was no difference in the perception of teachers of phase-1 and phase-2 schools. Interaction with Teachers • All responding teachers said the CAL programme is good. • Having an external agency such as APF was crucial for the programme. Many teachers contrasted APF run CAL programme with other Government run programmes. • Periodic monitoring during the project period ensured the programme ran smoothly. All support teachers were appreciative of the APF coordinators who visited them regularly and attended to technical problems almost immediately. However, the regular teachers were not aware of such visits. But they did recognize that all problems were attended to during the project period. • All teachers felt that the presence of support teacher is absolutely necessary. • Students’ attendance has improved after CAL programme started, and teachers in many schools have specifically stated this factor as a direct benefit of CAL Programme. Parents and students themselves have also corroborated this. • The programme is helping slow learners (though this is a general perception of all teachers, we only came across one instance where CAL was being used for remedial teaching). • Having a variety of activities in the CD appeals to children. • All teachers felt that CDs should be aligned to the state syllabus/textbooks. • CDs for higher classes especially classes 7 and 8 are needed. Most teachers said the CDs cater to lower primary classes alone. • More CD titles are required (This was a universal refrain, although very few of them had actually seen all CDs. What differentiated the responses of those teachers who had seen the CDs from those who did not was that the former had a specific list of concepts/topics for which they felt multimedia context would be useful). • More training needed to learn basic computer skills. All of them felt what was provided by APF was insufficient. • More inputs are needed to help adopt CD content to lessons. A majority of the teachers felt that CAL was another programme like Edusat or radio programme. A few teachers said that all these programmes take away their teaching time and they are unable to ‘complete the portions’. We were also informed that where Edusat relay time clashes with computer period, preference is given for Edusat as their higher authorities vigilantly check records pertaining to Edusat programme. 4.2 Students Students were in general articulate and candid as we interacted with them in the absence of their teachers. This is what they felt about the programme: • Enjoy computers a lot. During our visit, we found this to be true in all schools. • Like using the Paint tool on computers. Almost all children said they like to draw and paint on computer. Quite a few of them demonstrated their skill to us. • Love the animated characters in the CDs. Most children could readily recall the names of CDs and the character in the title that they enjoyed most. • Appreciate the songs very much. In two schools, students sang songs from the CDs. Students singing for our benefit • Like listening to English in the CDs. A majority of the children said English pronunciation on the CD is better than their teachers’. Quite a few students said after having played the Kannada version a couple of times, they now opt for the English version. The children said they are able to read and comprehend English language better on their own. • Like to play games on computers. Many students said they pay Rs.5/- for half-an-hour to play games at the local cyber café’s/browsing centers. • In at least six schools, children explicitly said they liked their support teacher a lot and miss her. • Computer classes were conducted regularly when ‘computer’ teacher was there. Students engrossed with a CD Almost all children could easily recall the names of CDs they had viewed and describe the games they played. However, only a few (less than a dozen) were able to recollect the concepts learned, especially in mathematics and science. Interestingly most of the CD titles they could recall pertained to Kannada language learning. Also, the students had their prejudices – whenever we attempted to draw out the most shy and reticent girl/boy into conversation, the others would say she/he is peddi/pedda and cannot handle a computer. We learned that while working in a group on the computer, these children took on a passive role, letting others in the group operate the computer. 4.3 Parents, SDMC members and Community members Very few community members whose wards were not studying in the school were aware of the CAL programme. Interacting with Community Members in a school neighborhood But, the situation was different with parents. There was a general feeling of euphoria about their children learning computers and their expectations from the programme are rather naïve, but high. • The parents feel that their children’s level of confidence has increased, after they learned to handle computers. A few of them cited instances where their children were able to display their computer skills in their ‘richer relatives’ houses • Children have started showing more interest in school. Many parents said children refuse to miss school especially on days they have computer class. • They are very happy and grateful to the external agency for having given exposure in computers to children. In one school, as a direct result of APF’s initiative, the Panchayat provided an additional computer to the school. • Most parents feel exposure to CAL would be a passport for good job opportunities for their children, in the future. This sentiment was more pronounced in schools closer to Bangalore city. • A few parents also opined that what is taught should not be based only on their school subjects but should also be useful for the future. • In about five schools, parents said they would feel so proud of their children working on computers that during the initial days when the project was launched, they would stand outside the computer room watching their children. • Four or five parents said their children are pressuring them to buy computers at home. • In at least four phase-1 schools, parents suggested that all teachers be trained in the use of computers so that classes can go on as before. • Some of the parents have noticed that their children’s English language has improved over the past year. Two of them told us that their child is attempting to teach them English at home as well. SDMC Members/Parents in a school Most parents admitted that they have started taking more interest in what their children learn at school after the CAL programme. However, other than the SDMC President, none of the others had any clue about utilization of funds. The next chapter enumerates the possible implications of the programme. Chapter 5: Implications of the Programme The focus of this study was primarily on evaluating the processes involved in implementing the programme. However, some quantitative data on enrolment and achievement scores have been captured for the periods before the commencement of the CAL programme and after the end of the programme (in the case of phase-1 schools). This cannot be strictly treated as baseline and endline assessments and it would be tenuous to try to assess the impact of the programme, based on these figures. Nevertheless, they could be used as indicators on the implications of the programme, along with the qualitative data collected for the study. 5.1 Classroom Integration Integrating computers in classrooms is being studied in fair amount of detail, especially in the Western countries. Various researchers have come up with different models to understand the stages of development (Berenfeld, 1998; UNESCO, 2005; World Bank, 2003). What is common to all these models is that once the preliminary stage of learning to use computers is taken care of, the initial stage of integration would involve teachers: 1. Selecting multimedia content suitable to the cognitive level and interest of their students; and 2. Deciding when to use the content – for example, as an introduction to a new concept while teaching the concept; or as reinforcement after teaching. In the present study, we found during our visits that CDs were being used for reinforcement, without exception. We also found many instances where children were using CDs much below their cognitive level. Nowhere had the ideas/activities in CDs become infused with the regular classroom activities. Classroom and computer room were treated as two insular places. Computers have tremendous potential to help classrooms shift from learning by telling to learning by doing, as envisaged in the National Curriculum Framework (NCERT, 2005). If teachers are helped during the initial stages with some hand holding and they being to feel empowered, future possibilities using computers to transform classrooms can become real. 5.2 Gains for students As mentioned earlier, the reliance has primarily been on qualitative data to ascertain gains for students. We have been able to capture some quantitative data. We could not get reliable data for average students’ attendance at the time of launching the programme. Hence, it has not been included in this report. Quantitative data pertaining to enrolment are given below: Fig 5.1: Comparison of average enrolment of students in phase-1 schools 0.00 10.00 20.00 30.00 40.00 50.00 60.00 70.00 BGBGBGBGBG SC ST Minority Others Total % of students 2006-07 2008-09 It is interesting to note that there has been a marginal increase in enrolment of both boys and girls among ‘Scheduled Caste’ and ‘Minority’. Enrolment of ‘Scheduled Tribe’ girls has also increased. But, there is a dip in the enrolment of boys and girls belonging to the ‘others’ category. This has resulted in the total enrolment of boys and girls in 2006-07 and 2008-09 very nearly equal. Fig 5.2 Comparison of average enrolment of students in phase-2 schools 0.00 10.00 20.00 30.00 40.00 50.00 60.00 BGBGBGBGBG SC ST Minority Others Total % of students 2007-08 2008-09 While the trends in phase-2 schools are similar to that of phase-1 schools for ‘SC’ and ‘Minority,’ the enrolment of boys has increased in both ‘ST’ and ‘Others’ categories, unlike in phase-1 schools. However, it has to be borne in mind these trends in enrolment are affected by myriad factors and cannot be attributed directly to the CAL programme. During our school visits, in at least six schools, we were informed of specific instances wherein students have shifted from private schools, after the launch of CAL programme. We do not have reliable data on attendance and retention. But, all the teachers, parents and even children themselves informed us that attendance and retention have improved, after computers were introduced in the school. Data pertaining to scholastic achievement are the KSQAO scores of class 5 and class 7 students in 2005-06 and 2008-09. Karnataka School Quality Assessment Organisation has been conducting annual assessments of all government school children of classes 5 and 7, since 2005-06. Fig 5.3: Comparison of average KSQAO scores of students in Phase-1 schools 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 VVII 2005-06 2007-08 Fig 5.4: Comparison of average KSQAO scores of students in phase-2 schools 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 VVII 2006-07 2007-08 As can be seen from the figures, the scores have increased across the board. This has been the trend across the state and not peculiar to the CAL schools. Hence, it is not possible to draw conclusion about the programme’s impact on students learning achievement based on this data. Again during interactions with teachers and students and observation of students we were able to discern perceptible impact the CAL programme has made on students. • Children have picked up basic computer skills. All children are able to turn a computer on/off and handle a mouse. In a few schools, children have been taught to use word processer. • Their interest in school has piqued (though, this has not necessarily translated into learning). As mentioned earlier parents, teachers and children themselves reported that they do not like to miss school anymore. • Children display greater levels of confidence. Parents have cited specific anecdotes where they discerned that their children feel more confident. • Teachers of two schools said that their ability to comprehend instructions has increased. • They have learned to work and interact in a group. In three schools, teacher recognized that but for the CAL programme students would not have had an opportunity to work in a group. • Students have gained better exposure to English language. Many children and few parents felt the CDs helped improve English language proficiency. • Children are well disciplined inside the computer room – in some cases even without supervision. We observed this in all schools where children were using computers. 5.3 Gains for teachers CAL can support changes in pedagogy and there is documented evidence of such programmes changing the role of teachers – for example, in Chile, a CAL programme has helped create a more egalitarian relationship between teacher and students (Alvarezetal, 1998). Before this can happen, teachers need to see value in CAL. They would then be willing to invest time and effort to adopt to the technology. While there is no rigorous research-based evidence for this, experience points out that teachers need a lot of support and hand holding, till they are able to confidently use CAL meaningfully (Mythili.R, 2007). In the present study, we found that teachers in general are convinced that CAL programme is useful for students. Very few of these teachers have realized it could be a powerful tool for themselves. Among those who did, they listed the following as what they had gained from the CAL programme: • Helped improve their own knowledge of some Maths and Science concepts (Around eight teachers from two schools); • Gained fluency in English (two teachers from one school); • One teacher who is a resource person for other taluk teachers has used the CDs to get ideas for activities/preparing TLMs; • Computers have been used to enter time tables and marksheet (in two schools); and • Teacher of one school is preparing notes of lessons using computer. It is interesting to note that given similar inputs, these dozen or so teachers have been able to gain from the programme, while others do not seem to perceive gains for themselves. In the present study, we have not consciously attempted to probe this difference. But upon reflection, we felt the following factors that possibly could account for this difference: 1. Where support teacher had D.Ed qualification and has received training from APF, their understanding of the programme is far better and has been able to reach out effectively to the regular teachers; 2. Presence of at least three teachers in one school, who were self motivated enough to take initiatives to learn and use computers; and 3. Presence of either a pro-active head teacher or one who does not interfere with teachers’ work. It would be of academic interest to study these differences in greater detail in the future. The next and concluding chapter collates the findings of the study and the inherent lessons therein to make this and other such similar programmes more effective. Chapter 6: Conclusions This chapter looks at issues to consider for enhancing effectiveness of the programme in light of the findings of this study. In doing so, we have broadly focused on the program objectives. 6.1 Resources The implementing agency has been very diligent in providing resources, going beyond what was initially planned. Findings • In two of the twelve phase-1 schools, none of the computers are working. • In three more schools less than half of the computers provided are working. • In the remaining schools, either all computers are working (in four schools) or only one computer is under repair. • In the phase-2 schools, all computers were working in two schools. In the remaining schools, only two or three of the six computers were working. • Children tend to misuse headphones – biting on the wire or pulling it. Headphones were functional in only a few schools. • Each school was equipped with around 50-60 CDs. • Printer and computer meant for teachers remained unopened, in all but three schools. Man Power Resources • Support teachers are there in all of phase-2 schools. • Services of the support teachers have been continued in three of the twelve phase-1 schools where the programme has come to an end. Issues Maintenance of resources especially UPS and headphones proved to be a problem. Also, attending to repairs appears to be a major bottleneck despite the fact that APF has provided schools with a list of technicians and their contact numbers. As to the support teachers, there was an unanimous agreement that their presence is needed to continue the programme effectively. Generating funds for their salary and high levels of attrition among them are the two major issues facing schools. Suggestions Greater buy-in from the Government Department of Education may ensure better maintenance. Tying up with the Mahiti Sindhu Programme, where in operation, may help in attending to repairs. Greater synergy with the local Panchayat/self help groups would also be useful. As to support teachers, it would be untenable to continue paying them salary in the long run. Moreover, if the focus of CAL programme is on integration, it would be better if teachers were to manage the programme. Three or more regular teachers from a school could be identified, based on the interest they show, and trained for the programme. 6.2 Training A fairly rigorous five-day residential training has been provided by APF to support teachers. Findings • The support teachers have found the training very useful. They were all especially appreciative of the soft skills component of the training. • A majority of the regular teachers have not taken the on-site training provided by APF seriously. Issues High attrition among support teachers has led to some wastage of training. A majority of teachers have not taken ownership of the programme and are therefore apathetic towards it. Suggestion There is evidence from research that suggests a significant correlation between level of technical and pedagogical support for ICT use in teaching and increased ICT use by teachers (Nancy Law, 2006). Considering that the support teachers found APF training useful, the same may be offered to interested teachers. The training needs to be followed up with substantial hand holding over a period of time. After all, well trained teachers are the key to effectiveness of any programme in a classroom setting. 6.3 Follow-up and Feedback APF had appointed a coordinator for the monitoring of the programme. Findings • There has been frequent and periodic follow-up and feedback by the coordinators. • This has resulted in attending to repairs and other problems on an immediate basis. Issues The support teachers said they found the feedback from the coordinators helpful in keeping them motivated. But, there had been no interactions with the regular teachers. Suggestions During the project period, the coordinator can tie-up with the Cluster/Block Resource Person for monitoring visits. This could help build the capacity of the CRP and BRP whose job is to monitor the academic activities in schools and provide feedback to teachers. Once the externally funded project ends, the CRP/BRP can continue this role, without considering it an added burden. It is also imperative to talk to regular teachers, during each visit, even when the support teacher is there. Follow-up and feedback can also be brought under the purview of SDMC/Community. A framework can be evolved to identify components of the programme that can be effectively monitored by SDMC/Community Members. Such components that require pedagogic expertise can be left to the CRP/BRP. 6.4 Sustainability Sustainability has been the mainstay of this programme. Community was expected to match the external grant. Assurance was taken from them to continue the programme, after funding stops. Essential documentation was also factored in. Support teachers in schools were trained to maintain records. Findings • SDMC President in most schools is taking keen interest in trying to mobilize funds to continue the programme. • Schools have ceased maintaining records pertaining to the programme, even where APF trained support teachers are continuing. Issues Apart from parents, other community members in most schools are unaware of the programme. One school (Doddamaralwadi) has collected money from parents for the programme. Unfortunately none of the computers in the school are working for more than six months. Documentation utility has not been adequately perceived by either support teachers or regular teachers. Suggestion Networking with local youth/women groups that are active, could help mitigate the lack of visibility to an extent. These groups can also canvas for reallocation of some of government budgetary provisions for schools towards CAL programmes, as well as generate external funds, if necessary. Tying up with BRP/CRP for monitoring may ensure better compliance for maintenance of documents. 6.5 Classroom Environment Experts are unanimous that innovation cannot occur in a teacher centric environment. CAL has potential to directly improve learning and indirectly increase attendance by making school a more attractive place. Moreover, it offers opportunities for students to become co-operative learners, when they use a computer as a group. Findings • There is anecdotal evidence to suggest improved attendance, attributable to presence of computers in schools. • Teachers have noted that children have learned to work in groups while using computers. • There was not a single instance of grouping in regular classrooms. Issues While initial enthusiasm could have resulted in improved attendance, sustenance of students’ interest in the present set-up is going to be a challenge. There is little transfer of learning from computers to regular classrooms. What children learn through computers is not being adequately harnessed within classrooms. Suggestion Emphasis, during the training programme and other sensitization efforts should be on learning culture rather than use of computer per se. Repeated exposure and discussions with teachers may improve classroom environment in tune with that envisaged in NCF, 2005. 6.6 Pedagogic Approaches CAL provides students with opportunities to learn in new ways. While computers are pedagogically neutral, teachers can use multimedia content to help students interpret information and not receive it passively (NCERT, 2006). CAL can be imaginatively harnessed to promote reasoning, creativity and critical thinking in children. Findings • Classrooms continue to remain didactic. • There is heavy reliance on textbooks both by teachers and students. • CDs are used to reinforce what is taught in classrooms. • There was no evidence of hands-on, activity based learning as promulgated in CDs, being transferred to actual classroom teaching. Issues CDs, essentially designed for a single user, are being used by four or more children. Purpose of using CAL is unclear to most support teachers and regular teachers. Suggestion The pedagogic contribution of multimedia content, whose function is similar to that of classroom teaching, has to be reinterpreted in the light of newer evidence from cognitive science on how people learn. There has to be a shift from prescriptive to a more engaging approach to learning. Teachers need to be provided with adequate support and training to make this possible. Otherwise, there is the risk of computers becoming an expensive add-on, instead of an integral part of a new pedagogy as envisaged. External monitoring and evaluation had not been designed into this programme at the outset. It could presumably due to the fact that APF has experience in running such programmes. Nevertheless as experiences around the world show, engaging the services of an external agency to conduct formative evaluation and thence to monitor progress, goes a long way in improving the effectiveness of a programme. The larger value of CAL rests in their capacity to motivate students, increase equity of access and reduce time needed to accomplish a given set of objectives. CAL is not just a tool for learning but represents a new environment for teaching/learning. This can be realized only if the programme can ensure buy-in from teachers. This, in turn, can happen only if long term commitments in terms of time and efforts are made. References Azim Premji Foundation 2002. A Study to Evaluate the Impact of Community Initiative, www.azimpremjifoundation.org Linden et al. 2003. Computer-Assisted Learning: Evidence from a randomized Experiment. Poverty Action Lab Mythili.R, 2007. Integrating Computers in Classrooms Paper presented upon invitation at Rajiv Gandhi Foundation, New Delhi Nancy Law (Ed). 2006. The use of ICT in Teaching and Learning. IEA Second Information Technoogy in Education Study, www.sites2006.net NCERT, 2006. Position paper: National Focus Group on Educational Technology, www.ncert.nic.in Rishikesh, S.B. 2005. Computer Aided Learning Program - A Longitudinal study in Karnataka A Qualitative Report. Azim Premji Foundation, Bangalore Teachers' ICT Skills and Knowledge needs, Final Report to SOEID.2007. The Robert London University, United Kingdom UNESCO. 2003. Building Capacity of Teachers/Facilitators in Technology-Pedagogy Integration for Improved Teaching and Learning. www.unesco.org Venezky and Davis. 2002. Quo Vademus? The Transformation of Schooling in a Networked World. OECD/CERI World Bank Report. 2003. Challenges for developing countries. World Bank, Washington, D.C Appendix 1 Data Summary: Phase 1 Schools Data GHPS, Ajjanahalli GHPS, Bidaluru GHPS, Doddamaralwadi GHPS, Gudimaranahalli GMPS, Harohalli Situation Very far away from the main road Away from the main road Little away from the main road Few km. from the main road Near the main Road Building and school surroundings Surroundings not very clean (around the class) Sprawling campus. Fairly good Clean Very clean, spacious Good, Spacious Classroom Ventilated Okay Clean Clean & ventilated Ventilated Library Yes - not used regularly - Yes Yes - used regularly Yes Computer room Clean, very well maintained. - - Neat, well maintained Clean, well maintained Boys Girls Boys Girls Boys Girls Boys Girls Boys Girls No.of students enrol. 145 69 92 108 158 119 155 180 197 193 No.of students pres. On day of visit 103 61 81 97 142 108 150 172 172 170 Nature of the school Single grade Single grade Single grade Single grade Single grade KSQAO result V VII V VII V VII V VII V VII 2005-06 56% 50% 49% 40% 57% 58% 93% 94% 55% 57% 2007-2008 79% 86% 56% 54% 77.3% 77% 98% 97% 71% 74% CAL Yes Yes Yes Yes Yes Radio Yes Yes Yes Yes Yes TV No - - No No Data GHPS, Ajjanahalli GHPS, Bidaluru GHPS, Doddamaralwadi GHPS, Gudimaranahalli GMPS, Harohalli No.of computers provided 7 6 6 7 7 Working 5 0 0 5 6 Not working 2 6 6 2 1 Whether there is support teacher Yes No No No No If yes, whether the support teacher is APF trained Yes - - - - If no, who’s handling the computer classes Classes are not going on - One science teacher from the school got training from APF for a day and he helps the children The PT master with a computer course guides the students Other skills learned through computers by students Computer games, typing, drawing, parts of computers - - Kannada typing Painting, typing, on, off Utilization of computers by teachers One young teacher comes forwards to learn basics in computers - - Teacher learning to type and keep notes they require Time table, marks sheet Phase - 1 Schools Data GHPS, Kachuvanahalli GHPS Kaikondahalli Govt HPS, Karahalli GMPS, Madabal GHPS, Shivanahalli GHBS, Vijayapura GHPS, Vishwanathapura Situation Interior village, Far away from main rd. On the main road Near the main road Interior village Near the main rd. Main road Close to the main road Building and school surroundings Clean, ventilated Well maintained Construction was going on Clean, Spacious Clean, ventilated Old building. Fairly well maintained Painting was being done Classroom Well maintained, Garden in the compound Cramped Okay Clean & ventilated Clean Okay Under maintenance Library - Yes Yes Yes – not used regularly Yes – not used regularly - Good Computer room Clean, well maintained Good Clean Not very clean, too much noise from the UPS Clean, spacious and well maintained Clean Good Boys Girls Boys Girls Boys Girls Boys Girls Boys Girls Boys Girls Boys Girls No.of students enrol. 52 53 0 373 148 79 83 93 66 83 348 0 91 98 No.of students pres. On day of visit 42 47 326 132 74 81 89 65 80 323 - 86 91 Nature of the school Single grade Single grade Single grade Single grade Single grade(3- 7) Multigrade (1-2) Single grade KSQAO result V VII V VII V VII V VII V VII V VII V VII 2005-06 67% 68% 59% 49% 45% 46% 53% 60% 57% 73% 23 % 36% 71% 59% 2007-2008 88% 89% 67% 74% 84% 76% 75% 68% 73% 80% 69% 69.2% 72% 81% CAL Yes Yes Yes Yes Yes Yes Yes Radio Yes Yes Yes Yes Yes Yes Yes TV No Yes - Yes No - - Data GHPS, Kachuvanahalli GHPS Kaikondahalli Govt HPS, Karahalli GMPS, Madabal GHPS, Shivanahalli GHBS, Vijayapura GHPS, Vishwanathapura No.of computers provided 6 6 7 5 5 7 7 working 6 3 7 4 3 3 7 Not working - 3 0 1 2 4 0 Whether there is support teacher No No Yes No Yes No No If yes, whether the support teacher is APF trained - - Yes - - - - If no, who’s handling the computer classes All 5 teachers takes active interest and take turns in assisting children Librarian takes care of the class Subject Teacher The subject teacher accompany the students A teacher has been appointed by the HM to assist the children in the comp. class Subject Teacher Subject Teacher Other skills learned through computers by students 7- English pronunciation and sentence formation 5- Kagunitha and paint Drawing on/off, knows typing their names and painting Drawing, Switching on/off, MS paint, word Painting, drawing Typing, excel, paint Typing, on/off Paint, on/off, typing Utilisation of comp. by teachers - - PE teacher does office related work - - - Prepared TLMs after seeing CDs. Teacher is a RP and had used ideas/activities in the CD to train trs’ in the taluk. Mistake in the CD on tenses Phase-2 Schools Data GHPS, Arudi GHPS, Bashettihalli GHPS, Hosahalli GHPS, Gunjur GHPS, Kantanakunte GHPS, SS Ghati GHPS, Thippur GHPS,Varthur Ramagondanahalli Situation Interior village Near the main rd. Interior village on the main road Near the main rd. Situated in valley. Interior village on the main road Building and school surroundings Good Well maintained Okay Okay Well maintained Wet roof Well maintained Yes Classroom Clean & ventilated Entrance to be repaired Crack in VII class room Clean Kept clean Ventilated, spacious Clean & ventilated Ok Library Yes – not used regularly Yes – not used regularly Yes – not used regularly Yes Yes – not used regularly Yes – not used regularly Yes – used regularly Yes Computer room Clean, well maintained, UPS, Chairs, CDs Clean, well maintained,UPS, Chairs, CDs Clean, well maintained,UPS, Chairs, CDs Clean Clean, well maintained,UPS, Chairs, CDs Clean, well maintained,UPS, Chairs, CDs Clean, well maintained,UPS, Chairs, CDs Yes Boys Girls Boys Girls Boys Girls Boys Girls Boys Girls Boys Girls Boys Girls Boys Girls No.of students enrol. 129 129 230 211 119 92 132 151 117 106 96 119 114 139 130 176 No.of students pres. on day of visit 117 121 208 194 105 80 120 143 102 100 79 113 110 136 117 170 Nature of the school Single grade Single grade Single grade Single grade Single grade Single grade Single grade Single grade KSQAO result V VII V VII V VII V VII V VII V VII V VII V VII 2005-06 25% 50% 68% 52% 35% 35% 45% 46% 52% 55% 61% 68% 48% 47% 57% 49% 2007-2008 21% 46% 73% 83% 55% 62% 72% 65% 83% 77% 24% 27% 74% 77% 81% 85% CAL Yes Yes Yes Yes Yes Yes Yes Yes Data GHPS, Arudi GHPS, Bashettihalli GHPS, Hosahalli GHPS, Gunjur GHPS, Kantanakunte GHPS, SS Ghati GHPS, Thippur GHPS,Varthur Ramagondanahalli Radio - Yes Yes Yes - - Yes Yes TV - Yes Yes - - - - - No.of computers 7 7 7 6 7 7 7 7 working 2 with UPS 2 3 6 3 (Ups working) 3 3 7 Not working 4+1(not installed) 4 0 3 1 not installed 2 0 Whether there is support teacher Yes Yes Yes Yes Yes Yes Yes Yes If yes, whether the support teacher is APF trained Yes No No No Yes No Yes No If no, who’s handling the computer classes - - - - - - - - Other skills learned through computers by students Draw and handling computers Painting, game, type, shut down, on, save, inserting Type, drawing, game, shut down, painting - On, game, paint, shut down, open, close, typing, creating a file and save On/off paint type, shutdown some insert CD;s, games and save They like to handle computers On/Off, drawing, computer Utilization of computers by teachers - - - - - - - - Appendix-2 Brief Profile of the Schools Phase 1 Schools GHPS, Ajjanahalli (Magadi cluster) Ajjanahalli is a very small village in Magadi taluk. The school was not prepared for our visit. Some of its older students were not available as they had to work in their fields. The school has a few young teachers, who showed a lot of interest regarding CAL Programme. The teachers who had been here for more than two years saw lots of improvement in their students especially in pronouncing both Kannada and English. They believe the CDs are improving the learning capability of slow learners. The young teachers are ready to learn computers if provided with training. Children enjoy the guidance of the support teacher. In front of each computer a time table specifying the name of the student and classis displayed. This is helping the children to settle down quickly. Other than viewing CDs the children are taught painting and word. The children watch CDs based on the timetable, but sometimes they choose the CDs they wish to watch. Support teacher appears to enjoy his role as. He has taught the children the basics that was taught to him during APF training. The only problem he said he faces is a lack of service personnel to assist him in maintaining the computers. The support teacher has maintained a book which states the requirements by the regular teacher to improve computer based learning. SDMC members/parents showed lot of interest in CAL programme. They felt that their children are learning more from the CDs. One of the parents said that his child made him understand how earth rotates after he learned it from a CD. They are confident that their children are going to improve their learning capability because of these computers. They are ready to sell coconuts from their school to pay the support teacher when necessity arises. They see the children getting excited when they talk about the computers. Parents have been shown how a computer works once in the school. A few of them come to see their children using computers once a while. They said “instead of getting knowledge from one teacher these children are getting knowledge from two teachers (the second teacher being the computer). GHPS, Bidaluru (Devanahalli cluster) The school is away from the main road. The campus is sprawling but not maintained very clean. The in-charge HM appeared sincere but sounded helpless about attending to the electrical wiring problem, which has made the computers dysfunctional. The SDMC President was clueless about even the presence of CAL programme in school. A member is a local youth group also met with us. He assured the HM of attending to the problem through his organisation. GHPS, Doddamaralwadi (Kanakapura cluster) The school is a bit far away from the main road. The school premises were very clean. Apart from the computers provided by QUEST’s CAL programme, the Government has also provided the school with six computers. None of these twelve computers are working. The head master came across as completely disinclined towards academic matters. The SDMC President is a graduate and comes to the school everyday and teaches students of classes 5,6,7 for 2-3 hours. The school has collected Rs.100/- from every child in the school for the CAL Programme. There was a perceptible resentment among teachers against both the head master and the SDMC President. The teachers appeared to be committed and sincere. The students seem to be fond of their teachers. Students were articulate and well groomed. They informed us that an outside agency has set up a library in their school. The librarian keeps the library an hour before school opens and an hour after school closes. Most children said they are fond of reading books in the library. GHPS, Gudimaranahalli (Magadi cluster) It is a small but spacious school and very neatly maintained. The HM was prepared for us. The SDMC members and the parents were hand picked by the HM to interact with us. One of the parents specified that if they could teach practical skills with computers it would help the children fetch a job. The community/SDMC felt that the electricity bill is more than Rs.500/- which is difficult for them to pay and no community member is willing to pay or collect money for this cause. The library is housed in the computer room. Half of the class was made to use the computers and other half was made to sit in the centre referring to the library books. The financial support for the computer building was provided by an old student who has now settled in the United States. There is no support teacher now. But the children strongly felt that they were able to learn more from the support teacher as the classes were regular. One male teacher had attended one day training at APF. He is the one who is helping the children. The other teachers showed enthusiasm but completely lacked confidence about handling computers. They believed that the students are more knowledgeable when it comes to computers. They told us that they learn from their students. Some of them have prepared materials/activities after seeing the CDs. They strongly felt that one of their staff should be given proper training by APF which could permanently help the school. Few of these teachers have been provided school based training by APF, but they felt that it did not help them much. GMPS, Harohalli (Kanakapura cluster) This school is on the main road. The school surroundings were clean. Classrooms were well ventilated. The PE teacher has some basic training in computers, but lacks pedagogic knowledge. He is also not too well versed in the use of computers, beyond basic operations. Teachers appeared disinterested in CAL programme. When questioned what they liked about the programme, they said children enjoyed working on the computer while they got free time. Children appeared confident in handling computers thanks to the support teacher’s (Ms. Ramya) inputs. Parents/SDMC members were happy with the CAL programme and that their children were learning computers. The SDMC President has a specially abled child in the school. So, he has been taking special interest in the school affairs. Community members feel it is untenable to ask for continued donation to pay the salary of support teacher. GHPS, Kachuvanahalli (Kanakapura taluk) Kachuvanahall, is a small village tucked away in the paddy field about 20 kms away from the Kanakapura-Mysore road. We went to Kachuvanahalli unannounced. The school has a well maintained garden. The corridors were clean. Children were sitting in small groups revising their lessons for the upcoming semester I examination. The school has four male teachers and a headmaster. The teachers came across as a highly motivated group of individuals working together well. Computer room was well maintained. At the time of visit, class 7 students were being shown the CD “Shakthiyodane Sarasa”. The children were sitting on the floor and watching the CD on one monitor. Upon being asked to see children use the computer themselves, children were made to work in groups of threes and were made to watch CDs of different subjects. The students were quite well versed with handling computers. Teachers have made use of the computers to teach subjects like English, Mathematics and Science. They opined that the computers helped them teach children going beyond their syllabus. For example: speaking in English fluently; teachers have also been giving students the English equivalent terminologies in mathematics and science which they picked up from the CDs. The teachers also felt that the students had an opportunity to discuss in small groups of three about a problem/question which other wise they would not have got. Teachers are adept in handling computers and have been using computers to prepare question paper, for maintaining records as well as invitation cards for school function. They appreciated the APF staff for periodic monitoring and feedback. The overall impression was that even after the support teacher had stopped coming to Kachuvanahalli the teachers have continued good work. They appear to be making a conscious effort to link what is being taught in classroom with the available CDs. With more training hand holding and guidance, integration could be more successful and children would reap the benefits. Students were also excited to talk about what they learned from computers. The children on their own said that pronunciation of English in CDs sounded better than their teachers and they learned how to form sentences in English from CDs. Children have also learned MS Paint and were able to use word-pad for typing. They seemed to have liked and learned a lot from Ms.Shylaja, a APF trained support teacher. The students had come to school during the summer holidays from 10.00 am to 2.00 pm to play on the computer. Kachuvanahalli is a typical small village with a small farming community. Talking to the SDMC president, it was evident that most land lords were in the city while only landless labourers and very small farmers live in the village. Hence collecting money to pay Rs.500/- for electricity charges was an uphill talk for them. Even though both the teachers and the SDMC President felt a support teacher was necessary, they did not think they would be able to collect enough money to pay. The parents felt that CAL programme had helped their children compete with those from private schools. GHPS, Kaidondahalli (Bangalore South 4 cluster) This school is on the main road. Classrooms are rather cramped with three sections in one room. Teachers appeared interested in CAL programme. They said getting financial support is not a problem for them. (There are a number of IT companies in the vicinity). The support teacher was being paid Rs. 3,500/- per month. Even so, the support teacher was not willing to stay on. Children were extremely well behaved in the computer room and were handling the computers independently without any supervision by teachers. Students of class 7 informed us that they had used internet in a cyber café for their social science project. They had made notes and used that information in preparing a chart. Most children also know to type, save files and create folders. We interacted with one community member. He said he had attended APF’s meeting two years back but could not remember what the meeting was about. Surprisingly, the parents of four children who we met were not even aware of the CAL programme. GHPS, Karahalli (Devanahalli cluster) This school is on the main road. Classes 7 and 8 are conducted in a private house under construction opposite to the school building. While the school surroundings are reasonably clean, the house where classes 7 and 8 children attend classes has a big, open sump filled with water. When we entered the school at 10.15 a.m, we saw the PE teacher caning late comers. The school has been able to garner funds through a local contributor to continue the services of the support teacher. None of the regular teachers, except the PE teacher (who has undergone a certificate course in computers) have taken any interest in the computers supplied to their school. Only two teachers have seen a couple of CDs as the time APF coordinators had provided training in their school. Children were able to handle computers with confidence. The parents were happy that their children are getting computer education but were clueless about the financial implications involved. GMPS, Madabal (Magadi cluster) This school is a little away from the main road. The school is to celebrate its centenary year. The HM was not informed about our visit. The school has seven female teachers and two male teachers (including the HM). The computer room was very untidy. The children were asked to dust it as we entered it. A television set was placed in the middle of the room. One computer and printer supplied for the teachers’ use were not even opened from their cover. As long as the support teacher was there the computer classes were regular. The UPS was making shrilling noise which made it difficult to be in that room. But the children were not bothered with the noise and they were fully involved in watching their CDs. The HM told us that this is how it has been for the past few months. The support teacher had taught the students to operate the computers and they were doing so independently. The students have learned drawing and painting from their support teacher. They looked very confident in handling the computers. Only two subject teachers have watched the CDs that too, one or two. They were unable to recall the names of these CDs. They said group activity has increased among children as they helped each other. They appeared disinterested in guiding their students to computer room as they feel that their workload is already more. They also said that because of computers, Edusat and radio programme it is difficult for them to finish their syllabus! They wanted a support teacher who can take care of the computer room and handle computer classes. Teachers do know how to switch on/off a computer. They say they are ready to learn more but are not ready to undertake computer classes for the children on regular basis. One of the teachers said the CDs are helping the slow learners. Students enjoy the computer classes but they said the frequency of these classes had reduced drastically after the support teacher left the school. They enjoy the Maths CD as it has games and makes it easy for them. One of the students whom we met had pronunciation problem and we were told that with the introduction of computers, he has learned to speak more clearly. Community members other than parents are unaware of this computer programme. The parents are happy to see their children learning from computer and CDs. Their only concern was that they do not have funds to continue their APF trained teacher. GHPS, Shivanahalli (Kanakapura cluster) This school is situated on the main road near Kanakapura village. It is a spacious school and is well ventilated. The computer room is near the primary building. The school has appointed a teacher who has studied here. Due to erratic power supply children are not able to attend the classes regularly. Students of higher classes are not able to go to computer classes this year as their classrooms are away from the primary building where the computers are kept. They say they miss their computer class. The teachers enjoyed first and second standard CDs which helped the children to learn alphabets. They suggested that instead of dolls, human figures can be used in CDs, which could make it more realistic. The language CDs are enjoyed both by teachers and students. They said activity based learning through computers could help the students understand concepts clearly. According to the teachers, the CDs are helping children to improve their language skills. They felt that there are not adequate Social Science CDs. They require maths CD for the first and second standards. The teachers help the support teacher select CDs that have to be viewed by students. The children are taught to paint and draw. They are confidently handling the computers. Parents are ready to support the CAL Programme as they could see the children learning more from computers. One of the parents told us that her children try to draw the animated characters they see in the school computers and talk like them at home. GHBS, Vijayapura (Devanahalli cluster) This school is on the main road. Classrooms are spread around in separate rooms and some of them are cramped. Teachers appeared to show very little interest in computers or viewing CDs although one has undergone a five-day computer training programme sponsored by the Government. Children informed us that they have not been to the ‘computer class’ ever since the support teacher left a month back. Parents of only three children could come to meet us on the day we visited the school. Among them two fathers did not interact at all. One woman did all the talking. She was an alumnus of the school and felt rather proud of her alma mater. She said she helps her children with their studies at home. She also informed us that her children have learned to handle computers well, from the support teacher. GHPS, Vishwanathapura (Devanahalli cluster) This school is close to the main road. Building painting was being done. The classrooms are under maintenance. There are nine teachers with 200 plus students. One teacher who lives in the neighbourhood, shows keen interest in CAL Programme. Other teachers have also seen CDs pertaining to their subject. Two of them who have computers at home are proficient in the use of computers. Computer skills were taught to class 7 students, who were interested. A couple of them had done a project using power point. According to PT teacher, one girl had informed him that her prestige at home increased after she learned how to use computers and taught her convent educated brother at home. The maths teacher said she maps CDs with lessons she has taught. Phase 2 Schools GHPS, Arudi (Doddaballapur cluster) This school is in an interior village, not very connected to Andhra Pradesh border. School is well maintained. Support teacher has undergone software training and is a graduate. He has also been trained by the APF. He has written a book on computers for children. He wants to get it printed. Teachers said they do not have time to view CDs. Children are able to handle computers confidently. SDMC members are co-operative and try to help the school in whatever way they can. GHPS, Bashettihalli (Doddaballapur cluster) This school is very close to the main road and very well maintained. Within the school compound, there is huge tree, below which the school has built cemented steps. Food is served to children here and the teachers also conduct classes once in a while, below this tree. The in-charge HM is very enthusiastic and sincere and so are the teachers. The support teacher is D.Ed trained along with the computer training. He was very enthusiastic and sincere. He said he has scored 92% in the D.Ed and awaiting appointment soon. He was guiding children to learn through CDs. He appeared successful conveying concepts to children. Two teachers who have computer at home are making use of the CDs occasionally. Maths teacher taught place value on CDs. Children had already viewed CDs and the teacher used the examples from CD to teach. The social studies teacher taught the ‘weather’ lesson in similar fashion. The teachers opined that they need more CDs in social studies. They used CDs to be syllabus specific like radio lessons. Children enjoyed the CDs and seemed to have learned concepts using CDs. Mathematics teacher taught set theory and asked children to make sets using things that were outside the classroom. SDMC members and community members were proud of the school and supportive of the school. GHPS, Hosahalli (Doddaballapur cluster) This school is in an interior village. School surroundings were fairly clean. Teachers informed us that CDs were supplied to them twenty days ago and therefore had not time to see the CDs. Support teacher had PUC qualification but not trained by APF. He knows the basic operation of computer but not had formal training. Children have learned to open and shut down computers and were highly enthusiastic. Parents are happy with the programme and willing to continue. GHPS, Gunjur (Bangalore South 4 cluster) This school is away from the main road situated in a sprawling campus that houses the high school. Computer room provided with grills with community’s help as there was an incidence of theft. The support teacher appeared lackluster. She said computers to be provided with more games. She installed Tally in the school computer, as she wants to learn it. Class divided into 2 groups. One uses library while the other group goes to the computer class. All teachers have seen one or two CDs. Beyond this, they did not envince interest in CAL programme, except for the in-charge HM. Children appeared to be very fond of their Kannada teacher, also the in-charge HM who has shown videos on Savarkar to children. GHPS, Kantanakunte (Doddaballapur cluster) This school is close to a bus stop. School surroundings were clean. The in-charge HM did not appear to be cooperating with the support teacher. Support teacher is D.Ed trained and also has computer training. She said that she is waiting for another job. Teachers seem interested and most of them have seen atleast a few CDs. A few of the parents visit the computer room and watch their children operate the computer. GHPS, SS Ghati (Doddaballapur cluster) School is situated in a valley, away from the main road. The school surroundings were clean. Teachers claim to be interested but most have not viewed CDs. They said they do not have time to view CD. The support teacher has been appointed only a month prior to our visit. He has basic computer training but has no teaching background whatsoever. Children are enjoying their time at the computers. SDMC President is very active. Parents informed us that children’s interest in coming to school has increased, after computers were introduced. One mother who is a graduate volunteers her time at school. GHPS, Thippur (Doddaballapur cluster) School is situated close to Andhra Pradesh border and is in an interior village.School is maintained very clean. Support teacher is D.Ed trained and has also been trained by APF. Most teachers have not seen the CDs supplied by APF. Teachers opined that with Edusat, Radio programme and CAL programme they are not finding time to complete portions. They said that all three programmes are interfering with their work! However, the mathematics teacher has seen most CDs and is attempting to integrate in his classroom teaching. He has a Master in Kannada and has appeared for KAS. He is dedicated and sincere. Children operate computers confidently. Parents are quite happy, but did not have anything specific to add. GHPS, Ramagondanahalli (Bangalore South 4 cluster) School is located on main road and well connected by buses. The campus was huge and very well maintained. The head master told us that a local donor is very generous with financial support. The school has also been adopted by an NGO which has donated note books and stationery to all children. This NGO has also been conducting weekly English classes for an hour. All teachers of the school appeared very enthusiastic about CAL Programme. Two teachers had recently been trained by Intel for 45 days, sponsored by the Government of Karnataka. Children handle computer confidently. Parents said they have involved themselves more in school after computers were introduced. They informed us that many of them visit the school whenever they find time. A few parents told us that their children are asking them to buy computers at home. They also informed us that their children refuse to take leave especially when they have computer class. Appendix -3 R.V. Educational Consortium Rashtreeya Sikshana Samithi Trust Jayanagar, Bangalore-11 Tools: EVALUATION OF COMPUTER ASSISTED LEARNING PROGRAMME OF QUEST ALLIANCE Name of field assistant: Date of observation: Form 1: School Details 1. Name of the school : 2. Nature of school : Multi grade/single grade 3. Name of the cluster : 4. School Strength : Boys Girls No of students enrolled No. of students present on the date of visit 5. Students’ profile : 6. Class wise details of students’ strength: Class 4 Class 5 Class 6 Class 7 2006-07 2007-08 7. Performance of students in KSQAO’s assessment (% of marks): Class 5 Class 7 2005-06 2007-08 8. Kindly observe the school premises and fill in the table: 8.1 Is there adequate space around the school? Yes No 8.2 Are the classrooms spacious? Yes No 8.3. Are the school surroundings clean? Yes No 8.4 Are the classrooms clean? Yes No 8.5 Is there a library? Yes No 8.6 Is the library being utilized? Yes No 9.1 How many computers are there? 9.2 Are all the computers in working condition? SC ST MIN OTHERS Boys Girls 10. Kindly refer to the records maintained in the computer room and place a tick if these records are maintained: 10.1 Number of students using computers : 10.2 Class-wise time-table for computer time : 10.3 Subject-wise time allocation : 10.4 Details of CDs viewed by students : 10.5 Details of CDs viewed by teachers : Form 2: Guidelines for Classroom Observation 1. Whether the classroom is : Multi grade Single grade 2. Subject taught during observation : Kan Maths Science Social Science Eng 3. Class(es) handled by the teacher: --------------------- No. of students: ------------------------- 4. Gender of the teacher : Male Female 5. Classroom Environment 5.1 Is the classroom clean? Yes/No 5.2 Is there a seepage-free even flooring? Yes/No 5.3 Is there a secure, leak-proof roof? Yes/No 5.4 Is the classroom well ventilated? Yes/No 5.5 Are there benches / desks for children? Yes/No 5.6 If no, how are the children seated? In rows / In groups 5.7 Did the children keep their bags stacked Yes/No 5.8 Is there a table / chair for the teacher? If yes, how often did the teacher sit on the chair? Yes / No Rarely / often / very often 5.9 Are there visuals displayed? Yes/No If yes, are they: Student created/teacher created/purchased? 6. Classroom Management 6.1 Where was the class conducted? Inside the classroom / outside the classroom 6.2 If inside the class, were the students seated in one place throughout the class? Yes/No 6.3 Did the teacher attend to the seating arrangement of the children? Yes/No 6.4 How frequently did children write? Not all / once in a while / often 6.5 While the children wrote, did the teacher go around checking their work? Yes/No 6.6 Did the teacher expect complete silence in the classroom? Yes/No 6.7 Did the teacher spend extra time with slow learners? Yes/No 6.8 How did the teacher engage students who finished their work ahead of others? 7. Teaching (provide details for all items under this category) 7.1. Games played 7.2 Activities given 7.3 Use of : • Models • Charts • Textbook • Other books • Newspaper • Magazines • Computer / CD • Any other teaching aid(please specify) 7.4. Stories / anecdotes / examples given 7.5 Dramatization / Role play done 7.6 Use of blackboard 7.7 Drill work / practice given 7.8 Use of surrounding environment 7.9 Written / oral work given 7.10 Grouping of children 8. Integration with CDs (Please provide the following details if this is being done) 8.1 Link with previous CDs viewed 8.2 Instruction for viewing CD relevant to the topic being taught 8.3 Background information / Preparatory work for students before viewing CD 8.4 Suggestions/Ideas/Discussions based on CDs 8.5 Any Other Form 3: Guidelines for Observation of Computer Class 1. Class 2. Subject 3. No. of students 4. No. of students per computer 5. Seating arrangement of students 6. Title(s) of CD(s) being viewed 7. Stage at which CD is being used: (before, during or after a topic has been taught) 8. Whether all students are viewing the same CD simultaneously If yes, whether they are doing so in a single group or in small groups If in a single group, how the other children are occupied 9. Level of interaction between support teacher and students 10. Level of participation by students 11. Intervention by support teacher 12. Confidence shown by students in handling the computer 13. Interaction among students 14. Motivation shown by students In using CDs 15. Discipline maintained by students Form 4: Guidelines for Regular Teachers’ Interview 1. Name of the teacher : (Need not be filled, if so desired) 2. Gender : Male / Female 3. Age : 4. Academic / Professional Qualification: SSLC/PUC/Degree TCH/B.ED/M.ED If graduate, mention subject : __________ 5. Experience of the teacher : in LPS______yrs in HPS_____ yrs 6. Whether the teacher has had computer training: Yes / No Duration of training___ 7. How was it useful? Did they gain any new insights about how children learn and how they can be taught? 8. Did they have prior exposure to computers? If No, do they feel confident about Handling a computer now? 9. What CDs have they viewed? 10. Which ones did they like? Why? Which ones did they NOT like? Why? 11. Do they discuss with other teachers about the CDs? If yes, what is the nature of their discussion? 12. What inputs do they give the support teacher? 13. What are their ideas on integrating computers in classroom teaching? 14. What value do they see in the CAL programme: - For themselves - For students 15. What is their perception of CAL programme vis-à-vis : - its general utility - its impact on student learning - community involvement - role of support teacher - role of external agency 16. What were their expectations when the programme was launched? Have they been met? 17. What has been their learning in this programme? 18. How can this programme be improved to better cater to theirs and students needs? Form 5: Guidelines for Interviewing support teachers 1. How do they view their role as support teacher? 2. What was their experience at the training programme? 3. What support did they receive from APF? 4. How does the HM support them? 5. What interactions do they have with community members? 6. Do department officials (CRP, BRP’s or BEO’s) show interest in the programme? Have any of them visited the computer room and taken note of what is happening? 7. What kind of inputs do they receive from teachers? 8. Do teachers view CDs? If yes, how frequently? 9. Which CD titles are preferred by: - Students - Teachers Which are their personal favourite CDs? 10. Other than viewing CDs are students taught computer skills? 11. Are all students actively involved while using computers? 12. Do they ask questions? If yes, what kind? 13. How do they arrange for viewing CDs? 14. What are the challenges/problems they face? How do they overcome them? 15. Whom do they turn to when they run into problems? 16. What is their perception of CAL programme vis-à-vis: - its general utility - its impact on student learning - community involvement - role of support teacher - role of external agency 17. How can computers be better utilized? 18. What do they like most about this programme? Form 6: Guidelines for students’ Interview 1. Do they like coming to school? Why? 2. Do they like computers? 3. What do they like to do most in the computer room? 4. Which CDs do they like most? 5. What have they learned in computers? Form 7: Guidelines for Interviewing Parents/SDMC members 1. Are they satisfied with: (i) the schools? (ii) the teachers? - Why or why not? 2. Is the education provided by school relevant to their children? 3. What is their opinion on CAL programme? 4. What changes have they seen in their children, after computers were introduced in the school? 5. What were their expectations from the programme? Form 8: Guidelines for interviewing Community members 1. Are they happy with the CAL programme? Why? 2. What has been their role in the programme? 3. How do they plan to continue the programme in the absence of outside support? 4. How have the HM/teachers co-operating with them for this programme? 5. What has been the role of the department officials in the programme? 6. How do they monitor the use of computers and utilization of funds? Form 9: Leading questions for discussion with APF personnel 1. Why was this programme conceptualized? 2. How is it different from APF’s existing CAL programmes? 3. What is the rationale for appointing support teachers in lieu of involving regular teachers? 4. What were the criteria for appointing support teachers? 5. How were they selected? 6. What was the frequency, duration and nature of training the support teachers? 7. What was the content of the training programme? 8. What were the modalities for feedback/ follow-up of these programmes? 9. How were regular teachers involved in the programme? 10. What support was rendered to them? 11. What was the role envisaged for the community? 12. How was the community moblised? 13. What were the steps taken for resource mobilisation? 14. What was the process put in place for monitoring of the programme? 15. How was the evaluation of the programme conceptualized? 16. What were the challenges faced during implementation of this programme? - Launching stage - Implementation state - Monitoring and feedback An Evaluation Study for ‘Lifelines for Education’ – project in West Bengal Final Report Submitted by: ORG Centre for Social Research (The Nielsen Company) 2nd & 3rd Floor, Sree Rama Deevena, No. 21, Ulsoor Road, Bangalore – 42 Date: 8th December 2009 Table of Content Chapter One INTRODUCTION 1.0 Background 1.1 Objectives of the Study 1.2 Study Methodology 1.3 Scope of Work 1.4 Study Limitations 1.5 Sampling Framework 1.5.1 Selection of schools and teachers 1.5.2 Selection of other functionaries Chapter Two LIFELINES FOR EDUCATION – A BRIEF PROFILE 2.0 Background of the project 2.1 Implementation mechanism 2.1.1 Profile and the roles played by One World South Asia (OWSA) 2.1.2 Profile and the roles played by Vikramshila Educational Resource Society (VERS) 2.1.3 Profile and the roles played by Quest Alliance 2.2 Information flow process 2.3 Project activities 2.4 Cost aspect Chapter Three COVERAGE OF THE PROGRAMME AND PROFILE OF THE BENEFICIARIES AND FUNCTIONARIES 3.0 Coverage of the programme 3.1 Profile of the primary beneficiaries 3.1.1 Demographic profile 3.2 Educational qualification and experience 3.3 Trainings received 3.4 Availability and use of teaching aids 3.5 Challenges faced by the teachers 3.6 Profile of the functionaries Chapter-IV EXPERIENCE OF THE PRIMARY BENEFICIARIES – THE TEACHERS 4.0 Usage pattern of Lifeline for Education service 4.1 Number of questions raised by the teachers 4.2 Pattern of the queries raised by the teachers 4.3 Motivation for using the Lifelines Education service 4.4 Barriers of using Lifelines for Education 4.5 Experience of the teachers about Lifelines Education 4.6 Level of satisfaction among the teachers 4.7 Perceived benefits of Lifelines for education 4.8 Contribution of the Lifelines for Education 4.9 Future use of Lifelines for Education service 4.9.1 System related issues in future use 4.9.2 Triggers and barriers for future use of Lifelines for Education 4.9.3 Areas of improvement Chapter-V EXPERIENCE OF THE STAKEHOLDERS – FIELD INVESTIGATORS, KNOWLEDGE WORKERS, EXPERTS AND OTHERS 5.0 Profile and roles played by Field Infomediaries (FIs) 5.1 Experience of Field Infomediaries – challenges faced 5.2 Opinions about the service and suggestions for improvement 5.3 Profile and roles played by Knowledge Workers 5.4 Experience of the Knowledge Workers 5.5 Opinion about the service and suggestions for improvement 5.6 Experience of the Contributors / Experts 5.7 Experience of the Education Administrators Chapter-VI CONTENT ANALYSIS OF SELECTED QUESTIONS AND ANSWERS 6.1 Profile of users 6.2 Typical questions and answers 6.3 Discussion on quality of the response Chapter VII OUTCOME AND IMPACT 7.1 Key success factor 7.2 Challenges faced in implementation of the project 7.3 System related challenges 7.4 Project Outcome Chapter VIII SCALABILITY AND SUSTAINABILITY OPTION 8.1 Suggestions for improvement 8.2 Future model of Lifelines for Education service 8.3 Alternative model for self sustenance 8.4 Conclusion List of Tables Table No: 1.1: Study Sampling Table No: 1.2: Sample of schools Table No. 3.1: Coverage of the Programme Table No: 3.2: Age profile of the teachers Table No: 3.3: Educational qualification of the teachers Table No: 3.4: Numbers of years in service Table No: 3.5: Trainings received Table No: 3.6: Use of teaching aids Table No: 3.7: Problems of the teachers Table No 3.8: Profile of the functionaries Table No: 4.1: Number of questions raised by the teachers Table No: 4.2: Patterns of questions raised by teachers Table No: 4.3: Motivation for use of Lifelines for education Table No: 4.4: Average time of receiving answers Table No: 4.5: Triggers and Barriers Table No: 4.6: Areas of improvement Table No: 5.1: Activity schedule of the Knowledge Workers Table No: 6.1 – Selected Q & As for content analysis Table No: 6.2 Few typical questions in History and Geography Table No: 6.3: Fe typical questions and answers available in any text book Table No: 6.4: Examples of unsatisfactory responses Table No: 6.5 Examples of good answers Table No: 6.6 Examples of answers in contrast to good answers Table No. 6.7: Questions and answers related to policy and administrative issues List of Abbreviations OWSA One World South Asia VERS Vikramshila Educational Resource Society EDC Education Development Centre, Inc SSK Sishu Siksha Kendra MSK Madhyamik Siksha Kendra ICDS Integrated Child Development Scheme KW Knowledge Workers FI Field Infomediaries SSA Sarva Siksha Abhiyan CLRC Cluster Resource Coordinator BRCC Block Resource Coordinator SI School Inspector ICT Information and Communication Technologies GP Gram Panchayat Q&As Questions and Answers Executive Summary Background of the study ‘Lifelines for Education‘ was an innovative service implemented by One World South Asia in partnership with Quest Alliance and Vikramshila Educational Resource Society to provide critical academic support and instructional training to the teachers in rural areas. This first-of-its kind service used the power of voice as the primary means for knowledge dissemination and information exchange. The project envisaged to improve the quality of teaching and also developed a knowledge database in the form of questions and answers. The pilot project was implemented at Monteswar block of Bardhman district in West Bengal since October, 2007 to September, 2009. The project covered almost all primary, middle and high schools, SSKs/MSKs and ICDS centres in the block. Around 1,759 teachers across 571 schools availed the service of Lifelines for Education project and 12,139 questions and answers were generated during this period. The project involved eleven Field Infomediaries (FIs), four Knowledge Workers (KWs) and a panel of 150 experts to facilitate the implementation of the project. An evaluation of the Lifelines pilot project in West Bengal was essential to assess the ground realities of the pilot project, experiences and perspectives of the primary beneficiaries and other stakeholders, outcome and impact of the pilot project as well as to assess the sustainability and scalability options of the pilot project. The Education Development Centre, Inc. (EDC) invited ORG Centre for Social Research (a division of the Nielsen Company) to undertake the evaluation of Lifelines for Education program in West Bengal. It was expected that the findings of the study would identify the areas for midcourse correction and facilitate the Government and policy planners in formulating strategies for scaling up and assessing the sustainability options for the project. The specific objectives for the evaluation study were – (a) to evaluate the outcomes and impact of the pilot phase of the project in terms of originally stated objectives and expectations; (b) to identify the key success factors and challenges for scaled implementation and sustainability options; (c) to review the design aspects and operational processes, cost aspects and institutional mechanism to drive program modifications and enhancements; and (d) to consolidate the available data in terms of questions and answers and build the baseline for indicative quantitative assessment. The study findings analysed the information obtained from 315 teachers across 141 schools covered by the programme as well as other project functionaries. Among the 315 teachers interviewed, 58 percent were primary teachers, 23 percent belonged to middle and high schools, 17 percent were SSK/ MSK teachers and the rest from ICDS centers. Study Findings Evaluation study revealed that Lifelines service has reached almost all schools in the block and on an average number of users found in each school was around two or three users. It was found that teachers from primary schools, SSKs, and MSKs accepted the helpline service enthusiastically in comparison to middle and high schools. Teachers from ICDS centres too expressed their need to avail this service but the queries raised by these teachers were mostly related to general knowledge or health education. The concept of helpline to provide support to the rural teachers was well accepted and many teachers reported that they have used the service to resolve the classroom related issues or to understand innovative ways of teaching. But many of them also revealed that they have used the service on request of Field Infomediaries. All most all sample teachers reported using the service more than once. Average number of questions asked by a teacher was reported as around five or six. Some teachers have also used the service more than ten times. Large proportion of teachers reported asking subject related questions; few of them also reported asking questions related to pedagogy, classroom management, administrative and policy related issues. Average cycle time as experienced by the teachers was almost two weeks in lieu of the proposed cycle time of 48 hours. It was found that in most cases though the answers were prepared within a period of 48 hours but the teachers received the answer only after 10 -15 days. Nearly 90percent respondents reported either satisfied or somewhat satisfied with Lifelines service and only around 10.5 percent of the teachers reported their dissatisfaction. The reasons cited for this dissatisfaction were – poor content of the response, delay in getting response and brief response. A large proportion of teachers across all types of schools reported that Lifeline service has improved their knowledge and skills. Many respondents also felt that this helpline service has raised the confidence level of the teachers. Others perceived the impact of Lifelines service on inspiring the teacher towards critical thinking and also inspired them to utilize innovative ways of teaching. Around 94 percent teachers reported that they would like to utilize the service in future and speed and quality of response were reported as key concerns for not using this service in future. FIs played the pivotal role in implementing the project. FIs were primarily engaged to receive the questions from the teachers and provide them appropriate answers. The major challenges faced by the FIs include motivating the teachers to utilize Lifelines service and asking relevant questions related to curriculum, pedagogy or classroom management. FIs also felt the pressure of generating a large number of questions as many times they have been given a target which was difficult to achieve. As mentioned earlier about the average cycle time of 15 days; it as found that the cycle time was entirely governed by the second visit of the FIs to the school which took about 15 days since a large number of schools were assigned to a single FI. KWs were responsible for preparing appropriate answers for a specific question raised, converting the answers in audio format and posting the final answer on the server. They also maintained the database in the form of questions and answers. KWs faced number of system related challenges in addition to workload and not getting the response from experts in time. The experts were quite enthusiastic about the project and participated in the project voluntarily but they also expressed that they did not have any control over the final response. They also did not have any clue on the satisfaction level of the teachers. The Education Administrators perceived the service as innovative and also opinioned that the helpline service should be integrated with other education initiatives like teachers training and workshops at district and block level. Content Analysis A total of 250 questions were selected randomly from the server for the content analysis. Analysis of the profile of users revealed that 60 percent of these questions were asked by primary school teachers, around 20 percent question were from ICDS centers, 15 percent questions from SSK/MSK teachers and remaining (5 percent) questions from middle and high school teachers.1 A large number of questions had a little relationship with the curriculum, pedagogy or classroom management. It was found that teachers have asked questions mostly related to general knowledge or quiz type questions catering to individual interests and curiosities. Though the teachers seeking general information from the helpline service might not be restricted but as almost 80 percent question revealed this trend, this suggests that the pattern of Q&As generated through Lifeline service has not met the expected outcomes of the project. Outcome and Impact The concept of helpline in education was well accepted. The stakeholders across all quarters perceived that Lifeline for Education service provided a much needed academic support system to the teachers in remote rural areas. The service was well accepted due to its inbuilt system features as well as due to the quality of human resources involved in implementation of the project. The use of the local language and the voice mode has led towards the acceptance of the service. Vikramshila’s existence in the block helped in initial icebreaking and access to the teachers without much difficulty. Pre-project workshops, seminars and advocacy with government functionaries ensured hassle free implementation of the project. In addition hiring of local FIs, dedicated KWs and inclusion of well known teachers proved helpful towards implementation of the service. The profile of FIs was not accepted by the teachers, the FIs were not considered as peers by the teaching community; the teachers rarely came up with the real difficulties faced by them. This was prominent from the content analysis as well as the response from the teachers that many of them used the service on request from the FIs. Though the concept of the helpline was accepted but the service did not win the complete confidence of the teachers and very few teachers have actually sought help to improve their teaching. Many teachers reported that they were not aware about the phone number which revealed the need of strengthening the awareness campaign on the project. The project was initiated with lot of enthusiasm but initial vigor and enthusiasm was found missing towards the end of the project. The cycle time of 15 days was found as major constraint towards effective utilization of the service. Quality of response was also found as another concern; content analysis pointed out many answers as unsatisfactory without adequate explanations. The education functionaries (like CLRCs, BRCCs, and SIs) were not involved in implementation of the pilot project and which had affected the effective utilization of the service. There were also few good questions generated through this system and if some teachers have benefited through these responses the project should be considered as successful in providing support in quality education. Scalability and Sustainability options The evaluation study also assessed the viability options for the project and the scope of expansion across other districts and states. It was found that the teachers were not coming out with the real difficulties since many of them were apprehensive to use the service. So for the scaled implementation anonymity of the user would be useful for viability of the project, which would provoke teachers to expose their real difficulties. Toll-free numbers with direct dial-in facility would also ensure anonymity and FIs should act only as contact point, facilitators for implementation of the project. Involvement of educational functionaries (such as CLRCs, BRCCs and SIs) in the project would improve the confidence level of the teachers. Quality of response should be improved to provide more meaningful information to the teachers at the same time sharing of information using local media channel would improve the demand and the visibility of the programme. Though the willingness to pay was not exactly tested in this research but it was felt that many beneficiaries won’t mind paying for the call charges i.e. around Rs. 10/ for each question and answer 1 Complete information on profile of users were not available from the server provided that they are completely satisfied with the responses. It was also felt that if call charges can be reduced from STD call to local call, it would be definitely an added advantage. Conclusion The Lifelines for Education the pilot project was implemented for around two years across all schools in the block. Though a large number of questions and answers (12,139 Q&As) were generated through the system but most of these questions rarely pertained to course curriculum, pedagogy and classroom management. This implies that though the service was accepted by the teachers but in many cases it was accepted due its innovative approach, technology involved and also due to the repeated requests of the Field Infomediaries while the archive of Q&As failed to identify the true gaps in knowledge and skills of the teachers and the pilot project has not met its targeted objective fully. The experts also felt that the duration of the pilot project was too short to win the confidence level of the teachers and this service should be implemented in long run and teachers should experience that there is no victimization for using this service and they should be definitely inspired to utilize the service more effectively. The archive of the Q&As generated in the pilot project should be screened and all relevant questions and answers should be shared with larger audience using local media channel like local newspapers, education programme in radio and television for improving the quality of education in the state. The Lifelines for Education project has the potential; to act as effective academic support system for the teachers, and with appropriate midcourse corrections, the project can actually play an important role in improving overall quality of education. Chapter One INTRODUCTION 1.0 Background Quality remains at the core of education. It determines what students learn, how well they learn and what benefits they draw from their education. These issues have pushed the countries across the world to have on their agenda the mission to ensure student’s optimum learning outcomes that help them play a positive role in creating a civil society. Stakeholders of education have expressed concern about the quality of education being imparted in schools. While it is difficult to define the quality of education operationally due to its complex nature, it is believed that it may be viewed in terms of the quality of various components of input, process and out-put dimensions. Some of these components may be: need￾based curriculum, appropriate physical facilities, use of modern approaches to teaching, scholastic performance of students, professional preparation of teachers, and linking education to life skills and world of work. The story of India’s educational achievements is one of mixed success. On the down side, India has 22 percent of the world’s population but 46 percent of the world’s illiterates. The country is home to a high proportion of the world’s out of school children and youth. On the positive side, it has made encouraging progress in recent past in raising schooling participation but the base of India’s education pyramid remain week.2 The most crying need for the development of the nation is the quality of education. Several surveys, researches and reports have pointed to the current poor quality of education. It was felt across all sectors that teachers’ knowledge and skill set are the cornerstones of school education. The school teachers who determine the destiny of the nation in class rooms should have a deeper understanding of the subject matter as well as pedagogy. To improve the effectiveness and competencies of the school teachers is a concern of the policy planners and Government across the world. In India teachers’ education is supported by a network of national, state, district, block and cluster level resource institutions working together to enhance the quality and effectiveness of teachers through a series of programs. Recent years have seen the development of Information and Communication Technologies (ICTs) and its use in different spheres of life including education. Information and communication technologies, which 2 ‘The progress of school education in India’- Global Poverty Research Group, Website: http://www.gprg.org/ include radio and television, as well as computers and the Internet, have been touted as potentially powerful enabling tools for educational change and reform. When used appropriately, ICTs can help to expand the access to education, strengthen the relevance of education to the increasingly digital workplace and raise educational quality. While the impact of ICTs in learning outcomes is still debatable, and the effective integration of ICT into the educational system is also a complex, multifaceted process that involves not just technology, but also curriculum and pedagogy, institutional readiness, teacher competencies, and long-term financing. Research suggests that ICT can improve quality of education in several ways: by increasing learner motivation and engagement, by facilitating the acquisition of basic skills, and by enhancing teacher training. In case of India a system of education built on the premises of quality and equity has been considered as imperative to its rapid progress as a modern and dynamic society. Government of India stands committed to provide good and quality education to every child in our country, especially those who belong to the underprivileged sections of our society. The importance of good teachers cannot be over-emphasized. Good teachers not only educate- they also inspire. Good teachers make good students. Good teachers make good citizens. The ways and means of improving the quality of our teachers remain as a great challenge for the country. Lifelines for Education was an innovative service to provide critical academic support and continued instructional training to rural elementary school teachers in order to improve the quality of teaching. The pilot project was launched in Monteswar block, Bardhman district of West Bengal during September 2007. One World South Asia (OWSA) in partnership with the QUEST Alliance (a partnership between public, private and non-governmental organisations) and Vikramshila Educational Resource Society (VERS) has implemented the project in Field. Lifelines for education was a value added support service for teachers located in remote rural areas where technology played a key role to connect the teachers with (subject-matter and pedagogy) experts to respond to their day-to-day queries related to academics. Teachers using the Lifelines for Education service registered their subjects along with the queries through telephone, a group of Knowledge Workers and facilitators helped to send these queries to subject matter experts / resources persons and prepared the answers based on their response, all answers were to be intimated to respective teachers within 48 hours mostly over phone. At this juncture an evaluation of the pilot project was essential to assess the ground realities, perception of the beneficiaries, outcomes, impact and sustainability or scalability options for the project in other districts and states. The Education Development Centre, Inc. (EDC) an international non-profit organization, which also develops, delivers and evaluates programmes to address urgent challenges in education, health and economic development has felt the need to conduct an evaluation study of Lifelines for education project. EDC as a part of its dot-EDU India programme entitled “Technology Tool for Teaching and Training’ (T4), and with funding support from USAID (United States Agency for International Development), has commissioned an evaluation study to ORG Centre for Social Research (a division of the Nielsen Company) to assess the strategies and performance of Lifelines for Education project. 1.1 Objectives of the Study The main objective of the evaluation study was to conduct an in-depth assessment of design and operational aspects as well as outcomes and impact of the pilot project and set directions to facilitate sustainable scaled implementation of the project in future. The specific objectives of the study were as follows: ‰ To evaluate the outcomes and impact of the pilot phase of the project in terms of originally stated objectives and expectations; ‰ To identify the key success factors and challenges for scaled implementation and sustainability options; ‰ To review the design aspects, entire operational process of the Lifelines service, cost aspects, and institutional mechanism to drive program modifications and enhancements; and ‰ To consolidate available data in terms of Q&As and build the baseline against which the indicative quantitative assessment is done. 1.2 Study Methodology Following research tasks were undertaken as a part of the evaluation study. Research Task -1: Desk research, review of relevant project documents, project proposals and other documents The Project Implementation Plan of the Lifelines for Education project was thoroughly reviewed to develop an understanding on project design, the roles and responsibilities of each of the functionaries involved in the implementation of the project, method of supervision and monitoring the progress of the project. Also detailed discussions have been carried out with the project staff members and functionaries of Vikramshila Educational Resource Society (VERS), One World South Asia (OWSA) and Quest Alliance. Following were the main area of information collected from the secondary data and Project Implementation plan and depth discussion with project officials. ‰ Project design and key functionaries involved in the project ‰ Roles and responsibility of key functionaries ‰ Geographical coverage, type of schools, number of teachers ‰ Process followed for selection of schools and pilot area ‰ Type of campaign initiated to motivate the teachers to use the Lifelines for Education service ‰ Usage pattern of beneficiaries ‰ Qualifications and skill sets of functionaries of the project and ‰ Level of participation of the experts Research Task-2: In-depth interview with project beneficiaries/teachers The beneficiary survey was carried out among a sample of Lifelines service users randomly drawn from the selected schools across the project area. In order to understand the pattern of utilization of Lifelines service, only the teachers availing Lifelines service was contacted. More specifically, the teachers who availed Lifelines service to get answer to their specific queries at least once in the period from January 2008 to September 2009, constituted the universe for the evaluation study. The required information for the study was elicited by interviewing selected teachers at schools, using a semi-structured pre-tested questionnaire. The questionnaire was finalized after a series of discussions with the officials from Quest alliance, VERS and OWSA, with technical input from EDC. To obtain the required information for the study the questionnaire was designed to collect the information on the following areas: ‰ Key challenges faced while teaching in elementary schools and support system available through other sources (e.g. teaching learning materials and reference books made available as part of Sarva Siksha Abhiyan or SSA and in-service training organised by SSA and school authorities) ‰ Pattern of usage of Lifelines service (i.e. types of questions asked and frequency of usage) ‰ Main reasons for using academic support service or ‘influencers’ ‰ Level of satisfaction with the Lifelines service (quality of responses, spread and depth of knowledge, content clarity and relevance; use of input received from the system, e.g. whether information was disseminated across students or followed any particular method of classroom management as suggested by the experts, how effective or useful was such additional input to improve the overall quality of teaching; cycle time in receiving the response from the system; and level of satisfaction with the performance of field workers and knowledge workers who act as facilitators for this project) ‰ Overall experience of usage of academic support service, willingness to use such service in future, whether users have referred or want to refer such service to peers ‰ Suggestions to improve Lifeline India service in future Research Task-3: In-depth interview/group discussions with facilitators (Field Infomediaries and Knowledge Workers) and contributors (Experts) In-depth interviews and focus group discussions were conducted with Field Infomediaries, Knowledge Workers and subject matter or pedagogy experts to assess the efficiency of information flow process and outreach mechanisms. This task has helped to assess the critical ‘positives’ and ‘negatives’ of the system. In-depth interviews and focus group discussions with Field Infomediaries (FIs): Four in-depth discussions and one focus group discussion have been carried out with the FIs to review their roles and responsibilities, review the process followed for enrolment of the teachers in the system, key issues and challenges faced in the field while motivating teachers to avail such service, opinion on efficiency of the process, effectiveness of the system, cycle time and relevance of the responses received from the system. In-depth interviews with Knowledge Workers (KWs): Knowledge Workers were interviewed to assess their role and responsibilities and challenges faced by them to perform their day to day activities. KWs’ experience of handling the system was the key to assess the quality of the technical system from the user’s perspective. Focus Group Discussions with contributors/experts: Discussions with the contributors/experts revealed the opinions and suggestions of these key functionaries regarding Lifelines project. Efforts were made to assess the challenges faced by these experts to get involved in such project such as time constraints. Research Task-4: In-depth interviews with education administrators / government functionaries of the Education Department Efforts have also been made to know the perception and opinion of Block Education Officers, district level and state level officials of SSA, Panchayat members and other key decision makers in the community regarding Lifelines for Education project. The suggestions of these stakeholders on the sustainability and scalability options for Lifelines service would help to workout the future model for Lifelines for Education project. Research Task-5: Content analysis of selected questions and answers Lifelines for Education pilot project developed a knowledge repository in the form of questions and answers. Questions were raised by the teachers and answers – shared by the experts - were framed by Knowledge Workers. Around 12,000 Questions and Answers (Q&As) were generated in one and a half years of project period. As a part of the current evaluation study, approximately 200 Q&As were selected for detailed content analysis. The selection have been made through a stratified random selection from the entire universe of the Q&As. Efforts have been made to select sample Q&As across all subjects and all sub categories. These 200 Q &A were translated in English and shared with few domain experts/consultants from West Bengal and other parts of India. The sample Q&As were analysed as following: ‰ Classification of questions based on subjects covered (language/others), factual questions, curriculum related questions, pedagogy or classroom management/child psychology related questions and questions related to government policies/ directives or other similar issues; ‰ Profile of users vis-à-vis type of questions and cycle time; and ‰ Understand the challenges faced by teachers in rural school and actual value of the input received through Lifelines for Education service. 1.3 Scope of Work The scope of work for the present evaluation study included following: ‰ Review of the design aspect and operational mechanism of the project ‰ Assessment of the reach and coverage of the project and profile of beneficiaries ‰ Assessment of the information flow and outreach mechanism followed for the project ‰ Review of the experience and perception of project beneficiaries and other stakeholders ‰ Analysis of the capacity and the role played by Field Infomediaries and Knowledge Workers towards implementation of the project ‰ Review of the institutional mechanisms and sustainability option in place for the scaled implementation of the project ‰ Evaluate the options available for integration of Lifelines project with other state or district level education intervention program. The operational area of the project was Monteswar Block of Bardhman district, West Bengal. The intervention area was 13 Gram Panchayats and 571 government primary, middle and high schools and ICDS/Anganwadi3 centres in the block. 1.4 Study Limitations The present evaluation study is limited to the experiences of the beneficiary and other stakeholders of Lifelines for Education project. The study has also made an effort to evaluate the available questions and answers and relevance of such Q&As towards improving the quality of education. The study has not made any effort to assess the actual teaching learning processes, performance of the teachers in the classrooms after receiving the support through Lifelines for Education service. The study has not used any quantitative method to measure the knowledge level or the skill sets available with the teachers and more specifically the learning outcomes of the children, as this was not included in the scope of work for the present study. 1.5 Sampling Framework Following table 1.1 shows the sample size for the present study. Table No: 1.1: Study Sampling Sl. No. Research Tasks/ Respondents Research Tool Sample Size 1 Desk review/secondary data collection from all project partners, VERS, OWSA and QUEST Alliance Check list 2 Primary beneficiaries/Teachers used Lifelines Education service in the project area Semi-structured questionnaire 315 teachers 3 Field Coordinators/Field level Intermediaries implementing the project in Field In-depth interviews and Focus group discussions 4 IDIs and 1 FGD 4 Knowledge Workers Do Do 5 Experts/Resource Persons In-depth discussion guidelines 1 FGD 6 CLRC, BEO, State Education Department officials In-depth discussion guidelines 4 IDIs 3 Integrated Child Development Service Scheme – Programme for early childhood care and development. Others, 5 Primary schools , 178 Middle / High schools, 33 SSK, 90 ICDS centres, 257 MSK, 8 1.5.1 Selection of schools and teachers A stratified random sampling process was followed for selection of schools. All schools across 13 Gram Panchayats (GP) were stratified based on type of schools viz. primary, middle and high schools, SSK and MSK and ICDS centers. Though a large number of ICDS centres covered during the pilot project but the present evaluation study has focused largely on primary schools, SSK, MSKs, middle and high schools considering that ICDS centres are not providing any effective education input. A total of 148 sample schools were selected randomly from 13 Gram Panchayats, from the stratified list of types of schools. Selected numbers of schools were proportionate to total number of available schools in the GPs. Out of 148 sample schools, beneficiaries of Lifelines service was found in 141 schools. Beneficiaries and teachers from the remaining seven schools were not available as a few of them were absent on the date of the field visit. Others were attending training programmes and not available in the school. Table 1.2 below presents the number and type of schools across blocks. All teachers those who reported using Lifelines service at least once during the project lifecycle were selected for the interview. Table No: 1.2: Sample of schools Gram Panchayats Primary school (N) Middle & High school (N) SSK, MSK (N) ICDS Centre (N) Total (N) PUTSHURI 6 1 2 9 PIPLON 6 2 1 9 BAGHASON 10 4 1 1 16 MADHYAMGRAM 11 1 3 15 KUSUMGRAM 9 2 2 1 14 BHAGRA MULGRAM 5 2 3 10 MAMUDPUR 2 4 2 6 DENUR 2 2 2 1 7 JAMNA 13 1 2 1 17 SHUSHUNIA 9 3 1 1 14 BAMUNPARA 6 1 2 9 MAMUDPUR 1 4 3 7 MONTESWAR 5 1 2 8 Total 90 20 26 5 141 Figure No 1.1 – Study Universe and study sample Study Universe – 571 schools Study sample- 141 schools Primary schools , 90 Middle / High schools, 20 ICDS SSK, MSK, centres, 5 26 1.5.2 Selection of other functionaries Out of eleven Field Infomediaries, four were randomly selected for in-depth interviews and group discussions. In-depth interviews were conducted with all Knowledge Workers and a focus group discussion was conducted with a group of experts from different spheres. For other functionaries, two government officials from the Department of Education, one block level education officer and one CLRC were interviewed to assess their perceptions related to the project. Chapter Two LIFELINES FOR EDUCATION – A BRIEF PROFILE This chapter contains a brief profile of Lifelines for Education project, including the background of the project, implementation mechanisms, roles played by the project implementing organisations, and information flow process. This information was collected from the project status report and other secondary documents made available by VERS and OWSA. 2.0 Background of the project Lifelines for Education, an initiative of the One World South Asia office in addition to British Telecom and CISCO systems, was a service that facilitates exchange of critical and timely information using the power of voice as the primary means of knowledge dissemination and information exchange. The system was designed as a right mix of telephony and internet for knowledge delivery by providing connectivity, content and capability via a phone based service. In its initial pilot phase the project has been successfully implemented in north and central India to provide relevant information to the farmers for agribusiness development. The rural farming community was able to get answers to their queries related to their day to day life; their illiteracy was not a barrier to obtain such information. Lifelines for Education was the application of same technology platform with appropriate customization to provide academic support to the teachers in rural areas. There were a number of advantages of using this technology. This was a domain independent technology and can be customized for use to any knowledge domain, in any geographic location and target groups. The technology has the phone and web based features to facilitate seamless knowledge exchange in audio format. The best part of the technology was that it supported the development of a database in the question and answer format which can be analyzed to assess the specific knowledge gaps among the target groups. The application thus has the potential to address a large variety of knowledge and information challenges at the local level. Lifelines for Education pilot project was jointly implemented by One World South Asia office, Quest Alliance and Vikramshila Educational Resource Society (VERS). The project envisaged to provide academic support to the rural teachers by linking them with the experts through a phone based service. The pilot project was launched across all government schools (primary, middle, high schools, SSK and MSKS) in 13 Gram Panchayats of Monteswar block, Bardhman district of West Bengal. The main objectives of the Lifelines for Education project were as follows: ‰ To pilot a system to provide critical academic support and institutional training to rural elementary school teachers ‰ To improve quality of teaching ‰ To support and build a knowledge database in the question and answer format 2.1 Implementation mechanism The following sections detail the profile and different roles played by key project partners: 2.1.1 Profile and the roles played by One World South Asia (OWSA) One World South Asia (the South Asian Centre of the One World Network) aims to harness the democratic potential of Information and Communication Technologies to promote human rights and sustainable development. Towards realization of these goals, OWSA has been working towards building strong partnerships with civil society organisations, private sector, government and other grass root level organisations to position ICTs strategically to strengthen the voices of the poor and marginalized. The overall purpose of Lifelines for Education project was to promote the ICT-based information and knowledge provision to rural teachers and OWSA was responsible for basic setup, customization of the technology platform, and providing functional technology solution and support for carrying out the entire backend operation. OWSA also played an important role in training of the project implementing team, Field Infomediaries, Knowledge Workers and Experts in addition to overall project monitoring and evaluation. 2.1.2 Profile and the roles played by Vikramshila Educational Resource Society (VERS) Vikramshila functions with a motto of “Making Quality Education a Reality for All Children” and has been involved in various educational initiatives over last one decade. As the main implementing partner, VERS was responsible for promoting the Lifelines service among the teachers and encouraging them to use this service. VERS was entirely responsible for implementation and facilitation of the project on the ground. VERS recruited and trained a team of Field Infomediaries to facilitate Field implementation, and also provided and trained a team of Knowledge Workers and experts to ensure quality and timely answers. VERS also took part in IEC activities for the project, provided personal consultation as a part of marketing / advocacy strategy, and networked with local experts to provide localized answers. 2.1.3 Profile and the roles played by Quest Alliance The Quest Alliance is a multi-stakeholder partnership working across India towards a social commitment to improve education and employability among India’s disadvantaged youth. As a resource partner and donor Quest Alliance has provided both technical and financial support in implementation of Lifelines for Education project. Quest Alliance has also undertaken initial research design study with Quicksand design studio to recommend design interventions and also initiated the current evaluation study. 2.2 Information flow process As discussed earlier Lifelines for Education service was based on an interactive voice response system and also using an innovative mix of telephony and internet. The project was designed to answer the teachers’ queries related to subject matter, pedagogy, classroom management and administration within a period of 24 – 48 hours by a group of educational experts / administrators. The information flow process was connected to the computer and telephone infrastructure and also to the knowledge level and skill sets of Knowledge Workers and experts as and when required. The service was operated with a telephone number attended through an automatic voice response system and the query was recorded in the server. A query ID /reference number was provided to the teacher to track the query and receive the response. The Knowledge Workers at VERS office received the query from the server and prepared the answer for the query accessing different resources available with them e.g. text books, reference books, dictionaries, encyclopedia as well as internet. Knowledge Workers contacted the concerned experts in case of non￾availability of the answer from other sources. Once the answer was ready, the Knowledge Workers stored the answer as an audio clip in the server against the query reference ID so that IVR system can play back the answer to the users / teachers in this case. Knowledge Worker also added the question and answer in the FAQ database for future reference. The end users/teachers used the mobile phone provided to the Field Infomediaries to raise the question and receive the answer. Figure 2.1 shows the information flow process for the project. Figure No 2.1 Information Flow Process 2.3 Project activities The implementation of Lifelines for Education service included a series of workshops, seminars, and training programmes involving the project beneficiaries and other stakeholders. Vikramshila’s presence in the block as an education resource centre has also helped in implementation of the project. A pre-project kick-off workshop was organized at Monteswar for initial icebreaking. A viability study was conducted by Quicksand Design Studio to determine the model for Lifelines for Education project. A number of training programmes and workshops were organized by VERS to orient and train the Field Infomediaries and Knowledge Workers for the project. Teachers’ workshops were organised periodically to raise the interest of the teacher’s community. This helped in advocacy campaign for Lifelines for Education service at ground level. FAQ booklets and reading material were distributed among teachers as supplementary resources for academic reference. 2.4 Cost aspect The following calculation of the average cost per question and answer was calculated from the records available at VERS. The total operational cost of the program for one and half years was around 22 Lakhs. The remunerations paid to the Field Infomediaries and Knowledge Workers were Rs 3,000/ and Rs 8,000/ per month respectively. Average call charge for asking a question and receiving the answer under the program was worked out as less than Rs 10/. However the teachers were not charged the user fee for using the service under the pilot project. Cost of the pilot project should be assessed with reference to number of beneficiaries and type of outputs generated in the project. The pilot project at the cost of 22 lakhs have benefited around 1,000/1,200 teachers directly, many other teachers indirectly (while discussing with friends and colleagues), and a large number of children in the blocks. In addition, around 12,000 Q&As generated would benefit other teachers, trainers and text book writers in developing teaching/learning materials.4 4 Information on operational cost of the pilot project was collected from VERS, while other project partners could not provide such information specifically for Lifelines for Education project. Field Infomediaries help teachers to enable them to use the service, market the service, monitor to ensure quality service Experts support Knowledge Workers to ensure quality and timely answers Teachers register their queries related to subject matter, pedagogy and classroom management Knowledge Workers responsible for viewing the query finding an appropriate answer and answering the query / call SERVER 48 Hours Chapter Three COVERAGE OF THE PROGRAMME AND PROFILE OF THE BENEFICIARIES AND FUNCTIONARIES The following chapter includes the reach and penetration of the project and profile of the beneficiaries and other functionaries. 3.0 Coverage of the programme The pilot project has covered 571 schools, which includes ICDS centres, primary, middle and high schools across the block. Total coverage of beneficiaries as reported in the project document was 1,759 teachers;5 the total number of questions generated was 12,139 and the project was facilitated by 11 Field Infomediaries and four Knowledge Workers in addition to around 150 experts. Table No. 3.1: Coverage of the Programme District Bardhman Block and Gram Panchayats Monteswar Block, 13 Gram Panchayats, No. of schools covered 571 No. of teachers enrolled 1,759 No. of Field Infomediaries 11 No. of Knowledge Workers 4 No. of Experts 150 (eminent educationist in the state, approx) No. of Questions and Answers generated 12,139 The present evaluation study has contacted an aggregated 148 schools across the block; 141 schools reported having the beneficiaries of Lifelines for Education service on the date of visit. In the seven remaining schools, teachers were either absent or busy in attending a training programme or retired. All available beneficiaries were interviewed at all 141 schools. A total of 315 beneficiaries/teachers were covered in the survey. 3.1 Profile of the primary beneficiaries Out of the 315 teachers, 183 teachers were from primary schools (58 percent), 74 teachers belonged to middle and high schools (23 percent), 52 teachers from MSKs and SSKs (17 percent) and only six teachers from ICDS centers. The average number of students reported for primary schools was around 100. Middle and high schools reported having almost 600 students while SSKs and MSKs had 160 students on average. Primary schools reported having two to three teachers. SSK and MSK had about four teachers. Almost all middle and high schools reported having more than ten teachers. This also revealed that in primary schools and SSK, MSKs teachers are teaching all subjects in the curriculum while in case of middle and high schools teachers are mostly teaching one to two subjects based on their expertise. An average number of students per class was reported as 27 students for primary schools and 60 students for secondary schools. 3.1.1 Demographic profile Most of the teachers reported to be in the age group of 35-50 years (45 percent). Around 75 percent of the MSK/SSK teachers, 50 percent teachers from ICDS centres, 39 percent teachers from primary schools and 36 percent teachers from middle and high school found in this age group. Around 27 percent of all teachers reported in the age group of below 35 years and similar proportion of teachers also reported in the age group of above 50 years. 5 The study revealed that the total number of beneficiaries was in the range of 1,000 – 1,200. Figure No. 3.1: Age profile of the teachers Age profile of the teachers Less than 35, 27.90% Above 50, 27.30% 35-50, 44.80% Middle and high schools reported the highest proportion of teachers in the age group of below 35 years (38 percent), and SSKs / MSKs rarely reported teachers in the age group of below 35 years – see Table 3.2. Table No: 3.2: Age profile of the teachers Age Profile Type of school Primary school Middle school/ High school SSK, MSK ICDS Centre N % N % N % N % < 35 54 29.5 28 37.8 3 5.8 3 50.0 35-50 72 39.3 27 36.5 39 75.0 3 50.0 50+ 57 31.2 19 25.7 10 19.2 Total Teachers sampled 183 100.0 74 100.0 52 100.0 6 100.0 Average age of the Teachers 42.4 40.8 45.8 37.2 Around 71 percent of all sampled teachers were male. Predominance of male teachers was found in primary, middle and high school (above 80 percent) and female teachers were predominant in ICDS, SSKs and MSKs. All the teachers in ICDS centres and 75 percent of teachers In SSK/MSK were reported as female. 3.2 Educational qualification and experience Analysis of data revealed that there was a significant gap in education level among teachers in primary schools and SSK, MSK and teachers in middle and high school. More than 55 percent of primary school teachers, 79 percent SSK, MSK teachers and almost all ICDS teachers have either completed secondary or higher secondary level of education. While half of the middle and high school teachers were reported as graduates and remaining half were reported as postgraduates – see Table 3.3. Base – 315 teachers Table No: 3.3: Educational qualification of the teachers Educational Qualification Type of school Primary Total school Middle school/ High school SSK, MSK ICDS Centre N % N % N % N % N % Class 10 41 22.4 33 63.5 3 50.0 77 24.4 Class 12 60 32.8 8 15.4 3 50.0 71 22.5 Graduation (BA/ B.Sc / B.Com/ B.Ed / B.Tech / BBA) 66 36.1 37 50.0 7 13.5 110 34.9 Post Graduation (MA/M.Sc) 15 8.2 36 48.6 4 7.7 55 17.5 Others / refused 1 0.5 1 1.4 2 0.6 Total 183 100.0 74 100.0 52 100.1 6 100.0 315 100.0 Average experience or number of years in service was reported as around 12 years across all sample teachers. Nearly 50percent teachers in both primary schools and middle and high schools reported having teaching experience of less than 5 years. Teachers in SSK, MSK and ICDS centres reported having better experience profile. Around 75percent SSK teachers reported having more than five years of experience – see Table 3.3. Table No: 3.4: Numbers of years in service Years of teaching Type of school Primary school Middle school/ High school SSK, MSK ICDS Centre Total N % N % N % N % N % Less than 2 years 12 6.6 11 14.9 1 1.9 1 16.7 25 7.9 2 yrs - less than 5 years 71 38.8 25 33.8 12 23.1 108 34.3 5 yrs - less than 10 years 68 37.2 15 20.3 34 65.4 5 83.3 122 38.7 10 years - less than 20 years 12 6.6 9 12.2 5 9.6 26 8.3 More than 20 years 20 10.9 14 18.9 34 10.8 Total 183 100 74 100 52 100 6 100 315 100 3.3 Training received Out of the sampled teachers, 87 percent of SSK and MSK teachers, 43 percent of primary school teachers and 50 percent middle and high school teachers reported that they have not received any formal pre￾service training, while 41.5 percent teachers from primary schools also reported having pre-service training like Basic Training, Junior Basic Training, Primary Teachers Training and PGBT. Around 91 percent of all teachers reported received in-service training; the proportion was the highest (94 percent) for primary school teachers. Table No: 3.5: Trainings received Formal Pre-service training Type of training Primary School (%) Middle and High Schools (%) SSK and MSK (%) ICDS Centers (%) Total (%) Dp. Ed /B.Ed /M.Ed 8.2 36.5 9.6 14.9 Other trainings (Basic Training, Junior Basic Training, PTT, PGBT etc) 41.5 13.5 3.8 32.1 No formal training received 50.3 50.0 86.5 100.0 53.0 In-service training Received Training 94 86.4 90.5 66.7 91.1 Base: Primary schools – 183 teachers; Middle and High schools – 74 teachers; SSK, MSK – 52 teachers; ICDS – six teachers 3.4 Availability and use of teaching aids The following table presents availability of teaching aids in the school. Out of 141 schools, only five schools reported having computer facilities and none of the schools had internet. Total 46 schools reported having question bank and 69 schools reported having library facilities – see Table 3.6. Table No: 3.6: Use of teaching aids Teaching Aids Primary Schools (N) Middle and High Schools (N) SSK and MSK (N) ICDS centers (N) Total (N) Text Books Yes 90 2 26 1 137 Reference books/Dictionaries Yes 74 19 12 105 Questions Banks Yes 28 13 5 46 Library Yes 47 14 8 69 Teaching Learning Materials Yes 89 20 23 5 137 Computer Facilities Yes 5 5 Internet Nil 90 20 26 5 141 Total 90 20 26 5 141 Charts, pictures, cards, globe and models were reported as most commonly used teaching learning materials. Regarding usage of reference books, around 29 percent teachers reported using reference books very often; while around 51 percent reported using reference books occasionally and remaining 20 percent teachers reported not using reference books at all. Nearly 13 percent of all teachers have also expressed that reference books were not required for their teaching. 3.5 Challenges faced by the teachers Lack of physical infrastructure, students’ absenteeism and drop-out, large number of students in a class leading to difficult classroom management, less number of teachers and heavy workload are the major challenges faced by the teachers in rural areas. Issues faced while delivering subject matter due to inadequate skill sets and knowledge etc were reported by 35percent SSK/ MSK teachers, which suggests that these teachers do require academic support service while this issue was also reported by 13 percent primary school teachers and 16 percent middle and high school teachers. Non-availability of text books and adequate teaching learning materials and issues related to multi-grade teaching were other common challenges reported by the teachers across all types of schools – see Table 3.7. Table No: 3.7: Problems of the teachers Primary Schools (%) Middle and High Schools (%) SSK and MSK (%) ICDS centers (%) Total (%) Physical infrastructure – lack of space, toilet, drinking water 31.1 41.9 51.9 100 38.4 Student absenteeism and drop-out 36.1 23 40.4 33.3 33.7 Large number of students in one classroom and classroom management 29 35.1 28.8 66.7 31.1 Less teacher 26.8 21.6 3.8 33.3 21.3 Non-availability of text books / adequate teaching learning materials etc 16.9 13.5 25 17.8 Issues related to syllabus / subject matter / not having adequate skill sets / knowledge 12.6 16.2 34.6 16.8 Issues related to multi-grade teaching 13.7 8.1 1.9 10.2 Children poor in education 6.6 5.4 3.8 5.7 No problems 2.2 13.5 1.9 4.8 Base: Primary schools – 183 teachers; Middle and High schools – 74 teachers; SSK, MSK – 52 teachers; ICDS – six teachers The teachers were further probed whether they discussed these challenges with anyone or received any support to overcome these challenges. A majority of the teachers reported discussing school related issues with their colleagues, head teachers and school management. In connection to support received, most of the teachers reported having informed discussions with colleagues, head teacher or CLRCs which helped reduce such difficulties. Others reported that teachers training programme or workshops conducted by education functionaries or NGOs were beneficial. 3.6. Profile of the functionaries As discussed earlier Field Infomediaries, Knowledge Workers and experts are the key functionaries involved in implementation of Lifelines for Education programme. The following table provides the profile of these functionaries. Table No 3.8: Profile of the functionaries Total No. Age, Sex, Education and Experience Field Infomediaries (FIs) 11 All male in the age group of 25 – 40 years, 4 of them had qualification as Graduates / Post Graduate in Bengali / History. Others were having Secondary / Higher Secondary level of education with experience of working in Vikramshila Knowledge Workers (KWs) 4 Female – in the age group of 25 – 35 years, All of them were graduate / post graduate in humanities/ social sciences and also had experience of working in NGO sector Experts / Resource Persons 150 (actively involved around 20) Schools teachers from the reputed schools in Kolkata and College/ University lecturers and professors. Some of them were Education Consultants and Administrators / officials from SSA etc. Chapter-IV EXPERIENCE OF THE PRIMARY BENEFICIARIES – THE TEACHERS The following section includes the pattern of usage of Lifelines for Education service, the key influencers for using such service, the benefit of the service and the level of satisfaction. It also tries to capture the willingness of the teachers to avail the service in future and suggestions for improvement of Lifelines service. 4.0 Usage pattern of Lifeline for Education service Analysis of the pattern of usage of Lifelines for Education service revealed the acceptance of such academic support service by the beneficiaries. 4.1 Number of questions raised by the teachers Around 90 percent of the sampled teachers reported that they have been using the Lifelines service for more than six months and average number of questions asked was reported as around five. Nearly half of all teachers/beneficiaries reported asking less than four questions while about one fourth of the beneficiaries also reported asking more than ten questions. Across different types of schools, SSK, MSK teachers reported maximum usage of the service as around 40 percent of these teachers reported asking more than ten questions while middle and high school teachers reported minimum usage of Lifelines service-only 13 percent of these teachers reported asking more than ten questions. Around 28 percent primary school teachers and 17 percent ICDS teachers also asked more than ten questions through Lifelines service. (Refer fig 4.1 and table 4.1) Figure No. 4.1 Number of questions Table No: 4.1: Number of questions raised by the teachers No. of questions Type of school Primary school Middle school/ High school SSK, MSK ICDS Centre N % N % N % N % 1 14 7.7 17 23.0 1 1.9 2 – 4 69 37.7 30 40.5 18 34.6 3 50.0 5 – 7 32 17.5 15 20.3 3 5.8 2 33.3 8 – 10 17 9.3 2 2.7 9 17.3 More than 10 51 27.9 10 13.5 21 40.4 1 16.7 Total 183 100.0 74 100.0 52 100.0 6 100.0 Around 39 percent teachers also reported not asking any question in last six months. Pattern of usage in last six months also suggests that teachers from middle and high schools have rarely used the service during this period. The major reason cited by the teacher for not using the service during last six months was irregular visit by Field Infomediaries during this period. 1 Question 10% More than 10 Questions 26% 2-4 Questions 38% 8-10 Questions 9% 5-7 Questions 17% Base – 315 teachers Lifelines project was initiated in September 2007 and the project was continued till September 2009 but the last six months of implementation the project was operational in an extended mode and initial enthusiasm was found missing. The period March- September also considered as less active in terms of teaching activities since this period has got exams, summer vacations, promotions and new admissions etc. 4.2 Pattern of the queries raised by the teachers Most of the teachers (90 percent) asked questions related to subject matter while analysis of pattern of questions also revealed that some teachers have also posed their questions related to pedagogy, general knowledge and classroom management. Nearly 37 percent teachers from primary schools and 46 percent teachers from SSK/MSK asked questions related to pedagogy but only nine percent middle and high school teachers reported the same. Around 15 percent primary school teachers and 17 percent SSK/MSK teachers have asked questions related to policy matters but very few teachers from middle and high schools reported the same. It was found that ICDS teachers had more questions related to health issues and administration related issues. The purpose of ICDS programme is to improve the health, nutrition and development of children. The programme offers health, nutrition and hygiene education to mothers, non-formal preschool education to children aged three to six, supplementary feeding for all children and pregnant and nursing mothers, growth monitoring and promotion, and links to primary healthcare services such as immunization and vitamin A supplements. So most of these ICDS workers / Anganwadi Workers (AWWs)/ teachers in ICDS centres were found concerned with child health and related issues. (Refer table 4.2) Table No: 4.2: Patterns of questions raised by teachers Content of the question Topics Primary school (%) Middle school/ High school (%) SSK, MSK (%) ICDS Centre (%) Subject Matter 89.1 94.6 94.2 50.0 Pedagogy 36.6 9.5 46.2 50.0 Administration 7.7 4.1 3.8 33.3 Policy 14.8 5.4 17.3 50.0 Classroom Management 7.1 2.7 15.4 16.7 General Knowledge 5.5 1.9 Environment 1.9 Health Education 16.7 Base: Primary schools – 183 teachers; Middle & High schools – 74 teachers; SSK, MSK – 52 teachers; ICDS – 6 teachers Knowledge Workers and Field Infomediaries have also reported a large proportion of subject related questions. Questions related to English teaching was reported as quite common considering that the state has recently resumed English teaching in primary section with a revised curriculum and a large proportion of teacher are still facing challenges while teaching English through direct method in primary division. 4.3 Motivation for using the Lifelines Education service More than one third of all teachers have reported that the difficulties faced by them while teaching in classroom have actually motivated them to utilize this service. This top of mind response revealed that the challenges faced by the teachers in rural schools included not only the lack of physical infrastructure and resources like text books, reference books, teaching learning materials etc but also the gaps in teachers’ own knowledge and skills sets. So the help line in education to provide critical academic support and instructional training to the teachers in remote rural areas was extremely needed and well accepted by a significant proportion of beneficiaries. Other important motivating factors as reported by the teachers included the reasons as following: to access facts and figures related to curriculum but not available in the text book, and to understand innovative ways of teaching. These responses suggested the need of additional reference books and adequate teaching aids. In addition, many teachers also reported the following reasons for using Lifelines service: wanted to experience an innovative service; wanted to identify the printing errors in text books; required to cross check different version of facts and figures in different text books. Usage of Lifelines service to manage difficulties while teaching English was also found as one of the prominent responses revealed by the teachers. Figure No. 4.2 Top of the mind and prompted response (figs in %) Base: 315 Teachers These teachers were further asked on other ‘influencers’ for using Lifelines service while prompting various responses. During prompted responses many teachers / beneficiaries also came out with the response that they have also used the service on request of Field Infomediaries (40%) which suggests that FIs have played a critical role in initial icebreaking and many teachers also reported using Lifelines service to avail the suggestions given by the expert panel. An analysis of the motivating factors across different types of schools revealed that around 33 percent primary school teachers, 29 percent secondary school teachers and 24 percent SSK/ MSK teachers reported that their difficulties related to subject matter and lack of knowledge and skill sets have actually motivated them to utilize such service. Teachers across all types of schools universally reported that checking of facts and figures either not available in text book or different text books providing different versions have motivated them to use Lifelines service. Many primary school teachers (21 percent) reported that the difficulties faced while teaching English have motivated them to use Lifelines service. (Refer table 4.3) 36.5 32.2 27.4 25.1 23.1 20.2 17.3 17.3 15.3 15.3 14.7 14.7 16.7 19.2 23.5 23.8 10 17.1 12.1 21.7 17.8 39.5 13.9 27.8 Difficulty in subject matter Check facts and figures not available in texts Wanted to understand the innovative ways of teaching Experience innovative service Check correct facts and figures Difficulties while teaching English in primary division Question raised by a student Difficulties in classroom management Wanted to test the service FIs of Lifeline requested me Doubts / difference in opinion with our colleagues Suggestions from expert panel Prompted TOM Table No: 4.3: Motivation for use of Lifelines for education Issues Primary schools Middle & High schools SSK, MSK ICDS centres % % % % Wanted to check correct facts and figures (different version) 23.0 31.5 35.5 14.0 Wanted to check facts and figures, not available in text books 32.0 38.4 45.2 28.0 Faced some difficulties related to subject matter 33.1 28.8 24.2 60.0 Faced difficulties while teaching English 21.3 6.5 40.0 Doubts / difference in opinion 11.8 17.8 14.5 22.0 Confused due to errors in the text book 4.5 9.6 6.5 2.0 Question was raised by a student and I did not have any suitable answer 19.7 15.1 14.5 14.0 I found difficulties in classroom management / understanding child psychology 16.3 8.2 9.7 26.0 Wanted to understand the innovative ways of teaching / new methods etc 29.8 16.4 14.5 36.0 Innovative service and I wanted to experience the service 24.2 26.0 25.8 30.0 I wanted to test the service and see the reliability / quality of responses 16.3 9.6 9.7 22.0 As Lifelines India got a very well known expert panel, I wanted to know their suggestions 16.9 12.3 9.7 12.0 I wanted to raise my concerns questions related to policy matters 5.6 1.4 1.6 4.0 I was motivated to see my colleagues using the service 5.1 8.2 6.5 6.0 Field Investigators of Lifelines requested me to use such service 14.0 19.2 16.1 12.0 Base: Primary schools – 183 teachers; Middle & High schools – 74 teachers; SSK, MSK – 52 teachers; ICDS – 6 teachers 4.4 Barriers of using Lifelines for Education Teachers were always respected and admired in Indian society; unlike the modern concept of facilitators, teachers were always considered the providers of knowledge. The fact that a teacher might lack knowledge and skill sets etc was not well accepted. Even teachers were also apprehensive to use the service at the beginning and considered that asking questions would be a sign of weakness or this might be an attempt to test the level of competency of the teachers. Absence of culture of questioning or academic sharing and lack of skill sets to ask relevant questions were found as other barriers for using Lifelines service. The above information came out during the discussion with FIs and Project Coordinator. 4.5 Experience of the teachers about Lifelines Education About 28 percent of the teachers reported that they have not received the answers to their questions and the proportion was highest among the ICDS teachers (83 percent). The discussion with FIs revealed that the questions raised by ICDS teachers were mostly related to child health, nutrition and other similar issues. So during the last six months of project implementation FIs have actually reduced their visits to ICDS centres and which could be one of the reasons that a large proportion of ICDS teachers / workers reported that their questions remained unanswered. The average cycle time i.e. time period to receive an answer after asking a question was reported as more than two weeks (15.8 days) as reported by the teachers. Though the project has envisaged for a cycle time of 24 – 48 hours but this cycle time was not maintained due to various reasons. Figure No. 4.3: Average cycle time for receiving answers One week to 10 days 32% 2 weeks / more than that 33% Less than a week 18% Less than 48 hours 17% Base – 315 teachers Almost similar response on average cycle time was received across different type of schools. Around 67percent teachers from ICDS centres, 34 percent teachers from primary schools, 26 percent teachers from middle and high schools and 23 percent teachers from SSK, MSKs reported the received answers only after two weeks. Little more than one third of respondents for both primary schools and SSK and MSKs reported the cycle time experienced as less than a week, while this proportion was around 27 percent for middle and high schools. (Refer Table 4.4) E.g. a question asked by an ICDS teacher – ‘if the blood group of father is B+ and the blood group of mother is also B+, what would be the blood group of the child?’ – The KWs faced difficulty to respond this question. Table No: 4.4: Average time of receiving answers for last question Average time Primary School (%) Middle and High School (%) SSK and MSK (%) ICDS Centers (%) 2 weeks and more 33.9 25.7 23.1 66.7 One week to 10 days 24.6 41.9 34.6 16.7 Less than a week 18.0 14.9 19.2 Less than 48 hours 17.5 12.2 17.3 NA – not received the answer 6.0 5.3 5.8 16.6 Base: Primary schools – 183 teachers; Middle & High schools – 74 teachers; SSK, MSK – 52 teachers; ICDS – 6 teachers 4.6 Level of satisfaction among the teachers Nearly 90percent respondents reported either satisfied or somewhat satisfied with Lifelines service and only around 10.5 percent of the teachers reported their dissatisfaction. Around 12 percent middle and high school teachers and 9 percent primary teachers were not satisfied with the response. The reasons for such dissatisfaction were reported as following: (i) content of the response was not up to the mark (23 percent of those reported dissatisfaction), (ii) response was very brief (6 percent of those reported dissatisfaction) (iii) delay in getting response (23 percent of those reported dissatisfaction) and (iv) not received the response (23 percent of those reported dissatisfaction). 4.7 Perceived benefits of Lifelines for education Teachers those who reported satisfied or somewhat satisfied were further probed to assess the actual benefit of the response received from Lifelines service. A large proportion of teachers have felt that these responses have improved their knowledge level around 77 percent of those reported satisfied with the Lifelines service has expressed this. Nearly 49 percent of these respondents have also felt that their teaching skills were improved through Lifelines service. Many reported that their confidence level had improved and helpline service raised their ability of critical thinking and questioning. (Refer figure 4.4) This revealed that the teachers perceived the benefit of Lifelines service more towards their self improvement, improving their knowledge level and skill sets etc while the impact of such input on teaching learning process and the performance of the students need to be assessed in future. Figure No. 4.4 Perceived benefits of Lifelines Education (figs in %) The above analysis reflect that the majority of beneficiaries were satisfied with the service and also felt that such support service can help them to increase their knowledge and skill sets by improving their overall quality of teaching. 4.8 Contribution of the Lifelines for Education All sample beneficiaries were further asked their opinions related to contribution of Lifelines for Education service. More than two third of these beneficiaries across all types of schools reported that such service has improved the knowledge and skill sets. More than half of all respondents have also felt that this helpline service has raised the confidence level of the teachers. Many teachers opinioned that the lifeline service helped them to mange the unknown subjects in the class well and to face the difficult questions raised by the students. Many also perceived the impact of Lifelines service on inspiring the teacher towards critical thinking and also helped or inspired the teachers to utilize innovative ways of teaching. (Refer figure 4.5) Even FIs also reported that Lifeline for Education service improved the teachers’ knowledge and skill sets. FIs have noticed that teachers discussing and debating on the responses received from Lifelines service among themselves and a culture of education discourse was initiated. They also believed that teacher have started imparting these information to the students in the classroom. Though further analysis of FAQs in later sections actually revealed that a large number of questions were related to general knowledge or quiz type question and not much related to course curriculum or pedagogy relevant for the teachers at primary, middle or high school levels. 77.2 49.1 25.3 24.2 17 11.8 9.3 4.2 4.2 Improvement of knowledge Improve my abilities / teaching skill Raised my confidence Can face questions raised by students Improvement of thinking and questioning skill Manage/handle the classroom better Impart innovative ways of teaching Identification / reducing text book errors / printing mistakes No help received Base 267 teachers Figure No. 4.5 Contribution of Lifelines Education (figs in %) Base: Primary schools – 183 teachers; Middle & High schools – 74 teachers; SSK, MSK – 52 teachers; ICDS – 6 teachers 4.9 Future use of Lifelines for Education service Around 94 percent of the teachers expressed that they would like to use the Lifelines for Education service in future. This suggests that teachers in rural areas are actually in need of such service and accepted the helpline service eagerly. 4.9.1 System related issues in future use Respondents were further asked about their opinion related to system such as clarity of the voice and ease of handling the system. Regarding the clarity of voice of responses, over 90 percent of these beneficiaries reported that they did not find any issue with the quality of the voice of the response. While many of these teachers reported that they wanted to hear the response more than once. This might be related to the need for better comprehension of the response. Though none of these beneficiaries have received any training on using Lifelines service but many reported using the service on their own without any support of FIs. Many young and enthusiastic teachers have reported the same. Around 75 percent teachers considered the system as very user friendly and also felt that they can handle the helpline without any help from the FIs. 4.9.2 Triggers and barriers for future use of Lifelines for Education While motivating factors are which the service-providers should bank upon and strive for excellence, the de-motivating factors clearly signal to the service-provider’s shortcomings. Based on these factors the service-providers should plan their course of action. The sampled teachers were probed to find out the motivating and de-motivating factors for the future use of the service. The following table reflects the triggers and barriers as perceived by the teachers. Table No: 4.5: Triggers and Barriers Motivating Factors % De-motivating factors % Innovative service 50.2 Irregularity of the FIs 44.0 Speedy response Quality of response Involvement of experts Service at doorstep Voice-based service 19.2 15.8 7.4 6.4 1.0 Delayed response The service is not toll-free No need of the service 33.0 11.0 5.0 Base: Motivating factors – 297 teachers and de-motivating factors – 18 teachers 54.6 44.6 57.7 16.7 75.4 73 82.7 66.7 54.1 50 69.2 32.2 33.3 39.2 34.6 16.7 Primary schools Middle & High schools SSK, MSK ICDS Centre Inspired the teachers towards critical thinking and raise questions Improve the knowledge and skill sets of the teachers Raise the confidence level while handling non familiar subjects or different questions Utilize innovative methods for teaching / learning The key motivating factors to use such service in future include the innovativeness of helpline service and quality of response, involvement of panel of experts and availability of hassle free service etc were important. The main de-motivating factors were reported as irregular visit of FIs and delayed responses. Very few teachers were found as still not convinced with the service and felt that such service was not required. The table clearly indicates that although the teachers accepted this helpline as an innovative service but the speed and quality of response should improve to motivate the teachers for further use of the service. In addition to this many teachers have also felt that the service should be toll free and easily accessible. 4.9.3 Areas of improvement All beneficiaries were further asked about their suggestions for improvement of Lifelines for Education service, the responses include following: 1. Improve quality of response 2. Reduce the cycle-time 3. Direct communication with the experts 4. Increase the involvement of the experts 5. Improve the frequency of visits by the FIs 6. Implement a toll-free number About 48 percent of the teachers felt the Lifelines service should be available on a toll-free number so that they can access the service on their own. In absence of a toll free number, many teachers felt that FIs should visit the school more regularly to increase the use of such service. Similar proportion of teachers have also felt that the awareness campaign initiated for the pilot project was not adequate. It was suggested that active awareness campaign through leaflets, posters, local newspapers and other media channels would help increase the demand for this service. The cycle time was also found as critical towards future use of Lifeline service. According to these teachers, the optimal cycle time should be reduced to one to two days, i.e. within two days time the teachers should receive their answers to the questions. A few teachers have also expressed the need for improvement of quality of response. They expressed that the response should be more elaborate with adequate explanation and examples. Others have also suggested that such service should be made available to the parents or students. A few respondents also felt the need of two-way communication and reported that direct communication with the experts might be more beneficial. Figure 3.5 Areas of Improvement (figs in %) Base – 315 teachers 4.4 6.7 7 8.9 9.5 10.2 10.2 21.6 32.7 33 47.9 Assistance of Govt A phone can be provided in each school FI strength can be increased Two way communication A website can be open on this service so that teacher can access Parents / students should be included under service Quality of response should be improved Prompt response of question Need to increase campaign( news paper,tv etc.) FI should visit school on regular basis Need to be toll free An analysis of the teacher’s responses across different type of schools revealed that the demand for a toll￾free number is the unanimous response. Proportionately more number of primary teachers wanted a toll￾free line. This might be due to the fact that primary school teachers were most frequent user of this service. A large proportion of ICDS teachers recommended for regular visits of the FIs. While FIs also reported that they have reduced the frequency of visit in ICDS centre considering that the questions raised by these teachers in ICDS centre were not related to education. Increased campaigns were also found as one of the key areas of improvements as suggested the teachers. Even many teachers have reported during informal discussion that they were not aware of the toll free number of Lifelines service so they could not access the service on their own even if they wanted to use the service for immediately but they had to wait for the visit of FIs to raise their questions - see Table 4.6. Table No: 4.6: Areas of improvement Primary schools % Middle & High schools % SSK, MSK % ICDS Centre % Need to be toll free 57.4 37.8 30.8 33.3 FI should visit school on regular basis 25.1 44.6 42.3 50.0 Need to increase campaign (news paper, TV) 31.1 37.8 32.7 16.7 Prompt response of question 19.1 16.2 32.7 66.7 Parents / students should be included under service 10.9 6.8 11.5 16.7 Quality of response 9.8 10.8 11.5 A website can be open on this service so that teacher can access 12.0 4.1 7.7 16.7 Two way communication 9.8 8.1 5.8 16.7 FI strength can be increased 7.1 5.4 9.6 A phone can be provided in each school 3.8 18.9 Assistance of state / central Government 6.0 2.7 1.9 Feedback on answers from student 3.8 5.4 3.8 Base: Primary schools – 183 teachers; Middle & High schools – 74 teachers; SSK, MSK – 52 teachers; ICDS – 6 teachers Chapter-V EXPERIENCE OF THE STAKEHOLDERS – FIELD INVESTIGATORS, KNOWLEDGE WORKERS, EXPERTS AND OTHERS Lifelines for Education service involved a group of functionaries at the field level and at back end for effective implementation of the programme. Field Infomediaries recruited at the field level were the key players of the entire field operation. Knowledge Workers played the most critical role at backend. Lifelines for Education service also involved a pool of experts from renowned schools universities, child psychologists, officials from the Department of School Education and other think tanks in state education system. This chapter includes the roles played by these functionaries while implementing the programme and their experiences and suggestions. 5.0 Profile and roles played by Field Infomediaries (FIs) Field Infomediaries were recruited by VERS from the same block and district. Young enthusiastic male and female in the age group of 25 to 40 years having varied qualifications were recruited. Many FIs were working with VERS for many years as resource teachers in Bigha education resource centre in Monteswar block and due to their familiarity with the teachers in the block they were selected as Field Infomediaries. All Field Infomediaries have been trained thoroughly on essence of quality education and handling the system. Field Infomediaries played the pivotal role in the service. They were provided cell phones and they were responsible for motivating the teachers and convincing them to use the Lifelines service through these cell phones. These FIs were the grass-root level functionaries who were primarily engaged to receive the questions from the teachers and provide them the appropriate answer. FIs also played an important role in IEC activities, initial icebreaking, building the confidence level of the teachers and help them registering their queries. FIs facilitated the entire process of posting the questions in the server through the voice recording system and were also responsible to replay the recorded answer to the teachers. FIs also maintained regular communication with the knowledge workers to help them to understand the queries. Each FI was allocated one / two Gram Panchayat to provide the service to all schools located in the area. There were total 11 Field Infomediaries working in Lifelines for Education project. All most all FIs reported that they used to initiate an informal discussion while visiting the school, raise probable issues related to subject matter where the teachers might have questions and such discussions helped the teachers in framing their question. FIs also maintained a record book where all questions and answers along with the date and question ID numbers were recorded chronologically. This record book was considered as extremely helpful in case of system failure. For example, where the knowledge workers could not hear the recorded responses due to poor audio quality, or even in case of ID mismatch in the server where question IDs and answer IDs did not match with one another. This record book was also supposed to collect the feedback of the teacher on quality of response on a five point scale. But such information was not collated systematically and the evaluation team could not collect this information from the hardcopy database. 5.1 Experience of Field Infomediaries – challenges faced The major challenge as reported by the FIs was coverage of a large number of schools by a single person. Each FI had to cover 40-60 schools in a Gram Panchayat. Though around 50 percent of these schools were reported as ICDS centers but teachers in ICDS centres also demanded such service though their questions were rarely connected to the objectives of the project. FIs also spent time in each school to have a meaningful discussion, post which the questions were raised. Again, visits to high and middle schools to meet teachers were linked to the free periods of the teachers. Considering all these the average number of schools visited per day was around two to three and almost all FIs reported taking about two weeks to visit all the schools assigned to them. Thus, the second visit to a particular school was only possible after a period of 10-15 days and the teachers who had asked any question during first visit used to receive answer only during the second visit. It revealed that the average cycle time of around 15 days as reported by the beneficiaries was accurate. Average weekly productivity was found to be 25 – 30 questions per week for each FI. The second major challenge as revealed by the FIs was motivating the teachers to utilize Lifelines for Education service. Many teachers were reluctant to use such service initially as they were apprehensive to disclose their professional weaknesses. In addition the traditional rote method of teaching and complete absence of critical thinking or academic discourse has also affected the utilization of the service. Regular meeting and workshops with the teachers, involvement of education administrators, state, block and district level officials, involvement of teacher’s union / association etc have helped the FIs to overcome such difficulties. Though many FIs felt that there was a variation in the usage pattern among the teachers of different types of schools while primary teachers and teachers from SSK and MSKs were quite keen to use the service, but middle and high school teachers were not reported as quite enthusiastic. Further analysis revealed that middle and high school teachers reported having better education profile and better support system in the school like a group of colleagues, library, computer, internet etc and they did not require the service as desperately as their counterparts. Although middle and high school teachers were reported reluctant to use Lifelines service, a group of young teachers appointed in the high schools through the recent School Service Commission Examination conducted by state government were interested in the service. It was found that these young teachers were mostly keen to use Lifelines service as many of them were also aspirants for different competitive examinations and Lifelines service helped them to prepare for these examinations. The analysis of FAQs in next chapter have also reveled a large proportion of questions not directly related to course curriculum in school. These questions were either related to the curriculum at college level or related to general knowledge. Another reason why this service was usually preferred by the young teachers could be possibly due to the fact that young minds have a better adaptability for innovative technology. FIs accepted the fact that many teachers did not ask relevant questions like questions related to curriculum, pedagogy or classroom management etc. In contrary many asked questions related to their personal interest like career opportunity for themselves, family members or relatives or other problems they face in their life. Teachers also asked questions related to their teaching service as well as rules and regulations, for example, rules on number and types of leaves available and types of allowance available. FIs reported that one of the most important challenges was to improve the quality of questions and inspired the teachers to ask good/relevant questions. Few FIs also reported that teachers took more time while recording their questions through IVR system; sometimes the teachers fumbled and could not record the question properly. This was also evident from the Knowledge Workers response’ as they found some questions as long statement and they faced difficulties in understanding the actual query. Most of these teachers never used IVR service and were not at all familiar with such service resultant into such difficulty and this pointed out the need of handholding while recording the queries. System related issues were not reported as a major challenge but few FIs have reported facing issues like system failure e.g. not being able to access the server, save the question or replay the answer etc. Few FIs also reported facing problems where answers saved were not matching with the question due to some mismatch in question and answer ID. But communication with QWSA team helped them to resolve such difficulties. Many FIs have reported that the target of number of questions given to them was difficult to achieve. During the project implementation phase, twice they have received a target of around 1500 questions in a month. Considering the school timetables, different programmes in schools and the number of holidays, raising more than 800 questions in a month was practically difficult. 5.2 Opinions about the service and suggestions for improvement All most all FIs reported the feedback they received from the teachers regarding the answers was satisfactory, be it the content of the answer, the language or the style of answering. Few of them have also claimed that they have seen the teachers discussing the answers received from Lifelines with their colleagues and students. FIs also felt that the types of questions were changed with the time, and teachers started asking more relevant questions towards the end of the project; the questions became more curriculum / pedagogy related and less general knowledge/quiz type questions. All FIs felt that the project should be continued with the support from state/central government, a toll free number should be provided to improve demand and FIs should act as only facilitators or troubleshooters. 5.3 Profile and roles played by Knowledge Workers A team of four Knowledge Workers (KWs) were appointed by VERS in this project. All Knowledge Workers reported having graduation/post-graduation level of qualifications and the team has also received thorough training in handling the system and preparing appropriate answers. A number of workshops and training programmes were conducted by OWSA and VERS to orient the Knowledge Workers in the programme. Though during the evaluation study it was found that initial KWs were not available and a new team replaced the first team of KWs. The new team was also found as competent to handle the system. KWs were also responsible for preparing the appropriate answers for the questions raised by the teachers. KWs reported consulting the reference books, resource materials, surfing the internet, and consulting the group of experts including state level education administrators to prepare the answers. The Knowledge Workers were also responsible for converting the answer in digital voice format so that the teacher could access it by phone. Knowledge Workers also reported using in-house resources at VERS before posting the final answer in the web in digital audio format. Updating and maintaining the data base in the form of questions and answers was found as other responsibilities of Knowledge Workers. Knowledge Workers held the key to the final answers as the final answers were entirely prepared by them. 5.4 Experience of the Knowledge Workers The first and foremost challenge reported by almost all Knowledge Workers was workload or a large volume of questions received from Field. An analysis of major tasks of Knowledge Workers revealed that on an average preparing answer for a single question takes about thirty minutes. But this time varied depending on type of question and ease of availability of the answer and also availability of the experts. One Knowledge Worker can manage around 12-15 questions in a day and around 250-300 questions in a month. The following table 5.1 provides details of different activities taken up by the Knowledge Workers and average time spent on these activities. Table No: 5.1: Activity schedule of the Knowledge Workers Activities Time Spent per question Sorting questions from server Among themselves About 30 Minutes everyday Hosting them in the system About 30 Minutes everyday Finding answers to teacher’s queries from different sources (in house) 10 Minutes per question Talking to experts 5 Minutes per question Preparing the final answers 10 Minutes per question Loading answers to the server 5 Minutes per question Knowledge Workers have reported number of system related challenges which are detailed below: ‰ System running slow and uploading the questions and answers took time.6 ‰ The voice recording of the questions from the Field was not audible, most of the time there was lack of clarity in voice or lots of background noise and hence the teacher’s question was not clearly audible. In such circumstances, the KWs referred to the FIs to get the actual question asked by the teacher and this process increased the cycle-time of the response. ‰ KWs also reported that the FAQ search options in the website was not at all user friendly and to retrieve answers for similar questions, a lot of references had to be provided. 6 The Knowledge workers reported that during the implementation of the project, the server ran slowly from June to August, 2009 for which a large number of questions got accumulated. ‰ KWs have also reported that the time slot available for recording the response was not adequate specifically for answers which require more explanations or examples. Other challenges faced by the Knowledge Workers include difficulty faced in comprehending the questions. It was found that teachers were unable to pose a straight forward question and sometime they pose a question in a roundabout way which was difficult to understand. This issue was also connected to the pronunciation of the teachers or local dialect used in the questions, to answer such questions Knowledge Worker had to coordinate with Field Infomediaries. Non availability of experts due to their busy schedule was also reported as an area of concern by the KWs. Though around 150 experts were registered to provide service but only 20/25 of these experts were found as actively providing support and due to their other responsibilities like teaching etc. sometime these experts were not available or even if they were available they took long time to respond which raised the cycle time. Lastly many Knowledge Workers reported facing problems in questions related to science and mathematics as all of them were from humanities background. Knowledge Workers were asked about examples of questions which they found difficult to respond or which took more time. The following were provided as examples during in-dept discussions and they also reveal the knowledge and skill level of KWs. Box No. 5.1- Examples of difficult questions 5.5 Opinion about the service and suggestions for improvement Similar to Field Infomediaries, Knowledge Workers have also felt that the quality of questions have improved during last six-seven months towards the end of the project. KWs reported finding more and more questions related to the curriculum, subject matter or pedagogy as compared to earlier questions which were mostly general knowledge related or sometimes irrelevant questions. This shows that gradually teachers have gained confidence on Lifelines service and motivated towards using this service more effectively. The Knowledge Workers suggested that number of available experts should be increased further to reduce the cycle time of the response. Knowledge Workers have also felt the need of a more user friendly system with more flexible FAQ search options, the need of a toll-free number for the teachers to ask the questions and integration of the project with other education initiatives for scaling up of Lifelines service across other districts and states. 5.6 Experience of the Contributors / Experts The experts were found to be very enthusiastic about the project; many reported that they were involved in the project since inception. A few of them also reported difficulties in attending calls from Knowledge Workers due their busy schedule and other involvements. The number of questions received by an expert did not pose any major challenge for them. One expert received around 20-25 questions per month, i.e. one to two questions per day. Few experts reported that they can revert to these questions within half an hour but others reported that they require at least 24 hours to revert particularly for critical questions. Experts’ mode of communication with the KWs was entirely over phone and they also reported that they found some questions difficult to explain over phone particularly answers which require illustrations for better explanation. The experts also pointed out that they do not have any command over the final output i.e. the answers in the digital format loaded by the Knowledge Workers. According to them the quality of the final response was entirely dependent on the comprehension skills of the Knowledge Workers. Though the experts were found confident with the skills of the Knowledge Workers but personally they did not have any control on the output. Question 1: When we multiply with two or more digit numbers we leave blank spaces or zero in the second line at right hand side, what is the reason behind this? Comment- This is very basic mathematics related question as we multiply by 10 we add a zero at the right side, so when we multiply by two digit number we add zero in the second line at right hand side. The experts also perceived that sometimes direct interaction with the teachers/beneficiaries would have created greater impact. They did not have any clue on the satisfaction level of the primary beneficiaries as this information was not shared with the experts. 5.7 Experience of the Education Administrators Education Administrators were of the opinion that Lifelines service was a ‘technology-based Ready Reckoner’ for the teachers of rural areas. All most all of them expressed the need of such service for the teachers in primary schools and SSK, MSKs. They also felt that the project should be continued and extended to other areas with the financial support from state/central government. The compendium of the questions and answers on different subjects were highly appreciated. They felt that such compendiums would be very useful for teachers across all types of schools. The experts also pointed out the fact that the project should be integrated with other education programs, training programs and workshops etc. The need of a toll-free number and the direct interaction between the teachers and experts was considered as key towards the future of Lifelines programme. They also opined that the impact of this programme can be assessed only in long run. Chapter-VI CONTENT ANALYSIS OF SELECTED QUESTIONS AND ANSWERS The following chapter details the content analysis of selected Question and Answers across all subjects. As mentioned in the scope of work the entire archive of Q&As was reviewed by the researchers. Effort has been made to select around 250 questions and answers distributed across all subjects through a complete random selection. The selected list of Q&As were shared with VERS, OWSA and Quest Alliance to collect information on profile of users raising these questions and satisfaction level of the beneficiary with the response. This information was not available from the server or hard copy data available with these organisations. In absence of such background information, effort has been made to take up the content analysis of Q&As and under the challenges faced by teachers in rural schools. Table 6.1 shows how many questions and answers were selected for the content analysis: Table No: 6.1 – Selected Q&As for content analysis Subject / Topic Number of Questions in the Server Selected Questions for Content Analysis N % N % English 587 4.8 12 4.8 Bengali 1,269 10.5 26 10.4 Physical Science 582 4.8 12 4.8 Life Science 807 6.6 17 6.6 Mathematics 858 7.1 18 7.0 History 1,055 8.7 22 8.6 Geography 1,614 13.3 33 13.2 Pedagogy 1,959 16.1 40 16.0 Administrative 320 2.6 7 2.6 Policy 170 1.4 4 1.4 Others – General Knowledge/Quiz type questions 2,918 24.0 61 24.1 Total 12,139 100 252 100 6.1 Profile of users Information on type of schools was available for 86 questions only, 60 percent of these questions were asked by primary school teachers, 20 percent were from teachers from ICDS/Anganwadi centres, 15 percent questions were asked by the teachers from SSK/MSK and only five percent of all questions were asked by the teachers from high schools. It is likely that a large percentage of questions were from primary school teachers, the entire information for all selected questions and answers was not available. Question related to science was found as 11.4 percent of all questions. Around seven percent of all questions were related to Mathematics and close to five percent of all questions were related to teaching English. Questions related to pedagogy along with very specific questions related to classroom were found around 16 percent. 6.2 Typical questions and answers Fairly large proportion (close to 80 percent) of questions seems to have little relationship with the curriculum. While there may not be any restriction for teachers seeking general information, this may be seen in relation to the major objectives of the project. A large chunk of questions were asked about History and Geography and majority of these questions were not related to curriculum. For some questions information would easily available in the text books. Many questions read like quiz questions catering to individual interests and curiosities. Table No: 6.2 Few typical questions in History and Geography Sr. NO. Question ID Question Answer 1 22158 Who wrote the 'Dan Sagar' and 'Adbhut Sagar'? Ballal Sen wrote Dan sagar and Adbhut sagar and for that he was honoured in the society of poets and philosopher. 2 22065 The Mesopotamia civilization was related to which age? The Mesopotamia civilization related with copper and bronze age. 3 21975 Who was the first independent king of Bengal? Kutubuddin Iboke was the first independent king of India and for Bengal the first independent king was Murshid Kuli Khan. 4 21724 What is the name of the holy book for Jews? The name of the holy book for Jews is TORAH, it is also known as 'Hebrew Bible'. 5 20338 Which neighboring country of India is known as 'The Last Shangrila'? The neighboring country of India, Bhutan is known as 'The Last Shangrila'. 6 20269 Is Delhi a state or Union Territory? If it is a state then when it got affiliation as a state? In the year 1993 Delhi got affiliation as a state. 7 16744 Which state has highest density of population and which state has lowest density of population in India? The West Bengal is the state with highest density of population and Arunachal Pradesh is the state with lowest density of population. 8 15885 How many islands are there in Sundarban? Of which how many islands are there in Bangladesh and in India? In Sundarban region there are total 102 islands, of which around 54 islands are in India 54 and rest 48 islands are in Bangladesh. There are other specific questions, the answers to which are available in any text book. Such questions are given in the next table. Table No: 6.3: Few typical questions and answers available in any text book Question ID Question Type of school Remarks from consultants 19499 How can we use ‘not only’ and ‘but also’ in the same sentence? Primary school Likely to be available in the text book 21498 What is the smallest number that should be added to 127 such that the number obtained is divisible by 5? Do The logic is too simple 14277 What should be the % increase in rate at which an object can be sold such that there is 10% profit instead of 8% loss? Do Books in mathematics with section on profit & loss carry solutions of such questions 22445 What is the difference between Harappan and Vedic civilization? Do Any History book would have explained this in detail The above examples suggest that teachers were either asking questions not related to the curriculum, or asking very obvious type of questions for which explanation was available in any text book. It has to be understood by the teachers that help can be sought by them for improving their teaching -- which includes knowledge of subject matter; how to transact it; managing large or multigrade classes; understanding how to help children learn particularly children with different abilities; how to evaluate; and how to use information available from evaluation to improve their own teaching or learning by the people. The content analysis reveals that the teachers were asking questions through Lifelines service but these questions were not related to the subject they were teaching in the class, or the method followed for teaching. As experts, education administrators and teachers themselves reported that they do require academic support service this clearly points out that the teachers were afraid of exposing their professional weakness through Lifelines service. This issue should be dealt with utmost care during scaling up operation of Lifelines for Education project. 6.3 Discussion on quality of the response Analysis of answers revealed that Knowledge Workers too need to understand how to respond to these queries. We would discuss different types of response given by Knowledge Workers in the following section: • Answers which are too technical / difficult to comprehend • Inaccurate answers • Unsatisfactory answers • Good answers • In contrast to good answers • Questions related to policy and administrative issues A. Answers which are too technical / difficult to comprehend Example 1: Example 2: For both the questions and answers mentioned above, the questions could be general questions not related to curriculum and the answers given are too technical for any non-scientist to understand anything. DNA, for example, can be explained to non-technical people for its use in a much simple language. B. Examples of inaccurate answer Question ID Question Answers 22707 What is the difference between town and city? The place situated at a high level from the sea surface on a plain land and the place having comfortable atmosphere for living also having better transportation and communication facilities, near agricultural land, having electric supply facilities, better environment for business and trade, proximity of mining facilities, availability of adequate water, administration centre and popular historical places, is called town. And the town where the population is over 10 lakhs is called city. The above question is an example of inaccurate answer where the respondent has used his/her imagination to define the town. Question ID Question Answers 22538 What is DNA? DNA is De Oxiribonuicleic Acid. It is a kind of nucleic acid. The structural chemical element of an animal cellular gene two nucleotide chain develops upward holds each other with the help of an axis and then one atom of DNA is made. Each and every chain is made up by deoxiriboze carbohydrate, phosphoric acid and purine and perimidine acidic base. The nucleic acid of a chromosome is the functional and structural unit of heredity called DNA. Question ID Question Answers 22612 What is protozoa? Protozoa means the first animal. It is a unicellular microscopic animal having clear nucleus. Over 30,000 classes of protozoa live independently in sweet water, sea and in soil. Some of them live dependent on others like plasmodium. Few species of protozoa are able to photosynthesis like euglena. The body part responsible for movement is not visible in protozoa, e.g. ameba. The protozoa’s of sweet water has a hole in the cytoplasm. Generally they complete their reproduction by asexual process but in some species sexual reproduction can be found. C. Unsatisfactory answers Table No: 6.4: Examples of unsatisfactory responses Sr. No. Question ID Question Answers 1 21984 Why there are two times rainfalls in South India? When the north east monsoon winds travel over the Bay of Bengal that time it received some water moisture and become obstructed by Eastern Ghats and as a result it rains in Tamil Nadu, That is why there is two times rainfalls in South India 2 19939 What is the definition of Ancient Period? How can we explain the definition to the children of 4th standard? Approximately 4000 B.C to 476 AD time period is called Ancient Period. You can tell about some incidents/ give some facts and figures of this period to the children so that they can understand the period easily. 3 21428 The Golden Temple of Amritsar was constructed in which Century and in which kings ruling period? It to be said that on 1,574 The Mugal Emperor Akbar donated a land to build the Golden Temple of Amritsar. 4 11746 In mathematics why do we use unit, ten, hundred etc from the right hand side? In decimal system smaller numbers start from right hand side and we move left for larger numbers. This is the reason we have unit, ten and hundred from the right hand side. 5 14662 What is the difference between Punctuation comma (,) and a semi￾colon (;)? When we pause in a sentence we put a comma and when the pause is for a long time we use semi-colon. For ending a sentence we use full-stop. The first question was reasonable and pertaining to the curriculum. The answer does not clearly say why it rains twice. It mentions only one season of rain without giving month etc. The concept of old century would have been explained with some incidence of this time so that children can understand this easily. The Question ID 21428 asked about the year when the Golden Temple was built and the ruling period of which king while the answer talks about the year when the land was donated. For the question on decimal system, the answer does not explain anything. Similarly the answer on semi colon is not correct answers. Table No: 6.5. Examples of Good answers Question ID Question Answers 21201 Which is the oldest university of the world Nalanda or Takshyasila? Till now there is a debate among historians regarding the oldest university of the world between Nalanda and Takshyasila. Many believed that the Al-Azhar of Cairo in Egypt is the oldest university of the world. 21966 Why the king of Egypt is called Pharaoh? The original meaning of the word Pharaoh was ‘Great House’ but later it was used to address the King of Egypt with respect. Approximately from 1500 B.C to 343 B.C all kings of Egypt were called Pharaoh. People used to believe that Pharaohs were the ambassadors of God. These answers really added meaningful information in contrast of one point answer to the other questions. Table No: 6.6. Examples of answers in contrast to good answers Question ID Question Answer Remarks 22116 Which is the driest place or district of West Bengal? Mayureswar of Birbhum district is the driest place of West Bengal The answer is to the point and quiz type answer. Information on annual rainfall or why the area does not receive adequate rain would have made the answer more interesting. 22111 What is the hottest place in the world? Al Azizyah, Libya The answer could be completed with information on temperature, where exactly in the word map Libya is located. 21121 Where is the living place of Tungus Tribes? The northern part of Russia is the living place of Tungus Tribes More information about the Tribes should have added value to the answer. Table No. : 6.7 Questions and answers related to policy and administrative issues Question ID Question Answer 19634 After passing Secondary in 2000, I received an appointment as a primary teacher. I saw in a newspaper that the minimum eligibility is Higher Secondary. Will my appointment be cancelled? Please enquire at DPSC. 19554 One of my colleagues will be absent for four months. How will it be possible for me to handle the center alone? Please talk to the CDPO about your problem. 14895 Whether the Contributory Provident Fund Scheme is still optional or it has been closed? You can contact the District Primary School Council about this Contributory Provident Fund Scheme Many questions were found related to policy, rules and regulations, many of these questions remain unanswered with Knowledge Workers guiding teachers to consult local administrator. A decision need to be taken at the policy level whether such questions are to be entertained or replied to. Chapter VII OUTCOME AND IMPACT Lifelines for Education, the project was envisaged to provide critical academic support and continued instructional training to rural school teachers in order to improve the quality of teaching and learning in classroom and thereby improve students’ learning gains. The project was implemented towards achieving this goal by helping the teachers to be effective in what they do. The purpose of this evaluation study is to assess the outcome and impact of the project in terms of originally stated objectives and expectations. The evaluation study made an effort to assess the following: a) The functioning of the pilot project b) Quantum and quality of help asked for and provided c) Building of knowledge database (Q&As) The outcome and impact of the project in terms of improving quality and relevance of classroom instruction were not assessed as this was beyond the scope of work for this study. 6.4 Key success factor Lifelines for Education, a helpline to resolve the difficulties faced by the teachers in the classroom, is an innovative concept and well accepted both by the teachers and education administrators. The fact that teachers in rural areas face challenges related to subject knowledge and skill sets like the ways to transact the knowledge and help the students to learn was agreed and accepted. A large number of teachers having qualification only till secondary and higher secondary level, lack of adequate resources, library /reference materials / TLMs raise such difficulties manifold. The situation is not going to change in the near future. The evaluation study revealed that this innovative project was well accepted due to its inbuilt system features as well as due to the quality human resources involved in implementation of the programme. The factors which have influenced the acceptance of this helpline were following: 1. Innovative service available through telephone 2. Support available in local language 3. Transfer of information over voice mode 4. Authenticity of information with involvement of well known expert panel 5. Dedicated Field Infomediaries following up at regular intervals 6. Information available at the doorstep without any hassles Vikramshila Educational Resource Society, the field level implementing organisation for Lifelines project, reported their presence in the block with a resource centre in Bigha village Monteswar for more than a decade. VERS was also involved in other education initiatives in the district. So VERS as an implementing organisation for the pilot project in Monteswar block has actually helped the project in initial icebreaking and reach to the beneficiaries/teachers without much difficulty. The pre-project workshops, seminars conducted at different locations in addition to advocacy with the Government functionaries at different level have ensured hassles free implementation. The project was launched simultaneously from two locations, at the block level the project was inaugurated by DPSC chairman and at Kolkata it was initiated in presence of state Project Director SSA Inclusion of well-known teachers/education administrators in the expert panel raised the acceptance level, while a dedicated project implementing team and back office added value in the project. 6.4 Challenges faced in implementation of the project With the back drop of key success factor, there were number of challenges involved in effective implementation of Lifelines for education. a) The role of Field Infomediaries – The profile of Field Infomediaries was not accepted by the teachers. The profile of FI was found as following: young male mostly in the age group of 25 – 40 years with varied level of education and working in the NGO sector. These functionaries were not considered as a peer group by the teachers; they were considered at lower socio–economic level and the teachers rarely came out with the real difficulty with classroom management and other similar issues with the FIs. Though the evaluation study revealed that this team was extremely dedicated and tried their best to inspire the teachers to ask meaningful questions. Due to constant pressure of these FIs teachers sometime asked irrelevant questions just to do favours to these functionaries. Many teachers have revealed during the prompted responses that they have used the service only on request of FIs to help them perform well in their job. b) Involvement of functionaries inside the education system, e.g. BRCC, CRCC and SI, was found marginal. All these functionaries were aware of the project but they rarely played the role of effective Infomediaries in the pilot project. c) Anonymity was not maintained and to verify the usage pattern, the teachers were asked to reveal their name and school name which has also created some kind of apprehension on the part of the teachers and many of them considered the service as a system adopted by the government to measure the proficiency level of the teachers. d) Communication strategy adopted in the project was not found very effective and a large proportion of teachers reported not being aware of the helpline number. Many reported that they could not use the system on their own because they were not aware about the helpline number. e) Initial enthusiasm was missing during the end of the project, numbers of Field Infomediaries were reduced, the entire team of Knowledge Workers was replaced and irregular visits by FIs to many schools were reported as one of the main reasons for not using the service more effectively. f) Skill set and role assigned to facilitators – The average cycle time was rarely found as 48 hours or less. The average cycle time varied between 10-15 days as reported by most of the beneficiaries. The average cycle time was entirely dependent on the second visit of Field Infomediaries in the school. As discussed earlier the schools were assigned to FIs in such a way that it was rarely possible for FI to visit a particular school more than once in fifteen days. Field Investigators also reported not maintaining any systematic movement plan. Many time depending on the urgency of the response, FIs reported visiting a school more often. g) Knowledge Workers skill set was critical for project implementation. It was reported that almost all Knowledge Workers were reported more proficient in humanities and had difficulties answering queries related to science and mathematics. h) Knowledge Workers’ understanding on how to prepare a quality response with information collected from the experts/other sources, was considered important in implementation of the project. Many questions read like a quiz type questions and answers were found to be brief and one point answers, without adding any meaningful information as discussed earlier in the content analysis section. i) Experts also need to understand the level of comprehension of teachers having studied only 10th or 12th standard. Answers using many technical words could be difficult to comprehend by persons not having studied a particular subject, particularly Science. j) Involvement of college lecturers and professors in the expert panel might have created such responses which were technical in nature. Well experienced school teachers would be more appropriate to function as experts in this project. 7.3 System related challenges User friendly system with ease in operation is imperative for effective implementation of any technical system. As discussed in earlier section the Knowledge Workers have faced number of challenges while operating the system. A large number of questions were found in the server and were marked as voice not audible due to technical difficulties.7 But these difficulties were also partly related to the lack of awareness of the teachers’ on how to record a question, with clear pronunciation, proper pause, and less background noise. The system should sort out such issues. Knowledge Workers have also reported the need of increasing the time slot for recording the response. The present time slot was reported as around two or three minutes. The user friendly FAQ search option was also perceived as an area which needs immediate attention to reduce the duplication of work on part of Knowledge Workers. SMS alert to FI as soon as the response was ready was another requirement as expressed by many FIs and KWs and the system did not have such in-built option available. In the present system, FI had to call and verify whether answers were ready or not while such additional service might not be required if the cycle time can be reduced for all responses. The evaluation study also revealed that the pilot project did not follow any systematic data base management process which needs to be look upon for scaled implementation. The evaluation team has faced difficulties in tracking query ID number with the profile of the users. Additionally, mismatch in query ID vs. response ID was also reported. The evaluation study also tried to collect the profile of the user and satisfaction level with the response from backend information but such information was not available, even the number of calls received from a particular number, calls received from outside the coverage area were not available from the system. 7.4 Project Outcome Lifelines for Education service has helped the teachers to get answers related to their specific queries. Many teachers have considered the service as an easy way to get correct answers to their queries related to subject matter, general knowledge and even personal questions faced by them in their day to day life. Very few questions were actually related to the issues faced in the classroom. It seems many a times teachers were asking questions not related to the subject they teach in the class, a large proportion of questions had little relationship with the curriculum. There were also good questions generated through this system and if some teachers have benefited through the responses the project should be considered as successful in providing support in quality education. The pilot project has proved its authenticity as an additional referral system and it seems beneficiaries have used this system to raise their knowledge level, to meet their own aspiration, and to resolve their difficulties they face in their own life. However, how far such information was beneficial for the classroom teaching remain questionable. The true knowledge gaps of the teachers could not be identified through this system and the impact the system has to be assessed in the long run. 7 The exact number of such questions could not be determined. Question - One of my students knows all answers but he can't write, whenever he writes his answers do not match with what he speaks, how to improve this situation? Answer - The child has got poor writing skills, he should practice writing more often. Please speak to his guardian, so that they help him to practice writing more at home. But here a specific process needs to be followed. e.g. when you ask him to write on something, please ask him to write 2 lines first day, 3 lines second day, 4 lines next day and in this way if you develop his habit of writing based on his own thoughts while increasing the content gradually he would be benefited. In addition to this he might not be able to write during exam within the given time and he might have lost his confidence due to this. You have to raise his confidence level; he should believe that he would be able to write like other students. Chapter VIII SCALABILITY AND SUSTAINABILITY OPTION This chapter provides an overview of viability option for the project and the scope of expansion across other districts and other states. 8.1 Suggestions for improvement The major suggestions which has come out across all quarters were as following • Reduce cycle time for the response to 24-48 hours: Teachers, experts and other facilitators have suggested that the ideal cycle time for the helpline should be less than 48 hours. • Toll free numbers: Many helpful suggestions came out during this evaluation. Many respondents reported that they are quite confident in handling the system on their own and no training is required to help them use help line service. • Quality of the responses: The quality of response was reported as another key concern and as presented earlier, the success of this helpline in the future would depend a lot on the quality of responses generated by the system. More meaningful, relevant, easy to comprehend information is the requirement. • Involvement of BRCCs, CLRCs and SIs was reported as important. The role of FIs might not be entirely replaced by BRCCs, CLRCs and SIs given their other administrative responsibilities. Therefore, FIs should be there only for technical hand holding or as facilitators or might be responsible for overall monitoring while the functionaries from SSA should take the role to motivate the teachers to utilize the service to resolve the challenges they face in their classroom teaching. • In addition to voice mode response, answers should be available in written format or in the form of booklets, or can be discussed over other media channels. • Active awareness activity is required to improve the demand of such service. • Anonymity of the user is extremely important to avoid apprehension on part of teachers; toll free number direct dialing facility would ensure anonymity. • Usage of functionaries within the system would give the teachers more confidence to utilize the helpline service. • The programme needs to be effectively linked with other education related programmes and activities such as in-service training programme, seminars and workshops conducted under SSA. • Team of Knowledge Workers should have different backgrounds as subject experts. More persons knowing mathematics and sciences may be deployed. • Library and other resources available to the Knowledge workers should be strengthened. • System design need to be revisited to make the system more user friendly, like clarity in voice recording, ease in FAQ search options, increasing the time for recording the response. • Active involvement of experts in preparing the final response, experts should monitor the final response periodically to ensure the quality of response. • Experts should be allotted specific timeslots to answer the questions forwarded by Knowledge Workers. • Satisfaction level of the beneficiaries should be tracked and this information should be shared with all project functionaries at regular interval. 8.2 Future model of Lifelines for Education service The evaluation study has also proposed the future model for Lifelines for Education system as following: • The service should be extended only up to the level of middle school teachers since the teachers in high schools have alternative support system available to refer in case of difficulties. • ICDS centres might not be included in the project since these teachers were not providing any knowledge input. • Direct line (if possible a toll free number) should be the most preferred option, where teachers should ask questions directly without any intermediary and maintaining complete anonymity. • Though the willingness to pay was not exactly tested in this research, it was felt that many beneficiaries would not mind mind paying for the call charges which is around Rs. 10/ for each question and answer - provided that they are completely satisfied with the responses. It was also felt that if call charges can be reduced from STD call to local call it would be definitely an added advantage. • Involvement of SSA functionaries/CLRCs is proposed to motivate the teachers to use the service. CLRCs are also experienced teachers; they should also have the options available with them so that they can answer the question themselves but the questions should be forwarded to the helpline. • Teacher’s apprehension in expressing the real challenges faced in the classroom; should be handled carefully. Even in case of involvement of government machineries, they might not feel confident to utilize the service. To win their confidence, the project should be continued for a longer period and the teachers should be explained that they are not victimized for using this service. • FIs should act more as a contact point, as a facilitator/campaigner to improve the usage or as a troubleshooter for technical difficulties and should work without the pressure of generating a large number of questions. • The profile of FIs should include few senior / retired experienced school teachers instead of young graduates. • Knowledge workers should be recruited from diverse background and they should be thoroughly trained on how to prepare good answers. • The resource base available to the Knowledge Workers should be strengthened, text books and reference materials published by NCERT, CBSE board and others can be referred as additional resources. • If the resource base is strengthened, the involvement of experts can be reduced and cycle time can be improved. • Cycle time has to be less than 48 hours in all cases. • Publicity of the service using local media channel/newspapers and publication of interesting questions and answers in local newspaper would enhance the visibility and demand for such project. • Institutional mechanism of proposed model should involve a combination of government and voluntary organizations. • Archive of 12,000 questions and answers generated in the pilot project should be shared with broader audience. In addition to publication of booklets, such questions and answers can be published in local newspapers and discussed during radio education programmes. • This archive should be shared with the text book writers and education administrators. Future Model of Lifelines for Education service 8.3 Alternative model for self sustenance To address the sustainability option for the project, in the following section, the consultant has also proposed a few alternative models involving different target groups to cross subsidize the part of the project implementation cost. Sustainability option 1: Lifelines for Education service could be available to the students in higher classes with career counseling option. The students and parents can call the number and ask their questions related to various career options and ways to pursue those careers and other relevant information. The users would be paid while availing this service and this would cross subsidize the service available for teachers. The experts felt that such model would be in high demand across the community, this might not be directly linked with the project objectives and the regular updated information on career options need to collected and update at regular interval from reliable sources. Sustainability option 2: As a second option, it is suggested that the publishers and booksellers can be roped into the project and information on additional text book; reference book etc can be shared with the teachers in rural area. This should be built in as an additional system feature where information on different reference books would be recorded and stored in advance and teacher would select the require option to hear the recorded information on additional support system . This option is partially meeting the project objective. But the relevant changes in system would be required and information need to be collected from publishers and booksellers, name of the book, information on content, market price, publisher name and address should be available to the end users. The model can be cross subsidized with the help of publishers and booksellers. An in-depth study would be required to test the pros and cons for both these models. Teachers directly using the service of 'Lifeline for Education' Dialing a toll free number / IVR menu driven service and recording the Qn. Knowledge Workers accessing the server to retrieve the question Skill building for Knowledge Workers to frame the answer in most appropriate way CLRCs / SI working as facilitator / motivators Expert interacting closely with experts located in the same premises/ fix time schedule for experts Publish interesting topics / issues in local district level Newspaper/ magazine, Radio programme t Integrate the programme with other Education Initiatives FI should only support in campaign, hand holding, troubleshooting (technical) etc Primary schools – till middle school level 8.4 Conclusion • The pilot project was implemented for nearly two years, covered almost all government primary, middle and high schools in the block along with ICDS centres. Many ICDS centres were covered partly due to the pressure from the local community/Panchayat and also to meet the target given to the Field Infomediaries to raise a substantial number of questions in each month. • The average number of users per school was around two though a very large number of questions were generated through the system compared to the number of users and actual usage pattern reported by the users. Average cycle time proposed for the study was not maintained, there was ample scope of improvement on quality of response. • Few questions were actually related to course curriculum or pedagogy; the service could not identify the true gaps in knowledge and skills of the teachers. • The evaluation study revealed that the pilot project has not achieved its objective fully. It was felt that such projects should be given adequate time to overcome initial hurdle, to win the confidence of the teachers. This suggests that the project should be continued with mid-course corrections. • Involvement of education functionaries within the system was required, the system design need to be changed to adopt more flexibility and user friendly approach and involvement of teachers association and senior teachers is found as important. • It was felt that with all these midcourse corrections the project can actually play an important role in improving overall quality of education. VIDEOSHALA CONSOLIDATED EVALUATION REPORT July 2009 BY CENTER FOR MEDIA STUDIES (CMS) NIRANTAR Research House, Community Center B 64, Sarvodaya Enclave Saket, New Delhi 110017 New Delhi 110017 Phone: 011 2685 1660 Phone: 011 2696 6334 Email: info@cmsindia.org Email: nirantar@vsnl.com http://www.cmsindia.org http://www.nirantar.net TABLE OF CONTENTS 1 EXECUTIVE SUMMARY 2 INTRODUCTION 2.1 CHAPTER BREAKDOWN 3 BACKGROUND OF VIDEOSHALA 3.1 VIDEOSHALA PARTNERS 3.2 MAKE UP OF EACH ECVU 3.3 VIDEO PRODUCTION MODEL 3.4 COVERAGE OF VIDEOSHALA 4 EVALUATION METHODOLOGIES 4.1 CMS EVALUATION METHODOLOGY 4.2 NIRANTAR EVALUATION METHODOLOGY 5 CONTENT ANALYSIS OF VIDEO FILMS 5.1 HIND SWARAJ MANDAL (HSM) ECVU 5.2 MEGHDHANUSH ECVU 5.3 NAVSARJAN ECVU 5.4 SAHAYOG ECVU 6 TRANSACTION AND IMPACT ANALYSIS OF VIDEO FILMS 6.1 HIND SWARAJ MANDAL (HSM) ECVU 6.2 MEGHDHANUSH ECVU 6.3 NAVSARJAN ECVU 6.4 SAHYOG ECVU 7 PROCESS ANALYSIS OF VIDEO FILM MAKING 7.1 HIND SWARAJ MANDAL (HSM) ECVU 7.2 MEGHDHANUSH ECVU 7.3 NAVSARJAN ECVU 7.4 SAHAYOG ECVU 8 TECHNICAL ASSESSMENT OF VIDEO FILMS 8.1 HIND SWARAJ MANDAL (HSM) ECVU 8.2 MEGHDHANUSH ECVU 8.3 NAVSARJAN ECVU 8.4 SAHYOG ECVU 9 VIDEO FILMS LEARNING OUTCOMES TESTS 9.1 HIND SWARAJ MANDAL (HSM) ECVU 9.2 MEGHDHANUSH ECVU 9.3 NAVSARJAN ECVU 9.4 SAHYOG ECVU 9.5 SUMMING UP 10 ROLE OF LOCAL COMMUNITY IN VIDEOSHALA 10.1 INVOLVEMENT OF COMMUNITY VIDEO PRODUCERS 10.2 IMPACT ON COMMUNITY VIDEO PRODUCERS 10.3 CHALLENGES 10.4 VIEWS OF OTHER STAKEHOLDERS 11 VIDEO FILMS SCREENINGS AND THEIR IMPACT 11.1 CLASSROOM FACILITATORS VIEWS 11.2 STUDENTS VIEWS 11.3 TEACHERS VIEWS 11.4 PARENTS VIEWS 12 FINDINGS & RECOMMEDATIONS 12.1 CURRENT STATE OF VIDEOSHALA 12.2 RECOMMENDATIONS FOR THE FUTURE 12.3 PARTNERS VIEWS ON FUTURE STRATEGIES 12.4 SPECIFIC RECOMMENDATIONS FOR EACH ECVU APPENDIX I 1 EXECUTIVE SUMMARY The Videoshala project, implemented in Gujarat since 2007, was created by two NGOs: Drishti and Udaan, in partnership with the QUEST Alliance. Four other NGOs: Sahyog, Meghdhanush, Hind Swaraj Mandal, and Navsarjan were then chosen as partners to set up and manage four Education Community Video Units (ECVUs) that focus on producing relevant educational videos for local children in local schools. These are community videos – videos created by local people who are trained as full-time community video producers (CVPs), by the founding NGOs and external experts, and who are members of the ECVUs. The videos are on subjects that the students find hard to learn, teachers find difficult to teach, and are imbibed with values of democracy, citizenship and diversity. These videos are shown in classrooms to children in local schools. Following the screenings a series of activities and interactive sessions are carried out by classroom facilitators who are also members of the community as well as the ECVUs. Two organizations: the Center for Media Studies (CMS) and Nirantar were chosen to carry out separate evaluations on the Videoshala project in 2009. This executive summary provides an overview of the findings of the two evaluation reports. The objective of the CMS evaluation was to track the extent to which Videoshala has achieved its objectives; to examine the progress and growth of the participants/stakeholders in the project; to measure the learning gains of students in Videoshala schools compared to non Videoshala schools; carry out a technical analysis of the videos; to identify the factors exerting the strongest influence in effecting change among children, teachers; and the challenges faced. CMS’s research design consisted of face to face and focus group interviews with students, teachers, parents, ECVU members and other stakeholders; field visits to the communities, schools and ECVUs; and the administration of a quantitative test to select students in Videoshala and Non Videoshala schools. The objective of the Nirantar evaluation was to assess the content, quality and the screenings of the video films produced by the four ECVUs. The evaluation process included an in depth qualitative review of a select number of videos; extensive desk review of Videoshala proposals, quarterly program reports, worksheets and activity sheets; field visits to the four ECVUs and to select Videoshala schools where the video films screenings were observed; and face to face and focus group interviews with students, teachers, ECVUs and other stakeholders. Targets Achieved Reflecting upon the process of the project initiation and implementation, all the project partners were of the unanimous view that Videoshala project has been by and large successful in achieving its intended objectives. They felt that one of the key achievements of the project was that it was implemented in the manner in which it was intended during the project planning. Further during the overall project intervention over 21 months (since 2007) Videoshala: • Reached 7,957 children (between ages 6 and 14) in 194 schools • Provided alternative livelihood opportunities to about 24 young people as Community Video Producers • Built the capacity of 21 local Classroom Facilitators to participate in their community’s educational process • Developed 24 videos on various classroom subjects that integrate the values of democracy, citizenship and diversity • Screened the videos in over 640 classroom • Strengthened six local NGOs to create compelling and effective learning materials Total number of students participated Primary Secondary Total Male Female Male Female 3,060 2,570 1,464 863 7,957 Video Films The evaluation of the video films of the four ECVUs revealed: • The videos produced by all four ECVUs are child centric. All videos have children as the main protagonists. Their questions, inquisitiveness and anxieties are addressed in all films. Yet they are not shown as being ignorant or uninformed, but in many films are confident and articulate. It is either a child or a group of children who take the viewers through a film and learn new information and ideas along the way. This learner-centered strategy ensures that children – the main target audience of the videos – are involved and engaged as viewers and can relate to the content easily. • In most videos children are encouraged to learn by gathering information, going to different places, conducting interviews and doing things themselves. Seeking information is an effective way of learning and this is a method that most videos have adopted. This is in contrast to the conventional top-down approach of the teacher as the only provider of all answers to children’s questions. • The videos are based on chapters from Gujarat State Board School Textbooks. Core curricular areas in textbooks are referred to and yet in most cases there is an attempt made to broaden the horizons of learners by providing new information or by contextualizing the information. The information in textbooks is dull, detached, simplistic and in some cases, problematic. In contrast, several videos are refreshing as they are located in the lived realities of learners – they show their own geographical areas, people (and children) who are similar to them and speak the same language. • People from marginalized communities are shown in positive roles in most videos. There is no stereotyping of roles or professions. People from Dalit, Muslim and working-class backgrounds are adequately represented. Their knowledge and skills are treated with respect and their contribution to children’s learning process is recognized. This effort is well appreciated; however, in some films there appears to be a tokenistic representation of marginalized groups. • Most E-CVU videos are well produced. They have used different genres of film making. Both fictional as well as non fictional ploys have been used in videos and this effort is commendable. There are plots and sub plots which are woven together in a coherent manner. The visual medium has been used to its fullest potential in many films. There are a few aberrations to this – some videos are not well shot, have plots which are unrelated and end abruptly. • The videos produced by all four ECVUs are unique in that they provide information and address values of citizenship, diversity and democracy. This is unlike all other audio-visual educational resource material available to schools in the area. This is quite commendable for the program. Teachers and principals of both government as well as non-formal schools also appreciated this feature of the ECVU videos. The information provided enhances quality of education in schools and values incorporated into the subject areas enable learners to think critically. • Most videos have made an effort to incorporate values of equality, democracy and diversity in different ways, however, this is not done in a way where values implicit or embedded in the content. However, in the case of some videos on Science and Language, it has been challenging to incorporate values. Implementation of Video Films The following issues relate to the implementation and screening of the video films in the target schools by the ECVUs. • In schools where the videos are screened, facilitation is as important as the screening itself. In the absence of facilitation, the video becomes an interesting activity that does not necessarily stimulate critical thinking. In schools where facilitation was good, the video was very effective. Children were able to understand concepts and information provided. However, where facilitation was absent, the video was like an ‘interesting activity’ which was not linked to curricular issues. • In most ECVUs very little time and resources have been invested in training facilitators. As a result of this, it is up to each individual facilitator to decide how he/she will transact a video. It is based on their existing skills and interest that this is decided. In some cases, screening and transaction of films is not prioritized given the existing work commitments of facilitators. However, in ECVUs where orientation and inputs have been provided to the facilitators, the classroom transaction is creative and inspires children to engage more with the video and the activities. • Each E-CVU has worked out its own process of involving facilitators in the process of film production. In some ECVUs feedback is sought from facilitators at different stages. • There is no common point at which a video is introduced in the school syllabus. It is up to each E-CVU to decide whether a video will be screened before/after/during the time that the topic is being covered in school. This also depends on the availability of the facilitator in some cases. This lack of clarity and ad hoc manner of transaction has led to some amount of confusion in the ECVUs. In some places there is a long gap between the time that the topic is covered and the screening, making the impact or effectiveness of the video inconsistent. • More discussion is needed on values, imbibed in the videos, in the classrooms. Facilitation is usually around information and concepts that are introduced. While values are in the realm of the abstract and therefore difficult to discuss, more effort has to be made to raise questions or debates around values of diversity, citizenship and democracy. • In some ECVUs, teachers of schools are also involved in the process of hard spot selection and in developing scripts and providing feedback on the rough cut. However, in other ECVUs, the facilitators and teachers have no role whatsoever in the film-making process. Cost Viability of Videos The video production was found to be cost effective in the following manner: • The estimated cost of the production of the videos as reported by stakeholders and technical support partner (On an average Rs.1.5 lakhs per video) was kept in mind while evaluating the expenses incurred. The actual expense as reported by NGOs heads and technical partners was also close to the same amount. • The same job would have been done by professional at three times the above cost, which shows the viability of using CVPs. • The cost effectiveness of the project also increases due to the training of the CVPs as they continue to make additional videos and also contribute in training any new CVPs that are hired. • Based on analysis of the videos by technical evaluators, it was concluded that the final product meets the desired standard of video production given the costs incurred. Connections with the Community Regarding the connections between Videoshala and the community, the hiring of local community members to become community video producers (CVPs) in the ECVUs was a very enriching experience. All the CVPs were of the view that Videoshala had given them a chance to do something for their community in a very positive manner along with helping them build their own individual skills. The classroom facilitators (CFs), who were also chosen from the local communities, felt that the project was providing them a unique opportunity to help their community along with gaining them meaningful employment. Additionally each ECVU has a team including women and people from marginalized communities and this composition reflects the diversity that exists within the project, which also enables the ECVUs to bring in different experiences and learning, from the local community, into the film-making process. Learning Outcome Tests The objective of the learning outcomes tests, carried out by CMS, with groups of ten to twelve students in each of the selected Videoshala schools and non-Videoshala schools (where students had not viewed the Videoshala films) was to make an assessment of the extent to which the videos were helpful in facilitating and enhancing the mastery of hard spots. Overall, the learning outcomes tests of each ECVU did show that the level of understanding, as projected by performance on the post-test as well as in comparison to a matched control group, amongst the Videoshala students has improved. However, the extent of recall of the content of the video was not found to be very satisfactory on some questions. This could be due to four possible reasons: 1) the time gap between screening of the videos and conducting the post-test; 2) in spite of the clarity on the issues, students were not able to express it through words in the learning outcomes test; 3) the post-screening interactive sessions were not able to clear the doubts in the mind of the students; and 4) some of the videos and learning outcomes tests need to be reviewed for their reliability and validity of content. The graph below shows the percentage of students giving right answers for questions related to two videos, each, made by the four ECVUs. Percentage of students giving right answers 0% 10% 20% 30% 40% 50% 60% 70% 80% HSM Meghdhanush Navsarjan Sahyog ECVUs Learning outcome Videoshala students Non Videoshala students Baseline students Learning from Videos According to the community facilitators (CFs) who were the first people who could observe and receive the immediate reactions, responses and queries of the target audience (i.e. students and teachers), the videos have not only been beneficial in knowledge enhancement in the students but other positive impacts have also been observed in the children. After screening of one or two videos, the children shed their shyness in asking questions and became more interactive, and were able to infer and understand the content of the subsequent videos more clearly. The videos made the children more aware, thoughtful and proactive on issues such as pollution and environment. The CFs also felt that the teachers took the videos very positively, which not only proved helpful to them to understand subjects and themes difficult to transact but also in their teaching methodology. The analysis of the qualitative data gathered on students showed that nearly all of the students liked the videos however the recall level of the content of videos was greater among the students of higher grades than of lower grades. The students of higher grades were also more able to articulate that the videos not only provided information on the difficult themes to understand, but they also made an impact on their thinking and behavior. They reported that after watching some videos on Pradushan, Vanaspati, Kudrati etc they have become more serious towards keeping their surroundings clean and green. The video Sthanik Swaraj Ni Sansthano also made them aware about functions of local self-governance and rights of a citizen. Many students also felt that the use of the local language in the videos as well, as the familiarity with illustrations and examples, made it easy for them to identify with the content. One of the key objectives of the Videoshala project was to help imbibe the feeling of citizenship and democracy amongst the students, which was also more evident in the responses from students of higher grades. The interaction with the teachers in the Videoshala schools demonstrated that some teachers were involved in the identification of the hard spots to be addressed in the videos; however, this number was not a majority. The teachers’ involvement during the screening of the videos in their respective schools was also limited. Nonetheless, almost 100 percent of the teachers were convinced of the effectiveness of the videos in helping in their teaching. A good majority of the teachers rated the content of the videos as very good and relevant; presentation of the videos as attractive; language of the videos as easy to comprehend; and overall quality of the video as very good and attractive. The teachers further added that their own understanding of the topics/subject matter got enhanced. For example, one of the teachers reported after watching a video on Vanaspati, that it came as a new learning to him that bamboo was a type of grass, and that it is the longest grass. Regarding the change the teachers observed in the students as a result of watching the videos, a majority of the teachers mentioned that the students had more clarity on the issues that the videos tackled. Many teachers also noticed the change in behavior and conduct of the students towards their peers of other communities and social groups. The students, they felt, have also become conscious towards keeping the surroundings clean and green. Furthermore, in non-Videoshala schools, the teachers were informed of the model of Videoshala and they agreed that difficult subjects and themes in the curricula can be better addressed through videos. Though they had not watched any videos of Videoshala, they were of the view that the idea of developing videos on hard spots in the curriculum by the community members in the local language could be very effective and relevant. Interviews with the parents of Videoshala students revealed that most of them were aware that videos were being shown to their children in the school. All the parents were appreciative of the video-inputs being used to facilitate the learning of their children. According to many of the parents, when the children came home after watching the videos they shared this experience with them. Regarding whether the parents observed any change in their children’s behavior subsequent to watching the videos, most of the parents mentioned that they found many positive changes in their children. Mr. Pankaj Lalji Bhai mentioned that the thinking of his son towards Dalits and other communities has changed as now his son has friends from Dalit and other communities which were not there before. Recommendations for the Future The CMS and Nirantar evaluation teams, along with the other respondents they interviewed, also made some recommendations to expand and grow the project in the future stating: • It is important to develop future content of videos according to the specific age and learning levels of children. • The issue of integration of values in films on Science, Language and Math has to be addressed. These are topics which are most ‘difficult’ for learners; therefore it is all the more important to integrate values in these topics – a task that is challenging, but critical for the project. • There should be a strategic involvement of facilitators/teachers in the process of the production of videos, especially in the selection of hard spots. Further the role of teachers in the classroom screenings has to be expanded. If the project plans to scale up its operation and involve government school teachers as facilitators, this involvement has to be very strategic in nature. • Efforts should be made to get in touch with the school authorities at Block Resource Centers and Cluster Resource Centers to make them aware of the Videoshala concept. This would potentially help Videoshala in getting permissions to screen the videos in government schools. • In many schools the facilitators are the face of the program. The larger objectives of the Videoshala program and the local producers are not known to people in schools and to the larger community in many places. Since one of the objectives of the program is to empower local people to produce educational material to improve quality of learning, it is essential that this information be shared with various stakeholders. The E-CVU should look at additional forums where its work can be promoted and publicized. • ECVUs should also screen the videos in the community so that additional children and teachers of other schools as well as parents can view them. • More connections should be made with additional NGOs and other private organizations to become partners in the project. 2 INTRODUCTION A growing trend in India is the use of Information and Communication Technologies (ICTs) in various types of primary education interventions. However many of these projects relegate local communities to a position of passive receivers of externally made content, while never letting them be the creators of the content themselves. Hence there are few examples of ICTs based projects that are entirely owned by the community. Videoshala is thus a unique attempt in allowing the participation of local communities in creating educational content, imbibed with democratic values, while simultaneously providing them ownership over technology. The Videoshala project, implemented in Gujarat since 2007, was created by two NGOs Drishti and Udaan in partnership with the QUEST Alliance. The objective of Videoshala is to set up Education Community Video Units (ECVUs) that focus on producing relevant educational videos for local children in local schools. These are ‘community videos’ – videos created by local people who are trained as full-time community video producers. The videos are on subjects that the students find hard to learn, teachers find difficult to teach, and are inculcated with values of democracy, citizenship and diversity. These videos are shown in classrooms to children in local schools. Following the screenings a series of activities and interactive sessions are carried out by classroom facilitators who are also members of the community as well as the ECVUs. These sessions encourage children to think more broadly and holistically about the content of the videos which address values as well as the hard spots in the curriculum. Videoshala entails: • Use of locally trained community members to produce quality video based education content for local primary schools; • The inculcation of values of diversity, democracy and citizenship in the educational content; • Formulating a well-crafted pedagogical process as an instrument through which a child learns, forms opinions, beliefs and starts to understand the social realities of his/her environment; • Creating an educational process that supports values, positive attitudes, essential life skills, critical awareness, and an understanding of the environment in the context of diversity. This consolidated evaluation reports combines the result of two separate evaluations carried out by Center for Media Studies (CMS) and Nirantar on the Videoshala project in 2009. CMS is an organization working in the field of media and education and is based in New Delhi. Nirantar is an organization that serves as a center for gender and education and is also based in New Delhi. The broad objectives of the CMS evaluation were to track the progress of Videoshala in meeting its objectives, to identify the factors helping it achieve success and identify challenges and future strategies. The broad objectives of Nirantar were to evaluate the Videoshala films and to assess their content, quality and pertinence to the target populations. 2.1 CHAPTER BREAKDOWN The succeeding chapter provides a background on the Videoshala project by enumerating and describing its partners and their responsibilities, the make up and organizational structure of the ECVUs and the sequence of the video production model. Chapter 4 lists the methodologies employed by the CMS and Nirantar teams in collecting and analyzing their data for their evaluation reports. Chapter 5 provides a detailed content analysis of 7 Videoshala films made by the four ECVUs, carried out by the Nirantar evaluation team. Each film is analyzed based on the: the relevance of the film to the audience; the comprehensiveness of the subject matter; the implications of using video as a medium; creativity of the content and form; the kind of information communicated in the film; how the film integrates values in its content etc. Chapter 6 follows this up by providing an evaluation of the transaction and impact of the seven Videoshala videos. The analysis is structured around the video screenings the Nirantar evaluation team observed and is supplemented with data from interviews and group discussions with students, facilitators and teachers. Chapter 7 continues the analysis of the seven video films by concentrating on the process of making the seven video films. Chapter 8 provides a technical assessment of 23 video films produced by the four ECVUs, on a list of ten criteria, carried out by the CMS team. Appraisal of all the twenty-three videos was done by the CMS evaluation team and supported by a technical expert, who had rich experience working as a senior producer in a TV News Channels. Chapter 9 demonstrates the results of the learning outcomes tests carried out with groups of ten to twelve students in each of the selected Videoshala (experimental) and non-Videoshala (control) schools by the CMS evaluation team. The objective behind this learning outcomes test was to make an objective assessment of the extent to which the videos had been helpful in facilitating and enhancing the mastery of hard spots. Chapter 10 includes the results of the evaluation, carried out by CMS, on the involvement of the local community especially of the community video producers in the Videoshala project. This chapter lists the benefits and challenges the project has brought to the producers. Chapter 11 enumerates the responses obtained by CMS’s evaluation team from classroom facilitators, teachers, students and parents on the Videoshala video films screenings and their impacts. The concluding chapter, chapter 12, discusses the findings, recommendations and future strategies to consider for Videoshala emanating from the evaluations carried out by CMS and Nirantar. The CMS and Nirantar evaluation teams, along with other respondents they interviewed, deliberated upon important aspects of the project such as the relevance to the existing educational and school scenarios, feasibility of the modalities adopted, components of the program, outcomes realized, overcoming limitations, future strategies etc. This concluding chapter discusses the overall success and challenges of Videoshala and the scope for its future scalability and sustainability. 3 BACKGROUND OF VIDEOSHALA 3.1 VIDEOSHALA PARTNERS There are a variety of partners working to make Videoshala a success for students, teachers and the local communities in Gujarat. The first type is those who manage the education community video units (ECVUs) and implement the project in the field, and the second are those who provide technical support/skills to the ECVUs. Besides the implementing and technical support partners a Central Coordination Unit (CCU) has also been created to coordinate and support the ECUVs. The NGOs, who are the implementing partners and manage the ECVUs are Sahyog, Udaan-Meghdhanush, Hind Swaraj Mandal (HSM) and Navsarjan Trust. Sahyog (Urban ECVU): Sahyog is a grassroots organization based in Ahmedabad. It emerged after the communal violence in Gujarat in 2002, with the experience of working for the relief and rehabilitation of the affected community. It is working with Muslim children and youth to improve the learning levels of children and facilitate mainstreaming of children in this area. It currently runs six educational centers in Vatva region of Ahmedabad. More than 400 children are covered directly under its education program. Sahyog has implemented the Videoshala project in 30 schools of the Batwa slum of Ahmedabad city. Udaan-Meghdhanush (Rural ECVU): In addition to being a resource centre, Udaan is directly involved in improving the quality of education in Panchmahal district through the Meghdhanush program. Udaan is working with approximately 75 government schools in order to improve the quality of education, by setting up education centers in each school and has implemented the Videoshala project in these schools. Hind Swaraj Mandal (Ideology Based ECVU): Hind Swaraj Mandal is a registered voluntary organization working on issues of education, research, constructive work and social organization in Saurashtra region of Gujarat since 1988. One of the major interventions of Hind Swaraj Mandal is to modernize Nayi Talim education introduced by Mahatma Gandhi. HSM implemented the Videoshala projects in 30 Uttar Bunyadi and Nayi Taleem residential schools of Bhav Nagar, Surendra Nagar, Ambreli and Rajkot districts of Gujarat. Navsarjan Trust (Dalit Dignity ECVU): Navsarjan Trust, a leading Dalit human rights organization, works in over 3,000 villages in Gujarat. Its mission is to eradicate untouchability through legal remedies and struggles against forced occupations, such as manual scavenging, and by ensuring Dalits’ access to education and livelihood. Navsarjan runs 170 Bhim Shalas (learning centers where Dalit children come together in non-school hours) in Gujarat. In addition to this, Navsarjan also has three model schools for children from marginalized communities. Navsarjan has set up its own ECVU called Apna Malakma (in my country land) and has covered 43 schools of Patan, Mehsana and Gandhi Nagar districts of Gujarat. Navsarjan ECVU has also screened its videos in some of the Bhimshalas. The technical support partners in the Videoshala project are Drishti Media and Udaan. They are involved in providing the technical training and other assistance to the ECVUs. Drishti Media: Drishti was set up in 1992 by two filmmakers. It is a media, arts and human rights organization and has made documentaries on human rights issues by involving the community. It believes in the philosophy of media of the people, for the people and by the people. Having expertise in documentary filmmaking, Drishti Media had vital role in Videoshala at different levels in providing technical support. Udaan: Udaan is a separate cell of the NGO Janvikas and came into being after the 2002 Gujarat communal riots and works in the field of primary education. Udaan started working with the riot victim children residing in relief camps by providing them educational opportunities and now runs 75 of its own schools. Udaan provides education and pedagogy support to Videoshala. Central Coordination Unit (CCU): The CCU includes officials from the two founding organizations (Udaan and Drishti Media) and a program manager who oversees and manages each ECVU. The CCU provides constant direction and vision to building and strengthening the ECVUs. It also organizes a planning, review and sharing meeting every month to help different stakeholders understand the project’s changing needs better, to share implementation concerns of the ECVUs and to identify new areas of support of them. The CCU also has weekly phone conversations with each ECVUs trainer and coordinator to monitor their work and find out if any additional support is needed. Based on the individual needs of each ECVUs producers, trainers, facilitators and coordinators, the CCU organizes specialized workshops as and when required. QUEST Alliance: The QUEST Alliance provides financial, technical and monitoring and evaluation support to the whole Videoshala project, and is actively involved in overseeing every facet of the project to enable it to become sustainable and scaleable. 3.2 MAKE UP OF EACH ECVU Each ECVU has a team of six producers, two to five classroom facilitators, one trainer and one coordinator. All the team members have been trained by Drishti, Udaan, Quest and other external experts through a series of workshops. Producers: The role of the producers also known as community video producers (CVP) is to identify hard spots for students and teachers in the local schools curriculum and to produce videos that address these hard spots, while also inculcating values of diversity, democracy and citizenship in the content. Towards this end the producers are trained in video film making, editing, scriptwriting, shooting, story telling, creating creative content, instructional design, educational pedagogies, learning from songs, dramas and illustrations, classroom facilitation and values of diversity and democracy. They have also been given guidebooks and other reference materials including textbooks and sample video films. Classroom Facilitators: The classroom facilitators’ main responsibility is to screen the videos in the selected schools and to conduct interactive session with the students during and after the screening. They have been trained in classroom facilitation and teaching techniques, on how children learn, administering worksheets, carrying out student activities and involving teachers in the video screening process. Trainer and the Coordinator: Each ECVU is assigned a trainer who has been trained in video production and can guide the team through the project cycle. A coordinator who has also been trained to organize and manage the production and screening of the videos is deputed to each ECVU. Most of the coordinators have prior experience working in the field of education and have a basic understanding and perspective on education and pedagogy. The trainer and the coordinator have attended workshops to understand the production of value centered educational content, how children learn, and information about knowledge change, and its analysis with documentation. 3.3 VIDEO PRODUCTION MODEL Brainstorming for Opportunities: The production of a video begins when the producers from each unit collectively identifying what students and teachers – in their target schools – are finding difficult to understand and teach. Based on this feedback, they start exploring topics and sub-themes for their videos. Research and Knowledge Creation: Producers then visit some target schools to test the students’ knowledge in the chosen subject and sub-themes of the video. Subsequently, they begin a research process to acquaint themselves with the relevant content addressing the chosen subject in class textbooks and other reference materials. They also interact with resource persons and subject experts, understand community perceptions on the topic and visit educational sites like the community science centre. Script Writing and Pre-production: Producers then think of the values that can be incorporated in the teaching of the subject, and prepare a preliminary script for the video. They also plan activities that will be carried out during the screenings, which reflect the knowledge as well as the values to be imparted in the film. Feedback workshops are then conducted by the CCU with an aim to sharply articulate objectives, review the content and reiterate scripts. Shooting plans and locations are decided by the producers, who then also select local actors from the community. Production: Once a rough cut of the video has been produced additional workshops are conducted by the CCU to provide feedback, and the producers implement the recommended changes before producing the final copy. The production process for each video is typically in the range of 40-50 days. Parallel Inputs: During the entire video production process additional workshops are held by the CCU with inside and outside experts, catering to the individual needs of each ECVU, in areas such as understanding hard spots, script writing and editing, choosing and designing activities, incorporating values in films, implementation in schools measuring learning, and knowledge gains of students and teachers. All of these strategies are supported by examples and references which are localized and contextualized to the realities of the students and teachers. Selection of Themes and Scripts: While each ECVU choose its own topics for its videos, each videos is based on the key pedagogical strategies to approach the hard spots from the lens of the values of inclusion and diversity; to target a dual audience of students and teachers; to make learning entertaining, interesting and fun; and to model instructional guidance for teachers by demonstrating examples of teaching methods. Screening Videos in Schools: Each ECVU has a schedule for screening its videos in its target schools, which has been formulated in agreement with the schools principals. The screenings, meant for students and teacher audiences of around forty in number, are made on a TV and a DVD player that the facilitators bring with them. Each video is accompanied by an activity guide which informs the facilitators about the organization of the session. During the screening, they stop to discuss topics or carry out any activities as designated by the pauses in the videos. Post-screening discussions and additional activities with the students help initiate discussions and engage students in activities to explore and learn, while teachers are encouraged to pay attention to teaching techniques and interact with mixed gender groups of students. 3.4 COVERAGE OF VIDEOSHALA The Videoshala Project has been implemented in different parts of Gujarat and the coverage of the Videoshala project under each ECVU and the videos produced by them on the issues are shown in Table 3.1 below. Table 3.1: Coverage of Videoshala Name of the ECVU Names of the Videos produced (issues of the videos) Name of the districts where videos were screened Number of schools where the videos were screened Hind Swaraj Mandal Bhumi 1.Bhav Nagar 2. Surendra Nagar 3.Ambreli 4. Rajkot 30 Urja Aapna Hako Lokshahi Ahar Ane Arogya Paryavaran Aney Teni Kudrati Sampada Navsarjan Vanaspati 1. Patan 2. Mehsana 3. Gandhi Nagar 43 Schools Pradushan Paryavaran Aneythenu Kudrati Santulan Sthanik Swaraj Ni Sansthano Gujraat Ni Lokjivan Sahyog Vanaspati Etle Shu 1.Ahmedabad 33 Juda Juda Kamo Aapna Dharmo Khushi Karo Ramakda Kuch Kadam Aa Ane Ee Ki Matra Udaan Meghdhanush Water 1.Panchmahal District 75 Earth Communication Respiratary System Pupu Learning Additions 4 EVALUATION METHODOLOGIES In the second year of its existence, 2009, Videoshala was evaluated by two separate organizations: Center for Media Studies (CMS) & Nirantar. 4.1 CMS EVALUATION METHODOLOGY The broad objectives of the CMS evaluation were: • To track the extent to which the Videoshala program has achieved its objectives • To track the progress and growth of the participants/stakeholders in the program by focusing on the factors contributing to growth/change/transformation • To identify the factors exerting the strongest influence in effecting change among children, teachers and the community producers • To identify the challenges encountered and how they can be addressed • Wherever possible to capture concrete examples of positive outcomes and impact of the project activities CMS’s research design consisted of: • Descriptive analysis and interpretation of findings regarding the impact of operational and procedural steps followed in Videoshala; • Quasi – experimental research design providing (i) pre-post- test (ii) experimental-control group data. Both quantitative and qualitative approaches in collection and analysis of data were adopted to show the impact of Videoshala. Scope of the Study: Multi-stage sampling of schools from all four ECVUs that have implemented the Videoshala project was carried out. In the first stage, it was envisaged to cover all the Videoshala intervention districts. However, Rajkot district could not be covered as the only one school taken under Videoshala was not serving as a centre for Board examinations during the fieldwork of the study. In the second stage, 20 percent of the total number of Videoshala intervention schools was selected. As a comparative sample under the quasi-experimental design of the investigation, a proportion of Non￾Videoshala schools from the same localities as the Videoshala schools were also selected. The number of Non-Videoshala schools varied for each ECVUs. Effort was made to have a good representation of these Non-Videoshala schools which was almost 1:1 for HSM and Navsarjan, about 2:1 for Sahyog and about 3: 1 for Udaan-Meghdhanush (video vs. non-video). Respondent Group: Information was collected from stakeholders of various categories involved in the Videoshala project at different stages and levels. This respondent group of the study included NGO representatives/ Program Directors, District Coordinators, Community Video Producers, Classroom Facilitators, Teachers, Parents and Students. Apart from this, information was generated in interactions with the Program Directors of all four ECVUs, representative of Drishti Media, Udaan, the CCU and Janvikas. Sample Details: The details of the ECVUs schools comprising the sample of the evaluation and the samples of different categories of respondents are given in Table 4.1 below. Table 4.1: Sample Schools in CMS Evaluation Name of ECVU Videoshala Schools Non-Videoshala schools Schools Covered FGDs Students Teachers Community Video Producers Classroom facilitators Parent School Covered FGDs Student Teachers HSM 6 6 74 12 5 2 5 5 1 65 10 Navsarjan 9 9 105 14 5 3 5 5 5 61 10 Sahyog 7 7 79 13 5 3 4 3 1 60 6 Udaan 15 15 150 30 5 4 6 5 2 60 10 Tools of the Study: Different research tools were used in the evaluation, including: • Guidelines for focus group discussions (FGD) with children: The FGD guidelines for children aimed at eliciting information on the children’s opinion on different aspects of the videos. The FGD contained around twenty-five questions. Some of the questions were general in nature and related to what subjects/topics they like and which subject/topics they felt difficult. • Guidelines for In-depth interview with District Coordinator: The guidelines for the in-depth interviews with the District Coordinators contained a number of queries like the personal profile of the Coordinators, their association with Videoshala, their role and responsibility in Videoshala in the process from identifying hard spots to video production and screening. • Guidelines for in-depth interview with Community Video Producers (CVPs): The guidelines for in-depth interview of the CVPs had more than 35 questions. The interview guidelines had questions on how the CVPs developed an association with Videoshala, information on their personal profile etc. Apart from this, the interview guidelines had a number of questions on the process followed from identifying hard spots to the production of films. The questions were also aimed at studying the impact of Videoshala on the local community. • Guidelines for in-depth interview with Class-room Facilitators (CFs): Since the role of the CFs was to screen the videos in the Videoshala schools, the interview questions for CFs aimed at studying the process of screening of the videos in the Videoshala schools; the responses of the children and teachers on the videos; the impact the CF’s observed the videos has on the children and teachers; and problem faced in screening the videos. There were also questions on the personal profile of the CFs and their learning experiences as a CF. • Guidelines for in-depth interviews with Teachers: The guidelines for discussion with teachers had queries on hard spot areas, the process of screening of the videos, their role and responsibility in screening of the videos in the schools etc. There were also queries on the relevance of the videos on hard spots, quality of the videos in terms of content and presentation, the impact of the videos on children and on themselves. Separate guidelines had been prepared for discussion with teachers of Non-Videoshala schools. The specific questions related to videos of Videoshala were not posed to the teachers of non-Videoshala schools. • Guidelines for in-depth interviews with Parents: The guidelines had questions on the parents’ observations on the impact and change in their children as a result of exposure to Videoshala. Some additional questions for parents who had worked under Videoshala project as actors were also included. • Guidelines for discussion with Project Directors and other key partners of the project: The guidelines contained questions on the background related to the association of the concerned NGOs with the Videoshala project, the process of implementation of Videoshala, the limitations of the Videoshala, response and impact of the Videoshala on the schools, and target groups. The guidelines also focused on information on future plans for Videoshala. • Learning Outcome Tests: Learning outcome tests were conducted with the help of the test papers used by the concerned ECVUs as the baseline study on the themes on which the videos were shown. The above-mentioned tools were developed by CMS and shared with QUEST Alliance. The test papers used for this purpose were the same as had been used by ECVUs for pre-testing the issues covered in the videos. (All the tools used in the evaluation are included in Appendix I.) The Field Team and their Orientation: Persons having wide experience of collecting qualitative and quantitative data in the field of education were divided into two teams. Before going to the field, members of both the teams were shown the videos used in the Videoshala project in order to acquaint them with the videos and issues covered therein. The team members were also given a three-day orientation on the concept of the Videoshala project and the kind of information to be collected from the field. During the orientation, each of the tools and related questioning to be done with the respondents were thoroughly discussed. Each of the two teams was working as a unit; however, different members of the teams had different roles and responsibilities. Each team consisted of one researcher, two field researchers and two local field investigators. Researchers of the respective teams had the responsibility of supervision and guidance of their teams in the field. The researchers were also involved in data collection. The researchers interviewed District Coordinators, Community Video Producers (CVPs), Classroom Facilitators (CFs) of their respective ECVUs. The field researchers conducted tests and FGDs with the children, interviewed teachers and parents. The local field investigators worked as modulators/narrators wherever there was language barrier between the interviewee and the interviewer. Each team collected research data from two ECVUs. The fieldwork started on March 16, 2009 and continued till April 1, 2009. Apart from this, two senior officials from CMS also visited Ahmedabad (April 22nd and 23rd, 2009) and interacted with the Project Directors of all four ECVUs, representative of Drishti Media and Janvikas and CVP of Sahyog ECVU to have a better understanding of the process of implementation, achievements, limitations and future plans of Videoshala. Method of Data Collection: The respondent groups were interviewed using the in-depth interview guidelines mentioned above and for each of the respondent groups, the guidelines had been prepared keeping in view their roles and responsibilities in the Videoshala project. The respondents were approached by the CMS team with prior appointment and at the place of their convenience (ECVU office / residence of the respondent). Data collection on students of the sampled Videoshala and Non-Videoshala schools was done using mixed groups of ten to twelve boys and girls in each school. For Videoshala schools, both group discussions and testing was done. The test was conducted on two issues out of the six videos (issues) they had watched; pre-test data to serve as baseline information was available for these two issues. For the post-test, the participating students were given question papers and answer sheets and were given sufficient time to write the answers. The proportion of boys and girls was almost 50:50 in every sampled school of all the four ECVUs, except one of the schools of HSM where only boys participated. The break-up of ECVU-wise students sample appears in Table 4.2 below: Table 4.2: Class-wise Students’ Sample (By ECVU) for CMS Evaluation HSM Navsarjan Sahyog Meghdhanush-Udaan Class IX V-VII III-IV III-IV Number 74 105 79 150 In non-Videoshala schools, the test was conducted using the same test paper on students of the same standard as the Videoshala School. However, in some of these schools, a mini FGD (not the regular one) was also conducted with the students to identify the subjects and issues they felt were difficult. Out of the total 18 non-Videoshala schools covered in the four ECVUs for the learning tests of students, FGD with the students was conducted in 10 schools. Like Videoshala schools, the FGD was conducted with the same students who took the learning tests. However, unlike Videoshala schools the questions were restricted to a few and covered general information on subjects they felt were difficult and how they tackled the difficulties. Since they were not exposed to Videoshala, no questions on the Videoshala were asked. However, they were probed whether they would like additional help on hard spots with the help of videos and, if so, why? The interviews of the teachers of both the sampled schools (Videoshala and non-Videoshala) were also conducted in the said schools. Analysis of Data: Since the purpose of the evaluation was to understand the process, interventional strategy and lessons learned during implementation of the project; it was deemed more desirable to adopt a descriptive and qualitative approach in generating and analyzing the data. However, to gauge the learning, outcomes tests were administered and compared with the baseline data. A matrix table was also developed where in responses of different respondent groups were put against the identified parameters, to analyze the information gathered. Limitations of Study: The field team got full cooperation from all the four ECVUs and respondent groups. However, there were some limitations of the study. The field team had to change nearly fifty percent of the sampled Videoshala Schools for various reasons such as schools serving as the centre for Board examination, the schools being in the process of preparation for annual examination, etc., making it difficult for the teachers to spare comfortable time for discussion. However, the team managed to hold discussions to the satisfaction of its members. Again, since the videos had been screened in different phases (some of the videos were screened more than one year back), the students and teachers had difficulty in recalling some aspects of the videos, which might have affected the findings in some manner. There were plans to collect data on attendance and retention of the students from the Videoshala schools, but in most of the schools the school authorities did not provide this information to the study team. 4.2 NIRANTAR EVALUATION METHODOLOGY The objective of Nirantar was to evaluate the Videoshala content and assess the quality of the video kits produced by the four ECVUs. The evaluation of the content focused on the following areas: • Curricular areas of video kits • Perspective on concepts key to the Videoshala program • Pedagogy • Process of transaction of video kits The evaluation process included: • Review of videos produced by each ECVU; • Desk appraisal of documents – Videoshala proposals, quarterly program reports, worksheets and activity sheets, Gujarat State Board School textbooks, central learning event reports and documents provided by the ECVU (which includes concept notes, the list of resource materials and information on team members); • A five-day field visit to the four ECVUs from March 26-30, 2009. During the visit interactions and discussions were held with ECVU members as well as the CCU core team. Visits to schools where the videos were being screened were also part of the fieldwork; • The evaluation team consisted of four members from Nirantar. The overall design of this evaluation exercise included: Content analysis: to examine subject matter accuracy, comprehensiveness, reflection of values of citizenship, democracy, diversity and gender and overall product. Classroom observation: of film screenings using guidelines and a framework of analysis to examine the physical arrangement and environment in the classrooms, as well as human interactions - between facilitators and learners. FGDs and interviews: with the purpose of covering the following areas: For classroom facilitators: • Nature and experience of training inputs • Preparation for classroom transactions • Analysis of the differences in teaching through textbooks vis-à-vis textbook-video combination • Analysis of impact on learners • Analysis of the accuracy and comprehensiveness of the subject matter • Analysis of representation of values in the material • Role in selection of topics and material creation For learners: • Discussion on videos that they liked, and reasons for it • Analysis of the differences in teaching through textbooks vis-à-vis textbook-video combination • Views on language and content in the videos • Role in selection of topics and material creation For producers: • Selection process of members from the community • Role in selection of topics and material creation • Resource materials and inputs provided to them • Representation and integration of local context • Representation of core values of citizenship, diversity and democracy in the videos The tools used were as follows: • Content analysis of video kits – to assess the content and quality of video kits, understand the context and needs of participants in the program and assess ways in which the program has been able to address these. Learning from the reviews and assessment reports were used to strengthen the areas of enquiry of the current evaluation study. • Classroom observations – covering schools in each ECVU. Classroom observations were held pre, during and post the video screenings. The observations allowed for the study of the effectiveness of video as a medium for teaching children. This method provided detailed and precise information on students’ engagement with videos and their content. The observations also enabled greater learning on the transaction of videos and facilitation around key concepts and values in the classroom. The nature of facilitation and instruction in the classroom was useful in evaluating methods used by facilitators to address the varying needs of students and the challenges faced by them in teaching. • Focus group discussions (FGDs) – with producers, classroom facilitators and learners in each ECVU. FGDs enabled for the capture of the nature of people’s engagement with the Videoshala program. FGDs enabled the assessment of the program’s effectiveness in improving the quality of education. This was also a tool for exploring people’s perspectives on key concepts and values identified during the project. • In-depth Interviews – with producers, classroom facilitators and learners in each ECVU. Interviews helped gauge the depth and nuances of perceptions of community members and students on values of citizenship, democracy and diversity. Conducting interviews was also useful to ascertain factors that have influenced learning in schools and in the community. In addition Nirantar watched 22 films made by the various ECVUs and analyzed seven films that were screened during the field visit to Gujarat. Their analysis is based on benchmarks of curricular areas and pedagogy. The schools visited as part of this exercise are listed in Table 4.3 below. Table 4.3: Schools Visited by Nirantar Name of Org Nature of School Name of School/Village Block District Hind Swaraj Mandal Run by Satubaba trust (govt aided) Satubaba high School Palitana Bhavnagar Lokshala based on Gandhian ideology (govt aided) Trivenitirth uttar buniyadi School (Vill- Kansar) - Bhavnagar Meghdhanush Run by Meghdhanush (govt aided) Kalol School Kalol Panchmahaals Government school Arad Halol Panchmahaals Navsarjan Run by Navsarjan (govt aided) Sami Mehsana Patan Government school Shobhasar Kalol Gandhinagar Sahayog Activity Centre (run by Sahyog) Sahyog Activity Centre Nawabpura Cluster, Vatwa Ahmedabad Private school Ashirwad School Vatwa Ahmedabad Data collection was done through interviews, FGDs, and classroom observations. This has been summarized in Table 4.4 below. Table 4.4: Methods of Data Collection by Nirantar Name of Organisation Head of org Coordinator of ECVU/ Prog leaders Facilitators Teachers Producers Learners Community Classroom observations Hind Swaraj Mandal 1 interview 1 interview 1 interview 1 interview 1 FGD 2 interviews 1 FGD 3 interviews 1 FGD 1 Meghdhanush 1 interview 1 FGD 1 FGD 1 interview 1 disc 1 interview 1 FGD 2 interviews 2 FGDs 2 Navsarjan N/A 1 interview 1 FGD 1 interview 2 FGDs 1 FGD 2 interview 2 FGDs 3 interviews 2 Sahayog 1 interview I interview 1 FGD + teacher 1 FGD 2 interview 1 FGD 2 interview 2 interviews 1 Total 3 interviews 3 interviews + 2 discussions 3 interviews + 3 FGDs 2 interviews + 2 FGDs 8 interviews + 4 FGDs 8 interviews + 6 FGDs 2 interviews + 1 FGD 6 5 CONTENT ANALYSIS OF VIDEO FILMS This chapter provides a detailed content analysis of seven films, made by the four ECVUs, carried out by the Nirantar evaluation team. The analysis of the Nirantar evaluation team is based on the relevance of the film to the audience; the comprehensiveness of the subject matter; the implications of using video as a medium; creativity of the content and form; the kind of information communicated in the film; how it compares with textbook knowledge; how the film integrates values in its content; and the how the local context and the context of the particular ECVU impacts the content. 5.1 HIND SWARAJ MANDAL (HSM) ECVU • Urja This video deals with renewable and non-renewable energy and their various sources. The video begins with a puppet show. Some animal are driving a car, and when the petrol finishes, the car is pulled with bullocks. It then goes on to show consumption of energy in various forms – e.g. cooking food on stoves and lighting lanterns when electric supply is cut. In this video, students are encouraged to come up with drawings of the various forms of energy and asked to classify them into Renewable and Non-Renewable forms. They are shown solar and Gobar Gas plants and Windmills. The aspects of some sources of energy are explained to them through the question-answer method and also through experiments. The video uses various mediums creatively. The attempt to explain solar and gobar gas energy by actual field visits is an effective strategy. However, while the distinctions between various sources of energy can be seen, the film does not go into details of how energy is created. The level of knowledge about energy which Standard VIII and IX students normally have is much higher than that shown in the films. The information about of energy displayed in the films is rather superficial, limited to technical terms and definitions. The concept of energy appears as in the text books, without delving into understanding what energy is, how we see it in different forms, how it is created. The different sources of energy have been identified only by their names and relevant pictures. The way in which energy is talked about in the film assumes that the children have certain knowledge already. For example no information has been provided as to what a Silicon Plate is and how does it work. It simply says that these plates store solar energy. Similarly it has been stated that water energy comes only through turbines and it is a form of converted energy. However, the film doesn’t explain what a turbine is and what is meant by converted energy. The significance of a Dynamo in the context of a windmill and how it transforms wind energy into power has not been explained in scientific terms. The use of Kerosene as a form of fuel has been explained adequately through experiments. However, the video does not explain how the fuel is obtained from the bowels of the earth, the scientific methods and processes of fuel extraction. The film has the potential to be a good source of information on energy, because the level of knowledge of Class VIII and IX students is limited. The film could have added to textbook knowledge to make the concept of energy formation, and renewable and non-renewable energy clearer. It is only through expanding the scope of the students’ understanding of the various sources of energy in our country, their various processes of formation and use that it is possible to show how it is being misused. Also, while speaking about what energy is and its sources, the emphasis should be on local resources as well as others. Using puppets to explaining the concept of energy seemed a little childish for Class VIII and IX students. The film integrates into its content the value of saving of energy. At the end of the film, the children organize a protest for saving energy, which is well done. The film shows a distinct awareness of gender and religious differences among students. However, the value of diversity is communicated through a song – ‘Hum Hind Desh Ke Niwasi’– which does not seem to have any bearing on the subject matter of the film. Also, this medium rather blurs the complexity of diversity, addressing it in a superficial manner. • Lokshahi Lokshahi deals with democracy, specifically the election process. This falls under the subject of Social Science of Class VIII and IX. This was the fourth film that HSM made. This film covers topics like – What is Democracy? What is the process of Elections? It raises questions about governmental responsibilities towards development in villages, especially water supply. Through this it attempts an understanding of the responsibilities and accountability of the government, and responsibilities of people to raise questions of the government. This film also raises the issue of plastic bags. As plastic does not decompose like other materials, it harms the environment. As an alternative to these bags, paper bags are made in the film. The discussion about democracy happens whilst the children are shown how to make paper bags, which makes the film an absorbing one. Discussions on democracy happen in different households and with various people –ranging from grandparents, school teacher and ‘expert’. The information and knowledge thus generated is not based on any single person as happens in schools where the teacher alone is considered the storehouse of knowledge. The methods of collecting information are also quite novel. Different modes have been adopted to provide relief from a dry discussion or lecture, to see democracy in different contexts. For example, Govind Dadajee, a Gandhian, interprets democracy in Gandhiji’s language. Sabir, a craft teacher talks about democracy with the creativity and balance shown in his teaching of the crafts. Devikant Bhai presents it in a question-answer form. The voice and concerns of women, also a part of the democratic structure, are heard through discussions in which they talk about their recurring water-problems. Students in the film participate actively in the discussions, so that the information is not provided in a convention, top-down manner. Normally such subjects are dealt, at best, through the medium of stories or case studies. In this film attempts have been made to show their relevance with day-to-day life. An attempt has been made to involve not only students of different religions, but also different types of children in this video. 5.2 MEGDDHANUSH ECVU • Pupu Pupu is a well-made film, using a creative story-line. The entire story is child-centered. The main character is a little girl called Revli, and the film follows the story of her lost toy ‘pupu’, which some of her little friends – both boys and girls – help her to find. Through the story about the search for the toy, the film covers concepts like states of matter and properties of solids. The children ask Revli what kind of toy she’s lost, and Revli subsequently explains its properties. Through this, various properties of solids are discussed – like hard, soft, smooth, brittle, transparent, etc. The plot creates curiosity amongst the children regarding what sort of a toy ‘Pupu’ is, whether Revli will find her toy or not, and what will come next. In terms of subject matter, it was quite relevant and comprehensive, and visually provided examples to children, which helped them understand states of matter and properties of solids. The subject matter was accurate and appropriate for the learners of this age group (Class III students), and the curricular requirements of their syllabus. In terms of concepts, the film was a little heavy. Properties of solids were discussed at length at the beginning, and the various states of matter – solid, liquid and gas – were taken on later. This may have made it more difficult for children to understand. The properties are done wonderfully well and contextualized within the search for the lost toy. The film could have omitted the discussion on solids, liquids and gases and focused more on properties. In the attempt to talk about different states of matter, smoke is mentioned as a form of gas. In the film, this is followed by a quick discussion on air pollution caused by smoke from a factory, and what needs to be done in order to prevent it. This seemed to be an unnecessary conversation to include, and it seemed as if the discussion on values was forced – superficial as well as instrumental. Another example of attempting to interweave values in the film is through Revli’s friends. From their names, it is evident that the children belong to different communities. The producers wished to represent the value of diversity through this aspect, but it seemed a bit tokenistic. Other values that are reiterated through this film are cooperation and mutual support, demonstrated by Revli’s friends helping her look for her toy ‘pupu’ and giving her support. An element of caste-based discrimination is also shown in the film. Revli is shown to belong to a Dalit family. This is represented in a scene where Revli is shown to sit on the ground, while the other children sit on the bed. In the same scene, Revli cups her hands while drinking water, showing that she belongs to a so-called ‘untouchable’ caste, and can’t drink from the same vessel as the other children. The other children, however, tell her that she should sit on top of the bed and drink water from the same vessel. An observation we had about the films in general was that, while they do represent and showcase the ‘ideal’, highlighting values of equality and non-discrimination – albeit in a limited sense – there is no reflection of more real-life conflicts. Things seem too simplistic in the way they are represented and not as nuanced as they truly are in our daily lives. This is especially true of how caste-based discrimination is dealt with in the film. Overall, we feel it is significant even to make a symbolic reference to certain values, but it may not be understood entirely by learners if it is not linked more concretely with the context of the area and the lived realities of the children and their experience of marginalization. 5.3 NAVSARJAN ECVU • Gujarat nu Lokjivan This is the fifth film made by the Navsarjan ECVU. The film is based on the social and cultural life of people in Gujarat. It takes the students to different parts of the state. The film has children as its main protagonists. These children are transported to different parts of Gujarat through something like a time machine (called bulbul) which has been made by the children themselves. Children operate the machine and choose to go to a different region each time. The machine transports each child to a different region (Northern Gujarat, Southern Gujarat, Kutch, Saurashtra). The child gets to experience the region and culture and is then transported back. He/she shares then with others the experience of the region. The film shows different types of houses that people live in, clothes worn by men and women, food eaten by people, handicrafts, geographical features and animal and plant life of the region, languages spoken, places of historical interest, industry and work done by people. At the end of the film children are also exposed to problems faced by people in Gujarat. Discrimination in wages, exploitation, migration, manual scavenging and the status of tribal people are some of the issues that children learn about in the film. The film also presents well contradictions that exist in Gujarat. It shows that the Narmada canal has provided water to many farmers, yet the dam on the river has caused large-scale displacement of people who have not been rehabilitated. The film also shows that while fishery is an industry in the state, the status of fisher people is not an issue of concern for the state. The film addresses the issue of diversity in a positive way. It celebrates the multifariousness that exists in Gujarat. Information on social and cultural life of people in other regions is a learning for children who may never get to travel to these regions. The film builds on the local context: made by the Navsarjan team in Mehsana, the film takes the viewers to Patan and shows the famous Patola sari, the Sun temple in the region, dairy industry and fertile land which appeals to children from the Mehsana area. Viewers also get to compare and contrast their own region with other regions in the state. The film has women and girls in significant roles: the actors are both boys and girls, and it is a woman who helps the children and provides information throughout the film. Popular songs and good quality of visuals ensure that viewers are engrossed in the film. The idea of the time machine is highly appealing to children and is an interesting medium to transport them to different regions. The film has the possibility of opening up a discussion on structural and social inequalities and the exploitative role of the rich and ‘upper castes’. However, there are some issues which have not been dealt with adequately. While women are part of the film, it does not raise the issue of gender-based discrimination. It shows women as manual scavengers (the reality being that it is women who are mainly in this profession) and as laborers. However, the issue of discrimination at work based on gender is not mentioned. The film also does not represent non-Hindu traditions and religions adequately. The adivasi culture is represented in the film, but there is no mention of Muslims or Jains or other religions that form a significant part of the state population. Gujarat nu Lokjivan uses the medium of video to the fullest. The film is technically sound and its visual quality is good. There are well-taken shots of houses, clothes, food, places of interest and physical features of different regions. The medium also inventively uses the time machine idea – a ploy that is highly appealing to children and that only a video could have shown so well. The film provides children information that broadens their horizons. Textbooks are often limited to providing information on life styles of people; they do not highlight issues of exploitation and discrimination, which the film does. Textbooks also present idealized situations and show diversity without differences or disparities. Gujarat nu Lokjivan does show heterogeneity and glorifies it, yet it does not shy away from showing conflict. The perspective of the producers and the E-CVU involved is well incorporated. Their understanding of Dalit issues and rights of marginalized people comes across well in many places in the film. A Glimpse into the Textbook ‘Gujarat: Social Life’ in Social Science Textbook for Standard 5 The Gujarat State Board school textbook glorifies the diversity that exists in the state. The book has a four-page chapter which includes brief information on food habits, costumes, dwellings, celebrations, caste, languages and folk dances. The chapter presents the people of the state as being “by nature humble, persevering and hard working.” ‘Social Life’ has a few illustrations and four photographs that show a fair, a temple and folk dances. There is an attempt to show both modernity and tradition – in addition to local food, clothes and styles of living, there are statements that tell the readers that men and women in Gujarat have also adopted modern clothes, there are multi-storey buildings and people are expanding their business to other states and countries. There are also statements like -“the people of Gujarat love celebrations”, “generally the life style of Gujarat is varied”. Despite being a chapter about people and their culture, the chapter is written in a style that is dull and distant. Information provided on each region is very brief. Adivasis are mentioned only in the context of their language. The focus is on fairs and festivals – this section is the longest in the chapter. The illustrations don’t give a real sense to learners about the types of houses and clothes. Unlike in the film, there is no attempt in the textbook to present oppression or even talk about issues related to marginalized communities. The sense that a reader gets is that Gujarat is a great state where people with diverse practices live in harmony. This is a highly simplistic and problematic representation of reality. • Sthanik Swarajya Ni Sansthao (Gramya) The film is about institutions of local self governance (rural). The film begins with the local Panchayati Raj elections. It shows a local woman, Sakri Ben, contesting elections to the post of sarpanch. Sakri Ben wins the elections and is among the few elected women representatives in the area. She is ridiculed and challenged by men, who are in the majority in the panchayat structure. After winning, Sakri Ben does what few other people have done in the past – she revives the Gram Sabha and challenges discriminatory practices. She calls for the Gram Sabha meeting, gets a Dalit man to sit on the chair in the panchayat office, mobilizes women to come for the Gram Sabha meeting – including the other elected representative – going door to door and speaking to people. She also promises to raise people’s issues at the meeting. The film talks about the gram panchayat structure as Sakri Ben explains this to her daughter Roshni and her friends. At the Gram Sabha meeting, Sakri Ben addresses issues raised by people – most of these are issues of development. Roshni and Sakri also address the issue of caste-based discrimination at school. Sakri, with her transparent and democratic style of working, gets development work started in the village. The village is electrified, construction of a pucca road begins and drinking water is supplied to taps. Sakri Ben’s efforts are appreciated by everyone and she receives praise from all quarters. The children learn about the Taluka panchayat through another man, Imran Bhai. The film does not show idealistic situations. The challenge faced by women in politics – a male-dominated arena, is well represented in the film. The film also shows lived realities of people – the reluctance of women and other people to come to the Gram Sabha meeting because of the total lack of interest among panchayat members to address their issues, the domineering attitude of men who have been in power and the discriminatory behavior towards Dalit children in school. Information on the tiers of the panchayat structure and work of the gram panchayat is provided in a simple and interesting manner. Ideas of local self governance, democracy and equality are shared in Sthanik Swarajya Ni Sansthao and yet the film is not message-loaded or instructive. The film has women as its main characters – Sakri, Kanta and Roshni are the lead characters who take the viewers through the film and provide information along the way. Children’s questions and their inquisitiveness push the film forward. There is additional information provided on the taluka panchayat in the film. Sthanik Swarajya Ni Sansthao breaks a gender stereotype by showing women in a public domain. Yet, the struggles of women like Sakri Ben are not highlighted adequately in the film. Despite facing hostility from other panchayat members, Sakri Ben appears to be a self-confident, articulate and assured woman who manages to gather support and initiate work within no time, and few obstructions. This is a simplistic representation of reality. While this does break the stereotypic image of women in panchayats being only dummy candidates for males, yet the scope of highlighting struggles of women could have been explored to a greater extent. The film stops short of building upon the value of citizenship. While the idea of women as citizens, as active participants who have agency in the process of democracy is quite clear in the film, it is not clearly stated, as a result of which it is easy for someone to miss the point completely. While it is important to show the linkages of the different tiers of local self governance, the information on taluka panchayat appears to be an add-on in the film. The information is provided by a Muslim character. The effort to show people from marginalized communities as providers of information is well appreciated; however, the representation in this form in the film, with no real integration into the plot, appears to be tokenistic. The choice of topic for the film is relevant and useful. Explaining the idea of local self governance by showing the challenges related to participation of women and other people is a very innovative and yet realistic method. Unlike textbooks which are distant from local realities, this film, based on the experiences of Sakri Ben, Kanta and Roshni, is very close to children’s contexts. Locating the story in such a context helps learners understand information and issues quite clearly. The medium of video is used well in the film; however, Sakri as the provider of information sometimes appears to be doing what a teacher would do in a classroom situation. While this process of learning is more interesting for the children as they travel with Sakri and get to understand her work as a sarpanch, and the functions of the Gram Sabha, yet the medium of video lends itself to more creative use which has not been fully explored in the film. The influence of Navsarjan and its values is quite apparent in the film, as it looks at issues of gender, caste and realities of rural life. The low participation of women in panchayati raj is also a local reality which is well captured by producers of the film. A Glimpse into the Textbook ‘Local Self Government Organizations - Rural Area’ in Social Science Textbook for Standard 5 The chapter in the Gujarat State textbook provides brief information on Panchayati Raj (“Panchayati Raj is a very important method to run the self government at the local level. It solves the problems related to the planning, development and locality quickly as per the expectations of the people. The organizations of Panchayati Raj are important for the economic and social development of the village”). Information on the three tiers of Panchayati Raj is mentioned in a few lines. The chapter has information on formation of the gram panchayat and the taluka panchayat – their functions, sources of income and administration. The chapter on local self governance is highly de contextualised. Its content is dry, dull and distant from the local context. There is an overload of information. It does not talk about the situation in Gujarat; there are no examples, no photographs or illustrations. The institution of Panchayati Raj is described in a simplistic manner, with no challenges or complexities. 5.4 SAHAYOG ECVU • Bhasha Gyan This was the last of the six films that Sahyog has made. The hard spot here is language, more specifically consonants and vowels – vyanjan and swar; aa, ee and oo vowel sounds are focussed on. While attempting to teach these vowels, the film – a fiction/feature – also follows the story of Rani, a secondary school girl who is prevented from continuing her education because her parents think it unnecessary to study beyond secondary school. Another story – or motif/metaphor – in the film is of a parrot in a cage, Bittu, which belongs to Khushbu, one of the protagonists of the story. Khushbu tries to understand why Bittu is kept in a cage, unlike all the birds she sees. The video has a plotline that is quite interesting and absorbing. Different threads of the story are tied together in a way that is not forced. The film begins with shots of Khushbu telling her mother about Rani dropping out of school, because her parents want her to. Rani appears in the film not long after, as the ‘teacher’ who explains vowels and consonants to the children. The parrot reappears at the end of the story, when Rani’s parents have been persuaded to let her go back to school. Khusbhu, convinces her mother Rekha ben that Bittu be left free to fly with her friends, and opens the door of the cage and lets her out. The film thus includes both content and values’ component, and weaves them together with the help of a multi-layered fictional plot. However, the actual teaching of language doesn’t differ much from classroom language teaching without a video. After a quick recall of the consonants, children enunciate the vowel (aa, then ee, then oo) – first silently, and then with sound. After this, the children make words combining consonants and the vowel being taught – this happens through different games. Then the vowels are drawn on slates, or on the ground; the difference between how vowels appear in different forms by themselves or attached to a consonant is explained. This is clearly difficult for the learners to grasp – that the vowel appears in a different form on its own, and when it is attached to a consonant to produce a certain sound. Games like hopscotch, and word building in different teams, are played for the learners themselves to join vowels and consonants, and make words, or their names. The video demonstrates a child-friendly language pedagogy, though not necessarily a very innovative one. Parts of the film, in fact, seem like the audio-visual version of a language class – for instance, when vowels are attached to different consonants on a slate. Also, though the person teaching the children is not a conventional teacher, the method of teaching is still top down – either by Rani, or by the gardener who works in their school. So even though an effort is made to question who the conventional giver of knowledge is, knowledge is still given from above. Much of this language teaching in the video could be achieved by the teacher in the classroom, in fact be even more effective if the learners were given a few fun exercises to do. The video itself adds little to the information in the textbook. It is relevant and comparable to the work that the students do in the classroom, but it achieves a similar distancing that textbook or conventional classroom teaching does. Also, though elements like humour, and learning from peers (each of the children in the film makes a word for the different vowel sounds) are used to teach, language is still taught as broken down into consonants and vowels, which is difficult for learners to grasp. There is no attempt to teach language in a way that roots it in the everyday lives of the learners, or as it is used, rather than the way it is written. Where the film is clearly effective is in the subplot about girls’ education, and the undesirability of dropouts. This part of the film differs in style also from the rest of the film (which is much more like an educational video). It has characters – the girls’ parents, members of the community who become involved in the incident. It takes place in peoples’ houses; it screens the dynamics between parents and children, and between different members of the community. There is an effort to locate both the film and the issue it is raising within the local community itself, and ask the local community to come up with a response, rather than an ‘expert’. This is true of many of Sahyog’s films, and maybe a reflection of the kind of work they do in the community, and the kind of relationships they build within it. Ironically, this ‘fictional’ part of the film is perhaps most easy to relate to for the audience, and most true. For instance, the value of equal right to education and a future for a girl is clearly understood and related to by even the primary school audience that the film has. The other subplot, of the undesirability of keeping a bird in a cage, links quite well to the story of the girl who is being held back from a future she is entitled to, and so the value of sensitivity and kindness to animals ties in the end to the value of gender equality. So the film has a content component which does not quite do justice to the medium of video, and a values component which manages to capture the imagination of the learner and communicate the importance of the value, and its place in their everyday lives, quite well. The language component covers the hard spot it sets out to, but lacks in creative research on language pedagogy which could have set it apart from the language textbook, or which could have made better use of the video format to teach language, for example, a more experiential understanding of vowels. In terms of the values the film communicates – these have little connection to the subject content of the film, which is language. In this sense, the subplot about girls’ education, or even about the parrot in the cage, could have been part of any other film. Although the genre of a feature film gives the filmmakers the freedom to use subplots, and link them on the level of character – it would have been interesting and perhaps more effective to link language and values in some other, more organic way. Technically, the film leaves quite a lot to be desired. Some of the shots are quite jerky, and when a shot begins, it often seems like the actors are not entirely ready for it. Often, one is not sure why a certain shot ends so abruptly and is followed by another, unrelated one. • Juda Juda Kaamo This is the second of the films that Sahyog has made. It is on the different forms of work that people in the community and around them do. Forms of work are discussed and then the children go and learn in more detail about some of these different occupations. This content is in the social sciences curriculum, but some information on occupations is new to the learners, as is the questioning of certain stereotypes associated with some occupations in the textbook. The video makes use of the medium in creative ways. It is shot really well, cutting from one image to another imaginatively – quickly yet without being jerky. It also makes use of music well, making you feel like you are watching a feature (almost Hollywood!), when in fact it’s a documentary. The camera is highly mobile – in the sense that shooting happens in different locations and with a range of different actors. This range of people and places is not something one feels when reading a textbook – however good the textbook is. While some textbooks do attempt to cover different forms of work and the people who do them, they never delve into the structural reasons of why certain people only have access to certain kinds of work (why the same people are not journalists and sweepers). In this video, there is an attempt to ask questions about who does what work, and the discrimination that some workers face. However, at the end, it leans towards a positivist equalizing of all kinds of work, highlighting all labor as a service of some kind, without an analysis of the unequal structures which determine the ‘choice’ of one form of labor over another. In short, the video showcases a certain progressive perspective of labor of the organization and the filmmakers, and the value of diversity and citizenship, but stops short of a structural analysis of labor. The video begins with cuts from one location to another, showing different forms of labor in places familiar to the audience. Shortly after, in a discussion format that forms the main mode of communication in the film (rather than the instructional mode, used in many films of other ECVUs), the facilitator (interestingly, Munira, the Sahyog ECVU coordinator) asks a group of children to name different kinds of work they see around them. A variety of forms of work come up – rickshaw puller, chai shop, kinara shop, chappal shop, toy shop, carpenter and so on. It’s interesting to see – at this point, and afterwards in the film – the kinds of work that get shown. Quite a few different forms of work that are generally invisible or part of the children’s everyday lives are made visible in the film – cobblers, barbers, construction workers. After this, through the format of a riddle, different forms of work/workers are visually depicted and the children guess what the occupation is. This is an effective pedagogy, and also makes use of the medium of video to the maximum, as different aspects of the work are represented, in a way that is not boring or repetitive, and in locales that are familiar to the children (Bibi ka Talab) raising curiosity in the minds of the children. The latter part of the film involves the children going – individually or in pairs or groups – to interview different people in different occupations. This as well seems like a much more effective learning strategy than just giving children information in a top-down manner. The children go to meet a construction worker, a farmer, a journalist, a radio jockey, a sweeper and a housewife. The choice of occupations depicted are urban, but they combine occupations that the learners’ families may be engaged in, and occupations that the learners may be curious or excited to know more about (RJs, journalists). The language shifts from Gujarati to Hindi depending on the person and the location – for instance the conversation with the RJ has a lot of Hindi, which the children must also hear on the radio. Questions asked are a combination of factual ones about the nature of work – even technical details, which are quite interesting – and everyday experiences of the different people interviewed, as well as questions like who takes care of the (female) construction worker’s children while she works, and where she comes from (raising the issue of migration); who can become a journalist; whether the sweeper faces resistance from her husband about her work. The recap with Munira not only reiterates the fact of different forms of work that exist around us, but also urge the children to raise questions about gender and caste discrimination in different forms of work. Why, for instance, are only men associated with some work (like construction work) when in fact there are many women who do it? Why is there so much stigma around the job of the sweeper, when it is such an essential job? The questions don’t necessarily provide an analysis of the status quo in society, but definitely push the children to question what they see around them, which is a huge difference from the textbook. The effect of local shooting, and with places and occupations that the learners are familiar with increases the impact of the video – the distance between the learner and the information is quite small, and there is an ownership over the information that the learners get in the film, that is rare to see in other teaching/learning materials. This film manages an integration of content (different forms of work) and values (diversity, equality, citizenship) in a deeper and more effective sense than many other educational videos. The values are recalled and reiterated by Munira in the film, but through the video they occur in the images of different people – men and women – from different backgrounds, and more importantly in the questions to these men and women about their lives and the challenges and discriminations they face. The people, locations, language and occupations seem familiar and local and immediately relatable. This makes the content and values have more impact and effectiveness. 6 TRANSACTION AND IMPACT ANALYSIS OF VIDEO FILMS This chapter consists of Nirantar’s evaluation of the transaction and impact of the Videoshala video kits. Their analysis is structured around the screenings the Nirantar evaluation team observed, the facilitation of the sessions, the activities and worksheets used, and the overall pedagogy of the transaction. Their observations have been supplemented with data from interviews and group discussions with learners, class room facilitators and teachers. 6.1 HIND SWARAJ MANDAL (HSM) ECVU Classroom Observations Urja was screened at the Triveniteerth Post Basic School, Kansar village, and run by the Sadbhawana Trust. It is based on the Nai Talim and Uttar Buniyadi Shiksha idea of education. It is a co-ed residential school, upto Class 12. This screening was limited to students of Class 9. Fifty students (20 girls and 30 boys) were selected for the screening. Six screenings had already taken place in this school. Since the classroom observation of the screening and session was an integral part of this evaluation we asked for a repeat screening, and the producers selected the film Urja for this purpose. In the introductory speech, the Coordinator briefly elaborated the concept of Videoshala and then referred to the contents of the film. This latter was necessary since there had been no contact with the school for over two months. We visited the school on a Monday, and learned that it was a weekly holiday so no teacher would be available. The students were also busy with their activities. However, within fifteen minutes the students collected. As they sat down on the ground, we noticed that there did not appear to be any mutual communication between the boys and the girls – it was as if they didn’t know each other at all. They sat at opposite ends of the room. The atmosphere of the class did not seem conducive to an interactive and innovative session. Neither the coordinator nor the facilitator appeared to notice this classroom environment, or respond in anyway to the students. During the screening, the children seemed to be watching the video, but without much enthusiasm. However, it is quite possible that this was because it was the second screening of this film. During the screening, there was no activity or discussion. After screening, the facilitator plunged right into activities he was meant to do, without any discussion about the film. The entire atmosphere was of a convention classroom, where questions are asked and answers given. Also, the facilitator was asking questions to the same – all boys – over and over. He faced the side of the room where the boys sat, and talked to them only. This seemed contradictory to a progressive value-based system of education, enhancing student interest and facilitating learning. The worksheet on Urja has questions which test only the information in the video. While there should certainly be some questions which reiterate the knowledge already gained, there should also be some which are analytical in nature, which push the learners to process the information. The questions seemed too simple for 14-year-old students. We felt that the values displayed in the video film would certainly be taken up for discussion, and were disappointed that no discussion happened. However, when we discussed the impact of the values component of this film with the producers, Yogesh said that the issue of discrimination against children in the supply of kerosene has provoked many discussions with students. Discussions with Facilitators and Learners When we spoke to the facilitators at HSM, they said that they learned everything they knew about facilitation from the ECVU coordinator, or accompanying others for screenings. None of them had attended a training specifically for facilitation skills. The facilitators had no grounding in education or pedagogy, and knew about the content and pedagogy of the Videoshala films only by informal interaction with the producers (with whom they had been friends much before the program started). So their grasp of the content or the values in the film was limited, and explained the low energy and quality of the video kit transaction. The students of another school we visited, Satua Baba School, admitted that observing the pictures on the video imprints the content in their minds and aids in recall of information. One girl told us that when a video is screened before the topic is covered in the classroom, it aids in understanding new concepts when they are taught. She also claimed that her marks had improved considerably after the Videoshala program had started at their school, as she could remember things she had watched in the video while writing her exams! She related the story of how she played with a child from another community after watching the film on Aapno Hak, as it built her understanding of the value of diversity. The local content of the video also helps them to absorb the information in the video. When they saw windmill in the film, they were able to relate to it, as there was one in their area also. 6.2 MEGHDHANUSH ECVU Classroom Observations We observed two screenings of the film Pupu. One was at the Meghdhanush School in Kalol, where about 30 children – both boys and girls – of Class III were present at the screening. All these children belonged to Casimabad – the semi-urban Muslim colony in Kalol – and nearby areas, where Udaan-Meghdhanush had been working after the riots of 2002. All but one child in the classroom belonged to the Muslim community. The second screening was at a government primary school in Arad village, which had also been affected by riots and violence in the 2002 carnage. Udaan-Meghdhanush has placed a volunteer in the school here to support quality primary education for children. Most of the children studying in this school belonged to the OBC community, and there were about 40 children present at the screening of the film Pupu that we attended. One of the first things that struck us at the beginning of the session on Pupu at the Meghdhanush Kalol School, was that the teachers sang and danced along with the Videoshala team that was transacting this session and every child also danced along. The song was about ‘What is this object like?’ and was a good beginning to the session. In a sense, it gave an indication of what was the subject matter for the upcoming session and was a cheerful introduction to the film. This activity also indicated a great sense of engagement and involvement, and certain values of equal participation in the transaction process, because everyone – teachers, students, facilitators alike – were singing and dancing to the same song. There was a great deal of repetition of concepts during the session. This was done by using a combination of various media – video, oral repetition, through activities done in between the film screening by pausing the film, and at the end of the film through the activity sheet. This helped to strengthen the concepts related to states of matter and properties of solids. The first activity involved making the children sit around in a circle; one child was called at a time and blindfolded. Then she was asked to feel the object provided to her and explain its properties. The facilitator called girls and boys turn by turn to the centre. It seemed like the children enjoyed this activity. And after each example, the facilitators asked the children to look around the classroom and identify other objects that have similar properties. We felt this exercise was important to build the children’s actual understanding about the concept – because they were able to apply the same principles and properties to other objects as well. In fact, they were responding so well to these questions, we asked them if they had done this particular session before – but both the facilitators and teachers said that this was the first screening of Pupu with this group. The second activity was done at the end of the film. Three chart papers, saying ‘solid’, ‘liquid’ and ‘gas’, were put on three different walls in the classroom. The learners were given different items, which they were expected to identify as ‘solid’, ‘liquid’ and ‘gas’, and stand under the appropriate chart paper. This was a good activity, but was a bit hurried. For instance, when a learner was given a glass full of water and asked to identify which state of matter it represents, it was a bit confusing, probably because s/he identified the glass as solid and the water as liquid, so there were two responses which s/he could give. The teacher did not give the child the space to clarify his confusion and asked him to stand under the chart paper that said ‘liquid’. The activities were creative and effective, but we felt that facilitators were in a hurry to finish the session and get the children to fill the worksheet. This might mean that though recall might be good – and can be gauged from the filled worksheets, the children may not get time for any substantive discussions to clarify any conceptual confusion. A worksheet was given to the children at the end of the session. Their main task was to identify states of matter and properties of solids and color them with crayons. The teachers assisted the children a lot in this activity. All the exercises were to do with identifying and coloring the objects. This seemed to be monotonous after a point. There was no creativity in these worksheets. The screening at the Arad government school was also quite good. Facilitators were patient and very organized in the messages they were giving the children, even though they did not stop the film for the scheduled ‘pause’ to do some activities in between. At the end of the film, the facilitators explained the concepts to the children, but did not encourage them to explore different objects and their properties in their immediate environment. They however gave many more examples orally than had been done in the previous screening in Kalol. For instance, an interesting example that was enjoyed by the children was that of ice lollies. The facilitators explained how the same ice lolly can take on various forms – solid when frozen and liquid when it melts. They also gave step-by-step instructions to the children to fill the worksheets at the end of the film. The children watched the film with rapt attention. It was evident that they were not used to the audiovisual medium being used, and they thoroughly enjoyed the film Pupu. On the whole, however, the session was not as lively as it had been in the Meghdhanush Kalol School. Part of the reason could have been the environment of the government school, where children were much less receptive and the teachers sat on chairs on one side of the room. In the coloring exercise, there were no color pencils or crayons made available to the children in the government school, but they used their lead pencils to color the desired objects grey. It seemed as if the children had understood the concepts discussed. What was interesting was that the Meghdhanush teacher who was placed in the Arad school seemed to know very well the levels of literacy and skills of the class, so she personally helped out some of the students who were ‘slower’ to pick up the instructions. There was one area in which more could have been done in the session – both in Kalol as well as in Arad. The whole discussion on why Revli was sitting on the ground, or why she would not drink water from the same vessel as the other children, could have been taken forward by talking a bit about caste and caste￾based discrimination. The facilitators merely said that we should not practice discrimination; that we should treat all equally. They did not mention the word caste, or that it is the basis of discrimination for Revli and many others like her. When we asked the children after the screening, they could also not identify the issue. They merely said that Revli was upset that her toy – ‘pupu’ – was lost and therefore, she seemed to be low. One thing that struck us during the transaction – both in the Meghdhanush school, as well as in the government school in Arad – was that there were some delays and tensions due to technical snags. For instance, in the Meghdhanush Kalol school, the DVD got stuck and it took some time for the facilitators and teachers to get it in order. However, to some extent the process was smooth because one of the facilitators continued to keep the learners engaged by doing activities with them, and by repeating some of the things they had learned. It seemed like overkill, but given the technical problem, there was probably no other alternative. During this time the other facilitator and the teacher sorted out the technical problem and got the DVD going again. Similarly, in the Arad school, no previous arrangement had been made for a television and the TV that belonged to the school was allegedly out of order. It was only after the facilitators checked it that they realized the TV was in order, and only a DVD player was required from outside, which had already been got from the Meghdhanush office. These technical issues caused some delays, and the producers and facilitators shared that sometimes there are other issues related to screenings – like lack of electricity, especially in interior villages; TV not being available in the village; absence of remotes to run the TV and DVD player, in addition to other technical problems that may emerge. Discussions with Facilitators Through the discussions and interviews with facilitators, we learned that the facilitators feel that to influence the effectiveness of the transactions and film screenings, it is important to build a rapport with the teachers, school principals and others beforehand and they see this as their role. This includes getting the technical things in order as well. They shared that this kind of logistical work is sometimes very challenging, especially when they have to screen films in villages that are distant and interior. And more often than not, teachers and principals in government schools are quite resistant to outsiders coming and taking sessions using audio-visual media. Overall, the facilitation in both schools was good. In terms of skills of the facilitators, the Meghdhanush group was very diverse, and this seemed to influence the effectiveness of the transaction. Most of the facilitators said that they do not do much thinking about the activities and worksheets. They rely on the manual prepared by the ECVU. Among the screenings that we saw, in the Kalol School the two facilitators were very new to the program – they had been in the organization only 21 days. But their skills of teaching and interacting with the children were very good, and this was probably influenced by their previous professional training in primary school teaching. Discussions with teachers The teacher in the Meghdhanush class where Pupu was screened mentioned that he had not covered the chapter on properties of matter in class yet, and would teach it in class within the next 10 days. He was of the opinion that if he did it after a small gap, the children would learn better, as otherwise their understanding of the film was limited to a recall of detail, not conceptual clarity. We had a discussion with the other teachers at the Meghdhanush Kalol School also, regarding how they use the films and what value it adds to their ongoing classroom teaching. It emerged that the teachers here already use many interesting teaching-learning tools and methods in the classroom everyday. They were highly trained teachers of the Udaan-Meghdhanush program. Every six months they have intensive internal trainings of 15 days or more – on pedagogy, materials and other elements of teaching-learning. As a result of this, they were already using interesting and creative activities, tools and media in class. Therefore, it was not too clear what value it added to have this film screened in this school, or how it was different from the other sessions that the teachers took. The teachers themselves gave an example that, while the film Prithvi was good, they felt that they would have anyway done the activities shown in the film, since in their regular trainings they were encouraged to do more and more activities with children. However, they did say that video as a medium is quite novel and therefore catches the fancy of the children. The teachers at the Arad government school also said that the concepts covered in the film were relevant, accurate and well explained. The curricular areas corresponded with the syllabus of Class III. They also said that since the audio-visual medium is so appealing, the children remember what they have seen for a longer time, as compared to what they remember from textbooks. The Kalol School teachers also mentioned that they sit with the children – in formal and informal spaces – to discuss values of equality, justice, diversity, citizenship and mutual support. They mentioned that, more than discussion on these values, what helped them was the demonstration of these values in their daily work. For instance, the teachers themselves clean the toilets and classrooms in the Kalol School. Through this action, they try and demonstrate that there is dignity in all kinds of work, and that we must do our work ourselves. Everyone in class sits together on the ground. This action also reflects the value of equality of all, and tries to do away with possible unequal power relations between various people. The Kalol School teachers also shared that they feel the children in this school are different from children in other schools. For instance, they said that children of Class III here ask more questions. They have the curiosity and interest in learning new things. They felt that these children would probably perform better in comparison to children in mainstream schools where they might go to study beyond Class V. The facilitators also shared that the power of video is that it is an audio-visual medium, and lends itself to representation of real-life situations. This also helps better retention and understanding. 6.3 NAVSARJAN ECVU Classroom Observations: Gujarat Nu Lokjivan The film was screened at the Navsarjan School in Sami village (district Patan). It was shown to all 74 students (16 girls and 58 boys) of this residential school. There were about 5-6 staff members present during the screening. Mahesh Bhai, the producer of the film, introduced Gujarat Nu Lok Jivan. He asked children if they had studied this topic before. The children said that they had learned about it in the textbook. There was a high level of interest among the students in watching the film. At the end of the film teachers and students were given worksheets. The worksheets had a range of activities – those that enabled students to recall information and also analyze it. The students took some time to fill in the worksheets; this was an indication of the fact that they were aware of the significance of this process. At the end of the film Mahesh Bhai asked the children what they had seen in the film. The children said they had seen houses, life styles of people, languages and oppression faced by people. They said that Gujarat was a great state, yet there were problems as well. Mahesh Bhai did not push the discussion beyond this. The children were amazed on seeing the description of Patola sarees of Patan. Mahesh Bhai provided more information and said that he had taken shots of weavers and of sarees to show the superior quality of its thread and weaving. The producer of the film had been a teacher for several years. He was well aware of the needs of learners. He was also invited by the School on many occasions to conduct sessions on various subjects. This process of interaction was useful to both the production process as well as the teaching –learning process. What was missing during the screening was a process of facilitation. No attempt was made to pause in between to assess if the children had been enjoying the film or to allow them to ask questions. The ECVU should consider including one or two activities or discussion points in each film to allow children to engage more with the video. Discussions with Learners The children really liked the film Gujarat Nu Lok Jivan. The pace of the film, the shots that transported them to various parts of the state, the music and the ploys used were highly appealing to the students. Recall was easy; the students could remember clearly what they had seen in the film. Among the lost list of things that they could recall, they listed – Patola sarees, Bhungas of Kutch, the sea shore (“dariya kinara kabhi kitaab mein nahin dikhta hai”), oppression of people, festivals and food. They also said that the film gave them new information on adivasi culture, dairy industry (they did not know that machines were used to wash and milk livestock) and girls’ education in Kutch. The students were well informed about the ECVU and its work. The context of the film, according to them, was real. They said that the location of the film was somewhere in Gujarat, some students said that the film was probably shot at the Dalit Shakti Kendra. Students said that Gujarat Nu Lok Jivan and also other films made by the ECVU were more explanatory than textbooks. They said that the story format was very appealing. However, they said that it was not possible for them to ask questions while the screening was going on: unlike classrooms, where they could ask questions at any point, the film did not provide this space. The students at Sami school said clearly that Gujarat Nu Lok Jivan is an attempt to show the diversity that exists in our lives. There was a high degree of awareness among students on values. They spoke about each film that they had seen in the past and shared the values that the films had incorporated. They also said that these values were clear in the films, but they were absent in textbooks. The discussion with students at Sami School was of a very evolved nature. The pedagogy followed in this school was very different from other schools. Critical thinking was encouraged in the school. The teaching-learning environment was non-threatening. Children were provided new information – national and international as well. All these factors have enhanced the learning process. Children were highly interested and engaged with each video kit. The videos in turn helped in furthering their learning. Discussion with teachers There were four teachers at the Sami Residential School at the time of the group discussion. Two out of the four were women, one of them being the mukhya shikshak. The teachers said that they were involved in the process of film making. They help in the selection of topics, provide feedback when asked for and are also involved at the stage of the rough cut. They said that when the film Pradushan was made, they did not like certain shots, which were later edited. They also shared that before the films Vanaspati and Sthanik Swarajya Ni Sansthao were made, the producers had come to them and asked them to share ideas and concepts that were tough for teaching and learning. They had also taken some important suggestions from learners. The teachers said that they screened films after teaching a chapter. There is a gap of about 10-15 days between the time that the chapter is transacted and the film screening. They said that learners can relate well with the topic if such a process if followed. The film provides visual aid and also deepens their understanding. In addition to this it also talks about social issues that textbooks don’t address. The teachers said that ECVU videos were better than other audio-visual material as topics are selected from textbooks and then dealt with in a manner that all the gaps and problem areas are addressed. Issues of gender and caste are integrated in these videos and these are very relevant, according to the teachers. They also said that songs, music, location and language add to the effectiveness of the ECVU films. The teachers also suggested that the ECVU make films on topics such as discrimination and inequality in educational institutions. They said that in villages this is a serious issue. They were also more than willing to help in the process of selection of topics, songs and characters for the films. The involvement of teachers in the film-making process is ideal. The ECVU should consider how this can be regularized (and not remain ad hoc). Involving teachers in the selection of topics is a good way of engaging them and also ensuring that they are later interested in using the film as supplementary teaching material, not an add-on. In the future, when the program goes on scale, this level of involvement will ensure that teachers play a greater role in the screening and transaction of the videos. Classroom Observations: Sthanik Swarajya Ni Sansthao The film Sthanik Swarajya Ni Sansthao was screened at Shobhasar Primary School (Kalol taluka, Gandhinagar district). The film was shown to students of Standard V-VII. The classroom had about 70 students, five teachers and the head mistress of the school. The children watched the film with rapt attention. The teachers also seemed interested in the content of the film. Once the film was over, fifth standard students were asked to stay on as the film was especially for this group of learners. About 28 students stayed back. One of the facilitators asked them and the teachers to fill up worksheets which were provided to them. The facilitator said that the worksheets were for them to get feedback from students and teachers. No introduction or orientation was provided before the film. The facilitators switched on the computer in order to show the film and immediately left the classroom. Once the film ended, there was no discussion whatsoever. The scope of the worksheet provided was exhaustive. It covered details of the content of the film and also included activities on a topic which had not been covered adequately (taluka panchayat). Discussions with Learners The children were able to recall issues covered in the film when they were asked what it was about. They said that the film was about Sakri Ben, the sarpanch of a village. When asked whether they liked the film, they responded in the affirmative and referred to the content that they liked. According to them the solutions to problems of roads, electricity and water were good. The children were not familiar with the Gram Sabha. On being probed to respond to what a Gram Sabha is meant to do, they said that its work includes collecting garbage, repairing roads and collecting water tax. The immediate recall of learners was based on the experience of the main character, Sakri Ben. Learners could also recall core information about the Gram Sabha. Yet, they were not able to recall information related to the tiers of the panchayat structure. They could not respond when asked if they had or had not seen women as panchayat members. In order to comprehend their understanding of values, we asked them if there was any kind of discrimination shown in the film. One learner said that in the film a boy in school was not allowed to drink water because he was from a different caste. We tried to probe this further, but she could not respond beyond this. Most children said they liked the film, and on being asked why, they said that it was because of the visuals. When we asked them if they would be interested in watching films on other topics, one girl said that she was extremely interested in social science and religious stories like that of Eklavya and Shabri and would love to see films on these topics. To the children, Sthanik Swarajya Ni Sansthao appeared to be an interesting film – visually appealing, but they could not relate it to what they had already learned through textbooks. This was also corroborated by government school teachers. Discussion with Facilitators During the discussion with Community Facilitators (CFs) at the Navsarjan ECVU, they shared their concerns about the pressure of work. These Community Facilitators are engaged in the work of the organization on issues of primary education, panchayati raj, women’s rights and rights of Dalits. As a result of this they are unable to fix time and plans for screening of video kits. According to them, the work on rights issues is always more urgent and demanding, therefore, screening of video kits is low on their priority list. The CFs said that they were not involved in the selection of topics for the video kits. According to them they only come to know about the topics at the stage of the rough cut. Prior to this they are not involved in any way. One of them said that they were not involved at the stage of the rough cut and that was because they did not have the time. They are required to travel as part of their work, work as para-legal workers, organize big events (sammelans) and follow up on cases that come to the organization. There was absolutely no time for them to engage with the process of film making. They organize screenings, collect worksheets and hand these to the producers for analysis. Discussion with teachers Two teachers who were part of the discussion said that the film had immense potential to build on the discussions that had happened on local self governance. The fifth and seventh standard teachers said that the screening was like a missed opportunity as it was not linked to teaching or textbooks, and children would not be able to understand the topic fully as there was no discussion before or after the screening. They said that children would only be able to recall information from the film if it was screened in this manner. The screening according to them was ineffective. They said that it was important to screen the film before a chapter is taught in the classroom: this would create an enabling learning environment for children and would help the teachers in transacting information. They also said that each film should be screened more than once in order to reiterate important concepts and ideas. The teachers said that the film was well made; it explained concepts and ideas in an in-depth manner compared to the textbook chapter which was too brief. They also said that the film was located in the children’s own context and therefore it had a different impact from textbooks. On being asked if values integrated in the film were relevant to learners, they said that gender (“difference between ladies and gents and caste discrimination”) was important – this would help young girls gain confidence and also “solve the caste problem”. They also said that other films that are shown to students have information, but do not incorporate values. Only one out of the two teachers had seen another film made by the Videoshala producers. Sthanik Swarajya Ni Sansthao, according to her, was better than the other film that she had seen, Pradushan. The first film used difficult language and was more complex, this film was clear and concise. The teachers were not involved in any manner in the film-making/ transaction process, yet they said that they could contribute by giving suggestions on topics during the selection stage (identification of hard spots) and also in developing the concept for each film. The head mistress Urvashiben Patel was not aware of the Videoshala program. She had been in the school for over ten years (and had been head mistress since 2007), but neither had she seen other films made by the Navsarjan ECVU nor did she know about the program. She spoke about other audio-visual material produced by the Sarva Shiksha Abhiyan, which was also part of the school syllabus. She also spoke about the Mina Manch CDs which were in three parts and provided a lot of information. However, she had to be oriented about the Videoshala films and their content. 6.4 SAHYOG ECVU Sahyog had already completed its cycle of film screening and classroom transaction when we visited the organization. We watched a repeat screening in one of the organization’s activity centers – in Maliwada, Vatwa. The transaction of the films and their impact is based on the observation of that screening, and discussions with learners and facilitators in that activity centre and a private school that the films are screened at (Ashirvad School). Classroom Observation of Bhasha Gyan The film that was screened was Bhasha Gyan, the last film that Sahyog made. The group of 17 children in the Maliwada (Nawabpura cluster) activity centre was mixed, between 6-9 years, classes 1-4; boys and girls, Hindus and Muslims. There were 3 facilitators – Dina, Arpita and Amita – and 3 teachers from the activity centre. From the ECVU coordinator Munira we learned that they came from similar backgrounds, their parents were laborers, rickshaw pullers, or worked in factories – making agarbattis or dying dupattas. The Hindu children were mostly SC, the Muslims were mixed caste. This cluster was the first that Sahyog worked in, and is one of the oldest settlements in Vatwa. Families have been staying here for 500-600 years. There is not much of a focus on education in the families in this area, so the Sahyog activity centers run support classes for weaker children. Before the screening, since we were waiting for the DVD to arrive, Munira, the ECVU coordinator who had accompanied us, sung an interactive song with one of the centre teachers. This was thoroughly enjoyed, and participated in. By the time the facilitator began discussing the film, the children’s attention had been caught. After this, as an introduction, the facilitator asked the children which films they had seen and what they remembered from all of them. Quite strikingly, the children recalled all the films and bits and pieces from the plot lines of all and recited them quite enthusiastically. A more detailed introduction to the content of the film was not done, but seeing the age group of the audience and the content of the film, perhaps this was not necessary. One could see, before the film and during the screening, that the facilitators were highly engaged with the group, making sure they drew the learners in to the discussion, and then keeping a hawk-eye on the children while the film was going on. The use of humour as a way of connecting with the children, and also helping them to recall the films seemed like an important and effective strategy. All these strategies ultimately showed up in the success of the transaction of the video. The students watched Bhasha Gyan raptly, even though at times the film is similar to a language classroom, and definitely less animated than the atmosphere we saw in the activity centre. A combination of the video format, and the fictional narrative of the film seemed to work to absorb the learners into the film for its entire duration. From the discussions with the children later also, it was clear that a combination of the medium and the engagement of the facilitators with the learners meant that the recall of detail of each video was quite high. Subject matter, storyline, characters and values were able to be recalled by the children – not just for Bhasha Gyan, but for even the first film they saw – Vanaspati Etle Shu, and especially Khushi. However, on pointed questioning, the children said that both textbooks and videos were important mediums of learning (“padhenge nahi toh kaise seekhenge?”). What was interesting for us was that the children showed little knowledge (a few seemed to know the Sahyog office) about who made the videos, even though Munira, the coordinator of the ECVU was a regular and popular visitor, and even a familiar face in the films. A later conversation with a member of the community whose son acted in one of the Sahyog films (Aapno Dharm) also reiterated that little was known about the producers of the films. More of an introduction about the producers may increase the interest level of the learners and make the transaction even more effective. After the video, the facilitators had a discussion with the learners about their response to the film – whether they liked it, and what they learned. They were quick to rattle off what they had learned, both in terms of the vowels, and the story of the girl whose parents made her drop out of school. Discussions – about girls’ education, keeping birds in cages, or about vowels – did not really happen in full, according to the guidelines in the activity manual. The session seemed to be happening at a very fast pace. In a circle, the facilitators made the children play a game where they made words combining vowels and consonants, which were written on different cards in front of them. The children had varying levels, but showed great enthusiasm and at least a basic level of understanding – which could have come from their school lessons, but had definitely been enhanced by the video. The session ended with the very rapid completion of worksheets. These included exercises where learners had to recall the symbols representing various vowels, or make words. Children filled in the worksheets with great interest (even though they were doing it for the second time, at least), and perhaps the worksheets could have been a little longer or more challenging, even for these primary learners. Other worksheets we saw had very visual exercises, which combined oral and writing skills in creative ways: for instance the pictures representing opposites, or different situations that the learners had to choose from, in the Khushi worksheet; different forms of work in the Juda Juda Kaamo worksheet. Discussions with Learners Conversations with children at another school, where a film was not screened (Ashirvad School) showed, again, the strong impact that the films had had on the learners there. In spite of being dragged out of an exam revision class, the children were eager to chat about the films. As favorites, they mentioned Khushi, Bhasha Gyan, Vanaspati Etle Shu. However, further probing revealed them to be more able to talk about the values that different films had highlighted – cooperation, freedom, secularism, environmentalism, gender discrimination. While these learners emphasized that the visual quality of the videos aided in recalling the content (especially at exam time!), they also pointed out that many of the films were similar to the textbook – only different in that textbooks have single pictures, whereas videos have many. Children we spoke to individually pointed out that it would be good if science experiments, which they felt were difficult, were shown on video. Discussion with Facilitators The facilitators at were bright and creative, and this showed on the levels of involvement of the children with the videos, their grasp of the content and the values. Even though the facilitators and the teachers at the Maliwada activity centre claimed to not have received a separate training on facilitating the videos, or any training on education or pedagogy, their involvement with Sahyog, an organization that has worked on education for many years, clearly shows. They were able to use the video format to its fullest potential, and recognize what kind of impact it had on the education of the children they worked with. For instance, they said that the film Khushi, a fictional video based on the need for a spirit of cooperation between people of different religious communities, had a very positive effect on the children and the environment they worked in. Even though discussions on communalism, labor and other issues were part of the sessions in the Sahyog activity centers, a film like Khushi made it possible to address and discuss these issues with a wider range of children, and even in government schools which Sahyog screened the films at. It aided in venting certain issues out that pervaded the still-strained area of Vatwa. Interestingly, the facilitators pointed out that the impact on the children in government schools was very noticeable – children who did not use to talk much in class, or express their opinions were now more likely to speak. These changes, the facilitators claimed, were even pointed out by the government teachers and principals, who thus assessed the quality and impact of the videos themselves and gave the facilitators feedback! The facilitators themselves admitted to a change/deepening in their understanding of values, and their openness to work with people from other communities. However, as mentioned earlier, in the screening of Bhasha Gyan that we observed, there was little or no discussion on values as per the activity manual provided to them. This may have been because it was a repeat screening of the film. Even though the facilitation of the video was effective, the facilitators and teachers couldn’t articulate what were the strengths of the video format over the textbook – beyond the fact that the videos were visual. In fact, since the facilitators were trained by Sahyog, they already had access to a certain progressive pedagogy, which also was reflected in the videos. We could see that a deeper engagement in the subject matter was necessary for the facilitators to make most use of the material. This could happen through a separate training of teachers who will facilitate the videos, or a higher level of participation of the facilitators in whatever research and training the producers go through while making the video. Discussion with Jasbir Kaur: A Parent A conversation with Jasbir Kaur, mother of the Sikh child Jasmeet in Aapno Dharm summarised the impact that the videos have had, and could have in the future quite succinctly. Jasbir said that the education that children were getting was too bookish, and required discussion of issues like religion and communinalism – otherwise ‘bacchon mein kattarvaad aa jata hai’. Jasmeet had taken his teacher by surprise when he stood up in class and told her knew what a synagogue was – and Jasbir was proud that he had learned enough about different religions to have a wider understanding of his own –‘dimaag mei kuch toh baithta hai’. She felt strongly that longer videos (and more) of this kind needed to be made and screened at as many schools as possible. 7 PROCESS ANALYSIS OF VIDEO FILM MAKING This chapter provides the analyses, carried out by the Nirantar evaluation team, of the process of making the seven video films from each ECVU. The Nirantar evaluation team spent considerable time with producers and the CCU to understand the method by which the topic of the video was selected, how the research around this topic was done, finally to arrive at a concept and then a script. The levels and kinds of support, input and feedback were also discussed – resource materials, workshops, technical and content related support and so on. The role of and relationships between the CCU and producers were analyzed, with regard to the final products – the video-kits. 7.1 HIND SWARAJ MANDAL (HSM) ECVU The video Bhoomi was prepared before Urja. Discussions with the producers shed light on the ‘hard spot’ research as a valuable way in which to identify areas which are difficult for students to learn. Science, Math and language were areas that the HSM repeatedly came to know as ‘hard spots’. At the hard spot workshop organized by the CCU for the producers, science was seen to be a boring subject. Thus the first two films that the ECVU made were on science topics – Bhoomi and Urja. CCU aided in the selection of these topics, and also narrowed down their focus. The producers received lot of help from the hard spot workshop, in fact they refer to the workshop when deciding the themes for the films. During our discussion with the producers, they said that when the video is screened, they discuss the topic for the next video with the learners. However, during our visit to two schools we found that neither the facilitator nor the coordinator posed this question to the children. The selection of the hard spot is the job of the producers only. The facilitator is not involved in this process. Research on the film is also done by the producers only, and this we saw as a gap between the product (the video) and its use. It was clear to us that that an involvement in the process of production would mean a better grasp of the content of the video. If the facilitators are involved in research and hard spot selection, then the session would also be facilitated better. We found during our visits that the facilitator completes the job in a very instrumental manner. The whole process, from selection of the hard spot to preparing the video, requires around two months. This period involves preparing a concept note based on the topic and how it is going to be made into a film, extensive research on the topic, preparing an understanding of the subject area, writing the story, preparing the script, deciding the characters and location, shooting and editing. The producers shared that they receive feedback from their own team, and the CCU at every stage. There were many occasions when the CCU suggested resource persons and relevant material for the themes. Most of the producers use the Sarva Shiksha Abhiyan textbooks as reference material. This is supplemented by a resource person who is an ‘expert’ in the area to prepare an understanding of the subject. For example, in the case of Lokshahi, they took the help of Vasudeo Bhai and Naseem Nikat (CSJ) while preparing an understanding on democracy. The producers felt that a value of the Videoshala program, as well as of their ECVU, was involving the community in various ways. We met with a member of the community who did the music for all the HSM videos. All of the HSM films have only original music, composed and sung by members of the community. Although Dhruv did not seem to know a lot about the program, he seemed committed and enjoyed working on the films tremendously. Finding skills and talent in the community to strengthen their films is something the team said they enjoyed, and that they saw as a value. Apart from this, the producers also saw the community as an important source of information for their films, especially the social science films like Lokshahi and Aapno Hak. We met the principal of the local panchayat school, who was a committed resource person for the producers, providing a wealth of information on local issues, education and also staying up all night to assist in editing during production of the final cut! Although we did feel like the pool of community that the ECVU accesses is quite small – except in few occasions when they have done screenings of their films among the community and facilitated discussions among them8 – they seemed to see garnering community support and involvement as a strong point of the program. 8 Screening of Aapno Hak in Bhatia village. Over 300 people attended. Both the producers as well as the CCU admitted that feedback was an integral aspect of the process of video production. The feedback given is at different levels, and on different aspects of the video – content (subject areas and values), script, technical skills. During the discussions with the producers it did seem that with the involvement of a number of people in the process of feedback does affect the efficiency of the process, and makes it hard to see the validity of all feedback received. They said that often they were confronted with a new person at every stage of feedback, and were confused as to which feedback should be taken. They suggested that the same person provide feedback through the process of one video production, so there was consistency in terms of the feedback given at each stage. The team said the feedback from the CCU in the first phase of the Videoshala project was helpful for ensuring an understanding of the producers in terms of content and pedagogy. The HSM ECVU is quite strong so far as technique is concerned. It can produce films independently. The team has benefited by the constant support of a technical trainer over the entire 18 months, and this kind of support may not be necessary in the next phase of the project. The major challenge the group has faced, in spite of regular feedback has been the integration of subject and values. They hope this will improve in the next phase of video production. Values have been understood in a superficial sense, and it is necessary for the ECVU to understand their complexity – for instance, a value like diversity. Some producers feel that they lack a proper understanding of the methodology of research in social issues. They hope to receive inputs on these issues. Another gap that they identified is a strong perspective and understanding of gender issues, and therefore, an independent workshop for this purpose is necessary. This was drawn from independent conversations with the producers, based on their own experiences of working in the ECVU for 18 months, the changes within them, as well as from the video kits themselves. 7.2 MEGHDHANUSH ECVU We presumed that the context of the facilitating organization would be a significant factor that impacts subject matter and content, representation of values and the entire process of film-making. Meghdhanush has been working in Panchmahaals district since the 2002 Gujarat riots that targeted the Muslim community, and displaced them from their villages by creating fear, using violence and terror. Large numbers of Muslims then settled in Kalol and Halol were organized first into camps, and subsequently into colonies. Udaan-Meghdhanush has been working in this area with children from the Muslim community. This, however, did not seem to be reflected in the Meghdhanush films, except in one film Aavyo Sandeso Vahan Vyavahar No. This is interesting because even the composition of the producer and facilitator teams shows that there are Muslim, Dalit and Adivasi youth from the community. Despite that, the representation of Muslim people, their context in Halol and Kalol, their politics, conflicts and struggles, as well as their spoken language (Hindi) – did not find place in any of the Meghdhanush films, and appeared only somewhat in Aavyo Sandeso Vahan Vyavahar No. In later conversations with the organization leaders, we gathered that this had been an issue for debate in the initial stages, and since the ultimate aim was to mainstream these videos into mainstream curriculum and pedagogy, it was decided that the films would continue to be made in Gujarati – the language in which the mainstream education system functioned. Teachers in the government school at Arad commented about the content of the film as good. They said the main advantage of these films is that the characters in it speak the same language and dialect as the children. Our observation is that in the Arad school, the majority of the children watching the films belonged to the OBC community. The major composition of students in that school was of this community, and their mother tongue is Gujarati. Therefore, they probably identified with the children in the film. By contrast, when we asked the children at the Kalol school whether the children in the film speak in the same language as them, they said ‘no’ – because these were Muslim children of Casimabad, whose spoken language is Hindi. The group of film-makers (producers) was well-trained – as part of the Videoshala program, as well as part of the Meghdhanush program before that. For instance, Jankhna, Mudassar, Neeta and Gulab had been with the Meghdhanush program as teachers in the primary education program before they were selected in a workshop to become community film producers. They said that this made it easier for them to understand various elements of the teaching-learning process which they could use in the making of the films. However, they also mentioned how difficult it sometimes is to put values into films like Pupu. It is basically a science film, and they said they found it challenging to think of ways in which to incorporate values into this subject matter. The sensitivity and skills of teachers who are now producers should be tapped. They also need to be given guidance in dealing with the challenge of representing certain values in science-based films. Also, in order to help producers make sound films on education, it would be desirable for them to undergo strong trainings on pedagogy, curriculum and other elements of the teaching-learning process. A large part of the success of the film screenings, the transaction and feeding into learning for children depends on how well the teachers who teach them regularly can integrate the Videoshala films into their ongoing teaching. In the Kalol school the teachers’ ownership of these films was quite strong. They mentioned that they had participated in the film-making process by being involved in hard-spot identification, being involved in research on the subject matter of the film, in giving regular feedback during the scripting process, suggesting actors for the films, and being involved in activity and worksheet development. For instance, in the film Pupu, the producers included the concept of transparency and explained it with the example of spectacles, after they got this suggestion from the teachers during a feedback session. But there was no such engagement with teachers in the government school. The program leaders of Udaan-Meghdhanush felt the importance of the Videoshala program was in that it involves members from the community in concept, plot and film development. They also felt it was valuable that the videos provide an opportunity to integrate values of equality and citizenship into school curriculum and teaching. They said it opened the window for Meghdhanush to engage with mainstream pedagogy, materials and ideology, and paves the way for bringing about structural changes in the education system. Since the program very consciously links the Videoshala program to its larger objectives, it seemed to be an organic part of the larger program. This was reflected in the team members’ ownership over the ECVU films, and their evident commitment to integrating it as teaching-learning materials into their ongoing educational work. It was also clearly articulated by the organizational head in the interview we had with her. We gathered from leaders of the Udaan-Meghdhanush program that the future plan was for ECVU films to be screened in a large number of government schools across Panchmahaals district. In fact, we also learned about how their strategies had changed over the past two years regarding the screening of the films in schools. For instance, in the beginning the focus was on doing film screenings in Meghdhanush and some government schools. In doing this, they realized that it would be valuable to speak to district officials before going into the schools. At the current stage, they are now even engaging with Block Resource Centers, through training of teachers, in order to ensure that qualitatively good film screenings take place, and organically build into the curriculum. On the whole, the presence of Meghdhanush seemed to really impact quality of transaction of the films – both in Kalol and in Arad. Since they have been working on primary education in that area for about 7 years now, and there is a clear linkage they make with the larger program goals, the use of the films in both schools seemed meaningful. One element that we felt requires some more reflection is the role being played by producers and facilitators. Programmatically, we understood that earlier there were more spaces for both to interact – whether it was in hard-spot identification, in the research process, scripting or other aspects of film￾making. This, however, probably began to change when the push came for doing film screenings on scale. Therefore, there is now a whole new team that is expected to transact the films in government schools. There are a few issues here that need to be addressed. One – the facilitators are not involved formally in film-making or in research to understand the subject matter in a more in-depth manner. Moreover, while they are the ‘face’ of the program in the schools, the creative work of the producers often goes unacknowledged. In both schools in Panchmahaals, the facilitators did not share with the teachers or learners about who has made the films and why. In our interviews and discussions, we found the facilitators also felt that the program had moved from a more organic approach towards greater fragmentation. Organizational leaders, however, felt that the issue of orienting facilitators or teachers to the films, or building an understanding on education could be tackled with good trainings for the facilitators. In any case the vision for the next phase is that government school teachers will take on the facilitation role. There was not much thinking on another important aspect of the program, namely, high turnover of women workers. Role of the CCU As far as the role of the CCU is concerned, we felt that the technical inputs over a period of a year and a half were quite solid. The Meghdhanush ECVU had been established as a community producers’ unit and they had grown in confidence regarding their own film-making skills. The CCU also managed to put out a good model of functioning – starting from identification of hard-spots through workshops and speaking to children and teachers. They also got valuable insights from government school teachers to help in hard￾spot identification. Subsequently the CCU provided technical inputs as well as substantive support by involving experts, besides guidance in scripting, rough cut and final cut. However, in terms of perspective and elements of education (curriculum, pedagogy, materials, etc) we felt the understanding of the ECVU teams is limited and needs to be strengthened in future. We learned from the CCU that this role was primarily to be played by the organization Directors. Some books and resource materials were provided by the CCU, and others were taken care of by individual organizations. Based on discussions with producers and other team leaders in the ECVU we gathered that a large part of the resource material in Meghdhanush was provided by an active program leader (Arti). Other than that, producers mainly relied on Gujarat State Board textbooks, and to some extent referred to material from local libraries. We felt that the producers did not refer to existing alternative materials or curricula as much as they could have. We feel this may have impacted the producers’ understanding of concepts, the research process, as well as content of the videos. It may also have implications on perspective – or the lack of it. 7.3 NAVSARJAN ECVU Sthanik Swarajya Ni Sansthao is the fourth video kit made by Navsarjan ECVU. The producers had made video kits on science subjects before this. A mapping of teaching and learning difficulties in Standards 5- 7 that was done by them made it clear that Social Science topics were also in the list of most difficult themes. 9 Institutions of local self governance were chosen as the topic for the fourth film. The coordinator of the ECVU, who has also been involved as a producer, suggested that this topic be taken up as it was challenging for teachers and learners and was closely linked to the local context. The topic also lent itself to raising issues of rights and discrimination. There was an intensive process followed by the producers before the film was finalized. In order to understand the issue better, they did the following: • Visited gram panchayats and taluka panchayats • Interacted with the sarpanch of Sujatpura village to understand the work and functions of the gram panchayat and elected representatives. • Analyzed the chapter on local self governance from the school textbook • Watched films such as Welcome to Sajjanpur, and documentaries made by National Children’s Film Society and Gaam Nathi Koi Panchnu. • The film was also based on the experiences of a woman sarpanch who was beaten up in her village. She had approached Navsarjan Trust. Her story was the motivation for this video kit. The preparatory work provided the team of producers with an in-depth understanding of the issue – the interaction with the sarpanch gave the team an overview of the functioning of the Gram Sabha and the challenges faced by panchayat members. This interaction and the analysis of films that had been used as resource material gave the producers an idea about the characters of the film and its location. The analysis of the school textbook also highlighted the limitations of existing material in providing clear, contextualized and concise information. This understanding and the experience of the woman sarpanch helped the producers develop the concept note script for the film. At the stage of the concept note, the team decided that it would provide information that was essential. This would include information on the rural institutions of self governance- as opposed to including rural and urban institutions. The team felt that covering both issues in a film that was 30 minutes long would be unfair as it would be too much information with little analysis – like the textbook chapter. Special attention was also paid to inclusion of values in the story at this stage. Feedback was sought from the CCU on the script and the rough cut. This feedback helped the ECVU strengthen the use of local 9 Videoshala Quarterly Report October- December 2008. knowledge, language and realities in the film. Feedback also enabled them to include songs and make the film technically sound. The producers mentioned during the discussion that the CCU had asked them to not include shots of violence faced by the woman sarpanch. This was because the CCU felt that showing her struggles in this manner would take the focus away from institutions of local self governance and would highlight her story. There is an intensive process of research that precedes the making of each video kit. For producers who are involved in this research, this is a process of significant learning. Not only does this provide in-depth information and clarity on themes, it also deepens their perspective on values that are core to the program and upgrades their technical skills as well. The producers have a deep sense of commitment towards the ECVU. For them each process of film making is energizing. In the case of the Navsarjan ECVU there seems to be a systematic process of seeking feedback from the CCU. At each stage the feedback has been useful and constructive. Critical feedback in some cases has also delayed the process of timely completion of video kits. Both the CCU and ECVU should take this into account as the pressure for completion of films is tremendous. 7.4 SAHAYOG ECVU Production of Juda Juda Kaamo Although we did not see the transaction of this film, it did come up for discussion with the children we spoke to, who said that it combined information that they had from various sources, and new information – on occupations like journalists, radio jockeys etc. We were interested to know what the process of making this film was, since it combined textbook and local knowledge in a way that was quite commendable. The research on the hard spot, and the narrowing down of this area to focus on occupations was quite exhaustive:10 • Interview with school children – in groups and individually, to determine their knowledge and understanding of different occupations. • Textbooks - Both in terms of analyzing what the textbook content on this subject was, but also to see what kind of stereotypical representations of different occupations exist in the textbooks, and to create content that challenged these stereotypes. • Other books: Especially those that had a strong gender and caste perspective, and enabled the producers to understand the way manual labor or women’s labor is represented. • Films: A range of documentary films on occupations were watched by the producers to see other content on occupations and the discriminatory attitude towards some of them, and also to explore the technical options in making a film on occupations – how interviews can be shot. • Conversations with interviewees: This was both to speak to the people featured in the film about the kind of involvement expected, and gauge what kind of responses they would give, and what kind of content the film would have. This process of research clearly shows in a well-development and clear concept – to communicate to children different kinds of occupations in the world around them, and inform this information with an understanding of why certain work is invisibilised or discriminated against. The genre of a documentary film worked well to maximize the video format – a range of people were chosen to feature in the film from the children’s everyday lives, and figures who the children were not familiar with but curious about. For a fairly mixed group of producers – age, religion, socio-economic background – many of whom had never handled a camera before, the video shows a certain level of technical skill and comfort with the medium. To the extent that the producers could clearly tell how to make the medium tell the story they wanted, in the best possible way. 10 Information from the Progress Report for the quarter January-March 2008. However details of which books and films were used as resource material were not available. Later discussions on Khushi and Bhasha Gyan (both of which use the fiction genre) showed that the producers were clear on when to use which genre, its strengths and weaknesses. Production of Bhasha Gyan This video seemed to have gone through quite a challenging process of production. The team had chosen to do a film around the hard spot of language – which many learners we also spoke to raised as a difficult subject. The first script and storyboard of Video 6 revolved around Television, as children had raised this as a medium they wanted to know more about. However, after the research, story and script had been finalized, and the rough cut produced, the CCU gave the team feedback that their film be based around the hard spot of language. The hard spot research, concept building and storyline was thus redone – a process both frustrating and challenging for the team. Inputs on Gujarati language and vowels were given by Udaan and an external resource person Mr. Keshav Chatterjee. The team saw these as essential in building an understanding of language concepts and pedagogy.11 From the discussion we had with the producers it was clear that the CCU’s feedback was sought and given at each stage of video production. The CCU’s suggestions were seen as valid and helpful. However, it was clear that the feedback process was desired to be more streamlined – in that the same people should give feedback at each level of production so the process doesn’t move back and forward till the end. Also that feedback is not sought from an external person too close to the end of the production cycle. There was seen to be the potential of compromise and discussion with the CCU, in case the team did not agree with the CCU’s feedback. However, the considerable decision-making power of the CCU seemed quite clear from the producers, as well as from the process of production of Bhasha Gyan. From both the discussions with producers and the films at Sahyog, it was felt that the team had a strong perspective on values – of diversity, citizenship and gender. The tendency was to move rather boldly away from curricula content, to other areas where they thought values could be interwoven. The producers – both individually and as a group – spoke about how their strength lay in being able to link values with the content of the films.12 They admitted that certain values – of equality of religions and gender – crept into their films almost unconsciously. They also spoke about the visible impact of the films on the community they worked in, in terms of the issues raised and the discussions in classrooms. They saw the video format as one among different mediums they used to spread a certain message in the community, which they were trying to spread through their other work. That the Videoshala program was one among the many innovative methods that Sahyog employed was clear from conversations with the coordinator of the ECVU and a trustee of Sahyog that we spoke with. The latter spoke about monthly meetings in which the entire Sahyog team met, and also joint trainings on pedagogy, education and values which all organization members received. The program seemed well knit into Sahyog’s work. At the same time, we got the impression from the producers that they had imbibed the strengths and values of the organization into a fairly independent video-production unit. The CCU feedback seemed to attempt, in many production cycles, to bring the team closer to curricula concepts (for instance in Bhasha Gyan). While it may have been relatively easy, and close to home, for the Sahyog ECVU to make films that generated discussions around values (because of the context in which they work), it is also important for them to push themselves to bring these discussions into films on science, geography and language. This emerged from discussions with learners as well. The Sahyog ECVU seemed a little inconsistent in terms of technical skills – some films were much superior to others. The need for more consistent technical support was raised by the producers and specifically the coordinator, who held that it was difficult to give her team critical feedback on technical aspects of the film when she hadn’t been trained in the technical part of filmmaking. The need for a longer and more research intensive production process was also articulated, which should differ with films produced for screening in schools, and for the community. The research material – textbooks, other films, some books – seemed limited, and the producers agreed that other innovative teaching/learning material produced by other organizations may push the level of their own material to be better. It was interesting to discuss the power of the video format with the Sahyog ECVU, as they had trainings in pedagogy, and the video kits were not the first time that they had experimented, or had discussion 11 Information from the Progress Report (January-March 2009). 12 Cf. interview with Ramesh bhai, producer from Devdi gaon near Vatwa. around values with learners. They still saw the video as a powerful medium to visually depict stories and situations which provoke discussion with children. They also saw the strength of the videos in their local, accessible language and visuals, which even sets them apart from other audio-visual mediums like the internet. At the end, though, the producers saw the effectiveness of the videos in a combination of strong content and facilitation, strong information and value components. 8 TECHNICAL ASSESSMENT OF VIDEO FILMS This chapter provides a technical assessment of the video films produced by the ECVUs, carried out by CMS. Appraisal of all the 23 videos produced under the Videoshala project was done by the CMS evaluation team and supported by a technical expert, who had rich experience of working as a senior producer in a TV News Channels. Except Navsarjan ECVU, six videos each produced by the three ECVUs were assessed. In case of Navsarjan, five videos were assessed. The Videos produced under Videoshala project have been evaluated on ten parameters. These are: 1) Beginning/Closing shots: Marks were given on the basis of how effective is the first and the ending sequence which set the tone and the pace of the film. Opening and last shots of the films leave an impression and that should be well thought of in advance. 2) Camera Composition: This suggests framing of shots. Shots should be: well framed, subjects should have proper head room and looking room, should not abruptly cut any body parts of the subject. Camera should focus well whatever is being talked about. Camera angles whether low angle, top angle or at an eye level forms important part of the camera composition. Proper cut aways and variety of shots form part of camera composition. 3) Camera Movement: Shots should be steady and smooth. Wherever required the tilting or pan or zoom in and zoom out should be well focused and steady, no jerks in between. Block shots should be neat and clear. 4) Picture Quality: This parameter suggests the quality of camera work done. Picture is underexposed or over exposed or the right exposure is maintained or not. Exposure should be consistent and should be maintained in the exterior and interior shots. Shots should be sharp and not out of focus. White balance has to be done to ensure right tone of the picture. 5) Sequencing of film/Direction: Characters of the film have to be guided to enact seriously and bring out the message of the film in an effective manner. The direction of the shots (meaning the way the film moves and the story line of the film evolves) has to be in a certain logical manner and so do the shots one after another. Shots have to be self-explanatory and have to be shot in a proper sequence and not break the flow of the story. Innovative ideas to explain simple things require good direction. 6) Editing: is a great tool where most of the look of the film is given shape. It takes care of the sequencing of the film and shots have to be put in a simple cut to cut edit. Wherever required add fade in and fade out is needed to bring out the proper mood of the story. Proper stay has to be given to the shot and moving shots should not be cut abruptly. 7) Graphic Plates: is used to enhance the quality of the film by adding special footnotes or registering certain important pointers in the film. Maps, or any visual presentation of the points or only text written on a simple black plate is called graphics and it can help register certain important points in a better way. They compliment the film shots. 8) Usage of Music: any music in the film should set the right pace and uplift the mood of the film. Background music should be chosen carefully. 9) Sound Quality: the sound of the film is a natural sound and used to bring originality of the surrounding, Sound bites used should be clear, sound levels in the film should be consistent and no sudden jerks in the sound should be there. Over all the sound levels should be at the right level. 10) Message clarity/Presentation: The message, whatever the subject is, in the film should come across clearly. Story line should be such that is seamless and communicative. Innovative ideas that bring out the ‘subject’ play an important role. The videos were rated against each of these parameters on a scale of 1-5, where 1= Very Poor, 2= poor, 3= Average, 4= Good and 5=Very Good. This was done to bring in more sharp comments in the process of evaluation, which in turn is expected to help the CVPs and the technical team to take into consideration the areas of improvement in the video production, in the future. 8.1 HIND SWARAJ MANDAL (HSM) ECVU Urja S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 5/5 2 Camera Composition 3 (More variety of shots would enhance the production quality) 3 Camera Movement 4 4 Picture quality 3 (Exposure changes suddenly in between the shots, background is burning in few shots) 5 Sequencing of film / direction 5 (Idea of puppet show to show the importance of fuel is well thought and brings freshness in the film. Breaks the monotony). 6 Editing 4 7 Graphics plates None 8 Usage of Music 4 9 Sound Quality 5 10 Message clarity/presentation 5 Overall Remarks: This film deserves special mention, well thought beginning/closing, original ideas and presentation style is good. Good angular shots and variety in shots can be seen. Characters chosen in the film were articulate and message of the film comes across effectively. Apna Hako S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 4/3 (Film takes off well with the Sadhu character setting the right tone through his song, ending shot could be better) 2 Camera Composition 4 (More cut aways could be used while teacher is interacting with the school kids to make the conversation interesting.) 3 Camera Movement 4 4 Picture quality 4 (Exposure in thana shots changes) 5 Sequencing of film / direction 4 6 Editing 3 (Moving shots are getting cut by another shot, more stay in few shots is needed) 7 Graphics plates 5 8 Usage of Music 4 (Sadhu character preaching through his songs is a good add on) 9 Sound Quality 4 10 Message clarity/presentation 3 (Message is clear but the presentation style could be more effective) Overall Remarks: Good attempt to instill confidence to fight against exploitation. Film should have right and consistent light exposure. Better-cut aways would increase the production quality. Bhoomi S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 5/5 2 Camera Composition 4 (good variety of shots at suitable chosen locations) 3 Camera Movement 4 (Good job while taking moving shots from the bus) 4 Picture quality 4 (Light exposure changes in between the shots in few places) 5 Sequencing of film / direction 4 6 Editing 4 7 Graphics plates None 8 Usage of Music 4 9 Sound Quality 4 10 Message clarity/presentation 4 (good ideas used to explain complicated subject) Overall Remarks: Neat film, shots are well thought, editing is good, music is appropriate. Aaharaney Aarogya S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 4/4 (Good establishing shots, smooth) 2 Camera Composition 4 (Neat block shots, jerks in between the shots) 3 Camera Movement 4 (good follow up of the characters and the subject, few shots not steady) 4 Picture quality 4 5 Sequencing of film / direction 4 (good activity on the location, efforts taken to explain food items, looses it’s magic towards the end) 6 Editing 4 7 Graphics plates None 8 Usage of Music 4 9 Sound Quality 4 10 Message clarity/presentation 4 (In the last sequence, children doing cleaning up and ending on these shots looks little out of context) Overall Remarks: Last shot should be well thought and planned. Avoid jerks in between the shots. Lokshani S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 5/4 2 Camera Composition 5 3 Camera Movement 5 4 Picture quality 3 (Footage overexposed, background is burning in most of the places) 5 Sequencing of film / direction 4 6 Editing 4 7 Graphics plates None 8 Usage of Music 3 (Better music needed to enhance the production quality) 9 Sound Quality 3 (Audio mixing to be done in a better way) 10 Message clarity/presentation 3 (ending is little abrupt) Overall Remarks: Audio mixing has to be smooth, more cut aways and well thought ending would improve the quality. Paryavaran Aney Tenu Kudrati Sampada S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 3/2 (Such a vast topic is not packed in tightly, beginning and end could be well thought) 2 Camera Composition 3 (Better framing of shots required, Bite frame could be better, more depth in the frame would bring better quality, avoid showing mike in the frame, use lapel mike instead) 3 Camera Movement 4 4 Picture quality 3 (Exposure fluctuates in between, suddenly light is thrown while children are coming to chat up with dadaji) 5 Sequencing of film / direction 2 (Repeat shots make the film boring, lack of variety of shots, more stock shots are used, fresh shots would have enhanced the quality) 6 Editing 2 (Moving shots are getting cut and replaced by another shot, avoid it, dissolves are not required while kids are talking, too many location jumps, continuity breaks while girl is serving water) 7 Graphics plates 3 (Too jazzy and distracting background, simple gfx would look neat, jerks while editing put them in the loop and use) 8 Usage of Music 4 9 Sound Quality 3 (Audio levels not consistent, keeps changing when dadaji is explaining and suddenly resonance increases when narrator comes in, mixing have to be smooth) 10 Message clarity/presentation 2 (This film has a great scope to explain things though the direction, sequencing takes a back seat in terms of shots used, lot of stock shots are dragging the film, invention of wheel and fire could be shown in sepia which can create effect of past days effectively) Overall Remarks: Vast subject could not be handled effectively; film got confused while using shots and sequences. More focused approach could improve the production quality. Swashan Tantra S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 4/4 2 Camera Composition 3 (Shots of the central character are not neat, his body parts are getting cut, more block shots and cut always needed) 3 Camera Movement 3 (Camera movement should have been better while girl is explaining the diagram on the mud, more focus on the diagram is needed as she explains) 4 Picture quality 4 5 Sequencing of film / direction 4 6 Editing 3 (slow motion of the act – while the boy throws stick and its repetition to bring impact is not needed) 7 Graphics plates 5 8 Usage of Music 3 (Music in the opening sequence could be better, bad music when the boy throws stick and decides to get in the river) 9 Sound Quality 4 10 Message clarity/presentation 5 Overall Remarks: Presentation of the film is good, original ideas to explain the Swashan tantra system. Effective conversation between the girl and the boy while explaining the process through diagram. Innovative style of boy to pick up stuff like dry leaves etc to make the diagram and bring clarity. Water S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 5/4 (Beginning shots are well thought of, sets the right mood, closing shots are good, use of natural ambience while kids moving with the message is preferable) 2 Camera Composition 4 (More variety of shots of kids in the class needed, close up of Dholak, children singing would enhance the production quality. Use more block shots while kids are near the well. Kid asking question to the lady, boys’ face is not visible. Looking room is missing in some shots) 3 Camera Movement 3 (Faces are getting cut, while camera is moving in the class) 4 Picture quality 3 (Pictures are not very sharp at few places, long shots specially) 5 Sequencing of film / direction 4.5 6 Editing 3 (Repeat shots to be avoided, lotus flower shot used twice, extreme close up of teacher to be avoided. Don’t change from one moving shot to another moving shot. Good use of slow motion, enhances the mood) 7 Graphics plates None 8 Usage of Music 4 (Starting music is good. Avoid music from films wherever not needed, use of natural ambience preferable) 9 Sound Quality 4 (Sound Mixing is good) 10 Message clarity/presentation 5 Overall Remarks: (Message of the film comes out very well. Interactive session of the students and teacher is very interesting. Pay attention to camera composition and avoid repetition of shots) • The Earth S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 5/4 Neat take off and ending was simple yet had an impact 2 Camera Composition 3 (Frame getting cut, no headroom in few shots and looking room should be better) 3 Camera Movement 5 (Good silhouettes, smooth and steady shots) 4 Picture quality 4 5 Sequencing of film / direction 4 6 Editing 4 7 Graphics plates None 8 Usage of Music 4 9 Sound Quality 5 10 Message clarity/presentation 4 Overall Remarks: (Innovative way used to send the message. Pay more attention to the framing of shots.) Pu Pu S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 4/4 2 Camera Composition 4 3 Camera Movement 4 4 Picture quality 3 (Footage over exposed while kids are searching Pu Pu) 5 Sequencing of film / direction 3 (Direction is loosing it’s magic in between, too stretched conversation, could be more interesting sequences) 6 Editing 4 (Good window split) 7 Graphics plates None 8 Usage of Music 4 9 Sound Quality 5 10 Message clarity/presentation 4 Overall Remarks: Nice film, explanation thro simple idea of Pu Pu is interesting. Better direction would improve the film quality. Sandesho Aavyo Vaahana S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 4/3 (Good establishing shots, mid-long shots used) 2 Camera Composition 4 3 Camera Movement 4 4 Picture quality 3 (Footage overexposed, light jump drastic when mother is lying ill) 5 Sequencing of film / direction 4 6 Editing 4 7 Graphics plates None 8 Usage of Music 3 (Better music to enhance the production quality) 9 Sound Quality 4 10 Message clarity/presentation 4 Overall Remarks: Simple shots, fresh shots, neat film, pay attention to the light jumps. Sarvada S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 4/4 2 Camera Composition 3 (More neat block shots could be used, more variety in the shots especially colorful and beautifully done up class could improve the production quality) 3 Camera Movement 4 4 Picture quality 4 5 Sequencing of film / direction 4 (Unnecessary cut aways used in between not jelling in the sequence) 6 Editing 2 (Avoid fast cuts, axis jump, lip sync of the girl – main character who is teaching is out ) 7 Graphics plates None 8 Usage of Music 4 9 Sound Quality 3 (Audio levels not consistent, audio dips at a few places, ambience is too loud at places) 10 Message clarity/presentation 4 Overall Remarks: Film comes up with a unique and interesting way of teaching addition. Also the idea of using comic bubble is interesting and registers well. Overall the story comes out effectively. Technically, right editing would improve the production quality. 8.3 NAVSARJAN ECVU Vanaspati S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 2/4 2 Camera Composition 3 3 Camera Movement 4 4 Picture quality 3 ( Light exposure needs to be corrected, more sharp images required) 5 Sequencing of film / direction 3 (Lack of activity in the shots) 6 Editing 2 (Editing is not smooth, one frame of a girl appears in between) 7 Graphics plates 2 (Do not need to write pause, slow fade out to black could be used to give the pause in between the sequence) 8 Usage of Music 2 9 Sound Quality 4 10 Message clarity/presentation 3 (Presentation could be more innovative an new ideas could be incorporated) Overall Remarks: Editing should be better, crisp and pay attention to the light exposure. Paryavaran Aney Thenu Kudrati Santulan S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 5/4 (Good establishing shots, not smooth though) 2 Camera Composition 3 (Variety of shots like Top angle break the monotony though Head room of the mother character is not appropriate while she is explaining her son.) 3 Camera Movement 4 4 Picture quality 2 (Footage is over exposed at many places so looking washed out and not sharp enough, White balancing is out not in most of the classroom shots) 5 Sequencing of film / direction 3 (Some un necessary shots in between, characters enacting the film need to look serious and give right expression) 6 Editing 3 (Jerks in between the shots, repeat shots to be avoided) 7 Graphics plates None 8 Usage of Music 4 (Background music is good, at one place, use of fast music and then suddenly regional music comes, mixing needs to be better) 9 Sound Quality 3 (Mixing to be done more smoothly) 10 Message clarity/presentation 4 Overall Remarks: Sound mixing to be done smoothly, use right cut aways to boost the film quality. Gujrat Nu Lokjivan S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 5/3 (Film takes off well, neat block shots of children) 2 Camera Composition 3 (Better framing of shots required) 3 Camera Movement 2 (Movement is not steady, it’s shaky in most of the shots, need neat shots to bring impact) 4 Picture quality 2 (Extreme long shots taken on tele look washed out and they are not required, lowers down the production quality, Exposure not consistent in most of the places.) 5 Sequencing of film / direction 2 (First sequence is too stretched and is not required, can cut it short, make it more crisp. Sequences at various locations are not flowing well. Aesthetics are missing out in shots. 6 Editing 2 (Shot changes while the first shot is moving, Repeat shots when referring to the same location, avoid it. Too many location jumps, More stay in some visuals required to bring out the impact) 7 Graphics plates Not used, maps with the location supers and the important points while referring to the location could be written by using graphics window for better comprehension. 8 Usage of Music 2 (Music changes abruptly, regional music could be used effectively to set the right mood) 9 Sound Quality 1 (Audio of children speaking is missing out in between, audio levels are not up to the mark, bad audio mixing) 10 Message clarity/presentation 2 Overall Remarks: Film takes off well but later on the production quality goes poor. Film keeps loosing its flow and most of the shots fail to bring an impact. This film should use neat, block shots, not much movement, and good music and bring out the rich culture of the region in nice colorful shots, maintain the picture quality. Pradushan S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 5/4 2 Camera Composition 3 (Too many long shots while kids interacting with teacher, mid long and close up shots could bring more quality and variety) 3 Camera Movement 3 (Shaky camera while kids taking out the procession, focus out at few places) 4 Picture quality 3 (sharpness missing in few shots) 5 Sequencing of film / direction 4 (Idea of tree speaking to the girl is a good idea) 6 Editing 3 (moving shots, pans shots are getting cut while moving) 7 Graphics plates None 8 Usage of Music 5 (Good background music, lifts up the mood) 9 Sound Quality 3 (Audio not clear in long shots) 10 Message clarity/presentation 4 Overall Remarks: Shots need to be more engaging. More clarity in shots and proper editing. Sthanik Swaraj Ni Sansthao S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 3/4 2 Camera Composition 3 (Establishing shots not impressive, shots of the Panchayat building exterior and interior shots not up to the mark. 3 Camera Movement 3 (Not smooth, jerky) 4 Picture quality 2 (Background burning- exposure is not correct, specially the interior shots , exposure keeps changing) 5 Sequencing of film / direction 3 (Few shots are staying for too long, few getting changed very fast, avoid it, bullock cart in the background has reached in the middle of the frame, shot changes and bullock cart disappears completely- avoid these jumps in the frame) 6 Editing 4 (Fast cuts, no stay, location jumps, interior to suddenly exterior shot, can use dissolves if required, Long shots then jump to extreme Close ups is avoidable, footage looks jerky) 7 Graphics plates None 8 Usage of Music 3 (Music in first sequence creates a mood of mishap, better music to be used) 9 Sound Quality 3 (Audio levels not consistent, mixing not good) 10 Message clarity/presentation 4 Overall Remarks: Good attempt but fix the right light exposure, right choice of music would be appreciated. 8.4 SAHYOG ECVU Aapna Dharmo S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 4/3 (Beginning shots are good, characters can be directed better to look happy and give right expressions while taking close ups, ending could be more effective) 2 Camera Composition 4 3 Camera Movement 4 4 Picture quality 3 (Light exposure changes at 2-3 places) 5 Sequencing of film / direction 6 Editing (Fade to Black out transition is not smooth, better cut aways needed, jump from exterior to interior shots and back to exterior shots while visiting Gurudwara should be avoided. More stay of shots required at some places while kids are interacting with Dharm gurus) 7 Graphics plates Avoid using Pause, it breaks the flow, fade to black out and black in could be used instead as a transition in between the two sequences. 8 Usage of Music 4 9 Sound Quality 2 (Ambience is too loud at most of the places, ambience is overshadowing the conversation of the kids and one bite of the person in the masjid) 10 Message clarity/presentation 5 Overall Remarks: Great attempt by the kids to bring out the message effectively, more angular shots and variety of cut aways to bring out the production quality is recommended. Juda Juda Kamo S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 3/4 (Opening sequence is too long, starting with night shots is not a very good idea, music is too loud, End shot is good) 2 Camera Composition 2 (Interactive shots of school kids with teacher need more variety, close ups, mid long shots from other angles would make it interesting) 3 Camera Movement 4 (Follow up shots of students e.g. getting into FM office are not smooth, jerky camera movement at many other places) 4 Picture quality 4 (Exposure change disturbs the picture quality) 5 Sequencing of film/direction 4.5 6 Editing 1(Too many jerks in between the shots, dissolves not neat at all, avoid stills in between – breaks the flow) 7 Graphics plates None 8 Usage of Music 3 (Too loud, could be soft at few places) 9 Sound Quality 3 (Ambience is good to use but too loud in most of the places and is disturbing the conversation) 10 Message clarity/presentation 5 Overall Remarks: This film has accommodated lot of professions to bring out the message, good attempt, especially use of original ideas to describe various professions. Innovative style. Put ambience sound under control, variety in shots could boost the film production quality. Khushi S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 4/4 2 Camera Composition 4 3 Camera Movement 4 4 Picture quality 4 5 Sequencing of film / direction 5 6 Editing 3 (More stay in shots, don’t change the previous shot while it’s moving) 7 Graphics plates None 8 Usage of Music 4 9 Sound Quality 4 10 Message clarity/presentation 5 Overall Remarks: Good ideas of presenting a film on a difficult and a sensitive subject. Both the representatives of Hindu-Muslim Dharma could be more emotive and articulate. Vanaspati Etle Shu S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 4/3 (Starting sequence is good, end shot could be better) 2 Camera Composition 4 3 Camera Movement 4 4 Picture quality 4 5 Sequencing of film / direction 4 (Girl, the main character of the film could be more expressive and articulate) 6 Editing 3 (long silent shots, more cut aways with good music could enhance the production quality) 7 Graphics plates 2 (Pause activity 1, 2,3 is not needed, especially after pause activity 1, there are 5-6 still shots which are not required, they break the flow, stay of plates is also too long) 8 Usage of Music 4 (Music is good wherever used) 9 Sound Quality 3 (Mixing could be better) 10 Message clarity/presentation 5 Overall Remarks: Gfx plate with pause written on it breaks the monotony, use fade ins and fade outs instead. Do not use still and silent shots. Bhasha Gyan S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 4/4 2 Camera Composition 3 (Head room and looking room needs to be corrected, though good close ups, variety of shots and angles are enhancing the production quality) 3 Camera Movement 4 4 Picture quality 3 (Shots inside the room are not as sharp as taken outside, use proper exposure) 5 Sequencing of film/direction 4 (well directed, characters are lively and make the learning easy and interesting) 6 Editing 4 (Fast cuts, no stay, location jumps, interior to suddenly exterior shot, can use dissolves if required, Long shots then jump to extreme Close ups is avoidable, footage looks jerky) 7 Graphics plates None 8 Usage of Music 4 9 Sound Quality 4 10 Message clarity/presentation 5 Overall Remarks: Film manages to leaves an impact though right exposure in the interior and exterior shots would enhance the film quality. Karo Ramakda Kuch Kadam S. No. PARAMETERS RATING (on a scale of 1-5) 1 Beginning /Closing shots 5/4 2 Camera Composition 5 3 Camera Movement 5 4 Picture quality 5 5 Sequencing of film / direction 5 6 Editing 4 7 Graphics plates None 8 Usage of Music 4 9 Sound Quality 5 10 Message clarity/presentation 5 Overall Remarks: This film is very good in terms of thoughts, idea, presentation, and good cut aways keeps you engaged. 9 VIDEO FILMS LEARNING OUTCOMES TESTS As part of CMS’s evaluation of Videoshala they carried out learning outcomes tests with groups of 10-12 students in each of the selected Videoshala (experimental) and non-Videoshala (control) schools. The objective behind this learning outcomes test was to make an objective assessment of the extent to which the videos had been helpful in facilitating and enhancing the mastery of hard spots. The non-Videoshala schools served as a comparative (control) sample not subjected to videos. Videoshala and non-Videoshala schools from the same vicinity and students of same grades were selected to ensure matched groups, with some exceptions where the cooperation of the desired schools was not available. Schools from a different location were selected in such cases. An important aspect, which needs to be kept in mind while gauging the students’ performance, is that these videos were shown at different point of time of the project cycle. This might have had an impact on the recall of the students. However, on a particular issue, the screening of videos in all the schools took place, more or less within the same time span (within a month or so). For example, if a video on Urjaa (Energy) was screened, then screening in all the schools under that particular ECVU was done within a month. As informed by the project functionaries, in each Videoshala school, on an average, four to five videos were screened. For the learning outcomes test, the CMS evaluation team selected two issues on which Baseline data (pre-test) was available. In each of the selected schools, test was conducted on these two identified issues. The students were given between forty-five minutes to an hour for writing the answers. However, the evaluation team ensured that test was held in a relaxed atmosphere and in case a student needed more time to write, it was given to her/him. As reported in the methodology section, the number of schools selected for evaluation was in proportion to the total number of schools covered by each ECVU under the Videoshala project. The number of schools and students per ECVU are reported in Table 9.1. Table 9.1: Learning Outcomes Tests Information As can be seen from the table above, on an average, from each school around 10-12 students were randomly selected. In a few schools, during the tests (in each school two issues were covered), one or two students dropped out after the first test and did not appear in the second test. Learning Outcomes Test Results The results of the learning outcomes test are discussed below for each ECVU. It is pertinent to mention that during the pre-production round of tests conducted by ECVUs (to identify the hard spots), the answers were categorized into correct or incorrect, although the answers for many questions were more subjective than objective in nature. In order to maintain uniformity with the baseline test, the same method was applied in the present evaluation round, which to some extent, the CMS evaluation team was not comfortable with. Further, the CMS evaluation team procured ‘model answers’ from each ECVU for questions of the test paper to know the expected answers. The external evaluators examined the answers with reference to the model answers. In the following sections, findings on the videos selected for post-test have been presented under the respective ECVUs. ECVU Number of schools Number of Students, who appeared in the Learning Outcomes test (post screening) Under Videoshala Project Sample schools Hind Swaraj Mandal (HSM) 30 6 74 Navsarjan 43 9 108 Sahyog 33 7 76 Meghdhanush-Udaan 75 15 151 9.1 HIND SWARAJ MANDAL (HSM) ECVU VIDEO FILMS In the schools under ECVU-HSM, two issues selected were - Urjaa (Energy) and Apna Hako (Our Rights). While Urjaa was screened in the schools during December 2007, Apna Hako was screened during June 2008. The videos had been shown to the students of grade VIII-XI. Urja The analysis of scores on Urja (Table 9.2) clearly shows that the level of understanding of the students of Videoshala schools on the topic has improved considerably, after watching the video on the ‘hard spot’. In general the percentage of students giving right answers to various questions in the post-test ranges between 50 and 93, which is outstanding. The percentage of students giving right answers is approximately double that of non-Videoshala schools in seven out of eleven questions, which goes towards establishing the effectiveness of the video intervention. Improvement in percentage of students giving right answers is even better when performance of pre-test is compared with post-test of the Videoshala group. For example, on some of the questions, the percentage of students responding correctly has gone up multiple times, i.e. from as low as 5-10 percent to as high as about 76 percent. This further substantiates the validity of the video on Urja. Table 9.2: Percentage of Students giving right answers (in %) for Urja Questions Videoshala (N=74) Non Videoshala (N=66) Baseline (N=40) 1. What do you understand by energy? 50 27 20 2. What are the different types of energy? 93 47 50 3. What do you understand by renewable and non￾renewable sources of energy? 76 76 10 4. Give examples of both types of sources of energy? 69 68 20 5. What do you understand by solar energy? 59 59 20 6. How is wind energy used? 76 45 5 7. In what form does wind energy come? 86 44 50 8. What do you understand by biogas? 55 24 13 9 What are the advantages of using biogas? 58 26 12 10. From where do we get energy? 69 29 5 11. How do we use energy in our daily life? 70 2 10 Apna Hako The results of Apna Hako (Table 9.3), too, indicate that the project was to a great extent able to meet one of its key objectives- to improve the level of knowledge of the students on the hard spots. Within the Videoshala post-test tallies, there is a big range (93-26 percent), across various questions. This shows that the video contents have been effective in case of some aspects of the topic covered, while not so for the other aspects such as question nos. 4 & 5. For 6 out of 8 questions, the right answers were either close to 50 or more than 50. In contrast, the baseline tallies are invariably low, and markedly so, there is a sharp contrast between performance on post-test and pre-test (baseline). Table 9.3: Percentage of Students giving right answers (in %) for Apna Hako Questions Videoshala (N=74) Non Videoshala (N=65) Baseline (N=40) 1. What do you understand by democracy? 73 31 25 2. What are the different rights given under democracy? 93 72 10 3. What do you understand by the right to equality? 72 35 10 4. What do you understand by right against governance? 39 8 5 5. What are the different types of freedoms we have been given? 26 20 5 6. What are the different duties of a citizen of India? 49 18 10 7. What are the human rights? 69 43 0 8. What has been included in the right to equality? 47 14 10 Also the Videoshala group, as compared to non-Videoshala group, showed markedly high performance on the test, going by the percentages of students who answered correctly. The non-Videoshala group shows extremely poor performance even in absolute terms, which means, performance on none of the questions is even touching 50 percent mark on right answers. 9.2 MEGHDHANUSH ECVU VIDEO FILMS In the schools under ECVU-Meghdhanush, the two issues selected were - Earth and Respiratory System. While Earth was screened in the schools during March 2008, Respiratory System was screened in December 2008. The test was administered to students of grade III and IV. Earth The analysis of the learning outcomes test on Earth (Table 9.4) shows that while both experimental (Videoshala) and control (non-Videoshala) students were able to answer the question on the shape of earth, they were not able to answer questions like, how the earth came into existence or how do day and night occur? Comparison with the Baseline could not be done because the data were not available. However, both the Videoshala the non-Videoshala data indicate that students need more clarity on 2 out of 3 questions the topic of earth. The findings on this, however, have a limited value as the test is too short in length, its reliability being affected for this reason. Table 9.4 Percentage of Students giving right answers (in %) for Earth Questions Videoshala (N=150) Non Videoshala (N=61) Baseline (N=) 1. What is the shape of earth? 75 69 2. How did the earth come into existence? 9 0 Not Available 3. How does day and night occur? 31 0 Respiratory System On the respiratory system (Table 9.5), which was one of the recently shown videos under the Videoshala project, in the post-test, performance was a mixed one across various questions. While, 76 percent students could answer the question-Is breath taken by nose; only 13 and 17 percent of the students could answer the questions like, ‘is the windpipe attached with the tongue’ and ‘where does the breathed air go’, respectively. Comparison with the Baseline data shows highly significant improvement in the performance of the students in post-test; though the rise in the level of performance varied from question to question. When drawing a comparison between Videoshala and non-Videoshala groups one fails to observe a significant trend of improvement as only three questions (4,5,8) show noteworthy higher performance. However, strikingly low performance in pre-testing provides a reason to doubt the validity of the test. Table 9.5: Percentage of Students giving right answers (in %) for Respiratory System Questions Videoshala (N=151) Non Videoshala (N=60) Baseline (N=175) 1. Which part of the body circulates the blood in the body? 53 47 14 2. Which part of the body keeps the blood clean in the body? 60 55 23 3. Is breath taken by nose? 76 75 55 4. Where does the breathed in air go? 17 0 3 5. What is the exhaled out air from the body called? 61 38 3 6. Which part of the body is attached to the windpipe? 13 32 0 7. Does the lungs clean the breath in air? 38 42 2 8. Does the windpipe, like a pump, circulate the blood in the body? 58 33 0 9.3 NAVSARJAN ECVU VIDEO FILMS In the schools under ECVU-Navsarjan, two issues selected were - Vanaspati (Vegetation) and Sthanik Swaraj Ki Sansthano (Institutions of Local Self governance). While Vanaspati was screened in the schools during December 2007, Sthanik Swaraj Ki Sansthano was screened in the month of September 2008. The screening was done for the students of primary and upper primary grades. Vanaspati The results of the test on Vanaspati (Table 9.6) show a mixed response. While on some questions in the post-test, the students of Videoshala performed better as compared to the pre-test, on other questions, the performance was not encouraging. However, performance of students of Videoshala schools was found to be markedly and consistently better than those of non-Videoshala schools. The percentage of Videoshala students answering correctly ranged between 64 and 4, with only two questions scoring below 10 percent. The performance of students from non-Videoshala schools, as such was very poor with most of the questions scoring zero to five percent. Only one question obtained right response from 45 percent of students. Hence, even though there were lots of inconsistencies of trends in comparing Videoshala students on pre-test and post-test they were found to be certainly better than non-Videoshala students. Table 9.6: Percentage of Students Giving Right Answers (in %) for Vanaspati Questions Videoshala (N=105) Non Videoshala (N=60) Baseline (N=121) 1. Draw a forest? 26 5 40 2. What do you understand by vegetation (Vanaspati)? 9 0 26 3. What are the different Types of vegetation? 60 7 48 4. What is bushes? 35 0 10 5. What is grass? 4 0 12 6. Can you identify a shrub? 64 40 10 7 What is a plant (chod)? 20 0 29 8. How can you identify a plant? 18 0 29 9. Can jasud be called a shrub? 38 5 47 Sthanik Swaraj Ni Sansthano Analysis of responses of the learning outcomes test (Table 9.7) shows that out that Videoshala students’ performance on Sthanik Swaraj Ki Sansthano (local self governance) was far more superior than non￾Videoshala group. Compared to the pre-test or Baseline data, the performance of the students in this post-test round was not found to be significantly different. Videoshala students’ responses, were found to be significantly better only in four out of eleven questions during post-test as compared to the pre-test. Performance of non-Videoshala students in comparison to the other two groups was alarmingly low on a majority of questions, which needs to be investigated. The non-Videoshala group and the baseline group, may at worst, be at par on performance which is not the case. As mentioned earlier also, there is a reason to doubt the method adopted to examine the baseline responses. Table 9.7: Percentage of Students Giving Right Answers (in %) for Sthanik Swaraj Ni Sansthano Questions Videoshala (N=108) Non Videoshala (N=58) Baseline (N=121) 1. Define “Sthanik Swaraj”? 3 0 42 2. What is Sthanik Swaraj’s Organization? 0 0 50 3. Which organizations are “Sthanik Swaraj Organizations”? 75 10 38 4. What kind of work does the “Gram Panchayat” do? 75 17 59 5. What are the sources of income of a “Gram Panchayat”? 50 17 52 6. Who manages the Accounts of a “Gram Panchayat”? 63 19 67 7. What is the term of a “Sthanik Swaraj Organization”? 62 33 44 8. At what age, a person can cast vote? 71 55 55 9. Who can become a Gram Panchayat Member? 37 0 43 10. Match the following 50 31 50 11. True & False 75 51 45 9.4 SAHYOG ECVU In the schools under ECVU-Sahyog, two issues selected were - Vanaspati Etle Shu (What is Vegetation?) and Juda Juda Kamo (Different types of occupations). While Vanaspati Etle Shu was screened in December 2007, Juda Juda Kamo was screened during February 2008. The screening was done for students of grade III-V. Vanaspati Etle Shu The analysis of test results of Vanaspati Etle Shu (Table 9.8) indicates that performance of students improved in the post –screening test compared to the baseline results. A very high percentage of students answered most of the questions correctly (ranging between 92 and 80), as compared to Baseline scores. Non-Videoshala schools were exceptionally poor on two questions (3 & 4), but were nearly at par with Videoshala students on four questions out of seven not lending thereby a sharp difference between the two groups. Here, too, the findings of impact on video are not very conclusive. Table 9.8: Percentage of Students giving right answers (in %) for Vanaspati Etle Shu Questions Videoshala (N=76) Non Videoshala (N=60) Baseline (N=78) 1. Have you heard the word “Vanaspati”? 92 95 63 2. What is “Vanaspati”? 80 28 58 3. What does a Gardner do? 64 3 10 4. What is a seed? 38 0 58 5. Have you seen climbers, bushes and shrubs? 89 95 18 6. Is bitter gourd a climber plant? 89 73 58 7. Does rose grow in bushes or plant? 91 73 38 Juda Juda Kamo The analysis of learning outcomes test on Juda Juda Kamo (Table 9.9) showed that the students were able to mention the names of only two occupations though the number of occupations mentioned in the video was eight. At the time of baseline test too, a high percentage was able to mention different types of occupations. Non-Videoshala percentages on both the questions given for the test were, however, markedly low (13 and 0) indicating strong differences between the experimental and control groups. However, the ECVU team was not able to clarify on the criteria followed (number of occupations) for marking an answer as ‘correct’. Also, it would be pertinent to understand the reason(s) for selecting such issues for hard spots, on which the pre-screening research results have shown a high percentage of students answering correctly (as per baseline norms). Table 9.9: Percentage of Students giving right answers (in %) for Juda Juda Kamo Questions Videoshala (N=76) Non Videoshala (N=61) Baseline (N=60) 1. What are the different types of occupations/vocations? 96 13 85 2. (To be asked only if child says yes about vocations) Can the work that your mother does be called an occupation/vocation? 55 0 88 The table below (Table 9.10) provides an overview of the average number of students (from Videoshala and non Videoshala schools along with the students who took the baseline tests) providing rights answers to the questions for each video film Table 9.10: Total Percentage of Students Giving Right Answers ECVU Name of Video Number of Questions Percentage of Videoshala Students Giving Right Answers Number of Videoshala Students Answ Questions Percentage of Non Video￾shala Students Giving Right Answers Number of Non Video￾shala Students Answering Questions Percentage of Baseline Students Giving Right Answer Number of Baseline Students Answering Questions Hind Swaraj Mandal (HSM) Urja 11 69% 74 40% 66 20% 40 Apna Hako 8 58% 74 30% 65 10% 40 Meghdhanush Earth 3 38% 150 23% 61 N/A N/A Respiratory System 8 47% 151 40% 60 13% 175 Navsarjan Vanaspati 9 30% 105 6% 60 27% 121 Sthanik Swaraj Ni Sansthano 11 48% 108 21% 58 49% 121 Sahyog Vanaspati Etle Shu 7 77% 76 52% 60 43% 78 Juda Juda Kamo 2 75% 71 7% 61 86% 60 9.5 SUMMING UP Overall, the learning outcomes tests of each ECVU did show that the level of understanding, as projected by performance on the post-test as well as in comparison to a matched control group, amongst the Videoshala students has improved. However, the extent of recall of the content of the video was not found to be very satisfactory on some questions. This could be due to four possible reasons-one, the time gap between screening of the videos and conducting the post-test; two, in spite of the clarity on the issues, students were not able to express it through words in the learning outcomes test; three, the post￾screening interactive sessions were not able to clear the doubts in the mind of the students and four, some of the videos and learning outcomes tests need to be reviewed for their reliability and validity of content. There are also a variety of factors which need to be controlled in order to arrive at some conclusive findings based on an intervention program like the one in question. Similar conditions may not have been available across various videos, schools, students etc. However, based on interaction and discussion with students and teachers, it may be concluded that they found the videos very informative, interesting and effective as a source of additional help in teaching-learning process. The pre-screening briefing or post￾screening discussion modalities may need to be looked into, in order to make Videoshala more beneficial. 10 ROLE OF LOCAL COMMUNITY IN VIDEOSHALA This chapter provides the result of CMS’s evaluation focusing on the participation of local community members in Videoshala. 10.1 INVOLVEMENT OF COMMUNITY VIDEO PRODUCERS The CVPs played a vital and key role in Videoshala project as they were the persons who not only produced the videos but also identified hard spots of the subjects taught in school curriculum; prepared scripts of the videos to be produced on the identified themes of the hard spots imbibed with values; identified actors and place of shooting; conducted the shooting; and edited the videos. Hence, their involvement as local community members in the project is very important. Mode of selection and profiles of Community Video Producers (CVPs) From each of the four ECVUs, five CVPs were interviewed by the CMS evaluation team. The CVPs were asked about the process of their selection, their motivation behind working as a CVP, the training they received under the project, their role and responsibility as a CVP, the process of content development of video kit and the process of producing videos. The interaction with the CVPs revealed that all the four ECVUs had followed a uniform procedure and criteria for the selection of the CVPs. The process consisted of the following steps: • Each ECVU invited applications from the community members for joining the project as CVP. • Initially, each of the four ECVUs short-listed twelve persons for the responsibility of the CVP. • Finally, six of the twelve were selected for the job of CVP. As informed by head of each ECVU, the selection of the CVPs was primarily done by taking into consideration the extent of the interest the persons showed to work as a CVP, their educational background and experience, their interest in the field of education and experience of working in the community. In addition to this, while selecting the CVPs, all the ECVUs except Navsarjan ensured that the team of CVPs should have representation of team members belonging to different religions and if possible, castes. Navsarjan works with the Dalit community and hence had the CVP team comprising members from Dalit community only. In the case of the other three ECVUs they work in areas with a mixed population. As one of the program partners said, “The idea behind having a CVP team with representation from different communities, particularly Hindus and Muslims was to present an example of communal harmony and to build a platform to bridge the gap between communities and understand each other better”. In addition to this, all the four ECVUs ensured that the CVP team had both male and female members. The brief profile of the interviewed CVPs presented is given in Table 10.1. Table 10.1: Profile of Community Video Producers Name of the ECVU Gender of the CVP (M-Male, F￾Female) Age (approx) of the CVP Educational Qualification of the CVP Social Group Meghdhanush￾Udaan M 42 B.A. ST F 42 B.A. General M 23 B.A OBC M 19 10th Muslim F 25 B.Com General HSM F 27 M.A. B.Ed General F 23 12th, PTC OBC M 26 B.A. General M 22 12th Muslim M 24 M.A. General Navsarjan M 31 M.A., B.Ed. SC F 19 11th SC M 21 B.A. SC M 35 M.A. SC M 35 M.A., B.Ed SC Sahyog M 19 12th Trained in Performing Arts Muslim F 22 B.Com Muslim F 28 12th Muslim F 40 12th OBC M 25 B.A. General The age profile of the CVPs suggests that a majority of them were youth. The average age was around 27 years, with the youngest being 19 years and the oldest 42 years. This, as shared by the program partners, was a result of wanting to have a mix of youth and experience for better understanding and transfer of knowledge in the script writing and video production process. The academic qualifications of the CVPs indicate that majority (13/20) of them were graduate or above. A few (3/12) held Bachelors in Education degree. As mentioned earlier, there was an effort to involve people interested in education process due to the fact that the videos were to serve as teaching-learning material for both teachers, and students. On the motivational front, the interaction with the CVPs revealed that most of them came to know about Videoshala and requirement of CVPs in the Videoshala through the volunteers of the NGOs (responsible for their respective ECVU) or through their friends. Some of the CVPs were already associated with the respective NGOs and showed their keen interest to work as a CVP. On being enquired about what motivated them to work as a CVP, one of them said, the innovative concept and the urge to learn a new skill and to contribute in the field of education was the motivating factor. For others, too, Videoshala’s concern with education of the children and opportunity for the CVPs to learn and produce videos was the most attractive part. Technologically speaking, none of these who joined as CVP had any knowledge and/or experience of technology (video-making). Some of them had handled a still camera but none of them had ever handled a video camera. However, some of the CVPs of ECVUs managed by Navsarjan, Udaan and HSM had experience in the field of education (had taught in schools and had undergone training in teaching). They could relate with the kinds of difficulties one faces in teaching some of the topics to the students. The CVPs of Sahyog were comparatively young and had no experience in the field of education. Training of CVPs: Modality and Adequacy After the selection of CVPs was completed, all of them were given an intensive and extensive training on identifying the hard spots, script writing, video production/film making. The training session was spread over a period of four to six weeks. However, since the female CVPs had problem in staying for so long at a stretch, more than one training (two to three) sessions of shorter duration were utilized. A seven-day residential training (from October 8-14, 2007) was held at Dalit Shakti Kendra Ahmedabad. The other trainings were held at each ECVU and were given by the trainer selected by the Central Coordination Committee (CCU). Along with the trainer, the Resource team from the CCU, too, interacted with the CVPs. They even visited a few schools along with the CVPs to facilitate the process of identification of hard spots on which a video was to be made. As shared by CVPs, the interaction with Resource Persons was very enriching as it helped them in understanding various aspects of video production. As one CVP from Sahyog said, “No specific task was assigned to any particular member. Instead, each member had to do all kinds of activities depending on the needs, which ranged from selecting the location and actors, picking and dropping the actors at their residences, to technical aspects like operating a video camera, and handle specifics related to light and sound. We even learned how to do editing of rough cuts on computer.” This know-how not only removed the initial hesitation and doubt in their minds about whether they will be able to perform their responsibilities as a CVP effectively but also gave them a sense of self-confidence. Such views were also expressed by CVPs of other ECVUs. As one CVP from Navsarjan shared that being part of the Videoshala project did help in fulfilling his dream of becoming a teacher. He added that ordinarily as a teacher, he would have used books to teach the subject, but fallowing his association with this project, he can now help even other teachers to use videos developed by him in order to teach the issues. To comment on the adequacy of training of CVPs, the dimensions of video-production covered during the training, the duration of four to six weeks and modalities adopted go towards making the venture a useful and successful one. Role of CVPs in Video Content Development The interaction with the CVPs showed that all the four ECVUs followed almost similar methods for identifying hard spots to produce videos on. All the four ECVUs did a rigorous exercise in identifying the hard spots. In the process of identification of hard spots the ECVUs organized a one-day workshop with teachers. Subsequently, the CVPs made school visits and interacted with teachers and students to identify portions/topics in the curriculum that students and teachers have difficulty in transacting. In the process, they arrived at more than one subject and within a subject more than one topic that both students and teachers found difficult. It was a difficult task for the CVPs to select one subject out of a number of subjects/themes mentioned by the students and teachers as hard spots. Sharing their experiences, the CVPs mentioned that teachers in general and students in particular mentioned English as difficult to understand. But it was difficult for the CVPs to make videos on the subject as the understanding of the CVPs themselves on the subject was not up to the mark and they were not comfortable dealing with the subject. Hence, among the difficult subjects or themes identified in the final selection videos were made on topics on which the CVPs were confident of their understanding and knowledge. The other aspects like availability of resources with the ECVUs for making the videos on the subject were also taken into consideration in final selection of the subject/themes. The CVPs shared their experience of going through the process of identifying the hard spots, which was enriching for them in many respects. This exercise made them aware of the difficulties of the students and teachers in teaching- learning process in general. Though most of the schools they visited for the purpose, welcomed the idea, some like one in Ahmedabad, as shared by a CVP from Sahyog, did not allow the team to interact with students and teachers, as the school authorities felt that it will disturb the classes. To know further about the extent of knowledge students have on the identified hard spots, the next step aimed at assessing the existing knowledge of the children on these topics. However, the method of assessment varied from topic to topic and ECVU to ECVU. For some of the topics/themes (identified hard spots) the ECVUs conducted group discussions with the students, while for some other topics, formal testing (pre-test) was done. Once the hard-spots had been identified, the CVP team held discussions and short-listed the topics for video-production. On the topics thus selected, the CVPs consulted textbooks, and organized workshops with subject experts to have better understanding. As reported by the CVPs, all these exercises improved their knowledge and understanding of the subjects and the topics selected for the videos, and they felt more confident towards video-production on these topics. For example, a CVP of Sahyog felt that the exercises he had to undergo during the process had brought about a sense of perfection and meticulousness in him. Video Production After finalization of the hard spots the next step was to prepare scripts for video production. To begin with each of the CVPs prepared a script on a given issue, which was taken up for discussion within the team. Various scripts on one issue were brought together and collated to generate a new script which would satisfy the criteria and requirements. Then the script was discussed with the CCU. In this process, as revealed by the CVPs, sometimes altogether a new script had to be prepared. According to the CVPs script writing was one of the most difficult tasks for them and as reported by them, while writing a script one has to keep in mind many things: it should be close to real life, should be locally contextualized, should have characters with which audience could relate or identify, clearly brings out the issue which needs to be highlighted. It was further added that to keep the audience glued to the video, stories should be interesting and have climax sequence as well. Once the scripts were finalized, the CVPs identified the location for the shooting and selected the actors from the community. The shoot plan was, then, prepared. While selecting the location, many aspects regarding the location had to be taken into consideration including the location should not be very far off. In order to provide an easy accessibility, the location should be relevant to the issue under consideration. An interesting incident shared by one of the CVPs (from Navsarjan) throws light on the kinds of problems that emerged during their work. The incident goes as follows: during shooting of one of the videos on local self-governance, the Panchayat members raised objection against the story sequence which showed a Dalit village head not being allowed to work freely and being dominated by a member of another caste (Patel). The objection of the Panchayat member was that since these kinds of things are not being practiced in their village, the village and their Panchayat building should not be used for the purpose. The CVP team had to clarify that they are just shooting a film which has no reference to their village. The shooting had to be stopped for a day and could start only after the CVP team pacified the Panchayat members. Identification and training of the actors was another important step of video production. It was a big challenge for the CVPs. In the Videoshala project, ‘actors’ who had never faced video camera before i.e., those who were not professional actors were identified. The other difficulty was felt that the girls were reluctant to take any roles for the videos. In some cases parents did not allow their child to take a role for the video. It may also be mentioned that not only the children were actors in the videos but adults were also required to take acting role. The adult actors were also selected from the community. So identification and selection of adult males and females; boys and girls who can take up acting roles for the videos and can give good performance as an actor, was a challenging task before the CVPs. However, as reported by the CVPs, identification and selection of the actors for the videos never came before them as a big problem. Those who were selected for acting roles were given three to five days training (depending upon their requirement of training and nature of the video). The training broadly covered the aspects like dialogue delivery, expression, emotion and timing. The selected actors were trained by the CVPs. As the CVPs reported, those who were selected for acting roles in the videos did not disappoint but performed to their satisfaction. After selection and training of the actors and selection of the location, the CVPs shot the videos and prepared rough cuts of the videos. Workshops of experts from Udaan, Drishti Media, and CVPs of other ECVUs were organized to provide feedback on the rough cuts. The rough cuts were critically reviewed and discussed and changes were made in the video before finalizing it. Some Interesting Experiences Shared Some interesting cases of parents/students who acted in the videos and whom the field researchers interviewed are presented below: • Sharing her experience, Ms. Rashida Hamid Sheikh mother of a student named Sonu Hamid Sheikh, mentioned that working in the video ‘Paani’ was a memorable experience for her. Though, in the video, she acted in a short still, it left an impact on her. She mentioned that working in the video brought confidence in her that she can do anything. It also brought about self-respect in her. She wished that if she had opportunity to work in other videos, she would definitely work. Regarding her son, Sonu, who had also acted in the video, she mentioned that her son also got motivated to go to school regularly as a result of working in the video. According to her, it was a matter of pride for her to see her son as an actor in the video. • Mr. Narendra Koteria - the father of a student of Hindswaraj Mandal Videoshala School, had acted in a video of Hindswaraj Mandal. According to him, working as an actor for educational video was a very good and exciting experience for him. He also mentioned that he enjoyed the company of the team members of the ECVU involved in video production. According to him, his participation in the video brought in him confidence and a sense of worth. His association as an actor in the video enabled him to come out of narrow thinking of caste, creed and gender. He feels that Videoshala has given him much more than he contributed to Videoshala. • Ms. Ramila Bharat Parmar acted in the video ‘Sthanik Swaraj Ni Sansthano’ of Navsarjan ECVU. She mentioned that though her role in the video was a very brief one, this was a great opportunity that made her a changed person. According to her, she came in contact with the team of CVPs (both males and females) and interaction with them brought confidence in her that a female can do what males do, besides her duties of household chores. Earlier she was fully dependent on the male members of the family for outdoor works. Now as a result of working in the video, she has become so confident that when she or her children fall ill she does not feel the need to take help of male family members to go to doctor/hospital. She also started visiting school to get teachers’ feedback on the child’s studies. She gives all the credit of this change in her to Videoshala. She wished that she got another chance to work in the video. 10.2 IMPACT ON COMMUNITY VIDEO PRODUCERS This section discusses gains and outcomes observed and reported by CVPs as they emerged during planning, development and execution of the video-content development. Technical Know-How and Skills Among CVPs According to the CVPs, every exercise and activity in the process of video-production was a new learning for them and added to their technical knowledge and skills, which got enhanced as the video - production progressed. By the time the CVPs had produced 6 videos to cover various topics of interest, they felt confident in (i) video-production technology, and (ii) their ability to understand the issues involved in our educational system. Sharing his views on technological gains as a result of association with Videoshala Mr. Mahesh, CVP, Navsarjan mentioned that now he is very confident of his technical-know how in video production. He not only learned the art of video production but also learned using Apple Computer. He feels that his skills in video production can be now adopted as a profession and a source of earning. Sharing her technical gains as a result of association as a CVP in Videoshala Ms. Jyoti Bhalabar mentioned that before her association with Videoshala she could not even hold a video camera but now she can make a video even. It’s a great achievement for her. She had never imagined that she could also become a skilled person. The other CVPs also had similar views about the technical gains as a result of their association with Videoshala. They were all confident that technically they had reached the level that they by themselves can produce a video. Community support available to CVPs The CVPs faced no resistance from their families or society for joining the project as a CVP, with a few exceptions in the initial stages of the implementation of the project. A case in question was that of a female CVP from a Muslim community who faced resistance from her community when she initially joined the Videoshala. They did not like her associating with male members, and that too, was from the Hindu community. However, she showed patience and, ultimately, her good work was recognized, and encouraged by her community. Another case of this kind was reported by that of a female CVP of Udaan￾Meghdhanush. Her family did not approve of her remaining outdoors for six to seven hours everyday. Such problems, however, maybe encountered in any other job. CVPs Day-to-Day Gains in Life Most of the CVPs reported that their association with the Videoshala has been gainful in day-to-day life. Working in a team made them learn good values from each other. It equipped them with problem - solving techniques and facilitated their problem - solving efforts in their personal-social life. Ms. Acharya Dipti of HSM, sharing her views mentioned that working as CVP has made her a disciplined person. Her association with Videoshala has brought a sense of responsibility and commitment to her responsibilities. She was of the view that she is now a changed person in attitude towards work and responsibility. Better Connecting with the Community For many of the CVPs, the association with Videoshala has made them think beyond their self-interests and contribute to the society. It has provided them a platform to come closer to community and understand community problems. It has been instrumental in bridging gaps in different castes and communities. For example one CVP from Sahyog ECVU reported that she had been nurturing wrong notions about the Muslim community and avoided going into Muslim localities. After joining the Videoshala and interacting with fellow CVPs and other community members during content development, she became aware of her misconception. Ms. Payal shared her experience that during production of the video ‘Aapna Hako’, her interaction with persons of different castes and creeds and the response she got from them made her realize that all communities are equally good. Now she has not only respect for all but she also interact sand exchanges views on issues of the community/village with all sections of society whether Dalits or others. Similarly, sharing his views, Mr. Kirti Kapadiya mentioned that working as a CVP has made him popular with the community/village. His interaction with the community has become frequent and his relationship with them has become closer. Now he is aware of the problems and issues of the community. He further mentioned that as a result of what he learned as a CVP, now he discusses the problems/issues of the community. Inculcation of a Sense of Pride It came to light during the interaction with the CVPs that not only the CVPs but the entire team of Videoshala got a sense of pride out of their association with the project. They thought that they have been instrumental in providing better education to children. The community members and the teachers on the receiving end have taken it very positively, too. The Program Directors of the ECVUs were of the opinion that the CVPs had gained reputation as persons committed to the betterment of school education. As many of the CVPs mentioned that apart from other gains of their association with the Videoshala as a CVP, they have imbibed a sense of satisfaction and pride that they are doing something for the benefit of the children of the community. The appreciation for the videos from teachers, students and villagers brought a sense of pride in them that their efforts are recognized and is liked by all. Effectiveness of the Videos As reported by the CVPs, teachers and students alike enjoyed watching the videos projecting the ‘hard spots‘, which were otherwise difficult to grasp. The teachers appreciated the innovative method utilized in the videos. An exception in this respect was noted as reported by a CVP of Udaan-Meghdhanush. According to the teacher in question watching the video said the video was not topic - specific. However, in general the videos were rated ‘high’ on learner-teacher involvement in the curriculum thus transacted. 10.3 CHALLENGES A variety of challenges were encountered by the CVPs. Across the ECVUs, similar problems and difficulties were reported. These problems related to script writing, procuring and training of actors, commuting to the shoot-site, restricted time frame, bias against media, shooting skills of the CVPs and rigours of shooting. These have been discussed below: • Script writing for the video was a daunting task for the CVPs. They had to undertake rigorous exercises in writing, revising, and refining the scripts. Workshops had been organized to review, discuss, and finalize these scripts. The extent of changes made in the process went up to 50 percent. Scripts of videos titled ‘Prakash’ of Navsarjan, ‘Earth’ of Sahyog were cited as examples. • Once the actors were identified, problems in convincing them and their parents to act in the videos were encountered. Parents were reluctant to send their wards to work for the videos as this would divert them from their studies. However, ultimately the CVPs were successful in convincing the parents. • All the CVPs reported it was a tough task for them to train persons, who had never faced video camera, to act. According to them, initially many of the actors became nervous while facing the camera. The CVPs had to consistently interact and boost their self-confidence to make them act in the video. • In all the ECVUs the CVPs reported that despite their extensive technical training on video production, they faced difficulties at all stages of shooting, particularly so while shooting the first video. The CVP of Sahyog informed that during the production of their first video the trainer was present from beginning to end of video-production. However, after the first video was completed, the CVPs were more confident and the trainer’s help was taken only when shooting for relatively difficult scenes. • Problems were also encountered with schools while taking away the children for the shoot. One such instance occurred during the shooting of ‘Pupu’ by Udaan-Meghdhanush. • Program-director of Sahyog pointed out that working in the urban slum was more challenging. Here, since the youth and families were exposed to media like TV, and to communal riots, the community was wary of video shooting and its purpose. • Operational problems like a restricted time frame, and the need to meet the deadline of 40-50 days had to be faced by video - producers and this made their schedule very hectic. • Lack of logistical support like facilities for commuting to the far off shooting site was experienced by Udaan-Meghdhanush while shooting their video titled ‘Vaahan Vyavaharo’. Introspecting on the video shooting and re-shooting, the CVPs reported that they did not have to face the situation of a total re-shoot of the rough cut of the Videos. However, in all the cases they had to make some changes and some parts of the videos had to re-shot As shared by CVPs as well as the technical resource persons, initially they felt bad when a video produced by the team was asked to be re-shot. They did not like it when they were asked to rework on the script or to re-shoot because the CVPs felt that resource persons were not ‘familiar’ with the ground reality.i.e. limitations and problems they face while shooting a particular shot or video. But later on they realized that it is for their good that the Resource Team is critical of their work. As shared by a representative of Drishti, to make the process participatory, the review of rough-cut was made in the presence of all CVPs. This not only helped them to learn from each other’s experience also enabled them to take the criticism in the right spirit. The CVPs were appreciative of the guidance and support received from the CCU which enabled them to produce the videos. However, in one ECVU some of the CVPs reported that initially the CCU was in constant touch with the ECVU but later their guidance and support was not very frequent. The overall experience of the CVPs was very enriching. They worked in their respective teams in a very amicable environment. They were fully satisfied with the outcome of their effort. The CVPs appreciated the rigours of the process followed in the production of videos and in the relevance of content projected therein. The fact that the topics were selected in consultation with the students and teachers, was impressive, and validated the content chosen. 10.4 VIEWS OF OTHER STAKEHOLDERS The representative of the CCU had a very high opinion about the CVPs. She appreciated the dedication and commitment of the CVPs towards the responsibilities assigned to them in the Videoshala program. The representative of the CCU mentioned that she had not imagined that the persons who were of poor educational background, the persons who had no any technical know-how prior to joining as a CVP would give such wonderful results. She appreciated the competence of the CVPs and their understanding about the project. She cited the example of their understanding about the Videoshala and application of their ingenuity when they used pot wheel in the video ‘Earth’ to describe rotation and revolution of earth. A Representative of Drishti Media was also full of appreciation of the understanding and dedication of the CVPs towards making videos. According to her working with them made Drishti Media team feel proud of the CVPs who, though, did not know anything about the technology of film production, learned everything in a short span of time. All the heads of NGOs (HSM, Sahyog, Navsarjan and Meghdhanush-Udaan) were happy with the entire team of their respective ECVUs. They were of the view that the team also enjoyed working in the Videoshala project. According to them it was new learning experience for them. In these ECVUs almost the whole team from the beginning to completion of six videos remained associated with the Videoshala project, that itself was a proof that the team members enjoyed working with Videoshala. However, the dead of Navsarjan had some dissatisfaction with some of the members particularly the Coordinator of the ECVU. But the dissatisfaction was not on incompetency of the team members but on ideological issues. 11 VIDEO FILMS SCREENINGS AND THEIR IMPACT This chapter enumerates the responses obtained by CMS’s evaluation from classroom facilitators (CF), teachers, students and parents on the Videoshala video screenings and their impacts. 11.1 CLASSROOM FACILITATORS VIEWS Screening of the videos had been executed by classroom facilitators (CFs). The role of the CFs was very important in the sense that they were the main agents through which Videoshala ultimately reached its target audience consisting of children and the teachers. To understand the specific role and responsibilities of the CFs and the impact of Videoshala on its target audience, the CFs from each ECVU were interviewed. Table 11.1 presents the ECVU- wise number of CFs interviewed, their educational background and gender. Table: 11.1 Profiles of Classroom Facilitators ECVU Gender of The CF Age Qualification Social Category Udaan Meghdhanush M 28 12th General Udaan Meghdhanush M 21 B.Com General Udaan Meghdhanush M 20 12th SC Sahyog F 22 10th General Sahyog F 23 B.A. General Sahyog F 20 11th General Navsarjan M 32 10th SC Navsarjan M 34 12th SC Navsarjan M 35 B.A SC HSM M 22 BRS + B.Ed OBC HSM M 22 B.A. General On the basis of discussion with CFs the investigating team came to the conclusion that broadly speaking, the process followed by all the ECVUs was more or less similar. In each Videoshala school, the videos were supposed to be screened for forty children. These forty children were to be selected from the classes for which the videos were produced. The classes chosen by different ECVUs varied. HSM had produced videos for class VIII and IX and Navsarjan had produced videos for classes IV-VII. Udaan and Sahyog had produced videos for class I –IV. Also, there were no fixed criteria for selection of forty children for whom the videos were screened. Each ECVU followed its own method. As the Program Director of Sahyog reported, the children who were comparatively weak in performance were selected for viewing the videos. In Navsarjan the videos were screened in the Bhimshalas. Bhimshala is a centre, set up by Navarsarjan in the villages where volunteers of Bhimshala teach and provide guidance to Dalit children - where only Dalit children viewed the videos. However, where there is no Bhimshala in the villages, Navsarjan screened the videos in schools where children of all social categories participated. The ECVU Hind Swaraj Mandal screened the videos in the residential schools of Gandhian Nai Taleem. Discussion with CFs revealed that for some video screenings, the number of children ranged between fifty and eighty. It also emerged that the role of the CFs in screening of the videos was very important. They had to arrange for a TV set and CD player to be used for the task of screening. Rapport building activities like songs, riddles, and plays were conducted with the children and briefing about the video was also done before screening of the video. Each video had an activity guide, which had information on how the session was to be conducted. The Facilitators followed the guidelines in screening of the video which included giving of pauses in places to hold discussions. Post-screening activities included discussion and filling up of activity sheets by the students. The whole process of screening of video took approximately two to two and a half hours. The interaction with the CFs revealed that the children liked watching videos on the hard spots. Watching the videos made them understand the topic better, and to have conceptual clarity. Some of the videos were liked more than others. For example, the CF of Sahyog reported that ‘Khushi’ was more popular than others as it happened to target social issues. Similarly, a CF from Navsarjan reported that ‘Pradushan’ was liked most as it was very interactive and informative. The video ‘Sthanik Swaraj Ni Sansthano’ screened at Navsarjan was liked because it provided information on functioning of the local self-governance. Gains for CFs Association with the Videoshala was gainful for the CFs in many respects, too. The CFs reported that working as CF provided them opportunity to learn many new things in the field of education. Some of the examples of impact on CFs as reported by them are presented here: • One of the CFs of Navsarjan mentioned that he had heard about the pollution but he was not aware of the impact of degradation in the environment. The CFs reported that their role and responsibility in the Videoshala improved their knowledge on the topics covered in the videos they screened. They also learned values on citizenship, diversity and democracy. As a result they not only practice the values they learned but try to convince others in the community to do away with beliefs regarding discrimination in caste and creed. The videos have made them understand the community, the problems the community faces and the causes of the problems. • Mr. Sanjay Kumar Parmar, describing the learning experience resulting from his work as a CF mentioned that after watching the video Vanaspati he stopped using polythene bags. • Describing the impact of association with Videoshala as a CF, Ms. Deena Bayas of Sahyog mentioned that before joining Videoshala she did not like to interact with members of Muslim community and she had very wrong notions about them. She avoided going to Muslim locality. But according to her, after joining as CF in Videoshala “I am completely changed. Now I have a number of Muslim friends and sit and eat with them”. • Mr. Bharat Bhai one of the Classroom Facilitators of Navsarjan mentioned that all the videos provided him knowledge on the respective topics they tapped. However, according to him, it was ‘Sthanik Swaraj Ni Sansthano’ that made him more aware about the structure and functioning of local self-government. The CFs of all ECVUs did not face any big challenge in executing their responsibilities. However, in ECVUs like Navsarjan and HSM, the CFs had to face the problem of carrying the TV sets and DVD players to remote schools. Power cut was also one of the problems that the CFs faced during video screenings. Impact of Videos on Students and Teachers- As Reported by CFs Classroom Facilitators were the first people who could observe and receive the immediate reactions, responses, queries of the target audience (i.e. students and teachers) on the videos. The CFs were of the view that children and teachers watched the videos very attentively. The videos facilitated the students’ understanding of the topics on which the videos were prepared. Some of the CFs mentioned that the children liked the videos so much that whenever they (the children) met them (the CFs) on the way or in the village they wanted to know if they would be shown more videos in future. According to the CFs, the videos have not only been beneficial in knowledge enhancement in the students but other positive impact has also been observed in the children. After screening of one or two videos, the children shed their shyness in asking questions and became more interactive. According to the CFs the understanding level of the children also increased. After watching the initial one or two videos the children were able to infer and understand the content of the subsequent videos more clearly. The CFs mentioned that the children became more disciplined while watching the videos in comparison to screening of the first videos. The videos made the children more aware, thoughtful and proactive on the issues like pollution, environment etc. It brought among them a sense of fraternity, equality and values and appreciation of the contribution of different people. The CFs reported that the teachers were very appreciative of the videos in terms of its content and the manner of production. According to the CFs the teachers took the videos very positively. They not only proved helpful to them to understand subjects and themes difficult to transact but also helped in their teaching methodology. The CFs reported that teachers shared their experiences that after the video screening the classes became more interactive and the children became more articulate in asking questions to clarify their doubts on the topics being taught. 11.2 STUDENTS VIEWS Content analysis of student data obtained through interactions and interviews gave rise to important findings about (i) effectiveness of video-screening on the student community as far as general impact on interests and attitudes towards various issues taught goes, and (ii) their learning outcomes on specific performance test related to difficult topics which could not be handled well by teachers. Findings thus elicited have been put under relevant heads below: Student groups were enquired about the subjects/ topics they found difficult. Most of the students of Hind Swaraj Mandal (HSM) and Navsarjan who were of upper grade (8-9 and 5-7 respectively) mentioned Mathematics, Science and English, as the difficult subjects. On the same question, majority of the children of ECVUs Sahyog and Udaan-Meghdhanush belonging to grade 3 and 4, mentioned Environmental Science and Mathematics as the difficult ones. On being asked about how they managed to cope up with the difficult subjects, the students gave different responses. However, majority of them shared that they worked hard and took teachers’ help to understand the topics difficult to comprehend. Some of them took help of their parents. While a few candidly informed that they did not make any special effort towards it. Recall of Videos It may be mentioned here that the recall level of the content of videos was higher among the students of HSM and Navsarjan (students of grade IX and grade V-VII respectively),. Even though the students from Sahyog and Udaan-Meghdhanush (grade III & IV) also responded well on different aspects of the videos, such as why they liked the videos, what they learned from the videos etc. The students of ECVU￾Navsarjan commented that all the videos were very informative on the respective topics. They mentioned the videos not only provided information on these difficult issues but they also changed their thinking and behavior. They reported that after watching the Videos on Pradushan, Vanaspati, Paryavaran Anethenu Kudrati Santulan, they have become more serious towards keeping their surroundings clean and green. The video Sthanik Swaraj Ni Sansthano has made them aware about functions of local self￾governance and rights of a citizen. Similarly, for the students of HSM ESVU, all the videos were very informative and knowledge giving. The students recalled the videos on Bhoomi, Urjaa, Apna Hako, Lokshahi, Aahaar Ane Arogyaa and on Nayesargik Sampaada. They informed that they learned about different kind of energy, and its use in our daily life. Through one of the videos on ‘own rights’, they understood the importance of exercising our rights. Some students also recalled the need to conserve our natural resources like land and its upkeep. With reference to one of the videos, they reported having learned the importance of taking balanced diet for keeping oneself healthy and fit. The students under ECVU-HSM were of upper grades (VIII-IX) and hence their recall on the content of videos was better than students covered from other ECVUs. In general, the students were successful in recalling the number of videos they had watched. The average number of videos recalled was 4-5, which compared with the actual number that the ECVU reported. The recall of the names of the videos screened was comparatively poor among the children of lower grade (like in ECVUs Sahyog and Meghdhanush-Udaan). However, the children were able to recall the content of the videos which is, perhaps the most important indicator of the impact the videos had on learning. Almost all of these students of Sahyog recalled watching the videos on issues like Vanaspati, Juda juda kaamo, Apna Dharmo. Some of the students also mentioned Khushi, Karo ramakda kuch kadam, and Bhaasha Gyaan. Extent to Which the Videos Were Liked On being enquired about whether they liked the videos, most of the children said that they liked the videos very much. They felt that the topics covered under the videos were very appropriate. Interaction with the students in the schools of the ECVU Udaan-Meghdhanush revealed that the topics like Earth and Respiratory System which were difficult to understand merely by reading books or as taught by the teacher, were understood well after they watched the videos. They further added that after watching the video on earth, their curiosity regarding the shape of earth was satisfied to a large extent. Similarly, video on respiratory system helped them clear their doubts and confusion. It facilitated their learning of the human respiratory system. They also liked other videos titled Water and Vahan Veyavharo, as they found them very informative. The learning outcomes test (discussed in detail in next section) did show that a majority of the students answered some questions correctly while for some other questions the performance was not up to the mark. When students were asked which topic they would like to watch videos in future, they mentioned the subjects such as English and Biology rather than mentioning topics or sub themes of the subjects of curriculum. Extent of Learning Good learning outcomes were reported by the students. They shared that after watching the video on Vanaspati they could understand the different types of vegetations and their importance. Similarly, the video on Juda Juda Kamo helped them understand that every kind of work is important. They learned that no work is insignificant and all work contributes in the development of the society. While mentioning about the gains from the video on Aapna Dharmo, they said that they could know more about different religions practiced in their community. Some of the children of Sahyog ECVU mentioned that from the video Karo Ramakda Kuch Kadam and Khushi they learned to walk like a soldier and help the people in need, respectively. Though all the six videos were reportedly screened by Sahyog, the recall level was higher for videos on Vanaspati Etle Shu, Juda Juda Kamo and Aapna Dharmo, as compared to the other three. Also the learning outcomes test results of the topics on vegetation, are in consonance with the views expressed by the students. However, in the test conducted on Juda Juda Kamo, the students were not able to recall all the occupations shown in the video. A majority could recall only two of them. Further the local language being the medium of expression in the videos as well as the familiarity with illustrations and examples, made it easy for the students to identify with the contents. Value Outcomes One of the key objectives of the Videoshala project was to help imbibe the feeling of citizenship and democracy amongst the students, which was evident in the students from higher grades. Students in general, expressed a keen interest in acting in future productions of such videos and urged that more hard spots should be identified for this purpose. 11.3 TEACHERS VIEWS Teachers’ Involvement The Videoshala project, besides empowering the local community members, was an approach towards providing a tool which will facilitate teaching in the classroom. The participation of schoolteachers right from the initial phase of the project to the screening and after was, therefore, very important. As informed by the project partners, teachers were involved right from the time of identification of ‘hard spots’. During the in-depth discussions with teachers in one of the Videoshala schools under ECVU Udaan￾Meghdhanush, two teachers reported that they were part of the process of identifying hard spots-‘earth’ and ‘water’. One teacher, each in Navsarjan and Sahyog schools reported that they participated in the discussion held in their respective schools to identify hard spots. It may be mentioned here that in the sampled Videoshala schools only a few of the teachers reported that they were part of the identification of hard spots. As reported by the CVPs and Coordinators, who visited on an average of four to five schools (for every hard spot), a few of the teachers and students were consulted in the process of identifying the hard spots. As reported by the program partners, initially the authorities in the government schools were not very supportive of the idea and did not show much interest. One reason reported was that they had to take permission from the higher authorities before allowing the Videoshala team to visit the schools. In a few private schools too, the school authorities were not very responsive to the idea and felt that this might disturb the academic session of the school. However, after watching the initial videos, they appreciated the concept and showed their willingness to participate in the project more actively. The next phase of teachers’ involvement was during screening of the videos in their respective schools. However, as reported by teachers their involvement during screening was limited. They were basically responsible for collecting the children to whom the video was to be shown and make them sit in a classroom where the video was to be shown. The main role during the screening of the videos in the class was that of the Classroom Facilitators (CFs). However, the teachers did watch the videos and as shared by them the videos could be useful tool in effectively transacting a particular topic to the students. Teachers’ Opinion on Videos Almost 100 percent of the teachers appreciated the concept of Videoshala and the videos thus produced. They were convinced of the effectiveness of the videos in teaching. A majority of the teachers were of the opinion that some issues/topics of the videos produced were of good quality. A good majority of the teachers have rated the content of the videos as very good and relevant; presentation of the videos as attractive; language of the videos as easy to comprehend; and overall quality of the video as very good and attractive. The teachers further added that the modality of teaching through videos is very effective as there is a possibility of using more examples and illustrations that facilitate teaching of a difficult topic. The teachers own understanding of the topics/subject matter got enhanced. For example, one of the teachers reported after watching a video on Vanaspati (vegetation) that it came as a new learning to him that bamboo was a type of grass, and that it is the longest grass. Similarly, using sticks and leaves for explaining respiratory system was an eye-opener for them and they learned that even the complicated issues can be simplified with the help of the things available in the surroundings. In non-Videoshala schools too, the teachers liked the concept of Videoshala and agreed with views of their counterparts in Vidoeshala schools that difficult subjects and issues in the curricula can be better addressed through videos. Though they had not watched any videos of Videoshala, they were of the view that the idea of developing videos on the difficult subjects and issues of the curricula by the community members in local language could be very effective and relevant. Teachers’ Observation on the Impact of Videos on the Students Regarding the change the teachers observed in the students as a result of watching the videos, a majority of the teachers mentioned that the students had more clarity on the issues that the videos tackled. Many teachers also noticed the change in behavior and conduct of the students towards their peers of other community and social groups. The students , they felt have also become conscious towards keeping the surroundings clean and green. More of such changes were noticed amongst the students of schools covered by the ECVUs- HSM and Navsarjan, possibly due to the reason that these students belonged to higher grades as compared to those of other two ECVUs (Meghdhanush-Udaan and Sahyog). In some schools of ECVU- Udaan-Meghdhanush, the teachers informed that after watching the video on ‘water’, the students stopped wasting water. On knowledge gained by students through the videos, the teachers across the four ECVUs reported that the videos had definitely been helpful in improving their knowledge on the issues covered. The learning outcomes tests results (reported herein) too supported their contention. Teachers’ Felt Involvement for Future There was mixed reaction from the teachers on the question whether would they like their involvement in the screening of the videos more. Some of the teachers wanted more involvement such as in the actual screening of the videos which was absent earlier, while others did not want to be given this responsibility. However, for majority of the teachers, the video has been beneficial in understanding hard spots and in learning to use examples and illustrations from the surroundings, making it easier to teach and to learn. The teachers mentioned a number of topics like digestive system, stages of human development, parts and functions of human body and universe etc. for future videos. Furthermore, all the teachers alike felt that the screening of the videos should be integrated with the initial teaching of the topics. According to them, the videos alone cannot be as useful when shown separately. 11.4 PARENTS VIEWS The interviews with the parents (either mother or father) of Videoshala students revealed that most of the parents were aware that videos were being shown to their wards in the school. Some of the parents interviewed had participated as actors in the videos. All the parents were appreciative of the video-inputs being used to facilitate learning of children. According to the parents, when the children came home after watching the videos in the school they shared it with them. Most of the parents also mentioned that the children came back home very happy the day they watched video in the school. Regarding whether the parents observed any change in their ward’s behavior subsequent to watching the videos, most of the parents mentioned that they found many positive changes in their wards. Two of the reported cases are: • Ms. Vinodini, mother of Pankit Trivedi of HSM Videoshala school mentioned that after watching the videos her daughter has become serious in her studies and has become more confident. According to the mother, it was the impact of the videos she watched in the school that motivated her to participate in the school function on the occasion of Republic Day. • Mr. Pankaj Lalji Bhai mentioned that the thinking of his son towards Dalits and other communities has changed. Now his son has friends from Dalit and other communities which was not there before. These are just examples of the positive change the parents mentioned that they observed in their wards after watching the videos in the school. They cited many more such changes. In brief the parents not only liked the idea of videos but most of them requested that videos on other subjects should be made and shown as this method, according to them, has been very beneficial for their children in many respects. 12 FINDINGS & RECOMMEDATIONS The insights in this chapter are based on the analyses carried out by the CMS and Nirantar evaluation teams, along with the responses they received from other stakeholders during the course of the evaluations. The CMS and Nirantar teams along with other respondents deliberated upon important aspects of the project such as relevance to the existing educational and school scenario, feasibility of the modalities adopted, components of the program, limitations of the scope and support, outcomes realized and future strategies. This section discusses the overall success and challenges of Videoshala and its future scalability and sustainability. 12.1 CURRENT STATE OF VIDEOSHALA Reflecting upon the process of the project initiation and implementation, all the project partners were of the unanimous view that Videoshala project has been by and large successful in achieving its intended objectives. They felt that one of the key achievements of the project was that it was implemented in the manner in which it was intended during the project planning. All the ECVUs had the Community Video Producers (CVPs) as well as the Classroom facilitators (CFs) selected from the local communities. Further, as desired, in spite of these CVPs not being experienced hands in video production, they were, as desired, able to not only identify and develop the content but also were successful in producing the videos of good quality within the stipulated time framework. The program partners were therefore, confident that in future efforts too, the idea of involving community members in producing videos could be retained. Regarding the production of the video films the CVPs went through an intensive process of research before each video was finally produced. This involved discussion, trainings, field visits, reading and referring to resource materials that included both watching documentaries and reading books etc. This was enriching for the CVPs, who are able to build an in-depth understanding on the topic before the film was made. Additional workshops on hard spots selection, on the process of script writing and on the development of concept notes were also useful for the producers. The ECVUs teams included women and people from marginalized communities (except for one E-CVU where there is only one person from a minority community). This composition of the team also reflected the diversity that exists within the program, enabling it to bring in different experiences and learning into the film-making process. Further the CCU also played a positive and rigorous role in the film-making process. Inputs, resource material and hand-holding provided to the producers yielded encouraging results. Additionally there is a structured process of feedback that exists between the CCU and the ECVUs. In terms of physical targets achieved, the Videoshala project had envisaged reaching out to about two hundred schools in eight districts of Gujarat. With the exception of the ECVU Navsarjan, the remaining three ECVUs: Udaan-Meghdhanush, HSM and Sahyog, were able to achieve their respective targets. Navsarjan could cover only forty-three of the stipulated seventy-five schools. The reason, as per the PDs version was attributed to the coordinator appointed by the ECVU who left in the middle of the project. Overall at the end of the two years of existence Videoshala had reached 7,957 children in 194 schools through 640 classroom video films screenings. At the same time the project had provided alternate livelihood opportunities to 24 people as CVPs and to 21 people as CFs, with all hailing from local communities. Video Films The following issues relate to the video films: • The videos produced by all four ECVUs are child centric. All videos have children as the main protagonists. Their questions, inquisitiveness and anxieties are addressed in all films. Yet they are not shown as being ignorant or uninformed, but in many films are confident and articulate. It is either a child or a group of children who take the viewers through a film and learn new information and ideas along the way. This learner-centered strategy ensures that children – the main target audience of the videos – are involved and engaged as viewers and can relate to the content easily. • In most videos children are encouraged to learn by gathering information, going to different places, conducting interviews and doing things themselves. Seeking information is an effective way of learning and this is a method that most videos have adopted. This is in contrast to the conventional top-down approach of the teacher as the only provider of all answers to children’s questions. • The videos produced by all four ECVUs are unique in that they provide information and address values of citizenship, diversity and democracy. This is unlike all other audio-visual educational resource material available to schools in the area. This is quite commendable for the program. Teachers and principals of both government as well as non-formal schools also appreciated this feature of the ECVU videos. The information provided enhances quality of education in schools. Values incorporated into the subject areas enable learners to think critically. • Most videos are not value-loaded, nor are there an overload of information. The selection process of information to be provided is thorough, as a result of which most videos are clear and succinct. • The videos are based on chapters from Gujarat State Board School Textbooks. Core curricular areas in textbooks are referred to and yet in most cases there is an attempt made to broaden the horizons of learners by providing new information or by contextualizing the information. The information in textbooks is dull, detached, simplistic and in some cases, problematic. In contrast, several videos are refreshing as they are located in the lived realities of learners – they show their own geographical areas, people (and children) who are similar to them and speak the same language. • There are a few exceptions to this. Some of the videos, while being located in the learners’ contexts, do not represent the complexities and nuances of this context adequately. In fact, these videos fall short of adding information or perspective that is beyond the textbook. There are some videos which place one character as the provider of information – like the textbook where the teacher gives knowledge to learners. • The language used in most videos is simple and accessible, unlike textbooks which are written in a more formal and difficult language. Simple Gujarati is the language spoken by most children. Children from Muslim backgrounds speak Hindi at home; however, they follow simple Gujarati. An effort has been made to show people speak Hindi in some videos. The local flavor of the language used is laudable, however, in some videos, concepts and definitions have not been explained. Difficult words have been used without being broken down into simpler, more accessible sentences. This is in the case of some videos on Science themes. • Most videos have made an effort to incorporate values of equality, democracy and diversity in different ways, however, this is not done in a way where values implicit or embedded in the content. In some videos there is a superficial understanding of values, as a result of which they appear to be forced or just an add-on to the topic. In mainstream textbooks idealized situations are presented to learners, chapters also present values in the form of messages which are flat as well as one-dimensional. In the videos, there is an attempt made to move away from this, however, the values presented are not multi-dimensional. In some videos there are only characters with names that suggest their background. While it is important to have such characters, their roles should not be limited. This is especially in a context where communal and caste-based discrimination is a serious issue and contributions of marginalized communities are not acknowledged. • The Social Science themes especially lend themselves to incorporation of values quite well. However, it has been challenging to incorporate values in videos that focus on Science and language. • An attempt has been made in most videos to show women as providers of information. They are shown in varied roles – as panchayat members, teachers, activists and women engaged in domestic work. This is a refreshing departure from conventional textbooks where women and their roles are invisibilised and where only a few people or ‘experts’ are shown as providers of information. In the ECVU videos, women have agency; they are aware of their rights; they are active citizens in public domains; and they are decision makers. • People from marginalized communities are also shown in positive roles in most videos. There is no stereotyping of roles or professions. People from Dalit, Muslim and working-class backgrounds are adequately represented. Their knowledge and skills are treated with respect and their contribution to children’s learning process is recognized. This effort is well appreciated; however, in some films there appears to be a tokenistic representation of marginalized groups. • Most videos produced by all four ECVUs show discrimination on the basis of gender, caste, religion and class. They indicate the power hierarchies that exist in society. This is unlike textbooks where there is no indication of any conflict. In some films, however, the story moves quickly to showing idealistic situations. While it is necessary to show positive examples to learners, there is a need to highlight struggles and complexities adequately. • Most E-CVU videos are well produced. They have used different genres of film making. Both fictional as well as non fictional ploys have been used in videos and this effort is commendable. There are plots and sub plots which are woven together in a coherent manner. The visual medium has been used to its fullest potential in many films. There are a few aberrations to this – some videos are not well shot, have plots which are unrelated and end abruptly. Implementation of Video Films The following issues relate to the implementation and screening of the video films in the target schools by the ECVUs. • In schools where the videos are screened, facilitation is as important as the screening itself. In the absence of facilitation, the video becomes an interesting activity that does not necessarily stimulate critical thinking. In schools where facilitation was good, the video was very effective. Children were able to understand concepts and information provided. However, where facilitation was absent, the video was like an ‘interesting activity’ which was not linked to curricular issues. • In most ECVUs, very little time and resources have been invested in training facilitators. As a result of this, it is upto each individual facilitator to decide how he/she will transact a video. It is based on their existing skills and interest that this is decided. In some cases, screening and transaction of films is not prioritized given the existing work commitments of facilitators. However, in ECVUs where orientation and inputs have been provided to the facilitators, the classroom transaction is creative and inspires children to engage more with the video and the activities. • There has also been a high turnover of facilitators in some ECVUs. While this is inevitable for any program, it has had an impact on the learning opportunities available for them and their overall engagement with the Videoshala program. • There is no common point at which a video is introduced in the school syllabus. It is upto each E￾CVU to decide whether a video will be screened before/after/during the time that the topic is being covered in school. This also depends on the availability of the facilitator in some cases. This lack of clarity and ad hoc manner of transaction has led to some amount of confusion in the ECVUs. In some places there is a long gap between the time that the topic is covered and the screening, making the impact or effectiveness of the video inconsistent. • There is almost no discussion on values that takes place in classrooms. Facilitation is usually around information and concepts that are introduced. While values are in the realm of the abstract and therefore difficult to discuss, no effort is made whatsoever to raise any question or debate around diversity, citizenship and democracy. • Each E-CVU has worked out its own process of involving facilitators in the process of film production. In some ECVUs feedback is sought from facilitators at different stages. Teachers of schools are also involved in the process of hard spot selection and in developing scripts and providing feedback on the rough cut. However, in other ECVUs, the facilitators and teachers have no role whatsoever in the film-making process. Their role begins at the time that the film is ready for screening. This has meant that the facilitators are detached from the entire process and therefore organize the screening in a similar manner. Cost Viability of Videos The video production was found to be cost effective in the following manner: • The estimated cost of the production of the videos as reported by stakeholders and technical support partner (average Rs.1.5 lakhs per video) was kept in mind while evaluating the expenses incurred. The actual expense as reported by NGOs heads and technical partners was also close to the same amount. • The same job would have been done by professional at three times the above cost, which shows the viability of using CVPs. • The cost effectiveness of the project also increases due to the training of the CVPs as they continue to make additional videos and also contribute in training any new CVPs that are hired. • Based on analysis of the videos by technical evaluators, it was concluded that the final product meets the desired standard of video production given the costs incurred. 12.2 RECOMMENDATIONS FOR THE FUTURE Videoshala nurtures quality of education at the very roots of the education system by relating it with contextual realities and the local community. The experience of implementing this project through the involvement of the community members has been, by and large, successful across various districts/schools and has inspired the ECVUs greatly. Videoshala, however, could be sustained, expanded and scaled up to reach larger sections of society. To realize this, the following recommendations are presented: General Considerations • Set-up a special cell at the state level to provide physical, financial, social support and resources to NGO’s and public-sector undertakings interested in participating in such activities as a concerted effort with definite goals. • Identify and enlist participation and cooperation of increasingly more committed voluntary agencies in the field of education and social work. • Prepare a pool of resources such as expertise, non-consumable items of use in video at State/District/Block levels to serve as a repository for teams interested in utilizing the resources. This will be more economical and save duplication of efforts. • Prepare, with the help of experts, a prototype program and pedagogy including guidelines for preparing audio-visual aids, provide training in audio-visual aids production, preparing training and other manuals, and preparing functionaries for using videos. • Prepare a pool of topics/issues/hard-spots needing extra attention, from various courses and curricula. • Prepare a minimally desirable qualification profile of community members and participants such as video-producers and classroom facilitators. • A parallel program utilizing the lessons learned from the present project and having an built-in follow-up plan may be conducted to enhance the effectiveness of the Videoshala concepts. • The present report should be disseminated in stakeholders, partners in the government at the policy level and others. Production of Videos • As reported by some of the CVPs the production of videos including all formalities of pre￾production has resulted in monotony and fatigue. In the future phase of Videoshala this aspect may be discussed to find out best ways of utilization of the resources. One possible way may be that at a time six CVPs should be involved in different phases of the production. Variety may be introduced in their work by rotating their assignments. • As reported by some CVPs and PDs more comfortable time for production should be made available. For some of the videos the time (40-50 days) may be adequate but taking the available resources and logistics with ECVUs in consideration, the time duration for production of a video is not adequate. • The producers have received a number of inputs; however, the inputs in technical aspects are more than those on conceptual issues or content. There needs to be a balance in the nature of inputs provided otherwise producers will largely be dependant on the CCU for feedback and suggestions on content and values. • The CCU can also consider developing benchmarks for assessing quality of videos at different stages of the film-making process. In the future, if the ECVUs function in an independent manner, the benchmarks will help them monitor quality and effectiveness of videos and the process of production. • Feedback from the CCU needs to be time bound. Any delay can cause delays in production of new films as a result of which ECVUs will not be able to meet targets. • The CCU and the E-CVU coordinators should enable producers to access and transact the resource material that is provided. Merely providing reference material is not enough as the ability to select and use this material is a skill which requires facilitation. Content of Videos • It is important to develop content of videos according to the age and learning levels of children. In some films mediums which are for children of a different age group have been used. This may make the medium ineffective for children who may be ready for more nuanced mediums. • It is important for the videos to explore the potential of opening up discussions on social and structural inequalities. While most videos do indicate these inequalities, sometimes in very subtle ways, they stop short of taking the discussion further, as a result of which it is quite possible for the viewer to miss the point completely. • The project should also develop an understanding on values by unpacking ideas and concepts. For example, while talking about equality, it is important to talk about core concepts behind the idea of equality and look at how inequality exists in the everyday life of the learner. Reflective workshops for producers of videos on values are also essential to build a deeper understanding. • It is also important for the E-CVU and the CCU to address the issue of integration of values in films on Science, Language and Math. These are topics which are most ‘difficult’ for learners. It is all the more important to integrate values in these topics – a task that is challenging, but critical for the program. • The E-CVU must look at approaches that will enable them to locate videos firmly in the local context. The realities of children from families affected by communal violence, caste-based discrimination, gender biases need to be articulated more strongly in the videos. Up scaling and mainstreaming is an objective of the program in the next phase and therefore highlighting conflict may not be well received by mainstreams institutions. However, in videos where idealistic situations are shown, it is important to make sure that conflict and disparity is also indicated. • The ECVUs and the CCU should look at the recent innovations in textbook writing. At the national level, efforts have been made by the National Council for Education, Research and Training (NCERT) to develop textbooks that focus on child-centered Screening of the Videos • The role of the teacher in screening of the videos was not more than arranging and controlling the students. The teachers can be a very important agent to meet the objectives of the Videoshala and hence their involvement should be more worthwhile. The responsibility of screening of the videos can be assigned to teachers of the respective Videoshala schools. This will not only curtail the budget of the project but they can also link the videos when they are teaching the subjects/topics. • To make the content of videos more clear to students, the same videos should be screened more than once, if required. • The CCU should intervene and suggest structured guidelines for facilitation which correspond with the videos. The CCU should, without being prescriptive, suggest to ECVUs the mandate for facilitators. In addition to this the CCU should also look at models that have been successful (in terms of facilitation) and make recommendations to ECVUs on the basis of this. • Inputs on pedagogy, values, creative teaching should be provided to facilitators. Each E-CVU should organize inputs for its own facilitators and the CCU can play a role in organizing an input especially designed for facilitators. • There should be a strategic involvement of facilitators in the process of production of videos. They could be involved in the selection of hard spots and in providing feedback at one stage of film making. If the program plans to scale up its operation and involve government school teachers as facilitators, this involvement has to be very strategic in nature. • Teachers can be involved in hard spot selection so that they engage with the videos as resource material that has been developed with their involvement and inputs. Otherwise, there is the danger of the distance between the facilitator/teacher and the video increasing further. • There has to be some logic to when a video should be introduced in schools. It cannot be left to the individual facilitator to decide. The impact of the video on teaching in classrooms can be maximized only if it supplements and enhances the quality of education. Therefore, a discussion between the ECVU and the CCU on this is extremely critical. • Worksheets should be developed for varied purposes. They should help students recall information from the videos and should also push them to think, analyze and reflect on the information and values. Worksheets can also help learners to reflect on values which are often not discussed in classroom situations. Scaling up & Sustainability • The ECVUs or the CCU (whichever may be possible) should get in touch with the school authorities at Block Resource Centers (Bras), Cluster Resource Centers, and made them aware of the Videoshala concept. This would help Videoshala in getting permissions to screen Videos in government schools. • Videoshala should target all the students of (for the grades focused upon) Videoshala schools. By showing the videos to all the students the utilization of the videos can be optimized. However, at a time, the maximum number of children of forty for the screening of the videos should be maintained for better impact of the videos on the students. • There should be up-to-date data base on Videoshala (related to the number of schools covered, dates of screening of the videos in the schools and other activities done by each ECVU) at a central place so that the information can be easily accessed as and when required. • In many schools the facilitators are the face of the program. The larger objectives of the Videoshala program and the local producers are not known to people in schools and to the larger community in many places. Since one of the objectives of the program is to empower local people to produce educational material to improve quality of learning, it is essential that this information be shared with various stakeholders. The ECVUs should look at forums where their work can be promoted and publicized. 12.3 PARTNERS VIEWS ON FUTURE STRATEGIES The partners of Videoshala also had some views regarding future strategies to consider: • As an alternative mechanism to boost Videoshala, it was suggested by the project director (PD) Meghdhanush ECVU that the Videoshala films content be integrated with the education system. For this, the ECVUs are already in touch with the State education department. On the other hand, they also want to fan out to more NGO’s to be partners in expanding the Videoshala project. • As another strategy of expansion, the PD of Navsarjan had a suggestion that there should be some independence to the ECVUs to implement the project. The ECVUs should not be guided and monitored by the CCU in this venture. The CCU should be more of a technical support partner. Then again, she was also of the opinion that the target of developing six videos per ECVU (as was fixed for the program period of one and a half year) and its screening should not be made too stringent as this would lead to compromise with the quality. The emphasis should be on outcome rather than output. She suggested that in the meetings of the CVPs at CCU, the project Directors and the representative/s of concerned NGOs of the CCU should also be invited so that the respective NGOs could keep pace with the developments of the ECVU. • The PD, Navsarjan ECVU felt that the Videoshala team should be technically so sound that in addition to producing videos for themselves, the team gets invited by others for the purpose. This could also be one of the possible ways of making the Videoshala self-sustainable. • Also, as a future strategy, the ECVU Navsarjan would like to screen the videos in Panchayat Bhavans so that the screening should cater not only the Dalit groups but also to a mix of social groups. • Some ECVUs felt that the CCU should visit the villages and interact with community, including children, more often so that a better understanding of limitations, challenges and issues can be developed to strengthen future efforts. • Citing the example of one of the CVPs who benefited from the Videoshala in enriching his thought process to enter higher education, and speaking highly of the Videoshala, the PD Sahyog ECVU recommended the necessity of dealing with government schools to strengthen future efforts. Speaking about his future plans for the video project, he revealed that he would like to screen the videos in the community including parents and children as well as teachers of different schools. He informed that in the next phase of the project, Sahyog would like to produce four videos for children and two videos for the community. • The PD of HSM ECVU contended that the Videoshala should be made part of the school curricula with a greater involvement of teachers. To make it more participatory, at local level, a committee of local people and teachers should be formed and they should decide on the topics and values to be incorporated in the videos. All of this would pave the way for making the Videoshala self￾sustainable. Marketing of the videos maybe adopted to generate a good resource for sustaining the project as the expansion of the program is very much needed and should not be confined to selected few schools. It was evident that all the key stakeholders had a positive experience with Videoshala; hence all of them felt the need to scale-up the project in future. Gagan Sethi, the head of Janvikas felt that the extent of acceptance and appreciation of Videoshala program reiterates the felt need of such programs in the present educational context. Sharing the thoughts on scaling up the project, he felt that setting up a CVP unit of Videoshala could retain the local contextualization of videos for two Talukas (sub-division) per district. This will help in preserving the basic essence of the concept of Videoshala-localization and could at the same time, be made available to all the schools in the region. He further mentioned that to integrate the Videoshala program with government schools, the dialogue at different levels of government is in progress and according to him some positive response are expected from government side. 12.4 SPECIFIC RECOMMENDATIONS FOR EACH ECVU The following are specific recommendations for each ECVU: HSM ECVU • Videos on scientific subjects should be made taking into consideration the level of knowledge of the children, and should add to the knowledge the children already have from other sources. The presentation of the science topics should not be merely in the form of definitions but explanation of scientific concepts. Language of the science films should be simple, in contrast to textbook transaction of this content. • The films should contain local information and also contain local context and issues. The context of the organization – its work on the environment and ecology of the Saurashtra region should be incorporated in the films. • Before the film is screened it should be ensured that the atmosphere in the class is conducive to an interactive and innovative teaching learning experience. The students should be in a position to link themselves with the subject and the video, and think critically about the content. Screening of the film should not be seen in an instrumental perspective, as an activity that has to be done. • The facilitator needs to be very active during the session, focusing on the needs of every student. The responsibility of the facilitator does not end with the screening of the video. S/he should initiate and facilitate discussion on the subject, and on the values in the film. The capacity of the facilitator should not be built merely to screen a film, but on educational activities and subject matter of the screening. • It is also necessary for the facilitator to participate in research and selection of hard spot. Both these processes can help enhance knowledge of the subject. • The questions in the worksheet should add to the knowledge gained from the film. Some questions should to revise what they have learned and some should be analytical questions. • Producers mostly refer to SSA material for the subject content. However, it is advisable to look up to alternative material such as Eklavya, Bodh, Digantar, and Avehi Abacus also. This would help in exposing the producers to different methods for understanding and teaching a complicated subject. • When contacting resource people, care should be taken that the ‘experts’ also include local people and people of different backgrounds from the community. Otherwise the process of filmmaking is comparable to the making of other Teaching Learning material, even textbooks. • HSM should look to how the community videos and the ECVU are creating a place and name for themselves, especially in the context of expansion. At the moment, few people in the community and even in the schools know who has produced the videos. • The HSM E-CVU is quite new. During the last one and half years, they have achieved high technical skills but their understanding of pedagogy and values of education are still weak. • Feedback is more effective if it is not too scattered over many people and points in the production process. UDAAN-MEGHDHANUSH ECVU • The E-CVU should try and make films that are positioned more within the local context, in terms of language, issues prevalent in the community, and build on local knowledge. • Producers to access more effective alternative resource and reference material of other educationists in order to enhance their understanding of concepts and to enable them to bring about creativity in film-making, activity and worksheet development. • More strategic involvement of facilitators and teachers to be encouraged – at suitable points in the film-making process – to bring to the table hard spots, help bridge the divide between film-making and its transaction, and ensure quality and continuity in curricular flow during classroom teaching. • More time to be allotted in the film-making process to substantive discussions and understanding of content and values, rather than giving more time to technical aspects, which are anyway strong. • Creative thinking about use of video as a medium in showing values within science-related films (e.g. - in 'Pupu'). NAVSARJAN ECVU • The analysis of the worksheets and activity sheets should be shared back with the teachers at some stage. The ECVU can consider sending a report to the teachers in order to inform them about the learning from each screening. • The facilitators at the Navsarjan ECVU are activists with a strong sense of commitment towards their work on rights issues; however, the role of the CF is new and fraught with tensions. The CFs clearly articulated that it was not possible for them to play this role. The CFs are skilled and have a lot of work experience; however, they are not trained as facilitators. As a result of this, their facilitation skills were poor (as we had seen at the Shobhasar Primary School). A short orientation provided was not very useful to their work. Facilitation is a specialized skill. An activist with other commitments should not be expected to work as a full-time facilitator. There is a need for the program to select a new set of facilitators who can work in a focused manner on organizing the screenings and transacting the video kits. • There is a huge distance that exists between the CFs and the producers at Navsarjan. We sensed some amount of tension in this relationship. The producers are those who work intensively on making the video kits and in the process learn a lot about the theme and related issues. The CFs are not involved in any manner in this process, however, they are the public face of the video kits. The producers feel that they are not recognized or acknowledged by teachers and learners because they are not part of the screening and facilitation. The CFs on the other hand typically feel detached from the film because their involvement in its making is minimal. The ECVU should address this immediately. • There already exists a distance between the teachers/ headmistress of government schools and the process of transaction of video kits. There is a danger of this distance increasing further in the absence of any classroom interaction or discussions with teachers. If reaching out to a larger number of state-sponsored educational institutions is an objective of the program in its next phase, it is of utmost importance that the ECVU facilitators interact with students and teachers. • There is no plan for the number of screenings that is to be organized by each CF. Some of them had done about six to seven screenings thus far and some had done about 25. It was clear that the number varied because it was up to each CF to decide when and where he/she could organize the screening. SAHYOG ECVU • Further technical training - building on the skills they already have. Working with actors is an area that could be improved, also editing. Basic technical training for the ECVU coordinator • A more rigorous process of research for films on curricular areas – like language, science. The team could do with workshops on creating innovative science teaching learning material, for instance, and this input should include discussions on how values can be made part of the content more seamlessly. Other creative T/L material should be provided to the team to raise the bar of their own materials. • The facilitators should be involved to a greater extent in the research and training that goes into a film, so their grasp of curricula content is stronger when transacting films. They should also be talked through the discussion points in the activity manual, or receive training on perspective building, so that they can facilitate a better discussion on both content and values. The facilitators –whether these are teachers or not - should receive a few rounds of intensive training on transaction, pedagogy, content and values. • The program, both Sahyog and Videoshala, needs to be better known in the community – beginning with the children who watch the films, to parents of the children who watch the videos, children who act in the video, and extending to the wider community in Vatwa. One way to do this is to do screenings with the community. Another way is to frame the screenings with a little discussion on who makes the videos, their context and so on. The impact of the program could be much more if it was better known. APPENDIX I I Guidelines for Focus Group Discussion (FGD) with Children Instruction for the field researchers: Before starting discussion with the children introduce yourself and brief the group on the purpose of the meeting/discussion, making a clear reference to the video project. Ensure that all the participating children are comfortable and in a mood to talk in the group without any fear or shyness) A. District: ___________________________________ B. Tehsil / Block: ___________________________________ C. Gram Panchayat: ___________________________________ D. Village: ___________________________________ E. Name of School: ___________________________________ F. Name of Field Researchers: ________________________ G. Date & Time of discussion: ________________________ Information on the Children attending the Group Discussion Sl. No. Name Age Gender Standard (Class) 1 M / F 2 M / F 3 M / F 4 M / F 5 M / F 6 M / F 7 M / F 8 M / F 9 M / F 10 M / F 1. Can you tell the name of the subjects/issues you read? 2. Which are the subjects that you like 3. Which are the subjects and topics you find difficult 4. How do you manage with the difficult subjects/issues? 5. Did you watch the video/s shown in the school (researchers can mention the date of the screening of the video/s in the school) 6. What were video/s about? What did you learn from the videos? 7. Did you like the video/s you watched on the issue? Why did you like or dislike the video/s you watched on the issue…………..? 8. How these videos contributed to your knowledge on the issue? 9. What message does it convey beyond the subject knowledge? 10. What activities did you do along the videos? 11. How those activities were helpful in understanding? 12. What role facilitators and teachers played while screening videos? 13. What changes this video has brought in your thinking, behavior and practice? (Probe if no answer is on values) 14. When and what did you, your friend or family last practice any of the message you received from the video? (Probe if no answer is on values) 15. What is your opinion on the language used in the videos? 16. Was the language used in the video/s easy to comprehend? 17. How do you like watching the video/s on the issue in local dialect? 18. How did you like or dislike the narrators/actors in the video? 19. What did you like the most in the video and why? 20. Can you tell how these videos (Videoshala videos) were different from other videos you have watched? 21. How this method of teaching (teaching by showing video/film) is different from other methods? 22. On what other issues would you like to watch videos? 23. What more information do you want to share about Videoshala project? II Guidelines for discussion with Local Coordinator Name of the Coordinator_______________________ Name of the NGO Associated with________________ Educational Qualifications ______________________ Social category: (1) SC/ST (2) OBC (3) General (4) Any other____(specify) Monthly income (in Rs.) ___________________ 1. For how long have you been associated with Videoshala project? 2. Kindly tell in brief about the background of your association with the project, 3. Your role and responsibility as a coordinator. 4. Can you give the details of the training you received from the day of your association with the project till now? S. no. Date of training Duration Content of training Trainer Place of training 5. What was the process to identify the hard spots? 6. Please tell us about the process of making videos and screening (Videoshala project)? 7. What kind of planning had been done before starting the production and during the production? 8. What kind of planning was done for screening? 9. How this video was different from other educational videos? 10. Rate the quality of the video produced by you on five point scale of (1) Excellent (2) Very good (3) Good (4) Average (5) Below average 11. Rate the relevance of the content of the video on three point scale: (1) Very relevant (2) Quite relevant (3) just relevant 12. What challenges did your team face in making videos and how did you tackle the challenges? 13. What was the response of the schools on the issues being tackled through videos? How receptive or critical were they? 14. Do you see any continuity or linkages in the videos developed in this project? What is your thought on the continuity of the videos developed and screened? Whether it is required and why? 15. How beneficial, if any, has been the videos for the teachers? 16. How beneficial, if any, has been the videos for the students? 17. How was the coordination between CCU and your unit (ECVU) ?What support did you get from CCU. What additional support did you expect from them? 18. How was the coordination between the team (ECVU) members? 19. In your opinion what has been the impact of the project on the project members (involved in producing the video kit). What changes did you observe? 20. What changes you observed in the community? 21. What’s your plan to expand and sustain this project? 22. What kind of support do you expect from other agencies including government in this endeavor? 23. What are the major challenges before you in expanding the idea of Videoshala and what have you planned to overcome them? 24. What other complex educational issues have been earmarked to be focused under Videoshala project? 25. What all have you to say about the project? III Guidelines for Classroom Facilitator Name of the Facilitator: Age: Educational Qualification: Social category: (1) SC/ST (2) OBC (3) General (4) Any Other _____(specify) Profession: Monthly Income (in Rs.): 1. Your experience in the field of education? 2. How did you come to know about the Videoshala Project? 3. What features of the project motivated you? 4. As classroom facilitator what was your role and responsibilities. 5. Did you receive any training for doing as a classroom facilitator? If yes, please give details. S. no. Date of training (Duration) Content of training Trainer Place of training 6. How beneficial the training you received was for your job responsibility? 7. How these videos were different from other educational videos (probe on community involvement…) 8. Kindly tell us the process of screening of the videos in the classrooms? 9. What activities you were involved while screening the videos and how those activities were helpful? 10. What additional role you would like to play? 11. What is your thought on number of students viewing the videos at one time. Is it ok, more or less? What’s your thought on carrying TV to screen videos. 12. In your opinion what is the optimum time required to complete one set of activities including screening videos, activities and filling activities sheet? And presently how much of time you are giving for one set of activity? 13. How the video screening on particular topic should be linked when teacher are actually teaching those topic? 14. How was it received by children? ⇒ Did they find it interesting or boring? ⇒ Were they attentive during the screening? ⇒ Did they raise questions during and post screening. 15. Do you feel that the videos helped in clarifying the concepts of the children and the teachers on hard spots? What is the basis of saying so? 16. Which video did they like most and why? 17. Rate the relevance of the content of the video on a three point scale of: 1) Very relevant; 2) Quite Relevant; 3) Just relevant 18. Could you recall any interesting incident during classroom screening of the vide/ 19. How was the teachers’ response during the screening of video/s? Did they actively participate? (probe: on their opinion on this approach of teaching, community involvement..., motivation to use the method) 20. What role school and teacher played during the screening? 21. What was their comment on the videos? (probe: can it be replicated at a higher level?) 22. What kind of problem, if any, did you face in screening the video? 23. If you look back at your involvement with the Videoshala project, what would you identify as significant change(s) in yourself? 24. What kind of changes, if any, did you notice in the attitudes of the family members/community members towards you after your association with the Videoshala project? 25. What changes did you observe in the: a) Children (probe: value, learning, regularity in classes, motivation to learn….) b) Teachers (their conduct in classroom, interaction with children, ) c) Community members 26. Did you enjoy working as a facilitator in the classroom for the Videoshala kit? If yes, kindly share your experience. 27. Do you think that your association with Videoshala project made you a skilled person and it has opened earning avenues for you? 28. What challenges did you face? 29. What all have you to say about the Videoshala project? IV Guidelines for Discussion with NGO Representatives A. Name of the NGO______________________________ B. Name of the representative ______________________ C. Gender (1) Male (2) Female D. Designation of the representative_________________ 1. Kindly tell in brief about the activities of the NGOs and work in the field of education 2. How did the idea of Videoshala originate? 3. How Videoshala videos are different from others educational videos 4. How do you think your ECVU is different from others. What role philosophy and objective of organization has played role in the Videoshala project (e.g. Identifying hard spot areas.) 5. Kindly tell about the entire process (from the inception of the idea to ultimately production and screening of videos). 6. What was your team for the project (on community involvement, can the model made sustainable….)? 7. What was the role and responsibility of each team members of the Videoshala project? 8. How frequently you used to meet to discuss the progress and solve the problem, if any, of the team? 9. Did everything work according to the plan or you faced challenges? 10. If faced challenges, what were those challenges and how did you tackle the challenges? 11. Do you think that the Videoshala project has been able to achieve its objectives? If yes, kindly elaborate. If no, what were the obstacles that came to the fore in achieving the intended targets? 12. What was the response of the schools on the videos? How receptive or critical were they? (…….possibility of replication …) 13. In your opinion how children liked it? 14. Do you see any continuity or linkages in the videos developed in this project? What is your thought on the continuity of the videos developed and screened? Whether it is required and why? 15. In your opinion what has been the impact of the project on the project members (involved in producing the video kit). What was the gain for them? 16. What changes did you observe with the community members? 17. What has the attrition rate of the producers and facilitators been? how has it affected the project? 18. Do you plan to expand the video project? If so, in what way? 19 Can the Videoshala model be used for higher grades and subject areas? (how ? why not?) 20. What kind of support do you expect from other agencies in this endeavor? 21. What role CCU plays and what are your other expectation from CCU? 22. What training provided by the resource organization and what additional training you felt is still required to enhance the team capacity? 23. Has your NGO ever approached the government authorities for the promotion of this concept of Videoshala? If yes, what has been their response and what is your expectation from them? 24. What are the major challenges before you in expanding the idea of Videoshala and what have you planned to overcome them? 25. What other hard spots have been earmarked to be focused under Videoshala project? 26. What is the budget you spend on an average for per video (cost effectiveness)? 27. What is your opinion on making animation videos in local language on hard spots? 28. What is your future plan of Videoshala project? V Guidelines for Discussion with Teachers (To be asked with the teachers of Non- Videoshala school) District: Block/Taluka: 1. Name of the School __________________________ 2. Name of the Teacher __________________________ 3.Gender (1) Male (2) Female 4.Years of experience _________ 5.Educational qualification _________________________ 6.Which subject(s) do you teach? (1) ____________ (2) ___________ (3) _____________ 7. In your opinion what are hard spots area? 8. How do you tackle hard sport at your level? 9. Do you think that using video as a mode of teaching instead of explaining through textbooks is more effective on hard spots ? Reason for yes or no ? 10. Have you seen any educational videos? On what issue/subject was it? 11. Have you heard about the Videoshala Project? If yes, what you know about Videoshala project? 12. What is your thought on local educational content developed by local community members? 13. Do you find any demerit in the present system of teaching? If yes, what? 14. What would you suggest to improve the present method of teaching? VII Guidelines for Community Video Producers Name of the Community Video Producer__________________________ Age: _______________ Educational Qualifications: ____________________ Profession: ______________________ Monthly Income: ______________________ Social Category: (1) General (2) SC/ST (3) OBC (4) Other (specify) 1. How did you come to know about the Videoshala Project? 2. What features of the project motivated you? 3. What has been your prior experience with technology (video making) and in the field of education? 4. As community video producer what was your role and responsibilities? 5. What was the role and responsibilities described by Videoshala prior to joining the Videoshala team? 6. Were you confident of fulfilling your role and responsibilities? If no, how did the Videoshala team support you in fulfilling your roles & responsibilities? 7. What has been the response of your family on your association with the project? 8. What was the response of the community when they came to know about your association/involvement with the project? (Have you to face any kind of resistance / comments of the community or they appreciated /encouraged you) 9.What kind of training did you undergo in order to produce the kit? Kindly give details of the same S. no. Date of training (Duration) Content of training Trainer Place of training 10. How much confidence did your training give you? 11. What kind of planning had been done before starting the production and during the production? 12 Rate the quality of the video produced by you on five point scale of (1) Excellent (2) Very good (3) Good (4) Average (5) Below average 13. Kindly elaborate the process of selection of content of the video? 14. Please tell us the process of video production? 15. Rate the relevance of the content of the video on three point scale: (1) Very relevant (2) Quite relevant (3) just relevant 16. How did the children, teachers like the new method of learning on the hard spots? 17. What new things did you learn (in the areas of technical, education ,film making)? 18. What change do you find in your behavior, thinking as result of working as a producer of Videoshala? 19. Do you think that your association with Videoshala project made you a skilled person and it has opened avenues for income generation? 20. Has this Videoshala project made you understand your community better? How? 21. How the Videoshala is contributing in education, community, society? 22. What role trainer played in this project? Do you need their involvement in future also? 23. What role the facilitator played in screening the videos and what role teachers played in it. Do you think any changes or improvement in their role to improve the impact of videos among children. 24. Do you see any continuity or linkages in the videos developed in this project? What is your thought on the continuity of the videos developed and screened? Whether it is required and why? 25. What changes did you observe in the children after viewing these videos. (Changes related to their knowledge on subjects and values as well) 26. What changes did you observe in the community members from this project? 27. Do you think that by producing video kit on education you have contributed something in the field of education to your society? Has it motivated you to do some more for your community? 28. What do you think the difference between the videos developed by you and other educational videos? 29. How was the coordination/atmosphere at pre and during production of video kit? 30. What opinion do you have on the existing team formation? 31. What were the major problems faced? 32. If you look back at your involvement with the Videoshala project, what kind of impact did it have on you in learning and attitudes? How is the Videoshala model impacting on work? 33. What would you identify as the limitation(s) of the Videoshala project? 34. What kind of support was available from the CCU? What additional support you are expecting from them? 35. Do you see any possibility to carry forward the project by local people without any external support? 36. How do you plan to take this project forward with respect to sustaining and scaling up? 37. What all have you to say about the project? VIII Guidelines for Discussion with Teachers (To be asked with the teachers of Videoshala School) District ______________ Block/Taluka__________________ 1.Name of the School __________________________ 2.Name of the Teacher __________________________ 3.Gender (1) Male (2) Female 4.Years of experience: _________ 5.Educational qualification: _________________________ 6.Which subject(s) do you teach? (1) ____________ (2) ________ (3) _________ 7.Are you aware of the Videoshala project? (1) Yes (2) No (3) DK/CS 8. If yes then, how have you participated in the Videoshala project? What are the different roles played by you in this project? 9. In your opinion what is hard spots area? 10. How do you tackle hard sport at your level? 11. If you participated in identifying hard spot areas for Videoshala project, kindly elaborate how you contributed in the same (probe the process of identifying hard spots, how did they arrive at a given concept for video content development, how values of citizenship and diversity incorporated into the curricular concept) 12. Could you please recall the issues, which were identified as the hard spots? 13. On what issue/s were the video kits shown in your school? 14. Can you describe the process of screening of the videos? 15. What was the role of the person (facilitator) who screened the video in the classroom? 16. How were the facilitators' interactions with the children? 17. What was your role in the screening? 18. Would you like your involvement in the screening to be more? 19. What would you say about the content, presentation, language, quality of the kit shown to the children? Video 1 2 3 4 5 Content Presentation Language Quality of the kit Duration 1 2 3 4 Code: Content: (1) Very good/ relevant (2) Good and relevant (3) Average/not very relevant (4) Poor relevant (5) Bad /Irrelevant (6) DK/CS Presentation: (1) Very attractive (2) Attractive (3) Average (4) Poor (5) Bad (6) DK/CS Language: (1) Local touch with easy to comprehend (2) Easy to comprehend but not local touch (3) Neither local touch nor easy to comprehend (4) DK/CS Quality of the Kit: (1) Very good/attractive (2) Good/attractive (3) Average (4) Poor (5) Bad (6) DK/CS Duration : (1) Adequate (2) Somewhat Adequate (3) Inadequate (4) DK/CS 20. Do you think that using video as a mode of teaching instead of explaining through textbooks is more effective on hard spots ? Reason for yes or no ? (probe on difference between using video vs traditional textbook based teaching: can it be replicated at higher level of education and on what subject areas) 21. In your opinion what changes have you noticed in the children after watching the video on hard spots ? (more than one response is possible) (1) Now students have clarity on a particular issue shown in the video. (2) Children are able to answer/respond correctly (3) It helped the children to think and understand other difficult issues (4) It has changed their behavior and conduct (5) It had nothing special for the children (6) Don’t know /can’t say 22. What change has been observed in children (probe: participation in classroom, regular in class/school)? 23. What changes have you observed in the behavior of children after watching the videos (probe citizenship and diversity value)? 24. How it has impacted on teachers (probe: change in teaching approach, their conduct in classroom, interaction in classroom, motivation to learn and teach) 25. If you have seen any other videos on education or any other videos, how these videos (Videoshala videos) different from other? 26. What’s your thought on local educational content developed by local people, helpful in (probe impact on learning on children) 27. How this video has been helpful for you? 28. What other issues do you feel should the Videoshala project take up in future for preparing the video for the children? (1) ____________ (2) ____________ (3) ____________ 29. What do you think the video screening on particular topic should be linked when you are actually teaching those topics? 30. Challenges and issues in translating the concepts into videos? 31. What all have you to say about the project? IX Guidelines for discussion with Parents/villagers 1. Do you think that some of the subjects/issues are not easy to comprehend for the children? If yes, can you please tell what are they? 2. How do you tackle with the problem at your level? What kind of support school and teachers provide in understanding those concepts 3. Do you know that educational videos are shown in the school? What are those videos and how are these videos different from other videos. 4. How these videos are contributing in solving these problems? 5. When did the children last share about the videos and what did they share about the videos? 6. What changes, if any, have you noticed in the thinking and behavior of the children after watching the educational videos? 7. What are the changes you observed in your family and community with the intervention of Videoshala 8. (Ask the questions if the parent has participated in the Videoshala project) How have you been associated with Videoshala project? 9. What was your experience with the Videoshala project? 10. How was that experience helpful in life? 11. Has your child acted in a Videoshala video? What was your reaction before and after the video? Why? 12. Was there any change in your child? 13. Would you let your child act again? Why? 14. To those whose children have not been in the videos- Will you allow/encourage your child to act in the video? Why/why not? 15. What more would you like to share about the Videoshala project with us? ©2010 International Youth Foundation. All rights reserved. No portion of this report may be reproduced in any form without the written permission of IYF. This project was made possible by the generous support of the American people through the United States Agency for International Development (USAID) under grant agreement no. RAN-A-00-04-00043-00 This report was funded in part under grant agreement no. EPP-A-00-08-00006-00. The International Youth Foundation (IYF) invests in the extraordinary potential of young people. Founded in 1990, IYF builds and maintains a worldwide community of businesses, governments, and civil-society organizations committed to empowering youth to be healthy, productive, and engaged citizens. IYF programs are catalysts of change that help young people obtain a quality education, gain employability skills, make healthy choices, and improve their communities. To learn more, visit www.iyfnet.org ANNEX I: EEA Global Evaluation Framework and Surveys EEA GLOBAL EVALUATION FRAMEWORK Indicators Definition Data Collection Instruments/ Source Level of Reporting Indicator 1.1: In-kind and cash resources leveraged for the project through private sector, government, civil society and other partners This indicator measures the effectiveness of alliance building, by measuring the amount of contributions raised for EEA projects. The contributions are to be measured using EEA’s Leveraging Policy, and can be in terms of in-kind donations or in cash. Examples of in-kind donations include: intellectual property, staff time, space, technical knowledge, and equipment. Simple matrices used to report figures for the quarterly financial report. Project, Country, Global Indicator 1.2: The effectiveness of the EEA Alliance Approach in building partnerships, and promoting sustainable and scalable youth employability programs This indicator measures the long-term effect of the EEA Alliance (put in place and managed through different mechanisms such as the EEA Secretariat or Steering Committee) in terms of sustainability, scalability, and cost-effectiveness. It specifically assesses the effectiveness of the EEA Alliance model in building partnerships that support youth employability, ensuring EEA project implementation and sustainability as well as the quality of projects. This indicator will be measured based on a set of criteria that include: 1. Alliance well established, developed and managed by a permanent body (association, steering or advisory committee, secretariat, or other), with staff dedicated to EEA work 2. Permanent body (association, steering or advisory committee, secretariat or other) established a grant making process, and guided the process of selecting best grantees/projects to receive EEA grants 3. Alliance members (steering or advisory board members, stakeholders, others) participate actively in EEA meetings and activities 4. An internal monitoring system is in place, sub-grants are being monitored regularly, and regular follow up is conducted on achieved activities versus planned through field visits and meetings with implementers 5. Alliance members (advisory or steering board members) are using their networking capacity to develop partnerships between the private sector, government and civil society actors to promote youth employability and facilitate youth placement 6. Communications system is in place to promote EEA programs and leverage funding 7. National and local communication and outreach events used by Alliance members to promote EEA programs and activities and advocate for youth employability initiatives 8. Alliance members have the capacity to advocate for the development of a policy framework that enhances youth employability (leadership) through a sound communications strategy 9. Alliance members are assisting implementing partners with start up of projects 10. Alliance members are providing monitoring and evaluation technical assistance to implementing partners 11. Alliance members help with trouble shooting problems and obstacles facing project implementation 12. The overall cost effectiveness of the Alliance strategy compared with single donor projects (which includes the different tiers of cost - both project-level and alliance building & support costs) 13. To what extent do partners find assistance from Alliance/Secretariats helpful (specific aspects that make the EEA program unique)? 14. To what extend did the Alliance influence partners to improve their institutional capacities? 15. Alliance partners demonstrate improved their capacities to work with youth and promote youth employability Questionaire, guided focus group meetings with partners, comparative analysis by partners of their experience with EEA Alliance vs. other traditional development project models that place less emphasis on multi-sectoral partnerships. Project, Country, Global Indicator 1.3: Number of private sector, government and civil society partnerships formed through the project to jointly offer employment and business development training activities in targeted areas This indicator measures the number of partnerships that have formed as a result of the innovative projects technically or financially supported by EEA. Partnerships are defined as any number or type of partners working together to achieve a common project objective. Partnerships are both formal and informal. Formal partnerships are established by funding commitments, signed MOUs, or letters of commitment. In-formal partnerships are verbal commitments backed by actual contributions (technical or financial) made to the project. Matrix listing partners. Typology of partners and contributions made to EEA projects. Project, Country, Global Intermediate Result I: Alliance Building and Leveraging Resources Indicators Definition Data Collection Instruments/ Source Level of Reporting Indicator 1.4: New partnerships, mechanisms and approaches created as a rippling effect of working with EEA programs to support youth employability This indicator assesses the success of the Alliance model in creating a sense of leadership for youth employability and a culture of working in partnership to improve youth employability This assessment will be done through tracking new alliances, partnerships, approaches and mechanisms that did not exist before, and have been created as a rippling effect the EEA programs. These include partnerships with local governments, donors, government and non-government institutions, and private sector actors. New partnerships considered can be created by the Secretariate or by the partners. The purpose of the the partnerships can include: - Direct support to youth (such as coaching, mentoring and financial aid) - Programs for training and employment develpment - Advocacy for youth employment - Policy making - Fundraising for youth training and employment - Youth-led business creation (entrepreneurship development) Data limitation: This indicator will be based on anecdotal facts and there may be an attribution issue given that -while linkages can be established, there is no scientific or systematic evidence that the creation of any new partnership is directly attributable to the effect of the EEA programs. Survey and focus group discussions with alliance members. 1. Question(s) related to this indicator to be inserted in the alliance questionnaire 2. Questions related to this indicator can also be included in Employer's satisfaction surveys 3. Focus group protocols Country level Indicator 1.5: Number of non-target institutions that request to adopt EEA promoted programs, models and tools as part of their activities This indicator is a measure of the project's long-term effect on other education institutions at the national level. By introducing EEA concepts to the target institutions, other institutions may be inclined to adopt the model in order to provide their graduates with the employability skills they need to enter the job market and succeed in competitive work environment today. Tracking the number of these non-target institutions will show the ultimate success of the program beyond EEA. Institutions might have intrest in EEA curricula of a certain program, methodology or ask directly of EEA assistance to replicate the model in their institutions. Records kept by implementers or Secretariat of all institutions that request EEA programs or adopt the training in part or in whole during the course of the year (starting 2007). If possible, conduct brief interviews with the representatives of requesting institutions to know how they intend to use the EEA supported program and what they think of it (optional). Project, Country Indicator 2.1: Number of young men and women participating in EEA projects This indicator measures the number of young men and women benefiting from EEA supported projects. These individuals maybe enrolled in any of the following programs: 1. Primary, secondary or high school programs that improve their skills to enter the job market in the future 2. Vocational or technical education courses for out of school youth 3. Career counseling services and skill development programs 4. Life skills and any other formal or in-formal training courses 5. Apprenticeships 6. Entrepreneurship development programs These programs maybe of part-time or full-time nature, and may be offered in a formal or an informal setting Participtant lists, project records. It is important to define the profile of each population in order to help in the analysis of the effect of various program components within a clearly defined context. Project, Country, Global Indicator 2.2: Number and percentage of young men and women completing EEA supported programs This indicator measures the number of young men and women who have benefited from EEA supported programs, and have completed the course, training, apprenticeship, counseling or any other form of intervention that they may have participated in. The term completion may differ between projects and countries. In some instances, completing may mean passing a test at the end of the program. In some cases, it may be completing a certain duration of time in a certain course. Participtant lists, project records. Project, Country, Global Indicator 2.3: Number of teachers, trainers, facilitators and counselors trained under EEA supported programs A proxy-indicator for measuring improved relevance of training and employability programs. Since most EEA supported programs are innovative, their success relies heavily on adequately training and counseling providers to use the new methodologies proposed by the EEA programs. Trained trainers and counselors will guarantee the quality of the training, motivation and retention of trainees. Training may include IT teaching methodologies, life-skills and career counseling. Attendance sheets/list of trained trainers. Project, Country, Global Intermediate Result II: Improved Access to and Relevance of Education and Employability Training Indicators Definition Data Collection Instruments/ Source Level of Reporting Indicator 2.4: Percentage of young men and women demonstrating improved capabilities through participation in EEA supported programs This indicator measures the effectiveness of EEA training and employability programs in better preparing young people to improving their personal life, getting quality employment and/or entrepreneurship, and becoming positively engaged and productive. This is a qualitative assessment of trainees’ capacities (skills, attitudes and behavior) to assess their ability to perform on the job and in life in general. "Capabilities" encompasses all the different dimensions of growth triggered by EEA training (professional, technical, personal and emotional) Combination of one or more: focus groups, observation, pre/post tests, interviews, and self reporting by beneficiaries. Project Indicator 2.5: Level of satisfaction of EEA trainees with the quality and relevance of the training, counseling and services received through EEA programs (for both employability and entrepreneurship development) Subjective and immediate feedback of trainees on the training they have received through EEA projects. This indicator will contribute to the assessment of which EEA supported training programs have responded to the needs of target young men and women. Evaluation of training by trainers (level 1 evalaution questionnaire). Project when applicable Indicator 2.6: Number and percentage of youth who completed EEA supported programs who report that they have referred at least another youth to the program This is a measure of project credibility and relevance to youth. If youth who have received the training refer a friend or a relative, this is a sign of confidence in the program, and also a sign that the program did respond to the expectation of the referring youth. End-of-training questionnaire, focus groups. Includes the youth views (qualitative). Project Indicator 2.7: Level of employers’ satisfaction with the soft skills and/or technical skills of EEA-trained youth High levels of employers’ satisfaction with young employees’ performance is a direct measure of the success of EEA programs. This indicator will measure the level of satisfaction of employers and business owners with the professionalism and personal capacities of young men and women who have been placed in their businesses/industries upon completion of EEA support programs. Data against this indicator will be collected through a sample survey of all programs improving employability skills. Project, Country, Global Indicator 3.1: Number and percentage of trainees who get a job within six months of completing the program This indicator measures the number of young men and women who have been placed in jobs within six months of completing an EEA sponsored training course, apprenticeship or career counseling program. Employment includes any full-time or part-time paid work, including any paid work assignment or paid internship that lasts for at least three months and provides a reasonable prospect for continued paid employment in the future. Follow up surveys and project records. The matrix will also track the kind of training that youth who got employed have received through EEA project. Project, Country Indicator 3.2: Number and percentage of trainees who continue education, obtain an internship or engage in further professional training within six months of completing the program This indicator measures the number of young men and women who continue their education, obtain an internship or engage in further professional training within six months of completing an EEA sponsored training course, apprenticeship or career counseling program. Follow up surveys and project records. The matrix will also track the kind of training that youth who got employed have received through EEA project. Project, Country Indicator 3.3: Number and percentage of EEA trainees starting their own businesses within six months of completing the training This indicator tracks the number of EEA trained young men and women who started their own businesses. They may partner with others. The indicator will count the youth who have actually established their business, but will also include information about those who are in the process of doing so in the narrative. This indicator will also make a preliminary assessment of the sustainability of the businesses created by the EEA youth using the following criteria: - Business capital - Value of sales/services during the first year after creation of business - Annual net benefits - Percentage of loans repaid - Number of jobs created and/or to be created This indicator will be measured based on a set of milestones already identified in the EEA M&E plan. Follow up surveys and project records. The matrix will also track the kind of trainings that youth who have started their business have received through EEA project. Project, Country Indicator 3.4: Number and percentage of youth with satisfactory internships or apprenticeships This indicator tracks the number and percantage of youth who rate their satisfaction with their current employment, internship or apprenticship as 4 or 5 from a scale of 1-5 where 1 is "least satisfactory: and 5 is "most statisfactory". The supporting tool of this indicator will analyze reasons behind any given rating. Follow up surveys, focus group meetings with youth in internships, phone interviews, spot checks and field visits. Project, Country Intermediate Result III: Improved Prospects for Employment and Successful Entrepreneurship Indicators Definition Data Collection Instruments/ Source Level of Reporting Indicator 3.5: Number and percentage of placed youth who declare having a satisfying and quality job This indicator measures the level of satisfaction of youth with the jobs they have acquired through EEA programs or upon completion of such programs. This assessment is mainly subjective as it is based on what the youth consider as "a satisfying and quality job". In order to make this measurement comparable throughout the projects and countries, youth will be asked to rate their level of satisfaction based on a number of criteria. These criteria include: (1) relevance of the job to the field of study of the participant; (2) average salary of the job compared to similar positions held by non-EEA graduates; (3) overall work environment; (4) possibilities for professional and personal growth. The rating will be based on a scale of 1-5 with 1 being very poor to 5 being highly statisfying. Follow up surveys, focus group meetings with placed youth, phone interviews, spot checks and field visits. Project, Country Indicator 3.6: Net salary increase of youth employed through the EEA programs (within six months after placement) This indicator will be calculated as percentage increase in net salaries of youth employed through EEA programs. Salary before employment could be pocket money, scholarship stipends or salary from part time job. Question included in the follow up survey/questionnaire. Project, Country Indicator 3.7: Percentage of youth (those who had jobs prior to joining the EEA program ) who are earning more income as a result of participation in the EEA program This indicator will help determine whether youth who previously worked are able to receive better employment and compensation as a result of participation in an EEA-supported skills training program. Question included in the youth follow-up survey. Project, Country Indicator 4.1: Number and percentage of employed youth financially helping their families for household, health and education expenses This indicator is a proxy measure of youth engagement and productivity at the level of their families. It is also a proxy measure of the positive effect of EEA programs on indirect beneficiaries (parents and siblings). Household survey, question included in the survey, anecdotal reporting - extracted from existing reports (sample basis). Project (where it is applicable), country Indicator 4.2: Number and percentage of youth who have reported improved financial and social conditions of their families It is also a proxy measure of the positive effect of EEA on indirect beneficiaries. Question included in the follow-up survey. Project (where it is applicable), country Intermediate Result IV: Positive Indirect Effect on Families and Communities of Youth Beneficiaries Page 1 EEA Indonesia: Survey A –Alliance Members This is one of three evaluation guides and tools that have been prepared to assess key stakeholders of EEA programs as part of our final evaluation of EEA operations from 2005 to 2008. This set focuses on the organization of Alliances for Youth Education & Employment and the management and technical support provided to it by Country Secretariats. The EEA Alliance Partners Survey & Evaluation seeks to build on the monitoring data that has been collected throughout programs implementation. It seeks to deepen our understanding of program results and how they have contributed to meeting the global objectives. It also hopes to capture the unique features and results of each country program. The survey of Alliance Partners will allow us to do cross￾country comparisons and were designed as generic instruments. Where possible, surveys should be paired with the conduct of focus group discussions among of Alliance members or partners and administered before or during the FGD. 1. Default Section 1. Date: 2. EEA Project: 3. Location of Interview: 4. Name of Interviewer: 5. Name of EEA Partner: 6. Name of Respondent and Position/Title (if different from above): 7. Member of: gfedc National Alliance gfedc Steering Committee gfedc Local Project-based Alliance Page 2 EEA Indonesia: Survey A –Alliance Members 8. Institutional Type (please check one) 9. Alliance Membership Date Joined the Alliance: 10. What is the reason for joining the Alliance? (please rank 1 to 6, with 1 being most important reason) 11. Please describe your role as a member of the Alliance and how you think it contributed to the project. 5 6 12. Did you donate your own resources to the project? nmlkj National Government Agency nmlkj Private Corporation nmlkj Local Government Agency nmlkj Private Institution nmlkj Private Individual nmlkj Private Non-profit gfedc Help youth learn relevant skills gfedc Help youth become employed gfedc Help community growth gfedc Youth is part of our institutions concern gfedc Invited by Reputable Institution gfedc Good Corporate Social Responsibility Other reasons (please specify) nmlkj Yes nmlkj No Page 3 EEA Indonesia: Survey A –Alliance Members 13. Please identify resources provided: 14. If your contribution could be monetized, what is your estimate of the monetized value of your contribution to the Alliance? 15. Was the Secretariat effective in supporting the work of the Alliance? 16. If yes, in what way? Please encircle the top 3 gfedc Funds gfedc Personal Time gfedc Trainers gfedc Goods & Materials gfedc Curriculum Design gfedc Job Opportunities gfedc Linkages & Networks Others Resources Provided: nmlkj Yes nmlkj No gfedc Assisted in developing programs for youth education and employment gfedc Implemented a transparent grant making process gfedc Assisted EEA project implementers start up their projects Implemented a regular project monitoring and evaluation process and communicated project status to members gfedc gfedc Implemented an alliance communications program on project learnings & results Provided technical assistance (such as monitoring and evaluation) on project implementation and alliance building gfedc gfedc Advocated for public support for youth education & employment projects Other (please specify) Page 4 EEA Indonesia: Survey A –Alliance Members 17. Has your involvement in the EEA changed your company/institution in any way? 18. What did your company/institution gain from involvement in the Alliance? 5 6 19. Do you have suggestions to improve implementation of Alliance programs? 5 6 nmlkj yes nmlkj No Please specify Page 1 EEA Indonesia: Survey B - Implementing Partners This is one of three evaluation guides and tools that have been prepared to assess key stakeholders of EEA programs as part of our final evaluation of EEA operations from 2005 to 2008. This set focuses on the organization of Alliances for Youth Education & Employment and the management and technical support provided to it by Country Secretariats. The EEA Alliance Partners Survey & Evaluation seeks to build on the monitoring data that has been collected throughout programs implementation. It seeks to deepen our understanding of program results and how they have contributed to meeting the global objectives. It also hopes to capture the unique features and results of each country program. The survey of Alliance Partners will allow us to do cross￾country comparisons and were designed as generic instruments. Where possible, surveys should be paired with the conduct of focus group discussions among of Alliance members or partners and administered before or during the FGD. 1. Default Section 1. Date: 2. EEA Project: 3. Location of Interview: 4. Name of Interviewer: 5. Name of EEA Partner: 6. Name of Respondent and Position/Title (if different from above): 7. What was the project you implemented? 5 6 8. How would you assess the results of the project? nmlkj Very Poor nmlkj Poor nmlkj Fair nmlkj Good nmlkj Excellent Please discuss 3 project results that support your assessment. Page 2 EEA Indonesia: Survey B - Implementing Partners 9. Please assess the impact of the Alliance approach on your project using a scale of 1-5 with 5 being the highest. 10. (If the implementing partner organized an Alliance in the community, kindly have him/her answer Template C1 – Assessment of the Alliance) Did you develop new partnerships with other institutions to promote youth employability outside of the Alliance structure? If yes, please identify these partnerships: 11. Did you receive technical assistance from the EEA Secretariat? Kindly identify assistance you received? nmlkj 1 nmlkj 2 nmlkj 3 nmlkj 4 nmlkj 5 Why? nmlkj Yes nmlkj No Partners Name Type of partner Purpose of the Partnership nmlkj Project Development nmlkj Project Management nmlkj Technical Training nmlkj Linkages nmlkj Institutional Development nmlkj Project Monitoring & Evaluation nmlkj Financial Management & Controls nmlkj Project Trouble Shooting Page 3 EEA Indonesia: Survey B - Implementing Partners 12. If your staff received technical training, kindly list all trainings received 5 6 13. Please assess the technical assistance you received from the Secretariat 14. What other technical assistance would have helped your project? 5 6 15. Has being involved in an alliance changed the way your organization works? 16. To what extent did the EEA Alliance project influence you to improve your capacities to work with youth? Or become an advocate for Youth Employability projects? 17. How has your organization benefited from your membership in the Alliance? Please identify top 3 benefits to your rganization. 5 6 gfedc Poor gfedc Fair gfedc Good gfedc Excellent Why? nmlkj Yes nmlkj No nmlkj Not at all nmlkj To some extent nmlkj To a great extent Please elaborate. Page 4 EEA Indonesia: Survey B - Implementing Partners 18. How have you personally benefited by your membership in the Alliance? Please identify top 3 benefits? 5 6 19. Have you implemented/or are implementing other development projects that do not use the Alliance approach? 20. How would you describe your experience with the Alliance approach in comparison to other traditional development projects? 21. Do you have suggestions on how to improve the implementation of Alliance programs for youth employability? 5 6 nmlkj Yes nmlkj No nmlkj Worse nmlkj Similar nmlkj Better nmlkj Much Better Please explain why Page 1 EEA Indonesia: Survey C – National EEA Secretariat 1. Default Section 1. Date: 2. Name of Respondent & Position/Title: 3. Name of the National Alliance: 4. Location: 5. Interviewed by: 6. When was the Alliance organized? 7. Number of Alliance Members: 8. Profile of Alliance Members (Please attach a list of all Alliance Members, their institutional affiliation and roles/responsibilities) 9. Organization of the Alliance: 10. If formal, did a Memorandum of Agreement define the roles of each member? nmlkj Private Business Sector nmlkj Government nmlkj Academic Other (please specify) nmlkj Formal nmlkj Informal nmlkj yes nmlkj No Page 2 EEA Indonesia: Survey C – National EEA Secretariat 11. If informal, were partners’ roles and responsibilities defined? Please elaborate how roles were defined. 5 6 12. What was the role of IYF-EEA? 5 6 13. Please assess their support to your operation as a National Secretariat. 14. Is the Alliance still organized and functioning? 15. If the Alliance is continuing to function, what are the top 3 reasons why the alliance work has been sustained and has been of value to project success? 5 6 16. If no, kindly elaborate on why it no longer functions. Would you still use the Alliance approach in implementing youth employability projects and why? 5 6 nmlkj Very Poor nmlkj Poor nmlkj Fair nmlkj Good nmlkj Excellent Why? nmlkj Yes nmlkj No Comments Page 3 EEA Indonesia: Survey C – National EEA Secretariat 17. Assessing the EEA Alliance: Please check the observation that applies to your experience in the EEA Alliance project. Please assess each Alliance that was organized through EEA, whether National or Local. Yes No Needs improvement The Alliance is well established, developed and managed by a permanent body (association, steering committee, secretariat, or other), with staff dedicated to EEA work nmlkj nmlkj nmlkj A grant making process was established that guided the process of selecting the best grantees/projects to receive EEA grants nmlkj nmlkj nmlkj Alliance members (Advisory Board Members, Stakeholders, other) participate actively to EEA meetings and activities nmlkj nmlkj nmlkj An internal monitoring system is in place, sub-grants are being monitored regularly, and regular follow up was conducted on achieved activities versus planned, through field visits and meetings with implementers nmlkj nmlkj nmlkj Alliance members are using their networking capacity to develop partnerships between the private sector, government and civil society actors to facilitate youth placement nmlkj nmlkj nmlkj A communication system is in place to promote EEA programs nmlkj nmlkj nmlkj Page 4 EEA Indonesia: Survey C – National EEA Secretariat and leverage funding National and local communication and outreach events are participated in by Alliance Members to promote EEA programs and activities and advocate for youth employability initiatives nmlkj nmlkj nmlkj Alliance members advocate for the development of a policy framework that enhances youth employability (Leadership) through a sound communication strategy nmlkj nmlkj nmlkj Alliance members assisted implementing partners with start up of projects nmlkj nmlkj nmlkj Alliance members provided monitoring and evaluation technical assistance to implementing partners nmlkj nmlkj nmlkj Alliance members helped with trouble shooting problems and obstacles facing project implementation nmlkj nmlkj nmlkj The Alliance strategy was more cost effectiveness compared with single donor projects nmlkj nmlkj nmlkj Implementing partners found assistance from Alliance/Secretariats helpful nmlkj nmlkj nmlkj The Alliance influenced partners to improve their institutional capacities nmlkj nmlkj nmlkj Alliance implementing improved their capacities to work with youth employability projects nmlkj nmlkj nmlkj Page 5 EEA Indonesia: Survey C – National EEA Secretariat 18. Have other organizations adopted the alliance approach for their youth employability programs? 19. Do you have suggestions to improve implementation of the future Alliance programs for youth employability? 5 6 Please specify what improvements are needed nmlkj Yes nmlkj No Please identify these organizations. Page 1 EEA Indonesia: Survey D - Youth Beneficiaries This instrument will help collect and organize information obtained from the observations in individual interviews or in a focus group meeting with youth who have participated in EEA supported programs. This section is targeted only to those youth who have completed EEA training programs (composed of one or more training courses, either in life skills, technical training, employability training or other). Depending on training received, questions may be dropped if not relevant. 1. SECTION I: PERSONAL DATA 1. Name of project 2. Participant's name 3. Gender 4. Age 5. Marital status 6. Education level 7. If you have left school before completing your studies, what are the reasons why you left school or college? 2. SECTION II: Training Evaluation nmlkj Male nmlkj Female nmlkj Below 18 nmlkj 18-24 nmlkj Above 24 nmlkj Single nmlkj Married with __children nmlkj Divorced/Widowed with __children Page 2 EEA Indonesia: Survey D - Youth Beneficiaries 1. Please rank your satisfaction with the program: Very poor Poor Fair Good Excellent N/A Overall content of the training program nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Quality of training materials nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Relevance of content and training materials to employment needs nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Interactive/practical exercises and activities (if any) nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Life Skills and attitudes learned from training (if any) nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Technical skills acquired from training nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj 2. Please list 3 or 4 most important skills or knowledge you have gained. If you do not think you learned new knowledge or skills, please explain why. 5 6 3. Please rate program trainers/counselors/facilitators in the following areas. Very poor Poor Fair Good Excellent N/A Overall performance nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Technical knowledge (if applicable) nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Counseling capacities (if applicable) nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Capacity to provide mentorship and career counseling nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Capacity to keep you motivated and engaged nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj 4. What is the aspect or component you liked most about this program? 5 6 Please explain your answers: 5 6 Other comments 5 6 Page 3 EEA Indonesia: Survey D - Youth Beneficiaries This section is targeted to youth who have completed at least one month internship as part of the EEA program. 5. What is the aspect or component you liked the least in this program 5 6 6. What changes do you think should be made to make the program more useful/efficient? 5 6 3. SECTION III: Internship Evaluation 1. Have you been placed in an internship or apprenticeship? 2. if yes, please rate the value of your internship or apprenticeship. Very poor Poor Fair Good Excellent N/A Overall internship experience nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Relevance of assignments to the training received nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Technical skills learned nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Coaching received during internship nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj 3. What are the most useful things you learned from your apprenticeship/internship. If your internship was not useful please explain why. 5 6 nmlkj Yes nmlkj No Other (please specify) 5 6 Please explain: Page 4 EEA Indonesia: Survey D - Youth Beneficiaries THIS SECTION IS ONLY FOR EMPLOYED, IF NOT WORKING OR SEFL-EMPLOYED, PLEASE SKIP TO NEXT SECTION. This section is targeted to youth who have completed EEA supported training programs or benefited from EEA supported employment support programs, and have either been placed in jobs through the EEA programs, or managed to find a job thanks to the skills and competencies gained through EEA programs. 4. How could your internship/apprenticeship be made more useful? 5 6 5. What are you doing now after your internship? 4. SECTION III-B: Job Evaluation 1. Did you receive any counseling, mentoring or job placement services from EEA program to help you find a job? 2. If yes, how would you rate the counseling/mentoring services you have received? Very poor poor Fair Good Excellent The services I received were: nmlkj nmlkj nmlkj nmlkj nmlkj 3. Did you have a job before participating in the EEA program? nmlkj Looking for a job nmlkj Got a job and currently working nmlkj Waiting to set up a small business nmlkj Currently working as an entrepreneur nmlkj Yes nmlkj No Please explain: 5 6 nmlkj Yes nmlkj No If yes, what kind of job did you have and how much did you earn? Page 5 EEA Indonesia: Survey D - Youth Beneficiaries 4. Do you earn more money as a result of your participation in the EEA program? 5. If you were working prior to your participation in the EEA program, and you are still working with the same employer, did your status or salary improve after you completed the training? 6. If you were working prior to your participation in the EEA program, and you changed your job after completion of the EEA program, how would you rate your new job in comparison to the previous one? Much worse Worse Similar Better Much Better N/A Overall job experience/satisfaction nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Salary level nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Other compensation nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Professional growth opportunities nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj 7. How many months after completing the training program did you find a job? nmlkj Yes nmlkj No If yes, how much more do you earn now? nmlkj Yes nmlkj No nmlkj N/A If you got a salary increase, please specify how much per month: nmlkj 1 to 3 months nmlkj 4 to 6 months nmlkj More than 6 months Page 6 EEA Indonesia: Survey D - Youth Beneficiaries 8. Are you happy with your current job and level of income and other types of compensation provided? Very unhappy Unhappy Happy Very happy Overall work environment nmlkj nmlkj nmlkj nmlkj Relevance of the job to the field of study nmlkj nmlkj nmlkj nmlkj Support from supervisor and/or peers nmlkj nmlkj nmlkj nmlkj Possibilities for professional growth nmlkj nmlkj nmlkj nmlkj Salary nmlkj nmlkj nmlkj nmlkj Other compensation nmlkj nmlkj nmlkj nmlkj 5. Section IV: Indirect Impact 1. If you are presently working, what do you do with your earnings? (mark all choices that apply) 2. Did your support help improve your family's financial and social situation? If you consider that your job is not satisfying, please state why gfedc Share them with my spouse and children gfedc Help my parents and siblings gfedc Spend them for my personal uses gfedc Save some for future uses gfedc No comments Other (please specify) nmlkj Yes nmlkj No nmlkj Somewhat If so, in what way? 5 6 Page 7 EEA Indonesia: Survey D - Youth Beneficiaries 3. Did you decide to continue your education or engage in further professional training as a result of participating in the EEA program? 4. Where did you hear about this program? 5. Have you referred another youth to the program? 6. Will you refer other youth to this program? 6. SECTION V: Change of participant's capabilities and attitudes nmlkj Yes nmlkj No please specify 5 6 nmlkj From a friend or a family member who participate(d) in the program nmlkj From local government nmlkj From my former school nmlkj From a youth association nmlkj From the media(newspaper, radio, TV, leaflets) nmlkj From the internet Other (please specify) nmlkj Yes nmlkj No If yes, how many? nmlkj Yes nmlkj No Why? Page 8 EEA Indonesia: Survey D - Youth Beneficiaries This section of the survey tracks the percentage of young men and women demonstrating improved capabilities through participation in EEA supported programs. This is based on a self assessment by the trainees themselves of the technical and/or soft skills gained through EEA programs. This section is to be completed by youth who have completed either life skills training, technical training, employability training, or a combination of or more of such training programs. Depending on the training received, some questions may be skipped if not relevant. 1. Did you gain any technical knowledge and skills needed for employment? 2. What are the most important technical skills or knowledge gained? List 3 or 4. 5 6 3. Have you received soft skills/life skills training through supported programs? If your response to this question is No, please skip to question # 17. 4. How would you describe your communication skills prior to the training? nmlkj Yes nmlkj No If not, specify why nmlkj Yes nmlkj No gfedc Very poor gfedc Poor gfedc Fair gfedc Good gfedc Excellent Other (please specify) 5 6 Page 9 EEA Indonesia: Survey D - Youth Beneficiaries 5. Did your communication skills improve as a result of the training? 6. How would you describe your ability to listen to people and to respond to them? nmlkj No, there has been no change nmlkj Yes, there has been some improvement nmlkj Yes, there has been good improvement nmlkj Yes, there has been a huge improvement nmlkj No comment Other (please specify) 5 6 nmlkj Very poor nmlkj poor nmlkj Fair nmlkj Good nmlkj Excellent Other (please specify) 5 6 Page 10 EEA Indonesia: Survey D - Youth Beneficiaries 7. Did your ability to listen to people and to respond to them improve as a result of the training? 8. How would you describe your self-confidence prior to the training? nmlkj No, there has been no change nmlkj Yes, there has been some improvement nmlkj Yes, there has been good improvement nmlkj Yes, there has been a huge improvement nmlkj No comment Other (please specify) 5 6 nmlkj Very weak nmlkj Weak nmlkj Fair nmlkj Good nmlkj Excellent Other (please specify) 5 6 Page 11 EEA Indonesia: Survey D - Youth Beneficiaries 9. Did your self-confidence improve as a result of participating in EEA supported programs? 10. How would you describe your sense of initiative prior to completion of training? nmlkj No, there has been no change nmlkj Yes, there has been some improvement nmlkj Yes, there has been good improvement nmlkj Yes, there has been a huge improvement nmlkj No comment Other (please specify) 5 6 nmlkj Very poor nmlkj poor nmlkj Fair nmlkj Good nmlkj Excellent Other (please specify) 5 6 Page 12 EEA Indonesia: Survey D - Youth Beneficiaries 11. Did your sense of initiative improve as a result of the training? 12. How would you describe your sense of responsibility prior to the training? nmlkj No, there has been no change nmlkj Yes, there has been some improvement nmlkj Yes, there has been good improvement nmlkj Yes, there has been a huge improvement nmlkj No comment Other (please specify) 5 6 nmlkj Very poor nmlkj poor nmlkj Fair nmlkj Good nmlkj Excellent Other (please specify) 5 6 Page 13 EEA Indonesia: Survey D - Youth Beneficiaries 13. Did your sense of responsibility improve as a result of the training? 14. How would you describe your ability to resolve conflicts before completing the training? nmlkj No, there has been no change nmlkj Yes, there has been some improvement nmlkj Yes, there has been good improvement nmlkj Yes, there has been a huge improvement nmlkj No comment Other (please specify) 5 6 nmlkj Very poor nmlkj poor nmlkj Fair nmlkj Good nmlkj Excellent Other (please specify) 5 6 Page 14 EEA Indonesia: Survey D - Youth Beneficiaries 15. Did your ability to resolve conflicts improve as a result of the training? 16. How would you describe your vision of your future prior to the training? 17. Do you feel more positive about your future as a result of participation in the EEA program? nmlkj No, there has been no change nmlkj Yes, there has been some improvement nmlkj Yes, there has been good improvement nmlkj Yes, there has been a huge improvement nmlkj No comment Other (please specify) 5 6 nmlkj Had no vision at all nmlkj Knew vaguely what I wanted to do/to be nmlkj Know well what I wanted, but did not think I could attain it nmlkj Knew well what I wanted, and knew how to attain it Other (please specify) 5 6 nmlkj Yes nmlkj No nmlkj Somewhat Changes noticed/reported by participants 5 6 Page 15 EEA Indonesia: Survey D - Youth Beneficiaries 18. Are you willing to pursue further education/vocational training? 19. How would you describe your relationships with family and friends prior to the training? gfedc No, I am already working gfedc No, I would like to work immediately gfedc No, I want to start my own business gfedc Yes, I want to pursue further education/vocational training exlusively gfedc Yes, I want to pursue further education/vocational training and work at the same time gfedc Yes, I am already registered to pursue further education/vocational training Other (please specify) 5 6 nmlkj Very poor nmlkj poor nmlkj Fair nmlkj Good nmlkj Excellent Other (please specify) 5 6 Page 16 EEA Indonesia: Survey D - Youth Beneficiaries This section applies only to those participants who have started a new business after completion of an EEA supported program. 20. Did you notice any changes in relationships with your family and friends after completing the program? 21. Had you not participated in this program, what would you have been doing instead? 7. SECTION VI: Entrepreneurship Development 1. Did you receive any entrepreneurship training and business support? nmlkj No, there has been no change nmlkj Yes, there has been some improvement nmlkj Yes, there has been a huge improvement nmlkj No comment If yes, in what way? 5 6 gfedc Nothing gfedc Would have gone to the street/stayed in the street gfedc Would have looked for other training opportunities gfedc Would have stayed at home and grown depressed gfedc Would have tried to go back to school gfedc Would have continued working in the informal sector gfedc No comment Other (please specify) 5 6 nmlkj Yes nmlkj No Page 17 EEA Indonesia: Survey D - Youth Beneficiaries 2. If yes, please rate the training/coaching services you have received: Very poor Poor Fair Good Excellent N/A Entrepreneurship development/business development training was: nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Entrepreneurship development/business development coaching was: nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj 3. Did you gain entrepreneurial knowledge and skills needed to establish a business from the EEA program? 4. Did the program help you feel prepared and empowered as a young entrepreneur? 5. What kind of financial and business coaching support did you receive to start your business? 5 6 If very poor or excellent, please explain why 5 6 nmlkj Yes nmlkj No If not, please explain why. If yes, list 3 or 4 skills you have gained or knowledge you have acquired 5 6 nmlkj Yes nmlkj No Please explain why. Page 18 EEA Indonesia: Survey D - Youth Beneficiaries 6. If you have received financial and business support to start a small business or group business, what do you think of the support you have received? Very poor Poor Fair Good Excellent N/A Financial support received was: nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Business support received was: nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj 7. Please indicate whether you started your own business or are a part of a business group. 8. What is your business capital (optional) 9. Are you able to cover your business expenses with your earnings? 10. Are you making a profit? Please explain why: 5 6 nmlkj My own business nmlkj Part of Business Group nmlkj Less than $500 nmlkj $500 - $750 nmlkj $750 - $1000 nmlkj Above $1000 nmlkj Yes nmlkj No Please elaborate: nmlkj Yes nmlkj No If yes, please specify your monthly net profits: Page 19 EEA Indonesia: Survey D - Youth Beneficiaries 11. Did you hire any employees? 12. Would you have been able to open your own business without EEA assistance and/or training? 13. Did you have a job before joining the EEA program? 14. If you had a job previously, are you making more money now with your new business? gfedc Yes gfedc No If yes, how many? nmlkj Yes nmlkj No Please explain why: nmlkj Yes nmlkj No If so, what kind of job did you have and how much did you earn? nmlkj Yes nmlkj No If yes, please explain how much more: Page 1 EEA Indonesia: Survey E - Employer The employer’s survey questions should preferably be submitted to the respondent ahead of the interview. During the interview itself, the interviewer can use the survey answers to generate a free discussion of their experience with EEA Alliance and with the EEA youth working in their company. Should the interviewer be unable to get the respondent to answer the questionnaire in advance, then he/she should allot sufficient time to be able to administer the questionnaire and generate a discussion with the respondent. The interviewer needs to take care not to bias the answers of the respondent and attempt to capture their responses in the most accurate way possible. 1. Default Section 1. Country: 2. Date: 3. Company name 4. Location 5. Type of Business: 6. Contact Person’s Name & Title: 7. Email or Phone: 8. How did you know about EEA program? 9. What do you know about EEA program? 5 6 nmlkj EEA partners approached my company directly nmlkj Advertisement in a newspaper/magazine nmlkj Project brochures and posters nmlkj Someone/company referred you to the program nmlkj Other Other (please specify) Page 2 EEA Indonesia: Survey E - Employer 10. Through EEA program, please tell us how many youth your company has employed: 11. Through EEA program, please tell us how many youth your company has offered internship/apprenticeship to? 12. How did your company recruit EEA youth for internships or employment opportunities? 13. Please rate the overall performance of EEA graduates Very poor Poor Fair Good Excellent N/A overall performance of EEA graduates nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj 14. How would you rate overall technical skills of EEA graduates who are employed by or have interned with your company? Very poor Poor Fair Good Excellent N/A overall technical skills of EEA graduates nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj 15. How would you rate overall life (soft) skills of EEA graduates who are employed by or have interned with your company? Very poor Poor Fair Good Excellent N/A life (soft) skills of EEA graduates nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj Through EEA partner NGO sending CVs of candidates nmlkj Through company outreach and competency tests given at the training site nmlkj Through internship or job fairs nmlkj Through internship or job applications sent by the youth directly nmlkj Through an NGO (EEA partner) website/portal Other (please specify) Please explain why Please explain why and list 3 or 4 most important or appreciated skills. Please explain why and list 3 or 4 most important soft skills for youth employability. 5 6 Page 3 EEA Indonesia: Survey E - Employer 16. If you are asked to compare EEA graduates with your other employees of similar age range and education background, how would you rate them? Much worse Worse Same Much better The best N/A Compared to other employees, EEA graduates are: nmlkj nmlkj nmlkj nmlkj nmlkj nmlkj 17. Do the technical skills and work attitudes of EEA graduates (knowledge hard/soft skills) meet your expectations or recruitment needs? 18. How can they improve their technical and soft skills if applicable? 5 6 19. In your opinion, do you think EEA training is closing the gap between employers’ needs and the employees’ qualifications? (i.e. providing qualified candidates to the business sector meeting the market needs). 20. Would you consider offering jobs, internships or apprenticeships to EEA graduates in the future? Please explain why: 5 6 nmlkj Yes nmlkj No Please explain why 5 6 nmlkj Yes nmlkj No Please explain how 5 6 nmlkj Yes nmlkj No Please explain why 5 6 Page 4 EEA Indonesia: Survey E - Employer 21. Please explain whether your EEA experience has any impact on your approach their HR approach in recruiting youth for internships or jobs. 22. What has your company gained from being involved in this program? 5 6 23. Do you have other suggestions to improve this program? 5 6 nmlkj The company plans to become more involved in alliance programs focusing on youth employability nmlkj The company is more open to the idea of providing internships and mentorship support to youth in general nmlkj The company is likely to recruit youth through EEA programs and other similar programs nmlkj No change in plans nmlkj Other Please explain why 5 6 ©2010 International Youth Foundation. All rights reserved. No portion of this report may be reproduced in any form without the written permission of IYF. This project was made possible by the generous support of the American people through the United States Agency for International Development (USAID) under grant agreement no. RAN-A-00-04-00043-00 This report was funded in part under grant agreement no. EPP-A-00-08-00006-00. The International Youth Foundation (IYF) invests in the extraordinary potential of young people. Founded in 1990, IYF builds and maintains a worldwide community of businesses, governments, and civil-society organizations committed to empowering youth to be healthy, productive, and engaged citizens. IYF programs are catalysts of change that help young people obtain a quality education, gain employability skills, make healthy choices, and improve their communities. To learn more, visit www.iyfnet.org ©2010 International Youth Foundation. All rights reserved. No portion of this report may be reproduced in any form without the written permission of IYF. This project was made possible by the generous support of the American people through the United States Agency for International Development (USAID) under grant agreement no. RAN-A-00-04-00043-00 This report was funded in part under grant agreement no. EPP-A-00-08-00006-00. The International Youth Foundation (IYF) invests in the extraordinary potential of young people. Founded in 1990, IYF builds and maintains a worldwide community of businesses, governments, and civil-society organizations committed to empowering youth to be healthy, productive, and engaged citizens. IYF programs are catalysts of change that help young people obtain a quality education, gain employability skills, make healthy choices, and improve their communities. To learn more, visit www.iyfnet.org