Fieldwork completed in June 2005 Mid-Term Evaluation Junior Achievement/Youth Enterprise Serbia Evaluation team Ilirjana Dana USAID/Albania Danijel Dasic USAID/Serbia and Montenegro Djordje Boljanovic USAID/Serbia and Montenegro July 21, 2005 USAID/Serbia Junior Achievement /Young Enterprise Evaluation 1 TABLE OF CONTEST EXECUTIVE SUMMARY.....................................................................................................3 INTRODUCTION ...................................................................................................................5 JUNIOR ACHIEVEMENT PROGRAM OBJECTIVES ....................................................7 PURPOSE OF EVALUATION ..............................................................................................9 METHODOLOGY..................................................................................................................9 FINDINGS..............................................................................................................................11 1. PROJECT ACHIEVEMENTS..............................................................................................11 2. MANAGEMENT AND IMPLEMENTATION.........................................................................15 CONCLUSIONS....................................................................................................................16 1. GENERAL CONCLUSIONS ................................................................................................16 2. PERFORMANCE AND IMPLEMENTATION.........................................................................16 RECOMMENDATIONS ......................................................................................................18 Annexes: A - Scope of Work B – Evaluation Plan C - Evaluation Agenda Phase II D - Junior Achievement Serbia – Survey Results E - List of Interviewees F – Evaluation Questions for survey with JA students G - Key documents reviewed USAID/Serbia Junior Achievement /Young Enterprise Evaluation 2 Acronyms ACDI/VOCA Agricultural Cooperatives Development International/Volunteers in Overseas Cooperative Assistance ADF America’s Development Fund AOR Area of Responsibility CC Community Committee LG Local Government EMIS Europian Movement in Serbia EEE Group Enabling Economic Environment Group for Local Economic Development - CHF AoR CHF Cooperative Housing Foundation CoP Chief of Party CRDA Community Revitalization through Democratic Action FO Field Offices GDO General Development Office IRD International Relief and Development JA &YE Junior Achievement and Youth Enterprise JAW Junior Achievement Worldwide MCI Mercy Corps International MWG Municipal Working Group SLGRP Serbian Local Government Reform Program SoW Scope of Work USAID US Agency for International Development CDC Community Development Council CDG Community Development Group DAI Development Alternatives International MZ Mesna Zajednica (Local Communities) IDP Internally Displaced Person PRS Project Reporting System RH Reproductive Health USAID/Serbia Junior Achievement /Young Enterprise Evaluation 3 Executive Summary USAID E&E bureau expressed interest in evaluating performance and values of Junior Achievement programs in various countries, as part of an evaluation training, in an effort to promote and revitalize the monitoring and evaluation expertise within the agency. This report represents findings, opinions and lessons learned on the effectiveness of the JA program in Serbia. The Junior Achievement Program in Serbia has some characteristics that limit the possibility of measuring the performance of the program as a whole, in all the regions, at all times. JA program in Serbia was launched in 2003 and it is a sub-grant under umbrella of CRDA – Community Revitalization through Democratic Action, a large-scale civil society program, divided geographically between five implementing partners. Three of five implementing partners are currently involved into JA activities, but all of them are in different stages of implementation. JA program in Southeast Serbia implemented by CHF is in its second year; in Central Serbia the program is implemented by ACDI-VOCA and is in the first year; and third implementing partner – Mercy Corps just started the JA program. JA Program is an extracurricular activity in high schools in Serbia. High-school students gain basic knowledge in market economy, as well as skills that will provide them with better understanding of private business, both in theory and in practice. The program has reached almost 2,650 students and teachers in the geographic areas mentioned in this report The program is being implemented through six phases: • JA introductory workshops • JA high schools’ selection • JA teachers’ trainings • JA Applied Economics • JA Company Program • MESE competition Evaluation field work was performed and completed during the week of June 12th -18th, 2005 and was focused on the CHF and ACDI/VOCA areas. The Evaluation Team focused on four municipalities due to time limitations - Vranje, Leskovac, Bujanovac and Presevo. The idea was to focus on these Central and Southern Serbia municipalities due to similarities in economic development. The selection process was based on multiple categorizing: JA status, economic status, schools’ technical capacity, ethnic, urban/rural differences. Additional criteria in selection of specific schools include: various performances, technical capacity, urban/rural, classical vs. vocational schools. The Evaluation Team conducted key informant interviews with almost 50 students and mini survey with 96 JA program students. The field work was divided in three phases: a) visit to USAID Serbia Program and GDO offices; b) interviews with direct beneficiaries in the field and; c) interviews with the Ministry of Education and JA Serbia representatives. This evaluation finds that JA program was very well received in high schools in Serbia and was perceived as substantial improvement when compared to the “old school” curricula and methodology. Students were highly satisfied with the JA approach to teaching and learning. However, the challenge for the JA program, given the way of funding is a better coordination with the implementers of CRDA and USAID mission in Serbia. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 4 Based on the evaluation findings, the JA Serbia team recommends the following: ÿ USAID should support the JA program through direct funding and address long term challenges instead of annualized specific focus. ÿ JA should engage in discussion with the Ministry of Education to consider the student’s associations as potential tool for sustainability of program. ÿ JA Serbia should initiate an awareness campaign of their program to create good relationship with local businesses and communities. ÿ Monitoring and reporting procedures should be improved especially in the fields of internal coordination, guidance, reporting and cooperation between JA representatives and CRDA implementing partners. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 5 Introduction In the 1990's, during Milosevic's rule the people of Serbia suffered cycles of economic and social instability that resulted in severely decreased production, low wages, high unemployment and under-employment, especially in the rural areas. Gross domestic product dropped by two-thirds over the period and unemployment grew dramatically. Rapid inflation surges during the past decade severely reduced the real value of financial assets. The elderly, the handicapped and single parent families, who were often without substantial financial or physical assets, were especially hard hit and particularly vulnerable, since pensions and other social welfare payments shrank to low levels and often were not paid on time. The quality and capacity of social service delivery mechanisms and institutions, such as the health system, public utilities and other public services in Serbia, have been undermined during the past decade by economic decline, a lack of investment, and the increased demand of a large refugee population. Insufficient and often inadequately trained health and social service personnel have had few resources with which to work. Infrastructure has deteriorated severely due to lack of investment and a social policy that kept public service fees unrealistically low and left communal enterprises with insufficient funds for proper operation and maintenance. Although the socioeconomic situation is serious, there is reason for cautious optimism. In the 1980s Yugoslavia's industrial and agricultural sectors were among the most sophisticated in Eastern Europe. These sectors still have significant assets, but need to be revitalized. Given the country’s rich natural and human resources, USAID/FRY/Serbia felt that community mobilization offered a promising opportunity to build on the democratic revolution in Serbia, and create a momentum for social and economic development at the local level. This drove the design of the Community Revitalization through Democratic Action – CRDA program. This report summarizes interviews and data analysis with project beneficiaries during field trip in June 2005 in Central and Southern Serbia as well as interviews conducted with representatives of USAID Serbia, implementing partners and Junior Achievement. These areas were selected because of the visible difference in economic development, different ethnic minorities present in visited communities and presence of two American PVOs (ACDI/VOCA and CHF) which are implementing USAID funded CRDA program in respective communities. The JA program is a sub grant to the CRDA program – a large scale civil society program that uses community development activities to build trust between different ethnic and religious groups, to demonstrate the value of citizen participation, to support grass roots democratic action and to bring immediate improvement in people’s living conditions. There are four categories of projects that are eligible for funding under the CRDA program. They are projects involving: 1) civic participation; 2) community infrastructure; 3) income generation and; 4) environmental improvement activities. The JA program contributes to the income generation category. The emphasis of the income generating activities is to put money in people’s pockets rather than to resolve macro-level economic policy issues. Training and technical assistance were defined as eligible activities that could be made available when necessary. The JA program in Serbia is mostly related to USAID CRDA economic pillar since it aims to produce a qualified workforce, future business people, who will in the long run, generate income and create jobs in their respective USAID/Serbia Junior Achievement /Young Enterprise Evaluation 6 communities. The Junior Achievement program prepares young people to be workforce ready, which endorses Serbia in making the transition to a market economy. Furthermore, Junior Achievement program creates a pro-business climate, which will stimulate economic development and the process of economic reforms. CRDA is divided geographically between five implementing partners. Three of five implementing partners are currently involved into JA activities, but all of them are in different stages of implementation. JA program in Southeast Serbia implemented by CHF is in its second year; in Central Serbia the program is implemented by ACDI-VOCA and is in the first year; and third implementing partner – Mercy Corps just started the JA program JA. The JA program started in 2003 for the first time and was implemented by CHF. The annual average funding is $60,000 per implementer. Currently, nearly 2,600 students and teachers actively participate in Junior Achievement program, implemented in 16 Central Serbian municipalities. Prior to Central Serbia, the Junior Achievement program has already been implemented in more than 30 high schools of Eastern and Southern Serbia. Through extra curricular activities, high-school students from thirty high schools gain basic knowledge in market economy, as well as skills that will provide them with better understanding of private business, both in theory and in practice. In collaboration with their teachers, high-school students establish student enterprises to, based on the simulation of doing business and adhering to all the rules, understand the way things function and to try to “survive” in the market. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 7 This report assesses USAID the program performance and implementing partners’ approaches and achievements in Junior Achievement/Youth Enterprise program that was initiated two years ago in CHF Area of Responsibility while JA has been present in ACDI/VOCA for one year only. The third CRDA implementing partner, Mercy Corps International was not part of this evaluation since their program is at start up phase. This report will present overall findings, conclusions, recommendations and lessons learned answering the following questions: 1. Does JA’s model meet the education/entrepreneurship needs of Serbia? 2. Are there differences in implementation according to urban/rural, geographic area, ethnic group, gender? 3. Have students attitudes/behaviors towards business/entrepreneurs changed as a result of their participation in JA and if so how? 4. What is a quality of relationship forged between business community and JA participants? 5. How does JA define program and financial sustainability and is sustainability being achieved? Junior Achievement Program Objectives Serbia’s transition from a closed and isolated economy to an open and market-driven one finally got underway in 2000. With the rapid changes that Serbia is going through in the process of democratization and transition to a market oriented economy, the demand for a better understanding of market economics is rising. However, current high school education does not provide the students with the necessary business skills and knowledge that would enable them to become a competitive workforce in the free market economies and the global market place. Consequently, young people are not capable of understanding and appreciating the reform process, the importance of entrepreneurship, business ethics and other values of a market-oriented economy and democratic society. In this situation it is difficult for a pro￾business environment to be created because critical economic thinking is not supported and the entire reform process is inhibited. Junior Achievement Serbia (JAS) started as an organization incubated by European Movement in Serbia (EMIS). It was established in partnership with EMIS through an Operating Agreement, which took effect in October 2002, between Junior Achievement International and EMIS. The purpose of JA program is to provide the students with an opportunity to participate in economic and entrepreneurial education and apply its principles in local, regional and international programs and competitions offered by Junior Achievement. Furthermore, by building competitive workforce, JA program will foster entrepreneurship, teach business literacy skills, endorse a market-oriented and democratic society, encourage work and business ethics and promote critical economic thinking on the part of the youth of Serbia. Hopefully, this will have a positive impact on improving the quality of life in Serbia, generate greater opportunities for the young people of Serbia, progress teaching methodologies, strengthen private sector and create a pro-business environment. Following is some indicators reflecting Serbia’s education situation. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 8 MAIN INDICATORS ON EDUCATION IN SERBIA Number Primary Schools Secondary (High) Schools Faculties & Junior Colleges Schools 3,967 528 207 Classes 35,578 11,304 / Students 826,812 325,845 182,209 Teachers 18,642 19,424 10,765 Source: Statistical Yearbooks of Serbia Junior Achievement Program Implementation Junior Achievement program is organized as an extracurricular activity in high schools in Serbia. It consists of two regular programs and two ad hoc programs. The two regular JA programs are Applied Economics and Company Program, while two ad hoc programs are MESE competition and Business in Schools. Classes are organized once a week and usually last 90 minutes. Junior Achievement program in Serbia has six phases in implementation: Phase 1 – Junior Achievement Introductory Workshops Phase 2 – Junior Achievement High Schools’ Selection Phase 3 - JA Teachers’ Trainings Phase 4 - Applied Economics Applied Economics is a one semester subject in which basic elements of free market economics are being presented to the students. The students learn more about demand and supply, forming of prices in the free market economy, marketing, financial markets, stocks and bonds, labor market etc. Phase 5 - Company Program The Company Program is a subject within which the theoretical knowledge acquired through studying Applied Economics is applied to real life business situations. The students form and legally register their own companies, select the management board, raise start-up funds, start production and place their product on the market, offer services etc. Phase 6 - MESE Competition MESE is a subject based on computer software which allows students to make strategic business decisions related to prices of their products, investing in marketing and development, increasing production capacities etc. Furthermore, MESE brings together Junior Achievement students from all around the world in a big international MESE competition that is organized every year. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 9 Purpose of Evaluation The purpose of this evaluation is to asses the performance of the JA program in Serbia and to see how it fits the countries needs as it relates to opening of markets worldwide and shifts from centrally planned economies to market-oriented ones. The report is designed to respond to 5 questions that were posed in the Scope of Work (Annex A). Analyses were carried out and recommendations were provided in response to those 5 questions. While addressing the questions the team took in consideration some more specific items that came out during the interviews and meetings. Following is a set of them: a) How does the JA program serve as a potential tool for conflict mitigation in south Serbia? This relates to the fact that the program offers a social perspective in its entrepreneurial spirit besides the business education one. Given that south Serbia area has potential for ethnic conflicts the program offers a social perspective that engages youth and forges the idea of this program being also a civil society one. b) Is the program structured and well designed to address the needs of students in different schools, general high schools, technical and vocational schools? c) Are summer schools a good way to improve the effectiveness of the JA program given time and space constrains? d) Is JA program in Serbia more oriented toward development of business or entrepreneurial skills? Methodology The team selected to undertake the evaluation of the JA programs was composed of three specialists with different professional backgrounds, program and project management. The team followed data collection and analysis methods common to USAID. The following is a set of tools used for data collection in the course of this evaluation: · Program and project documentation was carefully reviewed; · All relevant stakeholders that participated in the program were interviewed. These include: field staff of USAID, grant recipients (JA Serbia), regional directors and coordinators, teacher and students; · A survey questionnaire was developed and distributed to students to asses their attitudes and interests towards the program. · Briefings and key informant interviews were conducted with representatives of GDO and Program office to get their opinion on programmatic issues; · 5 municipalities outside Belgrade were visited where most of the interviews took place; Kragujevac, Vranje, Bujanovac, Presevo, Leskovac. Also, representatives of the business community were interviewed in Vranje. · Government counterparts from the Ministry of Education were consulted and interviewed. The selection process for choice of municipalities was based on multiple categorizing, such as: current status of JA program, economic status, technical capacity of schools for JA implementation, existence of ethnic, gender, urban/rural differences, etc… Also, the team tried to select schools with various performances in last period, and to create credible sample as a mixture of good and bad performance, proactive and indifferent teachers and principals, urban and rural environment, gymnasiums and vocational schools, etc. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 10 Nevertheless, questionnaires were disseminated in electronic form through JA Serbia network, and JA Serbia Evaluation Team received back 96 questionnaires from 10 schools. Four of them are in the second year of implementation – Leskovac and Presevo Schools and six are in the first year. Survey is well balanced also on the type of school issue: three classic High (Gymnasium), three Economic and four Vocational Schools as following: · High School in Presevo, South Serbia · Economic School in Leskovac, South Serbia · Agricultural School in Leskovac, South Serbia · High School in Leskovac, South Serbia · High School in Kragujevac, Central Serbia · Nursing School in Kragujevac, Central Serbia · Economic School in Kragujevac, Central Serbia · Economic School in Jagodina, Central Serbia · Technical School in Kraljevo, Central Serbia · Technical School in Kragujevac, Central Serbia # of Questionnaires Gender Type of School Regions Total Male Female Economic Vocational High Schools Southeast Serbia 30 13 17 6 3 21 Central Serbia 66 37 29 11 19 36 Total 96 50 46 17 22 57 As a result of pre-survey activities, the following schools were selected for interviews with JA students, teachers and principals: · Technical school in Vranje (excellent technical equipment, very proactive principal and indifferent JA teacher); · Economic and technical school in Bujanovac (very low technical capacity of school to participate in JA, enthusiastic JA teachers, sensitive inter-ethnic environment); · High School in Presevo (average technical capacity, inadequate space for JA students, enthusiastic JA teacher, sensitive inter-ethnic environment); · High School, Agricultural and Economical school in Leskovac (very good technical capacity for participation in JA, excellent results in last two years in all JA program components and capability to compare impact in different schools). The work was divided in three steps: a) first visit the USAID Belgrade office and meet with representatives of GDO and Program office to understand the programmatic rationale of implementing a JA program in Serbia; b) second interview all relevant beneficiaries and participants in selected municipalities in order to collect information about program implementation, and to be well informed and prepared for the last step; c) interview representatives of Ministry of Education and JA Serbia representatives in Belgrade. Limitations Implementation of Junior Achievement Program in Serbia has some characteristics mentioned in the introduction section that are relatively exclusive and limit the possibility of measuring the performance of the program as a whole, in all the regions, at all times. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 11 Therefore, most of the findings of this report refer to the JA program implemented under the CHF’s agreement given that CHF is already in the middle of program implementation and could provide information on performance. However, the questionnaire was also distributed in the schools involved with JA program in Central Serbia where ACDI-VOCA is the implementer. Given that the latter is in its first year of implementation the findings from that part of the program were limited to receptiveness of the program rather than performance. The survey was focused on direct beneficiaries – JA students and teachers in Central and Southeast Serbia. Time constraint was present during the survey, due to the fact that Phase II of this evaluation training was scheduled on the very end of school year. Some students finished classes, some of them were on vocational training, and all the rest were extremely busy with final exams, so all of that limited their ability to answer the questionnaire in a large scale. Findings The findings of this report are organized around the 5 questions raised in the SoW and are supported by findings on the management and implementation of the program per the state of JA situation as it relates to the relationship it has with USAID and the implementer of the program under which it is funded. 1. Project Achievements This section describes the findings per the questions laid out in the SOW. 1. Does JA’s model meet the education/entrepreneurship needs of Southern Serbia? ÿ The content of the curriculum materials used by JA is recognized as a substantial improvement compared to the “old school” curriculum. Teachers and students strongly believed that the materials used in the JA programs were very useful, up to date and very practical. Almost 94% of students claimed that JA curriculum is significantly improved, (63%) declared it is partially improved and (31%) declared it is better than standard curriculum. Comments of students regarding the curriculum include: “JA content is better and more interactive; JA is providing existing business information through a new and better way of learning; JA has better mixture of theory and practice; Classic curriculum is more detailed, but JA curriculum is more applicable to business”. 0% 20% 40% 60% 80% Quality of JA curriculum Significantly Better Better Have no opinion The quality of JA handbooks is without question one of the best-marked parts of JA in Serbia: 7% of students think that quality of handbooks is good, (51%) very good and (40%) excellent. The only suggestion for improvement in JAs handbooks in order to make it more USAID/Serbia Junior Achievement /Young Enterprise Evaluation 12 tangible is to adopt existing JA handbooks to local economy; example: change existing part of curricula related to US syndicates and explore role of local syndicates in Serbia. 0% 20% 40% 60% Quality of JA handbooks Excellent Very good Good Not good ÿ JA programs have influenced the teaching methods in regular subjects of schools by making them more open to interactive communication. Teachers were enthusiastic about the quality of the training they received but they felt that the amount of training provided was not enough. Teachers that attended the training sessions of the JA program appeared to have a strong professional grasp of the material while those with the long experience in traditional teaching are reluctant to accept new teaching methods which are to some part introduced by JA. However, new teachers have applied extensively the new teaching methods and engaged students in activities that have improved their communication skills. 0% 10% 20% 30% 40% 50% JA influence to "Old School" Significantly Partialy There is no influence ÿ USAID programmatic shift towards economic development and human resource and institutional development emphasizes the need for education programs. The JA model is perfectly in alignment with the new arising challenges in Serbia. The new strategy development process for Serbia has identified a shift in program direction toward economic development. The new focus on human and institutional capacity development opens the way for a better cooperation between USAID and JA to achieve the mutual goal of improving access to employment, develop a labor force and improve living conditions for people outside the labor market. JA program is considered a very good tool to assist the new generation embrace and apply the business concepts and the entrepreneurship spirit. 0% 20% 40% 60% 80% JA influence regards business Significantly Partially There is no influence USAID/Serbia Junior Achievement /Young Enterprise Evaluation 13 2. Are there differences in implementation according to urban/rural, geographic area, ethnic group, gender? The team visited and interviewed people in Central and Southern Serbia and covered a good deal of the territory where the JA program is implemented parts of which have different ethnicity. The team noticed that there were no differences in implementation of the program in favor or against any particular group or territory. The weaknesses of implementation such as late delivery of materials, was something caused due to technical difficulties and not any other particular reason related to geographic area or ethnicity. Also, all the areas that were visited faced almost the same issues as it relates to time and space constraints due to a limited favorable infrastructure. One of the indirect benefits of the program was to foster and promote interethnic relationships and even though both parties were open to discuss and cooperate nothing was obvious yet. Exchange activities are planned for the next year to engage both Albanian and Serbian student communities. 3. Have students attitudes/behaviors toward business/entrepreneurs changed as a result of their participation in JA and if so how? ÿ The JA program has started to influence extensively the student’s attitudes, aspirations and career planning. The practical skills that the program offers have made it very popular among students and teachers. It teaches them the rules of the business but at the same time it engages them in applying those business concepts in the real life. The students felt that the JA experience is offering them the right knowledge to succeed not only in business entrepreneurships but in every aspect of their life after school. Also, their exposure to the JA program has influenced their career decisions toward business. More than 93% of students think that JA knowledge will significantly influence their future education. For High School students, knowledge about economics is something that they need for further education on University. Economic School students think that JA business education orientation gives them reality-touch and that JA is excellent upgrade of their standard curricula. Vocational School students are more focused on topics like JA companies and other issues that can help them to start their own business. 0% 20% 40% 60% 80% JA influence to future education Significantly Partially There is no influence Comments: 1) “It will help me to start my own business; Now, I have better knowledge how to start business; I started to read pages about economy in daily newspapers”; 2) “I realized how hard is to run your own business, and I know now crucial importance of decision￾making process; I am even thinking to start my own small business to cover part of cost for my University studies and to ease-up burden for my parents; Now I know how system works.” USAID/Serbia Junior Achievement /Young Enterprise Evaluation 14 The MESE competition is also a very popular program in the JA courses. Almost 50% of JA students participated in the program and an additional 27% will participate next year. 23 % of the students never participated in MESE due to inadequate technical infrastructure of their schools (no or slow internet connection, inadequate number of computers, etc.) 0% 10% 20% 30% 40% 50% MESE Competition Yes, I participated I'm not but I will participate next year No JA companies are very attractive for students: 56% of the students claim that they are members of such companies and an additional 6% will be next year. 36% are not members of JA companies and they are all from Central Serbia, High School (Gymnasium) students. It will be interesting to explore that issue more and to find why they are not interested in one of “the most wanted” parts of JA program – could be because of very specific business orientation of JA companies, not familiar to students of classic High School. On the other hand, High School JA students from South Serbia are big fans of student’s companies. 0% 20% 40% 60% JA Companies I am a member I will be next year I am not member ÿ JA has not done a good job in well presenting its program and expand it where needed. JA program hasn’t been presented adequately to the schools and communities prior to the implementation. Almost 87% of students had no or very little knowledge about JA before they become JA students. However, their Self-initiative and eagerness to learn is one of the most recognizable characteristics given that that almost 90% students participated in JA program by their own decision; 7% claim that teachers selected them, and only 3% said that they joined JA on their parent’s initiative. On the other hand students themselves have done a good job in extending their knowledge about the program to their peers. Only a few of them thought that by doing this they would support competitors and as such they were not very supportive of the idea to let their friends know. 0% 20% 40% 60% 80% 100% Knowledge about JA I had knowledge before I become JA student I had no or very little knowledge about JA before I become JA student USAID/Serbia Junior Achievement /Young Enterprise Evaluation 15 4. What is the quality of relationship forged between business community and JA participants? ÿ The business community is not aware of the program and as such not very active in the JA activities and student mentoring programs. Given the short life of the program it is too early to talk about a relationship between business community and the JA program. This is not a result of non willingness by both parties more than lack of a structured cooperation. The umbrella agreement of the CRDA program has established in the communities EEE (Enabling Economic Environment) groups which include business leaders in the community. Some of the business community leaders interviewed said that they would be willing to invest in the program if the students would have contractual agreements to work for the investor’s companies. However, no efforts were noted from the JA representatives to sit and talk with business leaders. Neither the main implementer CHF did such thing. Every existing relationship is a merit of personal contacts and efforts of JA field coordinators rather than a structured plan for such intervention. On the other hand, students find visits to local business very attractive and worthwhile. 5. How does JA define program and financial sustainability and is sustainability being achieved? ÿ Students associations are perceived as potential entities for programmatic sustainability. ÿ JA headquarters believe that they will always need a sponsor for the life of their program, be this a donor or an established relationship with a business community group. All teachers and students were thrilled by the program and in various ways it has changed their attitudes towards business and better developed their entrepreneurship skills. For the next school year most of the schools had initiated plans and discussions with business community and associations to carry on the JA programs in their communities. Most students and teachers support the idea of summer camps to introduce and implement JA programs given that it could remove the constrains of time and space during the regular school year. 2. Management and Implementation ÿ Implementation of JA programs rely more on JA field coordinators. In the second year the trained teachers from the first year were expected to be coordinators for the region and take charge of the whole program as it relates to selection of schools, promotion of JA programs, lobbying in the LG and local business and cooperate with CHF and other partners. As such, the whole approach although very enthusiastic was too ambitious at this stage. In Technical school in Vranje one teacher was initially selected for JA program, he attended the workshops and trainings, and left program due to the various reasons (new job, etc). That causes serious problems with implementation. New teachers were selected late and not adequately trained. Internal process of selection and training in JA program is very vulnerable in the similar cases. ÿ Material for JA program was not disseminated on time in all schools. In some cases there were good reasons for that (translating JA books in Albanian), in others such as Serbian students in Bujanovac who were waiting for books for more than three USAID/Serbia Junior Achievement /Young Enterprise Evaluation 16 months, there are not any. When asked, JA teachers in Bujanovac claimed that books were in city of Leskovac – some 50 km away, but they were not informed on the books arrival. ÿ In Bujanovac school inadequate technical capacity for internet connection was also stressed as a mayor obstacle. Only computer connected to internet is in the principal office and locked. Participation in MESE competition was limited because students had to wait for principal and they are framed with principal’s schedule. ÿ Coordination with field representatives of CRDA is not on satisfactory level. CHF Field Office Vranje manager is claiming that he can not establish lines of communication with JA Coordinator for Pcinja and Jablanica region. He is not well informed about JA program that has been granted and implemented in his area of responsibility. Also, JA doesn’t look for feedback from the schools in order to improve its effectiveness. They rely extensively on the field coordinators for implementation of the program but they don’t necessarily get feedback from the schools as to how well the program is doing. The field coordinators are teachers that were trained in the JA seminars but this doesn’t make them experts in management. Conclusions 1. General Conclusions Why Junior Achievement? It includes entrepreneurship, job training and civic leadership as well as opportunities to interact with youth from whole Serbia through Junior Achievement activities. Work habits, values, aspirations and ideals have been affected. Doing something visible and practical for young people in the 15-25 cohort has a range of benefits. First, it can help young people develop the competencies required to become more productive in every aspect of their life. Second, it sends a powerful message to a broader Serb and Albanian public that America cares about the future of the youth. A partnership for youth development provides a framework on which to build a sense of joint purpose and solidarity. 2. Performance and Implementation 1. Does JA’s model meet the education/entrepreneurship needs of Southern Serbia? ÿ The content of the curriculum materials used by JA is recognized as a substantial improvement compared to the “old school” curriculum. ÿ The teachers believe that the quality of training is very good but that has to be provided more often. ÿ The JA program provides the means for achieving USAID’s and Serbia’s goal in fostering economic development. 2. Are there differences in implementation according to urban/rural, geographic area, ethnic group, gender? ÿ There are no differences in implementation of the program particular to any group or ethnicity. The aim of the program besides the business education still remains to foster interethnic tolerance. Both parties have to try to keep the line of the communications open and be as inclusive as possible in their activities. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 17 3. Have students attitudes/behaviors toward business/entrepreneurs changed as a result of their participation in JA and if so how? ÿ There is high satisfaction among students for the JA activities. In absence of any other similar program and due to good quality of the business education concepts associated with the entrepreneurial skills the JA program has influenced extensively the student’s attitudes, aspirations and career planning. ÿ The popularity of the JA program is related primarily to its hands-on training and activities that set it apart from the traditional teaching. 4. What is the quality of relationship forged between business community and JA participants? ÿ On the basis of field visits, the Evaluation team concludes that there is no cooperation between EEE (Enabling Economic Environment Group) and JA representatives to really forge a relationship. There is a very good potential though for creating alliances between the business community and the JA program given the already created structure (EEE group). All that is needed is a structured line of communications. 5. How does JA define program and financial sustainability and is sustainability being achieved? ÿ Even though the student’s associations are perceived as the means for programmatic sustainability there is a major obstacle that relates to their legal status that undermines the idea at this moment. ÿ While the JA program seems attractive to business that know of it still will be several years before it can reliably attract sustaining support from the private sector. In part this is also because of the very fragile state of this sector in Serbia. As such, the main source of financial resources will be the donors in country. The following reflects the conclusions as it relates to the management and implementation of the program: ÿ Management of JA program in Serbia is based on personal enthusiasm more than on solid and dependable internal structure. We noticed that overall performance of JA was more related to personal achievements and enthusiasm of JA teachers and students than to technical capability of school to carry on JA activities. ÿ Late delivery of materials in some areas delayed program implementation. ÿ The schools lack the proper technical infrastructure to make the program a full success. The selection of the schools not often coordinated with the technical capacity of the school to carry on the JA program. ÿ Based on the field visits team concludes that the organizational structure at the grass root level is at a very early and fragile stage of organizational development. They would benefit significantly from a more extensive technical competence and management skills. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 18 Recommendations The recommendations herein are based on the findings and conclusions of this report and are not listed in order of priority. 1. USAID should establish a direct relationship through direct funding and prepare a long –term strategy for the JA program in Serbia while including it under economic development SO. The evaluation team believes that the JA program is in alignment with the arising needs of the young population in Serbia for a better and more practical education policy. The program also complements perfectly USAID’s new shift in strategic direction toward economic development and human resource and institutional development. As such, the team not only believes that USAID should support the program but it should make an effort to include it in its strategic framework for direct funding in order to increase the efficiency and effectiveness of the program. In its relations with JA Serbia, it is important that USAID adopt a long term multi-year strategic approach based on a mutually agreed set of objectives designed to build overall organizational capacity. The Mission should avoid an annualized project specific focus that doesn’t address the long term underlying challenges. 2. JA should engage in discussion with the MoE to consider the student’s associations as a potential tool for sustainability of the program. Student’s associations would be a very good tool for sustainability of the program provided that law requirements are amended to prevent compromising such association by improving the legal framework. Quote: Leskovac, JA Student: “ I am afraid that while selling pancakes I will be visited by financial police and will be required to comply with current VAT legislation. When I was creating the price for my product I calculated VAT and I want to pay it, but the way how my business is created doesn’t support the legal framework.” 3. JA should either consider establishing summer schools in order to facilitate the implementation of its activities or engage in discussions with the MoE or other authorities to have the space and time required for such purpose. JA Serbia should consider supporting the summer schools in order to improve program efficiency. Given the schools situation the time and space constraints in the schools will not be overcome soon so alternative ways should be explored not only to improve efficiency but also to expand the program. 4. JA should better streamline the delivery of the materials needed for the courses so that students have them in time. The team received reports that sometimes translation of the materials delayed distribution and other times JA field coordinators were not informed of the materials arrival. It was not clear whether this is due to a shortage of supply, difficulties of forecasting demand, or both. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 19 Again, JAW specialists may be of some assistance in helping to identify and alleviate shortages of materials that are in very high demand. 5. JA should engage in discussions with EEE groups to create a relationship and to lobby the program as beneficiary to the future of this group’s members. JA Serbia should initiate activities such as large-scale conferences for Serbian Municipalities in Belgrade for promotion of JA programs and for fund-raising in order to establish financial ties with local community. In some areas we are informed that local community is covering part of costs for teachers (Cacak municipality is paying extra hours for JA teachers). 6. JA and CHF should better coordinate with each other and establish roles to better monitor the program implementation. Improvement in management should be focused on creation of direct links between field activities done by JA teachers and JA regional coordinators and local offices of CRDA implementing partners (CHF, ACDI/VOCA). The recommendation is based on noticed lack of coordination, guidance and reporting inside existing JA Serbia network, and on the other side absence of cooperation with representatives of CHF field offices. Main reason is internal structure of communication in JA Serbia and CHF. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 20 Anex A - Evaluation Scope of Work Evaluation Scope of Work Participant Team: ___SERBIA________________ USAID Project to be Evaluated: Junior Achievement Initial and Final Funding Years: 2003 – present Type Evaluation: __X__ Mid-Term (Formative) _____ Final (Summative) _____ Impact (Post-Facto) Purpose and Intended Uses of the Evaluation: To evaluation the performance of JA/Serbia and to advice on whether or not to continue funding. Brief Description of Project, the “Program Theory” that underlies it and it’s Intended Results: Junior Achievement in Serbia (JAS) was established in October 2002 after signing and agreement between Junior Achievement International (JAI) and the European Movement in Serbia (EMIS). This operating agreement has authorized EMIS to be the host of JAS and develop JA programs in Serbia. Junior Achievement Serbia (JAS) will serve as an independent unit within organizational framework of EMIS. It has been agreed that after certain period of incubation, JAS will be spun off and eventually establish itself as the separate non-profit organization registered with Serbia. Intended results are: 1) By creating a pro-business climate which in turn spurs economic development, international trade and attracts foreign investment and technology transfers; 2) By preparing young people to be workforce ready which will eventually assist Serbia in making the transition to market economies; and 3) By fostering democratic institution building through voting and teamwork exercises prevalent in the JA courses. Existing Performance Information Sources: PMP, quarterly reports, interviews, implementing partner’s reports, JA Worldwide reports, JA Russia evaluation report, cooperative agreements (sub grants) Evaluation Questions: 1. Does JA’s model meet the education/entrepreneurship needs of Serbia? 2. Are there differences in impact/implementation according to urban/rural, geographic area, ethnic group, gender? 3. Have students attitudes/behaviors towards business/entrepreneurs change as a result of their participation in JA, and if so how? 4. What is a quality of relationships forged between business community and JA participants? 5. How does JA define program and financial sustainability and is sustainability being achieved? Evaluation Methods: Participant teams are asked to propose methods. It is recommended that participants consult USAID’s Evaluation Methods section of the TIPS for Preparing an Evaluation SOW behind Tab 4, covering both (a) an overall design strategy and (b) a data collection and analysis plan. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 21 Deliverables: a. Evaluation Plan covering (a) the overall design strategy for the evaluation, (b) the data collection and analysis plan for the evaluation, (c) a list of the team members, and which one will serve as the team leader and primary contact (an e-mail and phone contact for the team leader should be provided) and (d) the team’s schedule for the evaluation. Due: 5/20/05 b. Draft Evaluation Report, consistent with guidance provided in Tab 14. Length of the report: Not to exceed 20 pages plus annexes and an Executive Summary of not more than 2 pages. Due: 7/21/05 c. Oral Presentation of the evaluation. Be ready on the first day of Phase III, including handouts. 8/1/05 d. Final Evaluation Report, incorporating advice you have been given by course instructors. Due: 8/12/05 Evaluation Team Composition: (For purposes of the course, include each team member’s name and key skills relevant to this evaluation) 1. Iliriana Dana, Team Leader: program office perspective, budget & analysis 2. Danijel Dasic, Team Member: project monitoring, analytical skills & logistic 3. Djordje Boljanovic, Team Member: project management and supervisory skills Procedures and Logistics: (Note anything special relevant to the work of this team) Reports and dissemination: (Be aware that the evaluations you do will become the property of USAID and JA and any distribution beyond that will require USAID E&E Bureau’s permission) Budget: (For purposes of the course identify any expenses your team might incur beyond those associated with your own travel.) USAID/Serbia Junior Achievement /Young Enterprise Evaluation 22 Annex B - Junior Achievement Serbia – Survey Results INTRODUCTION Implementation of Junior Achievement Program in Serbia has some characteristics that are relatively exclusive and could have significant impact to JA Serbia Team Survey: · JA Serbia is a sub-grant, under umbrella of CRDA – Community Revitalization through Democratic Action, a large-scale civil society program, divided geographically between five implementing partners. Three of five implementing partners are currently involved into JA activities, but all of them are in different stages of implementation. JA program in Southeast Serbia (CHF AoR) is in second year, in Central Serbia is in first year (ACDI-VOCA AoR) and third implementing partner – Mercy Corps just started with JA activites. · Different schools were selected for implementation of JA in Serbia: High Schools (Gymnasiums) – where students are preparing themselves generally for Universities; Economic schools – where students have some vocational training and they can apply for jobs after secondary school, as well as to continue with education on University: and Vocational schools – with students generally focused on specific education (technical, agricultural, etc.) and they are preparing for specific jobs. · Some regions of Serbia are targeted with other issues, like ethnic problems in South Serbia, under-development regions like South and Southeast Serbia, etc. Therefore our survey was focused on direct beneficiaries – JA Students in two areas, Central and Southeast Serbia, and on topics related to business education impact. Time constraint was also present during our survey, due to the fact that Phase II was scheduled on the very end of school year. Some students finished classes, some of them were on vocational training, and all the rest were extremely busy with final exams, so all of that limited their ability to answer our questionnaire. Nevertheless, questionnaires were disseminated in electronic form through JA Serbia network, and JA Serbia Evaluation Team received back 96 questionnaires from 10 schools. Four of them are in the second year of implementation – Leskovac and Presevo Schools and six are in the first year. Survey is well balanced also on the type of school issue: three classic High (Gymnasium), three Economic and four Vocational Schools · High School in Presevo, South Serbia · Economic School in Leskovac, South Serbia · Agricultural School in Leskovac, South Serbia · High School in Leskovac, South Serbia · High School in Kragujevac, Central Serbia · Nursing School in Kragujevac, Central Serbia · Economic School in Kragujevac, Central Serbia · Economic School in Jagodina, Central Serbia · Technical School in Kraljevo, Central Serbia · Technical School in Kragujevac, Central Serbia USAID/Serbia Junior Achievement /Young Enterprise Evaluation 23 # of Questionnaires Gender Type of School Regions Total Male Female Economic Vocational High Schools Southeast Serbia 30 13 17 6 3 21 Central Serbia 66 37 29 11 19 36 Total 96 50 46 17 22 57 FINDINGS Question #1: 78% of students have been informed about Junior Achievement Program through their teachers and 21% heard about JA from their friends. Only one student replied that he was informed through media. That suggest possible place for improvement – attract more students with proper media campaign. Outside selected schools JA program is not well known. 0% 20% 40% 60% 80% Informed about JA ThroughTeachers From Friends Through Media Question #2: Self-initiative is the one of the most recognizable characteristics for JA students due to the fact that almost 90% students answered that participation in JA program was their own decision. 7% claim that teachers selected them, and only 3% said that they joined JA on their parent’s initiative. 0% 20% 40% 60% 80% 100% Participation in JA It was my decision Teachers selection On my parents initiative Question #3: JA program hasn’t been presented adequately to the schools and communities prior to the implementation, due to the fact that more that 87% of students had no or very little knowledge about JA before they become JA students. Similar to the #1 recommendation – proper media campaign could attract more students to select JA Program as an extra curricula subject. 0% 20% 40% 60% 80% 100% Knowledge about JA I had knowledge before I become JA student I had no or very little knowledge about JA before I become JA student USAID/Serbia Junior Achievement /Young Enterprise Evaluation 24 Question #4: JA companies are very attractive for students: 56% claim that they are members additional 6% will be next year. 36% are not members of JA companies, and they all are from Central Serbia, High School (Gymnasium) students. It will be interesting to explore that issue more and to find why they are not interested for one of “the most wanted” parts of JA program – could be because of very specific business orientation of JA companies, not familiar to students of classic High School. On the other hand, High School JA students from South Serbia are big fans of students companies. 0% 20% 40% 60% JA Companies I am a member I will be next year I am not member Question #5: MESE competition is the most favorable part of JA program: almost 50% of JA students participated in MESE already and additional 27% will participate next year. 23% of the students never participated in MESE, due to the inadequate technical infrastructure of their schools (slow internet connection, inadequate number of computers, etc.) Improved environment for JA program implementation in schools is definitely one of the crucial places for improvement of JA program in future. 0% 10% 20% 30% 40% 50% MESE Competition Yes, I participated I'm not but I will participate next year No Question #6: Quality of JA handbooks is without question one of the best-marked parts of JA in Serbia: 99% of students think that quality of handbooks is good (7%), very good (51%) and excellent (40%). Only one JA student answered that handbooks are not comprehensible enough (Nursing School Kragujevac). 0% 20% 40% 60% Quality of JA handbooks Excellent Very good Good Not good Question #7: There is no doubt also, that JA teachers are selected and trained very well, because students ranked very good quality of JA teaching methods: Excellent 26%, very good 55% and good 18%. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 25 0% 20% 40% 60% JA Teaching Methods Excellent Very good Good Question #8: JA curriculum is also recognized as a substantial improvement comparing to the “old school” curriculum. Almost 94% of students claimed that JA curriculum is significantly (63%) and partially (31%) is better than standard curriculum. 6% of JA students have no opinion about it. Comments: “JA content is better and more interactive; JA is offering existing information about business, but through new better way of learning; JA has better mixture of theory and praxis; Classic curriculum is more detailed, but JA curriculum is more applicable to business”. Only suggestion for improvement is to adopt existing JA handbooks to local economy – example: change existing part of curricula related to US syndicates and explore role of local syndicates in Serbia. 0% 20% 40% 60% 80% Quality of JA curriculum Significantly Better Better Have no opinion Question #9: 78% JA students feel that “Old School” is influenced by JA program significantly (34%) and partially (44%), but 22% claim that there is no influence at all. In Southeast Serbia, after two years of implementation, almost 90% of JA students think that attitudes of teachers are changed. In Central Serbia, after only one year, students are more pessimistic. Comments: “Only younger teachers are willing to participate and support JA, old ones are thinking that JA is just a game”; After few months, some teachers, not part of JA, are trying to practice more flexible teaching methods”; We (JA students) are trying to advocate new teaching methods to our teachers.” 0% 10% 20% 30% 40% 50% JA influence to "Old School" Significantly Partialy There is no influence Question #10: More than 93% of students think that JA knowledge will significantly (61%) or partially (32%) influences their future education. For High School students, knowledge about economics is something that they need for further education on USAID/Serbia Junior Achievement /Young Enterprise Evaluation 26 University. Economic School students think that JA business education orientation gives them reality-touch and that JA is excellent upgrade of their standard curricula. Vocational School students are more focused on topics like JA companies and other issues that can help them to start their own business. Comments: “It will help me to start my own business; Now, I have better knowledge how to start business; I started to read pages about economy in daily newspapers.” 0% 20% 40% 60% 80% JA influence to future education Significantly Partially There is no influence Question #11: Similar to the previous question, 92% of students consider that JA program has influenced their attitude regards business significantly (68%) or partially (24%). But dissimilar to the previous question, there is no divergence in answers between JA students from different schools. Comments: “I realized how hard is to run your own business, and I know now crucial importance of decision-making process; I am even thinking to start my own small business to cover part of cost for my University studies and to ease-up burden for my parents; Now I know how system works;” 0% 20% 40% 60% 80% JA influence regards business Significantly Partially There is no influence Question #12: 86% of students have recommended JA program to their friends, and 9% will recommend. The most attractive component of JA program is definitely MESE, and all participants in survey without equal voted for it. On the other hand, 3% students don’t want to recommend JA to anybody because “I don’t want to support competitors”. 0% 20% 40% 60% 80% 100% Recommend JA to friends I have recommended I will recommend I don't want to recommend USAID/Serbia Junior Achievement /Young Enterprise Evaluation 27 Annex C. Agenda for Phase II Evaluation June 12th – June 18th 2005 Monday, June 13th 7:25 Iliriana Dana in Belgrade. Sasa Djordjevic, driver from Nis office will pick her up and drive to the Embassy. 8:30 Evaluation team gathering in USAID GDO Serbia premises. 10:00 – 11:00 Meeting with GDO Officers, Mark Pickett and Art Flanagan. 11:00 – 12:00 Meeting with Adriana Lazinica, Program Officer 12:00 Departure for Kragujevac. 14:00-15:00 Meeting with ACDI – VOCA representative responsible for JA program activities. 15:00 Departure for Nis. 16:30 Milica Spasic will join Evaluation Teamin Nis Office. Departure for Vranje. 18:00 Arrival to Vranje Hotel. Tuesday, June 14th 9:00 – 10:00 Meeting with JA students in Vranje. Preferable size of group is 3 to 5 students. Group should be mixture of students from Applied Economics, Student Company and MESE competition participants. We would like to have meeting in school premises – if possible. 10:15 – 11:00 Meeting with JA teachers in Vranje. 11:00 – 11:45 Meeting with principal in Vranje School. 12:00 – 13:00 Meeting with CHF Vranje staff responsible for JA program 13:00 – 14:30 Lunch in Vranje. Departure for Bujanovac. 15:00 – 16:00 Meeting with JA students in Bujanovac. Preferable size of group is 3 to 5 students. Group should be mixture of students from Applied Economics, Student Company and MESE competition participants. 16:15 – 17:00 Meeting with JA teachers Bujanovac. Departure for Vranje. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 28 20:00 No-host dinner with representatives of CHF field office in Vranje and Presevo, JA coordinator, UNDP, Municipality of Vranje (if there is a person connected with JA program), etc… Wednesday, June 15th 9:00 – 10:00 Meeting with JA students in Presevo. Preferable size of group is 3 to 5 students. Group should be mixture of students from Applied Economics, Student Company and MESE competition participants. We would like to have meeting in school premises – if possible. 10:15 – 11:00 Meeting with JA teachers in Presevo. 11:00 Departure for Leskovac 11:45 – 13:00 Lunch in Predejane. 14:00 – 15:00 Meeting with JA students in Leskovac. Preferable size of group is 3 to 5 students. Group should be mixture of students from Applied Economics, Student Company and MESE competition participants. 15:15 – 16:00 Meeting with Vesna Stoiljkovic, JA Coordinator for Leskovac and Vranje Region in JA Vranje (or CHF Vranje) premises. Departure for Belgrade. 19:00 Arrival to Belgrade Thursday, June 16th 9:00 – 10:00 Meeting with representatives of Ministry of Education (Ms. Silva Misljenovic) in their Belgrade premises. 10:30 – 11:30 Meeting with JA Program Financial Coordinator Irena Komazec. 11:45 – 12:45 Meeting with Zvonko Brnjas, Executive Director. 13:00 – 14:30 Lunch in Belgrade. 15:00 – 16:00 Meeting with CHF Belgrade Central Office representatives 16:30 – 17:30 Meeting with Mercy Corps Belgrade Central Office representatives USAID/Serbia Junior Achievement /Young Enterprise Evaluation 29 Annex D. - List of Interviewees USAID Serbia and Montenegro Mission 1. Mark Pickett, GDO officer 2. Art Flanagan, GDO officer 3. Adriana Lazinica, Senior Program Management Specialist ACDI-VOCA office in Kragujevac 1. Ivana Petrovic, coordinator for JA program Technical School in Vranje 1. Gordana Bliznakovski, principal 2. Sasa Bliznakovski, JA teacher 3. Marko Stosic, JA student CHF Vranje Field Office 1. Bata Stojkovic, CHF Vranje Office Manager 2. Miodrag Antic, EEE Vranje group representative 3. Brian Holst, CHF Serbia director 4. Rexep Ilazi, CHF Presevo Office Manager Economic-Technical School in Bujanovac 1. Jelena Nakic, II/3, JA student 2. Tanja Nikolic, II/3, JA student 3. Tijana Stosic, II/2, JA student 4. Milica Vasic, II/2, JA student, director of JA company 5. Milos Stanojkovic, II/2, JA student 6. Aleksandar Milanovic II/2, JA student 7. Stefan Taskovic II/1, JA student 8. Ivana Trajkovic, JA teacher 9. Sreten Jovic, JA teacher High School in Presevo 1. Limon Kadriu, JA teacher 2. Veli Mustafa, III grade, JA student 3. Enis Hyseni, III grade, JA student, 4. Liburn Mustafa, III grade, JA student 5. Premtim Jonuzi, III grade, JA student 6. Abdurrahman Zylfiu, principal High School in Leskovac 1. Cvetkovic Ivana, II/2, JA student 2. Cvetkovic Dobrivoje, III/8, JA student 3. Dejan Neskovic, III/8, JA student 4. Mladenovic Jelena, II/1, JA student 5. Mladenovic Aleksandra, III/3, JA student 6. Bojan Aleksic, JA teacher USAID/Serbia Junior Achievement /Young Enterprise Evaluation 30 Economical School in Leskovac 1. Radivojevic Miroslav, II/2, JA student 2. Ivancevic Aleksandra, II/2, JA student 3. Teodora Kocic, II/1, JA student 4. Aleksandra Aleksic, II/1, JA student 5. Dikic Irena, JA teacher Agricultural School in Leskovac 1. Markovic Sanja, II/6, JA student 2. Milica Nikolic, II/6, JA student 3. Jovana Andjelkovic, II/6, JA student 4. Aleksandar Veselinovic, II/6, JA student, director of JA company 5. Vesna Stoiljkovic, JA teacher & regional coordinator for Pcinja and Jablanica District JA Serbia premises in Belgrade 1. Silva Misljenovic, Adviser in Ministry of Education 2. Irena Komazec, JA Finance Coordinator 3. Zvonko Brnjas, JA Executive Director CHF Serbia Head Office in Belgrade 1. Darko Radicanin, Program officer 2. Nebojsa Nikolic, Program officer USAID/Serbia Junior Achievement /Young Enterprise Evaluation 31 Annex E. Questionarie 1. Do you recall how have you been informed about Junior Achievement program? a) from friends b) from teachers c) trough media campaign 2. How did you apply for JA program? a) it was my decision b) I was selected by teachers c) on my parents initiative 3. What was your level of knowledge about JA, in general, before you become a JA student? a) I was well informed b) just basics information c) I had no knowledge 4. Are you active member of JA student companies? a) Yes b) No c) No but I will be next year 5. Have you ever participated on JA MESE competition? a) Yes b) No c) No but I will be next year 6. What do you think about quality of JA handbooks? Excellent Very good Good Poor Inadequate 7. How do you rank quality of JA teaching methods? Excellent Very good Good Poor Inadequate 8. Do you think that JA curriculum is better than standard curriculum? Significantly Partially Have no opinion Barely Not at all If you want to add your comment, please do it in box below. 9. Do you think that JA program has influenced attitudes of teachers in your school? Significantly Partially Have no opinion Barely Not at all If you want to add your comment, please do it in box below. 10. Do you think that knowledge that you achieved through JA classes will contribute and/or influence your future education? Significantly Partially Have no opinion Barely Not at all USAID/Serbia Junior Achievement /Young Enterprise Evaluation 32 If you want to add your comment, please do it in box below. 11. Has JA program influenced your attitude regards business? Significantly Partially Have no opinion Barely Not at all If you want to add your comment, please do it in box below. 12. Have you recommended this program to any of your friends? a) Yes b) No c) Not yet – but I will recommend Which parts of curricula? School: Class: Gender: Male Female Age: 13. Date: USAID/Serbia Junior Achievement /Young Enterprise Evaluation 33 Annex L. CRDA Program Description THE COMMUNITY REVITALIZATION THROUGH DEMOCRATIC ACTION PROGRAM (CRDA) A. Introduction One of the primary programs of the USAID/FRY/Serbia mission is the Community Revitalization through Democratic Action (CRDA) Program. This is planned as a five-year, $200 million program covering all of Serbia except for metropolitan Belgrade (due to the Lautenberg amendment) and the province of Kosovo (which is under a United Nations mandate). It is a civil society program that uses community development activities to build trust between different ethnic and religious groups, to demonstrate the value of citizen participation, to support grass roots democratic action and to bring immediate improvement in people’s living conditions. B. Background In the 1990's, during Milosevic's rule the people of Serbia suffered cycles of economic and social instability that resulted in severely decreased production, low wages, high unemployment and under-employment, especially in the rural areas. Gross domestic product dropped by two-thirds over the period and unemployment grew dramatically. Rapid inflation surges during the past decade severely reduced the real value of financial assets. The elderly, the handicapped and single parent families, who were often without substantial financial or physical assets, were especially hard hit and particularly vulnerable, since pensions and other social welfare payments shrank to low levels and often were not paid on time. The quality and capacity of social service delivery mechanisms and institutions, such as the health system, public utilities and other public services in Serbia, have been undermined during the past decade by economic decline, a lack of investment, and the increased demand of a large refugee population. Insufficient and often inadequately trained health and social service personnel have had few resources with which to work. Infrastructure has deteriorated severely due to lack of investment and a social policy that kept public service fees unrealistically low and left communal enterprises with insufficient funds for proper operation and maintenance. Although the socioeconomic situation is serious, there is reason for cautious optimism. In the 1980s Yugoslavia's industrial and agricultural sectors were among the most sophisticated in Eastern Europe. These sectors still have significant assets, but need to be revitalized. Given the country’s rich natural and human resources, USAID/FRY/Serbia felt that community mobilization offered a promising opportunity to build on the democratic revolution in Serbia, and create a momentum for social and economic development at the local level. C. CRDA: The Concept and Approach In assessing the situation in Serbia, the mission concluded that it needed a program that would reach out directly to the local communities and produce rapid results. It was felt that the fledgling democratic movement could founder if people did not develop a serious commitment to the concept. Moreover, given the dismal economic situation, the mission USAID/Serbia Junior Achievement /Young Enterprise Evaluation 34 believed that action was needed to produce immediate improvements to local living conditions. These concepts drove the design of CRDA. In Lebanon, USAID had successfully launched a community-based development program called the Rural Community Development Clusters Project. This program, which had as one of its objectives promoting cooperation and trust between religious communities, seemed well suited for adaptation to Serbia where similar divisions existed within the population. The scale of the program in Serbia, however, would have to be far greater. The CRDA Program was therefore designed to focus on heavy community participation and rapid results. It is a citizen-driven program wherein local communities organize themselves and decide on priority development projects that they wish to implement. Although the cooperation of the municipal governments is sought, the program deliberately does not work directly through the local governments in order to reduce the possibilities of co-option, unrepresentative decision-making and bureaucratic delay. (The mission has a complementary activity, the Serbia Local Government Reform Program (SLGRP), which is strengthening the capacities of the municipalities.) The emphasis on rapid results was to alleviate suffering, to demonstrate that democratic action works and to buy time for implementation of major policy reforms at the national level. To ensure that citizen participation was broad-based and included all elements of the local community, the representation on each committee was required to reflect the local ethnic and religious mix as well as to have representation by women and youths. The concept was that by getting people to work together towards a common goal, the program would begin to build a sense of trust and cooperation between the different ethnic and religious groups. To ensure that the community was really committed to the project and took ownership of it, the Mission decided that participating communities would be required to contribute at least 25 percent of the total project cost. The contribution could be in land, labor, equipment or cash and also could come from any other source, including government, NGOs and other donor groups. Realizing that in many instances economic and social projects are best implemented on a regional basis, the CRDA Program was designed to include cluster committees (i.e. committees composed of representatives from a number of communities). The cluster committees are seen as providing a forum for reviewing economic and social development plans from a regional perspective and an opportunity for communities to cooperate to mutual advantage on shared economic and social priorities. Cost effectiveness can be improved, scarce resources shared more equitably, negative environmental impacts avoided, and environmental conditions improved through inter-community cooperation. The establishment of community and cluster committees provides a framework for community involvement and an opportunity to draw previously marginalized groups such as women and minorities into community action. Moreover, these committees constitute a forum representing broad-based community priorities that can form the basis for lobbying local and other government bodies for support. The community and cluster committees thus constitute the core vehicles for realization of the objectives of the program and are what make CRDA a civil society program and not simply a community development activity. D. CRDA Implementing Organizations and Regions CRDA is being implemented through cooperative agreements with five American NGOs. These organizations were selected through an open competition. Once the awards were made, USAID/FRY organized a meeting with the five grantees at which agreement was USAID/Serbia Junior Achievement /Young Enterprise Evaluation 35 reached as to which geographic area of Serbia each grantee would implement the program. The NGOs and their regions of operation or AORs (Areas of Responsibility) are: Cooperative Housing Federation (CHF) in collaboration with the South-East Consortium for International Development (SECID) and local partners is working in Eastern and Southeastern Serbia. Its regional office is located in Nis and satellite offices have been established in Vranje, Leskovac and Zajecar. Agricultural Cooperative Development International/ Volunteers in Overseas Cooperative Activities (ACDI/VOCA) with collaboration from the Urban Institute, Overseas Strategic Consulting Limited, International Orthodox Christian Charities, and local partners is operating in Central Serbia. Their regional office is located in Kragujevac. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 36 America’s Development Foundation (ADF) is collaborating with the University of Delaware/FLAG International, Arbeiter Samariter Bund Deutschland (ASB), the International Executive Service Corps, and local partners to implement CRDA in the Vojvodina region. Their regional office is located in Novi Sad. International Relief and Development, Inc. (IRD) has as its collaborating institutions the United Methodist Committee on Relief (UMCOR) and Camp, Dresser and McKee, Inc. (CDM) is working in Western Serbia. The regional office is located in Uzice with satellite offices in Sabac and Valjevo. Mercy Corps International in collaboration with Deloitte Touche Tohmatsu Emerging Markets is responsible for Southwestern Serbia. Their regional office has been established in Krusevac and satellite offices set up in Novi Pazar and Prokuplje. E. Program Implementation Once the areas of responsibility were assigned, the grantees were required to conduct an assessment of their region to identify communities for inclusion in the program. The factors that were used in community selection included need, community acceptance of the program concept, willingness to contribute, willingness to participate, municipal cooperation, and opportunities for exceptional progress in economic development or ethnic reconciliation. During this phase, local government officials were contacted to discuss the program and ask their advice concerning potentially eligible communities. After the initial selection of communities was made, the grantees held town meetings at which the local populations organized themselves into committees responsible for identifying and prioritizing local needs, mobilizing community and other resources, and monitoring the implementation of projects. As soon as the community committees were operational, the grantees began working with them to prioritize, plan, and implement projects to revitalize essential infrastructure, create income generating opportunities, address critical environmental problems, and promote civic participation. To assure accountability and transparency, the CRDA NGO implementers directly manage all procurement activities, although with the oversight of the community committee. Included on the grantee staffs are engineers and business advisors whose roles are to provide technical assistance and oversight for the activities being implemented. CRDA implementing NGOs will be working with their communities throughout the five-year life of the program, acting as a resource to them for increasing the quality and quantity of civic participation to achieve positive social and economic change. The grantees are supporting community action by facilitating community organization, providing targeted technical assistance and training, ensuring the technical, economic, and environmental viability of projects, and providing monitoring and oversight. Where possible the grantee will encourage inter-community solutions and will facilitate cooperation between communities and government to promote sustainability of projects. Within their areas of responsibility, the grantees are also expected to identify clusters of communities that constitute groupings with natural geographic, social, or economic ties. Cluster selection criteria will include groupings with natural geographic, social, or economic ties, a willingness of the constituent communities to engage each other regardless of ethnic, religious, or political differences and opportunities for exceptional progress in regional USAID/Serbia Junior Achievement /Young Enterprise Evaluation 37 cooperation. Cluster committees are then to be formed with representatives from the community committees to broaden participation on issues of regional concern and promote cooperation among communities in meeting social and economic needs. In areas where ethnic tension is undermining development, cluster committees are expected to provide a forum for initiating cooperation on shared interests, paving the way for finding common ground to overcome common problems. The CRDA Program believes that the cluster committees could offer regional solutions to problems that are beyond a community's capabilities, or offer cooperative exploitation of resources that are underutilized because of disputes, such as a shared water source. The CRDA grantees fund projects on the basis of their social or economic impact, their sustainability, broad based citizen participation in the identification and implementation of the project, and community, government, or other contributions. There are four categories of projects that are eligible for funding under the CRDA program. They are projects involving civic participation, community infrastructure, income generation and environmental improvement activities. Civic participation projects involve activities whose purpose is to engender a sense of community, reduce inter-ethnic tensions, and involve community populations in their own governance. While the community committees are considered to be the most important civil society activity, others activities could include community public hygiene and health campaigns, conflict resolution activities, multi-ethnic child and youth programs, development and support for parent-teacher organizations, etc. Community civil works projects are defined to include both the development of new and the rehabilitation of existing local infrastructure and facilities. This could include potable water, wastewater and irrigation systems, schools, health clinics, community centers, roads and other critical infrastructure. In general it was anticipated that CRDA funding would go into the development or renovation of works and facilities such as distribution and collection systems, well fields and pumping systems, storage systems, treatment systems, and buildings. Supplies, generic equipment such as desks, chairs, media equipment, medical equipment, as well as staff salaries and training, were seen to be the contribution of the community, government, or another international donor. Income generating projects are expected to focus on agribusiness, small-scale industry, local trade or service activities. The CRDA program could support agriculture or food processing activities such as a canning plant or sawmill, re-opening small-scale industrial enterprises such as a furniture or shoe factory, re-opening shops and service businesses or investment in home or cottage industries. The emphasis of the income generating activities is to put money in people’s pockets rather than to resolve macro-level economic policy issues. Training and technical assistance were defined as eligible activities that could be made available when necessary, but USAID did not include a credit element because of the complexity of doing so within the CRDA program. CRDA beneficiaries, however, may use the credit programs of USAID and other donors. Environmental improvement projects, the final category of eligible projects, include activities that mitigate pollution, protect health, or preserve natural resources. These could include reforestation, sanitary drainage, landfills, or erosion control. Environmental education programs could also be undertaken, either in conjunction with specific activities as mentioned above or as a stand-alone effort. USAID/Serbia Junior Achievement /Young Enterprise Evaluation 38 USAID Program Management The General Development Office of USAID/FRY is responsible for managing the CRDA. Because of the countrywide extent of CRDA program activities, the GDO has established field offices in Novi Sad, Uzice, Kragujevac and Nis to provide better implementation oversight. In addition, all grantees are to provide monthly input on implementation progress into an electronic management information system (PRS – Program Reporting System) that will facilitate tracking. CRDA Internet website has been established in order to provide access to the program results reporting data to all interested parties (http://www.sada.usaid.org.yu/).