EVALUATION OF THE PROGRAMS OF JUNIOR ACHIEVEMENT KYRGYZSTAN SUBMITTED TO: UNITED STATES AGENCY OF INTERNATIONAL DEVELOPMENT/CAR EVALUATION TEAM: LAUREN RUSSELL AND KHORLAN IZMAILOVA JULY 21, 2005 _______________________________________ TABLE OF CONTENTS PAGE NO. EXECUTIVE SUMMARY………………………………………………..……. 3 ACKNOWLEDGEMENT……………………………………………….………4 ACRONYMS………………………………………………………..…………4 INTRODUCTION………………...……………………………………………4 BACKGROUND……...………………………………………………………..5 PURPOSE OF EVALUATION…………………………………………………..6 CAVEATS………………………………………………………………….....7 FINDINGS AND CONCLUSIONS….…………………………………………....7 ADDITIONAL FINDINGS…...………………………………………..……....13 RECOMMENDATIONS……………………………………………………….15 2 ______________________________________ EVALUATION OF THE PROGRAMS OF JUNIOR ACHIEVEMENT KYRGYZSTAN EXECUTIVE SUMMARY The Junior Achievement (JA) program provides education materials to Kyrgyz students in the areas of business, management, and the market economy. Junior Achievement Kyrgyzstan (JAKg) also provides teacher training and prepares students to participate in national and international competitions where students apply economic theory to practical business situations. JA Kyrgyzstan was founded in 1995 by a joint effort between Peace Corps volunteers and teachers from Bishkek Academy of Finance and Economy. Today, JAKg’s JA Economics program has been translated into Kyrgyz and Russian as well as become fully adapted to Kyrgyzstan economic realities. The JA Economics program is taught in 9th, 10th, and 11th grades and boasts a 20% saturation rate due to support from USAID, Soros Foundation, Counterpart Consortium, and Eurasia Foundation. JAKg has also delivered training programs and workshops instructing elementary and secondary school teachers on the theory and methodology of JA Programs. For this evaluation, we conducted interviews and focus groups in the schools in two major urban centers-Bishkek, the capital, and Osh, one of the largest trading centers in the Central Asian Republics (CARs). The Team believes that we visited schools that were more advanced than typical schools in Kyrgyzstan. There are five main findings from the evaluation. First, everyone the Team interviewed, including the students themselves, agreed that students demonstrate new knowledge after participation in JA. Second, even after ten years in Kyrgyzstan, JA remains an unknown quantity in the Kyrgyz business community. Third, JA program is not yet optimally structured and implemented in terms of urban and rural differences and ethnic issues. Fourth, the JA teaching style is used outside the JA classroom. Finally, JA/Kyrgyzstan needs a plan for sustainability. On December 10, 2004, USAID/CAR suspended the USAID Cooperative Agreement with JA’s Central Asian Regional Office. This action was taken to safeguard U.S. Government funds and is based on serious deficiencies found in the financial management system and operations of the Central Asian Regional Office that is implementing this Agreement. 3 ACKNOWLEDGEMENT The Team is especially grateful for the invaluable logistical support and warm welcome of the CAR Mission, Junior Achievement Kyrgyzstan, especially Anna Gregoriava for the warm welcome and logistical support. ACRONYMS CAR Central Asian Republics JA Junior Achievement JAKg Junior Achievement Kyrgyzstan SO Strategic Objective U.S. United States USAID Unites States Agency for International Development INTRODUCTION This report summarizes the background and findings of the evaluation of Junior Achievement Kyrgyzstan (JAKg) funded by the Unites States Agency for International Development (USAID). It was prepared as part of the Europe and Eurasia Certificate Program in Evaluation by Lauren Russell, USAID/Russia Program Officer and Khorlan Izmailova, USAID/CAR Project Management Specialist, Office of Health and Education to be presented in Sophia, Bulgaria in August 2005. The field portion of the evaluation took place June 6 through June13, 2005. JAKg provides education materials to Kyrgyz students in the areas of business, management, and the market economy. JAKg also provides teacher training and prepares students to participate in national and international competitions where students apply economic theory to practical business situations. The JAKg program currently contributes to USAID/CAR Strategic Objective (SO) 1.3 “Improved Small and Medium Enterprise Environment.” 4 BACKGROUND Country Profile Kyrgyzstan is a poor, mountainous country with a predominantly agricultural economy. Cotton, tobacco, wool, and meat are the main agricultural products, although only tobacco and cotton are exported in any quantity. Industrial exports include gold, mercury, uranium, and natural gas and electricity. Kyrgyzstan has been fairly progressive in carrying out market reforms, such as an improved regulatory system and land reform. The adoption of relatively liberal economic policies has made Kyrgyzstan distinguish itself in the region. The population is over five million with one third of the population under 14 years of age and a median age of 23 years. Seventeen percent of the GDP is gross investment. The population below the poverty line is 40% and the unemployment rate is18% (2004). There are several sizable ethnic groups. Kyrgyz constitute 64.9%, Uzbek 13.8%, Russian 12.5% (1999 census). JA Kyrgyzstan was founded in 1995 by a joint effort between Peace Corps volunteers and teachers from Bishkek’s Academy of Finance and Economy. It has evolved into a program that impacts 14,425 students with two staff members and seven board members. JAKg has been delivering basic knowledge of economics and entrepreneurship to 5th through 11th grade students through several programs, which combine theory and practical experience, i.e. JA Applied Economics, Student Company Program, Management and Economic Simulation Exercise (MESE), and Global Business Ethics. These JA materials were the first hands-on, interactive, educational materials in the area of economics and business to appear in Kyrgyzstan and the only kind to provide practical activities. Today, JAKg’s JA Economics program has been translated into Kyrgyz and Russian as well as become fully adapted to Kyrgyzstan’s economic realities. The JA Economics program is taught in 9th, 10th, and 11th grades and boasts a 20% saturation rate due to support from USAID, Soros Foundation, Counterpart Consortium, and Eurasia Foundation. JAKg has also delivered training programs and workshops instructing teachers of elementary and secondary schools on the theory and methodology of JA programs. Translated and revised training materials in Kyrgyz and Russian languages are used to deliver both the theory of a free market economy and the methodology of teaching JA programs in a manner that engages students. 5 PURPOSE OF EVALUATION The evaluation assessed the overall JA Kyrgyzstan program. The report is designed to address the five questions raised in the Scope of Work (Appendix A). 1. How have student’s demonstrated new knowledge of economics and business because of their participation in JA compared to base-line (zero)? 2. What is the perceived value of JA in the business community, contrasting those working with JA compared to those not exposed to JA? 3. Is JA management optimally structured and implemented in Kyrgyzstan – in terms of assumptions about (a) urban versus rural differences, (b) gender inequities, and (c) ethnic issues? 4. Has the JA teaching style that involves greater student involvement been adopted outside the JA classroom? 5. What plans for sustainability does JA/Kyrgyzstan have if current funding sources evaporate? The Scope of Work called for an evaluation that would analyze and guide USAID and JA Kyrgyzstan. Data Collection and Methodology The Team used a combination of evaluation methods that included: - Reviewing quarterly reports, financial and audit documents, cooperative agreement etc; - Meeting with JA staff and members of the JA Board of Directors in Bishkek; - Meeting with Ministry of Education in Bishkek and Department of Education in Osh; - Individual interviews “key informant” with students in Osh and Bishkek; - Individual interviews “key informant” with teachers in Osh; - Focus group discussion with JA students in Bishkek; and - Focus group discussion with school directors in JA schools in Osh. 6 CAVEATS In view of the limited (two weeks) lead time to schedule interviews and focus groups for evaluation, the schools that were visited by the Evaluation Team are not representative￾all schools visited were in two major urban centers-Bishkek, the capital, and Osh, one of the largest trading centers in the CARs. In general, we believe that we visited schools that were more advanced than average. Students were taking final exams when we toured the schools. The “end-of-year” preparations were a distraction for the majority of individuals we interviewed. In six out of the seven schools we visited, students were busy cleaning the room around us during our interviews. Overall, the Team had modest access to students, teachers, and school administrators, but only little access to representatives from the business community. Of the two business leaders we talked with, neither had heard of JA, even though they had high school age children who were likely in schools with JA. The many focus group discussions with teachers, students, and administrators provided only a snapshot perspective of JA. A lack of historical documentation makes it impossible to provide a valid longitudinal assessment of student progress. FINDINGS AND CONCLUSIONS JAKg has been expanding its reach annually and has been making an impact in the field of capacity building among primary and secondary schools teachers in the methodologies of instructing market economics. All of this is due in part thanks to a dedicated and professional Board of Directors. Most of the board members are teachers and regulators of educational system who care to make a difference in the quality of education and delivery standards in Kyrgyzstan. 7 JAKg is working with the Ministry of Education and Science to endorse JA Economics into the 5th through 11th grade curriculum and to publish an adapted version of the JA Economics books. This adapted version will be published in both Kyrgyz and Russian. On December 10, 2004, USAID/CAR suspended the USAID Cooperative Agreement with JA’s Central Asian Regional Office. This action was taken to safeguard U.S. Government funds and is based on serious deficiencies found in the financial management system and operations of the Central Asian Regional Office that is implementing this Agreement. A recent disbursement review disclosed there is not a functioning system of internal control that will ensure the integrity of USAID funds and the risk of misuse is unacceptably high. In an interview with a financial management office specialist in USAID/CAR, we learned there were no job descriptions of JAKg staff and a driver had been promoted to a position that gave him great access to funds without the appropriate qualifications. Until USAID/CAR is satisfied that JA International has adequate financial and program management systems in place, they can not consider further funding. Currently USAID/CAR is reviewing an unsolicited proposal from JA/CAR for a 20- month cooperative agreement. In an interview with a member of the JA Board of Directors, the Team was informed that Kyrgyzstan does not have an active American Chambers of Commerce. Yet, the Team found businesses that JAKg could develop a strategy to approach. One such business is “Shoro” a beverage company that is becoming increasingly popular and gaining an international reputation. It was noted that JA would be better if there was a stronger connection between schools and the universities. Teachers and administrators told us that universities are aware of JA and find that JA students make more attractive candidates than non-JA students. In addition, university professors represent the majority of JA guest lecturers. Findings and Conclusions to the Five Main Evaluation Questions • Students demonstrate new knowledge after participation in JA. Students, in each of the different venues we met with, either in one-on-one interviews or focus groups, universally reported gaining a deeper and more accurate understanding of economics and business as a direct result of their participation in JA. The positive benefits described above are demonstrated by students choosing to elect to take the economics exam, a non-mandatory pre-college exam. In one school alone, the number of students electing to take the exam went from two to six in a single year. 8 One student we interviewed in Osh reported that JA instruction has helped her in everyday life. Now when she goes to the market she takes into account principles of supply and demand. If the supplies of her desired goods are plentiful, she knows she can bargain on the price! JAKg, teachers, and administrators do not track students’ progress upon secondary school graduation. The one exception to this is the consensus in the opinion that an increasing number of students are entering economics programs at universities. In the seven schools we held interviews, teachers and administrators reported only three examples of JA alumni who went on to use their JA background after graduation. One of these was a JA graduate who went on to start his own business; a T-shirt kiosk in the Osh Bazaar. On two occasions it was noted that JA graduates have returned to the JA classroom as guest speakers. Students reported back to their teachers that this knowledge has helped them in there everyday lives. Of the 28 students we met several gave us anecdotal evidence of how their JA instruction has helped them. This feedback ranged from understanding the principles of supply and demand in order to effectively bargain in the marketplace to learning how their parents budget their resources. Ministry of Education officials we met with said that JA students are by in large more work place savvy and are more skilled at seeking and getting jobs than their non-JA counterparts. • JA remains an unknown quantity in the Kyrgyz business community. JA remains virtually an unknown quantity in the Kyrgyz business community. JAKg does not have an outreach plan for gaining business community buy-in. JAKg arranged our interview with two members of the business community, yet neither had heard of JA, even though they had high school age children who were likely in schools with JA. All teachers and students interviewed showed great interest in wanting business leader participation in the classroom. But in the majority of schools that solicited volunteers, none could be found. One teacher said she had simply given up and will revert to asking only the student’s parents to volunteer in the future. Relatively successful schools only have guest speakers twice a year and these were usually professors from local universities. In Kyrgyzstan, due in part to prolonged hard economic times; there is little social awareness to give beyond the family. This could be the leading constraint to finding business leaders willing to volunteer their time in the JA classroom. 9 “Our school was just a regular school before JA. Now we are a JA/Economics School” - Administrator (Principle) of School # 62 • JA program is not yet optimally structured and implemented in terms of urban versus rural differences and ethnic issues. There were no gender inequities found. Urban versus Rural Differences It was widely noted that teachers and schools chosen for JA were picked as they were the low hanging fruit. Many were the original pilot schools and they were chosen because of their proximity to the JA offices so that progress could be easily monitored. A major handicap for the inclusion of rural schools is two fold: under-trained teaching staff and a lack of computers in the schools. (Computers and internet connection is a requirement for many advanced JA programs.) Many rural schools in Kyrgyzstan lack even a school telephone. Only being able to reach the school through visits hampers JA’s ability to implant the program in the school. Also, teachers are tested and must pass the exam at a certain aptitude level before being chosen to teach JA. Many rural school teachers lack the test taking/academic sophistication to be chosen to teach JA. A JA board member we interviewed said that in order to accept the majority of rural teachers they would need to lower the standards required for a JA instructor. The downside of the expanding JA program is that as the program includes more and more schools, the number of teacher trainings has decreased as a result of finite financial resources. Gender Inequities There were no gender inequities found save for reverse inequities in that girls in the majority of schools were noted as being far more eager to get involved in JA exercises and activities-especially advanced projects that require time out of school. Ethnic Issues All JA management staff we interviewed quickly said there were no ethnic issues in the implantation and structuring of JAKg. Yet, JAKg is implemented without including Uzbek and Tajik schools in Kyrgyzstan. Approximately 150,000 children study in Uzbek schools and 3,000 children study in Tajik schools. The JA program leaves out around 30,000 Uzbek speaking and around 1,000 Tajik speaking students between the grades of 10 9th to 11th from these schools, the majority of which are in Osh. The problems associated with implanting JA in Uzbek and Tajik schools are manifold. First, Kyrgyzstan’s Ministry of Education allows for Uzbek and Tajik schools to teach using the Uzbek language exclusively. Second, many of the Uzbek schools are private and outside of the Ministry of Education’s purview. Third, the majority of students and teachers in the Uzbek schools do not understand the Kyrgyz language enough to benefit from the Kyrgyz and Russian language versions of the JA materials. Frequently teachers, administrators, and JA staff would argue that JA/Uzbekistan should provide the materials to them, even though JA/Uzbekistan material would be fitted for Uzbekistan’s economic context and not Kyrgyzstan. Although this situation could be remedied by a decision from JAKg to translate the materials into Uzbek and Tajik languages; these populations are not viewed as a priority in JAKg’s expansion efforts. • The JA teaching style is used outside the JA classroom. Teachers and administrators in the majority of schools were quick to say that their teaching styles have always been modern and that they have always taught with a JA style of heavy student involvement even before JA came to their school. Interestingly, in these same schools, students we interviewed reported that JA instruction was different and involved greater student participation than non-JA classes. This is evident in the fact that many teachers report that students who do poorly in other subjects become energized and do well in economics as they find the hands on exercises and school company programs entertaining. Ministry of Education officials that we met with informed us that across the board, JA schools encouraged students to be more actively involved and mobile in the classroom than non-JA schools. One Ministry of Education official noted that this was one of the underlying reasons for their eagerness to take on JA in the beginning, as the previous mainstay of the Kyrgyz economics curriculum, “Introduction to Economics,” was lecture based. What the Team can conclude definitively from what we gained in the interviews is that by and large the JA teaching style and coursework was followed exactly as it was prescribed in the teaching materials. Therefore at least in the JA classroom, student involvement and team work was the norm. • JA/Kyrgyzstan needs a plan for sustainability. Over and over again we heard that it is simply too early to talk sustainability-even after ten years in-country. There are currently no plans in place for sustainability if JAKg funding sources evaporate. During an interview with a member of the JAKg Board of 11 Directors, we learned that he has been paying out of pocket for some of JA’s expenses due to the current hold on USAID funding of JAKg. The main handicap for sustainability of JAKg is that current tax laws in Kyrgyzstan provide no incentive to businesses to make charitable donations. Currently an amendment to the tax code is under consideration, though it is not expected that there will be any major changes in the near future. The July elections are hoped to bring stability in the tumultuous Kyrgyzstan political environment. In the meantime, JAKg is continuing to seek international support, in-kind donations and a continued good relationship with the Ministry of Education to sustain the organization and allow JAKg to continue delivering economic education training and materials to schools in Kyrgyzstan. JA management staff we talked with believe that many of the positive dynamics for sustainability were lost in the revolution. JA books were close to being accepted by the Ministry of Education prior to the revolution. Once accepted by the Ministry of Education, World Bank offered to pay for the printing. The program is currently in a holding pattern waiting for the July election results. “We are on a shoe string trying to stay around.” -JA management staffer Another handicap to sustainability is the continued belief by the Ministry of Education that economics and business education is secondary to other subjects. JA teachers are always primarily a teacher in a different subject, usually math or geography but sometimes even crafts and Russian language. In addition, Ministry of Education guidelines mandate that economics instruction in the schools is one hour a week. While this is an improvement over economics being left as an elective course, many teachers reported that this is simply not enough time to devote to the subject. JAKg has been effectively expanding its reach annually and has been making an impact in the field of capacity building among primary and secondary schools teachers in the methodologies of instructing market economics. All of this is due in part thanks to a dedicated and professional Board of Directors. Most of the board members are teachers and regulators of educational system who care to make a difference in the quality of education and delivery standards in Kyrgyzstan. JAKg is working with the Ministry of Education and Science to endorse JA Economics into the 5th through 11th grade curriculum and to publish an adapted version of the JA Economics books. This adapted version will be published in both Kyrgyz and Russian. 12 ADDITIONAL FINDINGS Students unanimously reported that they enjoyed their JA classes. In particular, the Kyrgyz examples and case studies in the JA materials caught the students’ attention and found to be very helpful. Many students noted that it was the knowledge gained from case studies that they will take with them after graduation. Furthermore, no previous textbooks on applied economics existed in Kyrgyzstan before JA, so the JA materials have filled a much needed niche. During our evaluation we found a number of problem areas and drawbacks from the way the program is currently being run. These fell into three main areas: instruction materials, teaching methods, and teacher training. Instruction Materials: • Not enough JA material for class. One teacher reported having to hand￾carry ten sets of books to and from school to ensure that they weren’t taken after hours. Other teachers manage time consuming check out procedures to ensure materials are returned. • Outside of Bishkek, the JA teaching materials were found to be very difficult for teachers not trained in economics. • Several schools with Kyrgyz-speaking student bodies received Russian language JA materials. This could have resulted from confusion over the fact that frequently the teachers tasked with teaching JA in these schools also happened to be Russian language teachers. The teachers were experiencing great difficulty translating contemporary economic terms found in the materials. • Some teachers received the materials without a teacher’s guide making the task of teaching the materials more difficult. • Only one audio tape that supplements the JA teaching materials is provided per school. The teachers worry what will happen if that tape breaks. • One school received dozens of glossy, expensive JA brochures and no extra books. The teachers asked the Team, “What are we supposed to do with all of these fancy brochures?” 13 Teaching Methods: • Very few guest speakers in the classroom, especially from the business community. • Teaching the Business Ethics class is optional for many teachers, even in schools that have the computers the course requires. • No acknowledgement is provided for students who excel. • JA students consistently requested practical opportunities to practice skills/lessons learned from JA materials. • Student Company Program is not widely adopted. • Economics instruction is mandated for one hour a week by the Ministry of Education. • Students frequently inquired about JA summer camps as a means for building on lessons learned. Teacher Training: • All teachers asked for more JA teachers conferences and trainings, so that teachers could learn from each other lessons learned and best practices. • A 9th grader, Denise Tsoy, in Bishkek won 7th place in a world wide on￾line competition. No certificate or acknowledgment of his achievement was ever sent to him-making his win very anticlimactic. • Teaching the Business Ethics class is optional for many teachers. Teachers are only paid an additional 400/som/mos ($10) to teach this class on their own time. • There was no real training for the business ethics course, only one round table with someone from India and Atlanta, GA. These individuals were unable to answer technical questions; especially regarding operation of the computer program. • Teachers want more “Training of the Trainer” courses so they can start teaching each other. • Teachers reported that they could gain a lot from a new training to go along with the new books they received. 14 • There are no full time secondary school economics teachers in Kyrgyzstan. • A representative from the Department of Education in Osh informed us that there is a greater need for basic economics training of JA teachers in rural areas. This was confirmed in subsequent interviews with JA teachers. RECOMMENDATIONS: In light of USAID/CAR’s December 10, 2004, suspension of the USAID Cooperative Agreement with JA/CAR, several serious deficiencies must be given priority. First, a functioning system of internal control needs to be established to ensure the integrity of USAID funds and prevent the risk of misuse. Second, job descriptions need to be created for JAKg staff vacancies and capable personnel need to be hired. The Team Recommends that JA/CAR: • Create a regional approach to soliciting business community buy-in. The Team Recommends that JAKg: • Develop a strategy to approach businesses for fundraising. One such business is “Shoro” a beverage company that is becoming increasingly popular and gaining an international reputation. • Conduct a needs assessment, in JA schools, of textbooks and teacher’s manuals to ensure the distribution of materials is in the correct language and quantity. • Host more training for teachers, including “train the trainers.” • Attract more guest lecturers to the classroom. • Consider additional means of teaching economics, such as summer camps. • Include in the job description for the JAKg Office Director a performance measure to work towards sustainability of the program. • Develop and implement more opportunities for JA teachers to share information and lessons learned, such as teacher’s conferences, regular meetings with the Board of Directors, training of trainers, etc. • Fund the translation of JAKg materials into Uzbek and Tajik to ensure that these large communities are not left out of the JAKg program. 15 • Encourage a local business outreach program to increase their awareness of and possible participation in the JA economics classes. • Develop a system for tracking alumni. At a minimum, one example would be polling JA students upon graduation as to their future plans. Another example would be working with universities and or economics faculty to poll incoming first-year students to track JA graduates. This data would be especially useful for evaluating future performance by JAKg. For instance, in determining “cost per student” who entered or graduated college. The Team Recommends that USAID: • Work with the Ministry of Education to elevate the importance of economics in the classroom. For instance by hiring professional economics instructors in the schools and providing adequate overtime pay for teachers who teach economics electives after hours. 16 Annex 1 SCHEDULE Evaluation of Junior Achievement Kyrgyzstan Lauren Russell, Khorlan Izmailova June 5-10, 2005 Monday, June 5 9:00-10:00 Office JA 10:30-11:30 School #70, Makarova Taiana Mikhailovna, Director, 219875 11:30 – 12:30 Ilim private school, Esenalieva Guldjamal Djusupbekovna, Director, Toktonalieva 6b, 541-379 12:30 – 13:30 Lunch 13:30 – 14:30 School #24, Alburina Ilmira Fanilovna, Director, Toktohula 110, 66-36-02 14:30 – 16:30 School #24, Gorolchuk Svetlana Anatolievna, teacher of JA Economics 17:00 – 18:00 Focus group discussion with students (22 people from JA program) 18:00- Dinner with Edward Edgaro, Acting Director of JAKg, Chairperson of the Board Tuesday, June 6 8:00-9:00 Eamon Doran, Activity Manager at USAID 10:00 – 11:00 School #62 Sharikova Lyudmila Alekseevna, teacher of JA Economics, Surekulova 8 11:00 – 12:30 Ministry of Education, Emma Valentinovna Kazarinova, Kirova-Tynystanova 12:30 – 13:30 Lunch 13:30 – 14:30 Soros 14:45 – 15:45 Counterpart, Erkin Kasybekov, Director, 0502 58-60-56, 66-21-88, Sovetskaya 204, 4th floor 16:00-18:00 Ilim private school, Elena Viktorovna Boiko, teacher of JA Economics Wednesday, June 7 9:00 – 10:15 Flight to Osh 11:00 – 12:00 Bazarov Bakhadyr Tolonbaevich, specialist of Osh Education Department, 21285, 75902 12:00 – 13:00 Lunch 13:00 – 14:00 School director that implement JA Economics, focus group discussion, individual interviews (8 people) 14:00 – 16:00 School #34, Arykbaev Abdylaziz 16:00 – 18:00 Working in the hotel Thursday, June 8 9:00-11:00 School #8, Focus group discussions with students 11:00 – 12:00 JA staff in Osh, Joldubai Karimovich, Adult Training Center 12:30-13:30 Lunch 13:30 – 16:30 Meetings with business representatives 17:00 – 19:00 Flight to Bishkek Friday, June 10 Trip to Almaty Annex 2 Focus Group: Students Questions to ask students individually, as possible: 1. How many years have you taken JA classes? a. One year b. 2-3 years c. 3 or more years 2. What year in school are you now? a. 11th b. 10th c. 9th d. 8th e. Other ________ 3. When attending your JA classes, how involved are you (participation versus just listening) compared to your other classes? a. More than other classes. b. Same as other classes. c. Less than other classes. 4. Who teaches your class? a. Your teacher b. A different teacher in your school c. Someone from the business community d. Other ___________________________ 5. Does this person teach the class: a. All of the time b. Most of the time c. Some of the time 6. When the main instructor is not teaching the class, who is? a. Your teacher b. A different teacher in your school c. Someone from the business community d. Other ___________________________ 7. Are any of your other subjects or classes taught like JA? a. Most (3 or more classes or subjects) b. Some (1-2 other subjects or classes) c. None 8. Do you think that JA is teaching you skills that you can use when you graduate? a. No b. Yes 9. How do you rate the skills you learned in your JA classes to those learned in other class? a. JA classes were less helpful compared to other classes. b. I learned about the same amount of skills in JA classes compared to other classes. c. JA classes were more helpful compared to other classes. 10. Did the JA Economics course change your understanding of business fundamentals? a. Yes b. No c. Uncertain 11. Did the JA Economics course change your understanding of economics? a. Yes b. No c. Uncertain Questions for group: 1. What have you learned from JA? 2. What are the positive things about JA? 3. What are the negative things about JA? 4. What would you like to change about JA? What do you think could improve teaching economics and business? 5. How do you think the skills you learned in JA will help you after graduation? 6. Is there anything else you would like to share with us about your experiences with JA? 7. What do you plan to do after you graduate? 8. Is your class involved in the business community? 9. Do you think the business community is aware of JA? Focus Group: Chamber of Commerce Business members 1. Do you know what JA is? 2. Do you know that secondary schools in your area are using JA teaching materials? 3. Do you personally consider economics instruction in secondary school important? 4. Do you personally consider business instruction in secondary school important? 5. Do you feel that students graduating with Junior Achievement have a better understanding of economics and business than other students? 6. If yes, how have students demonstrated their knowledge of economics and business because of their participation in JA? 7. Do you feel that JA is important in teaching business and economics or are their better means? Interviews: School Administrators and Ministry of Education Officials 1. What is your position? 2. Have you ever been a teacher? 3. Do you have guest speakers for the JA classes? a. Who determines the guest speakers for the JA classes? b. How often are they brought in? 4. How open are the teachers to following the JA curriculum? 5. Have you seen any effects by the JA courses on the business community? 6. Do you feel that JA is important in teaching business and economics or are their better means? 8. Do you see a difference between JA aned non JA classes and JA and non JA teachers-differences in their style of teaching? 9. Has the JA teaching style been used in other classes or in other schools that don’t have JA? 8. Have you heard or seen evidence of students graduating and using the skills they learn in JA? 9. If JA funding ended tomorrow, would this program (at some level) continue? Interviews: Teachers 1. How long have you been a teacher? 2. What do you teach? __________________________________________________________________ 3. How long have you been teaching JA? 4. Is your class participatory or is it a lecture format? 4. How often do you have a business guest lecturer? 5. How knowledgeable have been your guest lecturers? 6. How closely do you follow the JA curriculum? (we will need to translate this so that it does not sound accusatory!) 7. How much student involvement do you have in your JA classes? 8. How does this involvement compare to your other classes? 9. Has your involvement with JA caused you to change the way you teach your other classes? 10. How have your students gained a better understanding of economics and business, if any, from their involvement in JA? 11. How effective are they in involving students in the exercises? 12. What are the positive things about JA? 13. What are the negative things about JA? 14. What would you like to change about JA? 15. Have any gender issues been considered in implementing JAKg programs? 16. Have any ethnic issues been considered in implementing JAKg programs? Interviews: Teachers (non JA) 1. How long have you been a teacher? 2. What do you teach? 3. How long have you been teaching JA? 4. Have you heard of JA? 5. 6. Is your class participatory or is it a lecture format? 7. If participatory-how long? What influenced your teaching style? 8. Do you exchange ideas with JA teachers? 9. If yes, to hearing about JA-what have you heard-positive? Negative things? Interviews: JA Management Staff 1. What is your role in the organization? 2. How do you determine which schools receive JA? 3. What type of feedback do you receive about JA from the school? 4. What type of feedback do you receive about JA from businesses? 5. Has the implementation of JA been different between regions? 6. Is JAKg making any plans for sustainability if current funding ends? 7. Have any gender issues been considered in implementing JAKg programs? 8. Have any ethnic issues been considered in implementing JAKg programs? 9. Have any urban/rural issues been considered in implementing JAKg programs? Evaluation Plan Participant Team: Kyrgyzstan Project/Program to be Evaluated: Junior Achievement/Kyrgyzstan Evaluation Questions: 1. How have students demonstrated new knowledge of economics and business because of their participation in JA compared to base-line (zero)? 2. What is the perceived value of JA in the business community, contrasting those working with JA compared to those not exposed to JA? 3. Is JA management optimally structured and implemented in Kyrgyzstan – in terms of assumptions about (a) urban/rural differences, (b) gender inequities, and (c) ethnic issues? 4. Has the JA teaching style that involves greater student involvement been adopted outside the JA classroom? 5. What plans for sustainability does JA/Kyrgyzstan have if current funding sources evaporate? Evaluation Methods: Overall Design Strategy The design of the evaluation will encompass comparative theory using a range of data collection techniques to analyze the following: - Student knowledge of economics and business before and after involvement in the JA program. - Businesses’ perception of JA and JA’s perception of local business involvement. - Changes in economic and business knowledge of students in JA sites in Bishkek compared to other, more rural, locations. - Teaching style exhibited in non-JA classes in schools with JA programs (before and after). Evaluation Methods: Data Collection The team will use a combination of evaluation methods that include: - Reviewing, data combing, and analyzing documents related to the project (cooperative agreement, work plan, quarterly reports and the report from the 2004 evaluation, and the students’ pre-test and post-test results. - Focus group discussions - at least one with students in both Bishkek and Osh and another with members of the Chamber of Commerce. - Individual “key informant” interviews with teachers, business representatives, JA staff, Ministry of Education representatives, and school administrators. Team Contact Information: The team will be composed of the following experts: 1. Lauren Russell, Team Leader, lrussell@usaid.gov 2. Khorlan Izmailova, kizmailova@usaid.gov Proposed Schedule: Activity Description Location Approximate Dates Background Finalize schedule, review background documents including performance reports, development of interview and focus group questions, determining interview and focus group sites and scheduling times for interviews. Home Countries May 22 – June 5 Kyrgyzstan Interviews Visit selected sites - three days in Bishkek and two days in Osh (exact sites TBD). Kyrgyzstan June 5 – June 11 Kazakhstan Interviews Meeting with JA/CAR and USAID CTO Almaty, Kazakhstan June 13th Analysis, Report Draft final report design, additional interviews, if necessary. Home Countries June 14 – June 28 USAID/Almaty Comment Period Report draft submitted for circulation in USAID/Almaty. USAID/Almaty reviews and comments on final draft. Incorporate the comments into the report, finalize and submit to USAID/Russia. Home Countries June 29 – July 5 Submit Report Finalize report and submit to MSI Home Countries July 6 – July 21 Additional Planning Information for JA Estimate of JA Country Staff Time/Assistance Needed by Each Team: One 8-hour day of JAKg staff time assisting with site selection, providing reports, and logistical support. JAKg could assist in the evaluation by helping us gather for focus groups a cross section of: • Kyrgyzstan Chamber of Commerce; ideally at least 5 members participating. • Students; ideally as many as 15 students in each focus group. At least two focus groups will be held; one in Bishkek and one in Osh. JAKg could help us gather for key informant interviews a cross section of: • Teachers • Administrators • Ministry of Education representatives