Mid-Term Performance Evaluation of Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People USAID/ARMENIA FINAL REPORT This report was produced for the United States Agency for International Development (USAID) by Optimal Solutions Group, LLC as part of USAID’s Learning, Evaluation, and Analysis Project (Contract Number: AID-OAA-C-11-00169). USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT LEARNING, EVALUATION, AND ANALYSIS PROJECT (AID-OAA-C-11-00169) MID-TERM PERFORMANCE EVALUATION OF JUNIOR ACHIEVEMENT OF ARMENIA (JAA) ENTREPRENEURSHIP AND CIVIC ACTIVISM FOR YOUNG PEOPLE FINAL REPORT Prepared for: United States Agency for International Development/Armenia Contract Number: AID-OAA-C-11-00169 Prepared by: Optimal Solutions Group, LLC University of Maryland @ M Square Research Park 5825 University Research Court, Suite 2800 College Park, MD 20740, USA www.optimalsolutionsgroup.com Authors: Mark Turner, Chief of Party Alexandre Monnard, Lead Analyst Anna Yeritsyan, Local Consultant and Evaluation Specialist Arman Udumyan, Local Consultant and Evaluation Specialist January 31, 2014 Disclaimer The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States government. 2 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project CONTENTS Acknowledgments......................................................................................................................................... 6 Acronym List ................................................................................................................................................ 7 Exhibits List.................................................................................................................................................. 8 I. EXECUTIVE SUMMARY .................................................................................................................... 9 II. INTRODUCTION AND PROJECT DESCRIPTION ........................................................................ 11 JAA Entrepreneurship and Civic Activism for Young People Program ................................................. 12 Areas of Work..................................................................................................................................... 12 Project Activities................................................................................................................................. 12 Expected Results................................................................................................................................. 13 Mid-Term Performance Evaluation Purpose and Assignment ................................................................ 14 Project Objectives ............................................................................................................................... 14 Key Evaluation Questions................................................................................................................... 14 Report Structure ...................................................................................................................................... 15 III. METHODOLOGY ............................................................................................................................ 16 Limitations to the Evaluation .................................................................................................................. 16 IV. EVALUATION FINDINGS .............................................................................................................. 17 Project Design......................................................................................................................................... 17 Evidence that the Results Could be Sustained Beyond the Project..................................................... 18 Project Implementation ........................................................................................................................... 21 Programmatic Evolution of JAA......................................................................................................... 21 New Versus Existing Aspects of the JAA Core Program ................................................................... 21 Assessment of the JAA Program’s Foundational Components .......................................................... 22 Program Timeline ............................................................................................................................... 23 Program Coverage .............................................................................................................................. 23 Organization and Roles....................................................................................................................... 24 Achievement of Intermediate Outcomes ............................................................................................. 25 Program Efficiency ............................................................................................................................. 26 Project Results ........................................................................................................................................ 29 Business Projects ................................................................................................................................ 29 Student NGOs ..................................................................................................................................... 31 3 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project V. CONCLUSIONS ................................................................................................................................. 34 VI. RECOMMENDATIONS ................................................................................................................... 36 Project Design......................................................................................................................................... 36 Project Implementation ........................................................................................................................... 37 VII. APPENDICES .................................................................................................................................. 39 Appendix 1: USAID Evaluation Statement of Work .............................................................................. 39 Appendix 2: METHODOLOGY ............................................................................................................. 45 Initial Data Gathering ............................................................................................................................. 45 Document Review ............................................................................................................................... 45 Research Design...................................................................................................................................... 46 Sampling Framework.............................................................................................................................. 46 Sampling of Districts/Marzes.............................................................................................................. 46 Methods of Categorizing Cohorts....................................................................................................... 47 Methods of Selecting Schools ............................................................................................................. 47 Selected Implementation Districts and Schools .................................................................................. 48 Data Collection ....................................................................................................................................... 48 Site Visit to Armenia .......................................................................................................................... 48 Protocols and Guides .......................................................................................................................... 48 Coding Approach ................................................................................................................................ 49 Data Analysis .......................................................................................................................................... 49 Quantitative Data Analysis Components ............................................................................................ 49 Qualitative Data Analysis Components .............................................................................................. 50 Data Verification by Triangulation ..................................................................................................... 50 Appendix 3: Evolution of the JAA Program ........................................................................................... 52 Appendix 4: The Armenian Context....................................................................................................... 54 Appendix 5: Comparable Programs in Armenia ..................................................................................... 60 Appendix 6: Reference List .................................................................................................................... 64 Economics Education Activity ............................................................................................................ 64 Entrepreneurship and Civic Activism for Young People Project ....................................................... 65 Appendix 7: Statements of Differences .................................................................................................. 75 Appendix 8: Evaluation Design Matrix .................................................................................................. 76 Appendix 9: Sampling Framework and Site Visit Schedule ................................................................... 79 4 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project JAA Main Office Interview Schedule................................................................................................. 79 JAA Regional Offices Schedule ......................................................................................................... 79 Program School Schedule—Principal, Teachers, Students, and Alumni............................................ 80 Business Partnership Interview Schedule ........................................................................................... 81 Government Officials Schedule .......................................................................................................... 81 Appendix 10: Evaluation Tools .............................................................................................................. 82 Interview Guide for JAA Main Office #1 ........................................................................................... 82 Interview Guide for JAA Main Office #2 ........................................................................................... 85 Interview Guide for JAA Regional Coordinators ............................................................................... 88 Interview Guide for School Principals ................................................................................................ 90 Interview Guide for School Teachers ................................................................................................. 92 Focus Group Guide for School Students and Alumni ......................................................................... 96 Interview Guide for Business Consultants ........................................................................................ 100 Appendix 11: Template of Introductory Letter to JAA Program School Administrators ..................... 103 Appendix 12: Interview and Focus Group Participation Consent Form ............................................... 104 5 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Acknowledgments The evaluation team is grateful for the guidance provided by those at USAID/Armenia, in particular, Mariam Gevorgyan and Haikanush Bagratunyan, who were very responsive to the team’s questions and provided direction throughout the evaluation project. Special thanks are also extended to the JAA program staff, particularly Executive Director Armine Hovannisian, Project Coordinator Anahit Manukyan, and the regional coordinators, all of whom were very responsive to the LEAP team’s questions at the onset of the evaluation and provided valuable guidance in identifying project implementation locations. Finally, the project team leaders are grateful to Optimal’s Evaluation Specialist Jen Auer and LEAP Project Director Christabel Dadzie for their guidance in completing this evaluation. 6 Acronym List AMD Armenian Drams CIS Commonwealth of Independent States CSR Corporate Social Responsibility EV Economy and Values Research Center DEA Data Envelopment Analysis GDP Gross Domestic Product IMF International Monetary Fund ILO International Labor Organization JA Junior Achievement JAA Junior Achievement of Armenia LEAP Learning, Evaluation, and Analysis Project MoE Ministry of Education NGO Non-Governmental Organization PMP Performance Management Plans RA Republic of Armenia SOW Statement of Work TIMSS Trends in International Mathematics and Science Study USAID United States Agency for International Development USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project 7 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Exhibits List Table 1: Evaluation challenges and mitigation strategies........................................................................... 16 Table 2: Classification of Key Program Features as New or Legacy ......................................................... 21 Table 3. JAA Achievement of Intermediate Outcomes and Outputs.......................................................... 25 Table 4. Summary of program efficiency and costs metrics....................................................................... 27 Table 5. Program expenses ratio for several JA and selected chapters in the United States ......................28 Table 6. Types of products selected by student teams by region................................................................ 29 Table 7. Type of NGO implemented by student teams by region .............................................................. 32 Table 8: Selection of Schools across Regions ............................................................................................ 46 Table 9: Selection of Schools across Program Years.................................................................................. 47 Table 10: Sampling Framework and Final Selection.................................................................................. 47 Table 11: Gross Enrollment in Secondary and High School by Year and Gender (%) ..............................57 8 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project I. EXECUTIVE SUMMARY USAID/Armenia funded the four-year Entrepreneurship and Civic Activism for Young People program, which was launched in April 2011 by Junior Achievement of Armenia (JAA). The project combines a longer-standing effort to improve youth education in economics with the added goals of increasing entrepreneurship and community-based civic activities that address community needs by equipping Armenian youth with the skills and knowledge necessary to compete and succeed in tomorrow’s world. JAA operates a number of related programs to educate students on international business practices, ethics, and corporate social responsibility (CSR) issues. Under the USAID Learning, Evaluation, and Analysis Project contract (AID-OAA-C-11-00169), Optimal Solutions Group, LLC (Optimal) was tasked with planning, designing, and conducting a mid-term performance evaluation of the JAA program. The performance evaluation focused on the following questions: 1. Are the project goals ambitious enough, yet attainable and supported by context analysis (including alignment with the country’s economic, civic, education and youth development policies and strategies)? 2. Is the current project design based on sound development hypotheses leading to the achievement of sustainable results? 3. Has the replication of the past project been a justified approach for the implementation of the current project? 4. How efficient is the project in achieving its goals (e.g. how are resources/inputs [funds, expertise, time, etc.] converted to produce outputs? What is the balance of administrative vs. program costs? 5. How broad are the scopes and the ideas of the projects implemented by the students? In November and December 2013, Optimal conducted 31 site visits across 6 marzes (districts) and performed in-depth interviews and focus groups with key stakeholders and beneficiaries. The empirical evidence derived from this fieldwork and reviews of programmatic documentation supported the following findings: 1. The JAA project goals appear to be sufficiently ambitious. The stated short-term goals are attainable within the grant’s period of performance. However, it will likely take at least 10 to 20 years for the skills potentially acquired through the program and the outcomes (i.e., higher rates of entrepreneurship) to be fully realized. 2. JAA is in a good position to be financially sustainable because it has low overhead costs, and it utilizes volunteers and in-kind contributions from participating schools and teachers. In addition, the Armenian Ministry of Education continues to utilize and institutionalize JAA’s approach and instructional materials into high school curricula. However, unlike other worldwide Junior Achievement programs, JAA has not received sufficient operational funding from corporate sponsors. 3. Building upon past projects, JAA’s Entrepreneurship and Civic Activism for Young People program utilizes more than 1000 trained teachers across the country, has developed instructor and student manuals, and has a reputation, both among educators and NGOs, for achieving programmatic goals in a cost-effective manner. 4. There are no comparable programs in Armenia. As a result, it is methodologically difficult to directly assess JAA’s efficiency. Nonetheless, the overall per student cost per year appears to be reasonable given the sustained training provided to teachers and the program’s depth, intensity, duration, and added features like summer camps, the annual trade fair, and school trips to businesses. 9 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project 5. Student, teachers, and alumni in all schools visited were surveyed about student businesses. These surveys revealed that students tend to use available, easily attainable inputs and their own labor for the production. The availability of inputs, often free or donated, appeared to be a major determinant in product selection. Participating schools in urban areas exhibited significantly greater diversity in product selection than participating schools in rural areas did. Two broad recommendations flow from the conclusions of this performance evaluation—which the evaluation team perceived to be largely positive—and the findings that support these conclusions. Project Design: Student businesses (and products) should aim to fit within the main segments of the Armenian economy, such as information and communications technology (ICT), tourism, healthcare, education, and agriculture. The Armenian government has already adopted functioning strategies for each of the above sectors, which inherently increase student businesses’ and NGOs’ opportunities for guidance and access to funding from respective government bodies and private and public entities. Project Implementation: Enhance the breadth and substance of the program curriculum, provide more teacher training and technical assistance, and utilize teachers (and schools) who have already received JAA training for additional school years. These and other recommendations are described in detail in the full report. 10 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              II. INTRODUCTION AND PROJECT DESCRIPTION Since the fall of the Soviet Union at the end of 1991, the Republic of Armenia, a small, mountainous, and landlocked country in the Caucasus region, has been struggling to recover from the resulting economic and political shock, including a complete breakdown of the previous, centrally planned economy and intra-Soviet Union trade patterns. In 1994, Levon Ter-Petrosyan, the first democratically elected President of the Republic of Armenia, and his government launched International Monetary Fund (IMF)-sponsored, comprehensive measures to stabilize and reform the economy. These reforms helped transform the country into a liberal market economy and brought about impressive growth. Between 2001 and 2008, Armenia sustained an annual gross domestic product (GDP) growth ranging between 6.9 percent and 14 percent annually.1 However, this steady growth came to an abrupt end in 2009, when the country’s GDP declined by more than 14 percent, despite loans from multilateral institutions, before stabilizing around 3­ 4 percent. Since 1992, the U.S. Agency for International Development (USAID) has worked with Armenia to assist the country in transitioning towards creating a stable market economy, respecting democratic principles, and enhancing amicable relationships with its neighbors. Especially after contentious elections in 2008, USAID has focused its interventions in Armenia on preventing a simultaneous deterioration in fundamental freedoms and economic development, which threatens to erase past economic gains.2 To accomplish this, USAID has implemented programs to increase Armenia’s competitiveness, strengthen democratic governance, and assist in areas of energy security, access to primary health care, and protection of vulnerable populations. USAID has also fostered freedom and economic development by investing in the country’s young adults.3 Junior Achievement of Armenia (JAA), an affiliate of Junior Achievement (JA) Worldwide, was established in 1992 to assist Armenia’s transition to democratic governance and a free-market economy. Since its inception, JAA’s programs have focused on improving the lives of young Armenians by giving them the tools necessary to compete in the global economy and to foster democracy. JAA’s stated mission is to “give today’s Armenian youth the necessary skills and knowledge to compete and succeed in tomorrow’s world.” Specifically, the JAA Entrepreneurship and Civic Activism for Young People program achieves organizational goals by designing and implementing economics and civics education. The JAA Entrepreneurship and Civic Activism for Young People program was partly built on a model established in 1919 in the United States by Junior Achievement, which delivers financial literacy and educates young people about means of production and free enterprise through experiential learning. Today, JA Worldwide operates this same curriculum in more than 120 countries across the globe.4 1 The World Bank, GDP growth (annual percentage), available at http://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG/countries?page=2 2 USAID, Armenia Country Profile, available at http://armenia.usaid.gov/en/node/37 . 3 Civic Participation Project, Youth and Community Action Program evaluation available at http://pdf.usaid.gov/pdf_docs/pdacu499.pdf 4 Junior Achievement Worldwide website. 11 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              JAA Entrepreneurship and Civic Activism for Young People Program The USAID/Armenia-funded Entrepreneurship and Civic Activism for Young People program launched in April 2011 as a four-year project implemented by JAA.5 The project combines a longer-standing effort to improve youth education in economics with the added goals of increasing entrepreneurship and community-based civic activities that address community needs by equipping Armenian youth with the necessary skills and knowledge to compete and succeed in tomorrow’s world. JAA operates a number of related programs to educate students about international business practices, ethics, and corporate social responsibility (CSR) issues. The scope, purpose, and goals for this evaluation are described below. Areas of Work The Entrepreneurship and Civic Activism for Young People program focuses on the following five interconnected objectives:6 1. arm young people with entrepreneurial skills and prepare them for the work force of tomorrow; 2. teach young people to understand and appreciate work ethics and corporate social responsibility; 3. engage corporations in the education of young people and expand young people’s horizons with respect to their future careers; 4. empower and provide young people with skills and avenues such as non-governmental organizations (NGOs), which they can utilize to undertake projects benefiting their community; and 5. establish alumni clubs and networks and encourage them to continue their community service. Project Activities To achieve these goals, the program is comprised of two distinct but related project components: the business component and the NGO component. The business component involves each participating class creating a student business and studying CSR and its benefits for both businesses and communities. After the student businesses have been implemented and operated over a four-month period, the students shut them down, disperse the dividends, and decide what percentage of profits should be earmarked for community service. In the next school semester, the NGO component involves each participating class creating an NGO, with the objective of carrying out a community project funded in part by the profits reaped in the business component. Students are encouraged to conduct community outreach to leverage their profits, and all eligible classes compete for one of ten $300 grants awarded yearly by JAA to community projects deemed particularly compelling. The program is primarily designed to take place after school, requiring at least two meetings per week, and can be implemented in high schools, vocational schools, and colleges. Schools and other institutions seeking to participate in the JAA program can apply and are selected by JAA regional coordinators who maintain relationships with principals and teachers in their region. Examples of school selection criteria provided by JAA include the level of interest within the institution, its location, and the classes offered. JAA aims to include a total of 1,500 students in 100 institutions in each program cohort. The two components of the Entrepreneurship and Civic Activism for Young People program last a total of 11 months. Participating high school students normally start in September of 10th grade, which is the same time that these students typically cover economics in their social studies class. Participating students implement the NGO component in the following semester and then complete the program at the end of the academic year. 5 The USAID-JAA Cooperative Agreement was signed on April 1, 2011. 6 http://www.jaarmenia.org/Entrepreneurship%20and%20Civic%20Activism%20for%20Young%20People 12 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              JAA requires participating teachers who implement the program at the school level and guide the students in their endeavors to take part in a training session generally held in Yerevan during the month of August before the academic year begins. This training session covers active teaching methods, the pedagogical material used in the program, and program expectations and reporting requirements. This training is followed by a series of regional monthly training seminars that cover upcoming program material and provide a forum for questions and discussions. Student businesses are expected to be undertaken with guidance from a business consultant, generally a local volunteer who has hands-on experience with operating a business. In addition to providing teacher training, JAA also provides nearly all the material supporting the program, including textbooks about applied economics, ethics, and civics and company kits that include accounting sheets, manuals, and job descriptions for company officers. In order to enrich the experience of participating students, JAA also incorporates other activities into the program, often conducted in partnership with other organizations.7 For example, JAA has created the following programs: Children’s Festival and Trade Fair, organized in partnership with the Cafesjian Family Foundation; the JAA Summer Camps started by JAA in 1993; and Brain Ring competitions, organized in partnership with the Armenian Central Bank. The highest rated student companies are selected to display and sell their products at the Trade Fair. Schools are also encouraged to send their best students to take an exam on applied economics and civics to qualify for one of the 100 spots at the annual Summer Camp. JAA summer camps further reinforce the content of the curriculum by including activities such as stock market simulations, mock trials, and the production of commercial skits. The Brain Ring competitions focus on students testing and sharing their applied economics knowledge among their peers before competing as teams against other schools. Other activities offered by JAA include Skills@Work, an international enterprise competition for young people from vocational schools; JA More than Money, organized in partnership with HSBC Bank, which focuses on teaching students about earning, spending, sharing, and saving money; among others others. Expected Results According to the program implementer, the overall aim of the Entrepreneurship and Civic Activism for Young People program is to foster model citizens who8  are skilled at their job;  appreciate free enterprise;  value democracy; and,  are active and contributing citizens of their community. For this reason, USAID/Armenia requested this mid-term evaluation of the Entrepreneurship and Civic Activism for Young People program to determine if the program has been implemented as designed.9 7 Please note that these other activities will not be evaluated in this report, but the interplay between them and the Entrepreneurship and Civic Activism for Young People program will be discussed. 8 Ibid. 9 http://www.jaarmenia.org/Mission_and_History 13 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Mid-Term Performance Evaluation Purpose and Assignment Under the USAID Learning, Evaluation, and Analysis Project contract (AID-OAA-C-11-00169), Optimal Solutions Group, LLC (Optimal) was tasked with planning, designing, and conducting the mid-term performance evaluation of the JAA program. This study, which was planned, designed, and conducted by Optimal, aimed to help determine the effectiveness of the USAID/Armenian Mission’s investment in the program and inform decisions about continued funding. This evaluation adhered to the guidance provided by USAID Evaluation Policy, 2011. Specifically, the performance evaluation focused on highlighting the JAA project design, achievements, and progress details to produce a descriptive and normative understanding of the program and its ability to achieve expected results. The evaluation team, which consisted of local and U.S.-based staff, used quality standards during project planning, data collection, and analysis of JAA data to provide credible evidence that corresponds directly to the evaluation questions as proposed in the statement of work (SOW). In addition, the evaluation team took into consideration local contextual information in designing the data collection tools to ensure that the results of the evaluation are useful to the identified audience. The results of this evaluation are evidence-based and the evaluation provides detailed analyses of JAA project performance. Therefore, these results can be used to ensure accountability of JAA stakeholders and to improve the effectiveness of future USAID investments in youth entrepreneurship and civic engagement programs in Armenia. Project Objectives The JAA mid-term performance evaluation had the following objectives:  examine the effectiveness of the current JAA program design, intervention, implementation strategies, and approaches to arming Armenian youth with the skills and knowledge necessary to compete and succeed in tomorrow’s world and appreciate free enterprise, democracy, and civic engagement;  investigate the efficiency and sustainability of the current JAA program; and  document lessons learned, good practices, and factors that influence program effectiveness that can be disseminated to JAA program implementers. Key Evaluation Questions The evaluation focused on the following five guiding evaluation questions, as posed by USAID/Armenia in the SOW (see Appendix 1). 1. Are the project goals ambitious enough, yet attainable and supported by context analysis (including alignment with the country’s economic, civic, educational and youth development policies and strategies)? 2. Is the current project design based on sound development hypotheses leading to the achievement of sustainable results? 3. Has the replication of the past project been a justified approach for the implementation of the current project? 4. How efficient is the project in achieving its goals (e.g., How are resources/inputs [funds, expertise, time, etc.] converted to produce outputs)? What is the balance of administrative vs. program costs? 5. How broad are the scopes and the ideas of the projects implemented by the students? 14 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Report Structure The next section (Section III) provides an overview of the evaluation methodology, which included developing an evaluation design, conducting a document review, identifying a sampling framework, and conducting site visits to Armenia to perform interviews and hold focus groups with relevant stakeholders. Section IV presents detailed evaluation findings, and Section V provides conclusions and preliminary recommendations, specifically responding to USAID/Armenia’s five key evaluation questions. 15 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project III. METHODOLOGY This section details the methodology used to conduct the mid-term performance evaluation. The evaluation team worked within industry standards to ensure that the methods used were relevant to the project and would provide actionable results to inform USAID/Armenia’s decision making. The team did the following as part of the evaluation:  reviewed and verified project administrative documentation from the start of the program in 2011 to the start of the 2013 academic year;  collected and analyzed interview data from program administrators and stakeholders to evaluate the program design and implementation at the macro-level;  conducted 31 site visits across 6 marzes (districts) including Yerevan and performed in-depth interviews and focus groups with key stakeholders and beneficiaries to understand micro￾level results related to the business and NGO components of the program;  conducted quantitative analysis of the program’s efficiency and progress toward target goals; and  identified the project’s lessons learned and areas of sustainability to make data-driven recommendations for USAID to use in deciding the future of the JAA program. For a detailed description of the methodology undertaken, see Appendix 2. Limitations to the Evaluation The mid-term JAA program evaluation included some limitations, which are outlined in Exhibit 1 below. Exhibit 1: Evaluation challenges and mitigation strategies Challenge Mitigation Strategy or Explanation Non-representative sample of JAA participants and respondents None. Selection of respondents was not intended to yield representative samples. No comparison program in a comparable country None. This is an implementation evaluation, not an impact evaluation. Benchmarks are described when available. No comparison of non-participating schools in Armenia None. Insufficient resources to canvass non￾participating schools in Armenia. Incomplete documentation and information Conducted in-person interviews with a wide range of stakeholders to supplement documentation. Insufficient duration since program implementation None. Evaluation relies on short-term and intermediate outputs and outcomes. 16 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              IV. EVALUATION FINDINGS The findings provided below were derived using evidence compiled through document reviews, semi￾structured interviews with stakeholders, focus groups, and observation of JAA-supported activities, taking into account the hypotheses derived from the SOW and evaluations of comparable programs in other countries. Project Design Evaluation Question 1: Are the project goals ambitious enough, yet attainable and supported by context analysis (including alignment with the country’s economic, civic, education and youth development policies and strategies)? The following are the specific objectives of the Entrepreneurship and Civic Activism for Young People program: 1. arm young people with entrepreneurial skills and prepare them for the work force of tomorrow; 2. teach young people to understand and appreciate work ethics and corporate social responsibility; 3. engage corporations in the education of young people and expand young people’s horizons with respect to their future careers; 4. empower and provide young people with skills and avenues such as non-governmental organizations (NGOs), which they can utilize to undertake projects benefiting their community; and 5. establish alumni clubs and networks and encourage them to continue their community service. A literature review and an examination of expert interviews and country development strategic papers revealed a similar set of objectives for the country’s education system. The following extract from the State Curriculum of General Education10 states that a graduate of school must  have acquired the knowledge defined by the state education standards and an ability to apply acquired knowledge creatively in real life;  be an independent thinker and problem solver;  demonstrate qualities of understanding and cooperation with friends in the same age group, as well as with parents and all other members of society including both the old and the young;  appreciate both rights and responsibilities and be law-abiding, honest, humane, responsible, an initiator, and an active citizen with an interest in social affairs;  be able to assess personal capacities realistically, have confidence in personal abilities, be willing to participate in self-education, and be committed to life-long learning;  be able to achieve a profession and a job compatible with personal preferences, interests, skills, and abilities and be able to manage household affairs independently. The overlap of the JAA objectives and the country’s education objectives suggests that the JAA objectives are aligned with the goals of the Ministry of Education and Science of the Republic of Armenia. 10 National Curriculum for General Education, pp 4-5 http://www.ibe.unesco.org/curricula/armenia/ai_fw_2010_eng.pdf 17 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              In comparison to other developing countries, Armenia’s entrepreneurship rate among adults is low. Individual business ownership in Armenia is only 2 percent as compared to 15 percent in other developing countries. Entrepreneurship provides alternatives for employment and develops creativity, especially among young people, so it is important to develop entrepreneurial skills and interest among school-aged youth if there is a desire to increase the rate of entrepreneurship and business ownership. Additionally, research suggests that youth interest in entrepreneurship is high, but can quickly diminish if not nourished. According to the National Youth Report, 68 percent of young people interviewed preferred to establish their own business rather than be employed at a decent salary.11 The National Youth Report also found that running a business (as opposed to being a paid employee) provides a greater degree of freedom in terms of planning one’s own life, and contributes more to self-respect.12 In summary, the JAA project goals appear to be sufficiently ambitious. The stated short-term goals are attainable within the grant’s period of performance. However, utilizing the skills potentially acquired and the outcomes yielded (i.e., higher rates of entrepreneurship) will likely take at least 10 to 20 years to be realized. Evaluation Question 2: Is the current project design based on sound development hypotheses leading to the achievement of sustainable results? Evidence that the Results Could be Sustained Beyond the Project The sustainability of a project such as JAA’s depends on a variety of factors, including the level of commitment of implementers, community buy-in, and the ability of the organization to keep costs low, obtain both corporate and public sponsorship, and create strong alliances with other NGOs. Based on the data collected from site visits, there was consistent evidence that implementers have a strong commitment to the program. This commitment went beyond mere words; it was reflected in deliberate actions. Nearly 74 percent of the surveyed teachers mentioned that the resources provided by JAA were sufficient to operate the program; however, nearly 70 percent of teachers who answered this question reported that the school or the teacher was providing financial and/or in-kind support for the implementation of the project. In addition, all surveyed teachers mentioned that the time they spent on the project was significantly greater than what was allocated by the project timeline. Teachers themselves have little financial incentive for investing their time and effort into this project—they are paid 8,000 Armenian drams (AMD) per month to cover their travel expenses. By comparison, an Armenian teacher’s average salary is approximately 2,000 AMD (of course the numbers will be different depending on experience and credentials) per academic hour during regular classes. Comparing the ratio of hours worked per AMD, teachers engaged in JAA activities work 2 to 3 more hours per AMD than those participating in non-JAA activities. In other words, teachers engaged in JAA activities are consistently volunteering extra time, which has the potential to make the JAA project more financially viable both in the short and long term. Interviewed teachers nearly unanimously agreed that the JAA program, including the CSR component, enhanced students’ community engagement. JAA regional coordinators reported that projects undertaken under the JAA program attracted active students, irrespective of academic performance. For example, a regional coordinator indicated that the experiential nature of the JAA program allowed the participating 11 National Youth Report of Armenia, analytical part, RA Ministry of Youth and Sport ISBN 978-99941-2-856-3, Yerevan 2012, p. 58 12 National Youth Report of Armenia, analytical part, RA Ministry of Youth and Sport ISBN 978-99941-2-856-3, Yerevan 2012, p. 56 18 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project students to demonstrate the results of their efforts and to serve as visible role models for their peers and the community. Armine Ivanyan, an expert at the National Institute of Education, mentioned that “the program gets the student out of the school and greatly encourages community involvement.” Community buy-in was not assessed in this evaluation because community leaders were not directly interviewed. However, principal and teacher interviewees suggest that parents were generally enthusiastic about their children’s participation in the JAA program. Several teachers reported that students would routinely assist their parents with business-related challenges during and after program participation. These same interviewees reported that JAA students provided unsolicited advice to the designated business consultants during the program. Our findings suggest that business consultants’ JAA engagement may also be sustainable. More than 90 percent of the business consultants interviewed indicated that they chose to engage in the JAA project because they sought to support youth and community development or wanted to help enhance students’ entrepreneurial skills. Furthermore, nearly half of the business consultants interviewed believed that JAA students have enhanced business knowledge and are more mature compared to other students of the same age. They also mentioned that participating students are more proactive and have better communication skills. JAA regional coordinators also attributed these qualities to project participants. JAA Armavir/Aragatsotn Regional Coordinator Flora Khachatryan reported that JAA students have broader sets of interests and are happy to take the lead. These comments tie directly to several of the shortcomings identified in Armenian managers (discussed in the Armenian Context subsection of Project Design), suggesting that the program may have long-term value for students once they enter the workforce, if the gains are sustained. In combination with utilizing volunteers’ time and in-kind contributions, JAA’s overhead expenditures appear to be constrained. For example, JAA utilized shared office space with other organizations in Yerevan and in the marzes. JAA’s financial records indicate a historical tendency to constrain overhead expenditures. For example, the 2006 USAID grant was not completely expended before its official termination date, yet the program appeared to have completed its objectives. JAA then requested a no￾cost extension and continued to provide services. JAA’s efficient use of program funds is also demonstrated when comparing program expense ratios with JA chapters in the United States, based on the overall per year cost on a per school or per student basis. JAA has also demonstrated a strong record of accomplishment in creating sustainable and mutually beneficial partnerships with the private sector. For example, HSBC has been persistently involved in supporting the program, including providing multi-year grant awards for the JA More than Money program and through career day events. Other companies have recently followed suit, including VivaCell￾MTS. JAA has demonstrated similarly strong partnerships with public organizations, including the Armenian Central Bank, with which it has operated the Brain Ring contest for over 5 years. The partnership with the Ministry of Education (MoE) has also strengthened over time and yielded tangible results, although the journey has been slow and required significant effort according to the JAA main office. For example, the MoE has recently drafted an order to include applied economics in the 9th grade curricula along with instructional material on entrepreneurship and business. These alterations to the Armenian educational curricula can be directly traced to JAA’s demonstrated “proof of concept” and active engagement and partnership with the MoE. 19 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project JAA has made some efforts to diversify its funding sources and to collaborate with NGOs with comparable goals serving the same populations. An interview with World Vision, with whom JAA collaborated for several years, also illustrated that JAA made inroads with other NGOs, both domestic and international. For additional information, see Appendix 3. With regard to the question of the sustainability of student gains, 92 percent of business consultants mentioned that the JAA program positively affects the students’ business and/or civic mindset. Of these positive responses, 62 percent of the business consultants indicated that the students enhanced their entrepreneurial knowledge. However, 30 percent of the business consultants indicated that it was too soon to assess any changes in the students’ mindset. In addition, JAA regional coordinators mentioned that about 80 percent of students believe that “one can bring a positive change in his/her community” and that within the JAA program, the students are the primary actors, not the teacher. These same respondents indicated that marketing, defining the price, teamwork, and CSR contribute to the transformation of students’ mindset. This evidence, combined with the findings from other countries about the effect of the JA business program and the generally weak entrepreneurial ecosystem in Armenia suggest that the JAA program has the potential to affect youth’s entrepreneurial and business mindset and knowledge. In other words, the youth who participate in the JAA program may achieve sustained marginal improvements as compared to otherwise similar Armenian youth who did not participate in the JAA program. More broadly, given the national scope and relatively high penetration of the JAA program, there could be marked improvements in the Armenian entrepreneurial ecosystem in 10 to 20 years. True sustainability could be gauged by schools’ persistence in providing the JAA-designed program without JAA funding. Some schools have provided the program while not (or most often, no longer) formally involved and funded by JAA. In these cases, JAA monitors the students’ performance and provides program material (e.g., textbooks). A discussion with the JAA main office suggested that these schools tend to work very hard to obtain their own funding and tend to be more active in selling their product during trade fairs to cover their transportation costs. However, program implementation also involves complementary activities implemented by JAA, including the trade fairs and summer camps that schools are unlikely to be able to coordinate and sustain if JAA were to cease its operations. These activities, organized by JAA’s main office, are fairly central to the program’s design: trade fairs provide an important source of experiential learning and revenue for student businesses, and summer camps incentivize students to learn and excel, since attendance is contingent on scoring well on an economics exam. This evidence suggests that it may be possible for some schools to offer the JAA program (or at least components of the program) with little or no external support. However, it is important to remember that JAA’s efforts to promote, coordinate, expend, and improve the program have been instrumental in sustaining its momentum. Significantly reducing JAA funding or stopping JAA’s main office operations would likely result in substantial interruptions, if not the discontinuation, of school-based JAA programs. 20 o o o o USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic                                                              Project Implementation Evaluation question 3: Has the replication of the past project been a justified approach for the implementation of the current project? Programmatic Evolution of JAA The current JAA program is the result of a two-decade long evolution. Initially, JAA offered an after￾school elective class focused on the free market and economics in just eight Armenian schools. This program—Economics Education—grew over the years and is the precursor to the current Entrepreneurship and Civic Activism for Young People program. By 2004, JAA reported that 500 high schools in Armenia were participating in Economics Education and that the organization was already receiving some funding from USAID.13 In 2005, a comparable USAID grant was terminated and was subsequently renewed in September 2006 for $732,000 over three years, with the goal of training 900 Economics Education teachers, promoting economic education in Armenia by securing placement of economics in the secondary school curriculum, and expanding internet-based educational programs. To address which aspects of Entrepreneurship and Civic Activism for Young People are new and which are carryovers from previous programs, a brief overview of the program’s evolution is described in Appendix 3. New Versus Existing Aspects of the JAA Core Program Reviewing the activities that took place prior to the initiation of the current USAID grant allowed the new features of the Entrepreneurship and Civic Activism for Young People program to be distinguished from the program features that existed under the Economics Education program and other related programs. E2 identifies key program features as legacy and new. Exhibit 2: Classification of key program features as new or legacy 21 Activism for Young People Project  Theoretical aspects of applied economics were taught.  JA programs and material were widely used.  Each participating class developed and implemented a business. Students sold their businesses’ products at the JAA trade fair.  Teachers were trained in economics and civics education. Veteran teachers were used as trainers. New teachers participated in a 2-month, high￾intensity training, followed by monthly seminars.  Part-time regional coordinators selected schools and preformed other aspects of program administration.  Sites were visited frequently to assess program New Features:  Corporate Social Responsibilities (CSR) was added to program coursework.  Business consultants were added to guide the development of the student businesses and to discuss CSR.  Each participating class develops and implements an NGO. $300 grants are given for strong NGO ideas.  There is a greater focus on entrepreneurship and business operations than in prior program.  A business ethics component was integrated into the program curriculum, whereas before it was a separate class never widely adopted.  JAA alumni clubs were created and are leveraged.  Coursework focuses on human rights.  A customized curriculum beyond what was offered by JA was developed and customized for 13 This information was reported in a December 2004 JAA press release (available at http://www.armeniapedia.org/wiki/Junior_Achievement_of_Armenia). However, the specific terms and duration of this contract were not specified. Legacy Features: o o USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic 22 Legacy Features: New Features: delivery and teacher performance.  Pre/post questionnaires were used to assess the effectiveness and value of teacher training; and student learning.  Applied economics was included in the official social science curriculum.  JAA summer camps focused on economics and civics for top-scoring students.  Students took JAA “Career Day” field trips to established businesses.  Ancillary (e.g., Brain Ring, JA Titan, Skills@Work, etc.) and value-add (e.g., Aflatoun) programs were offered. the Armenian context. Overall, there are relatively few key features of the current Entrepreneurship and Civic Activism for Young People program that did not exist under the prior Economics Education program. However, it is important to note two things. First, the Economics Education program evolved markedly from 2006 to 2010, and this evolution was retained in the Entrepreneurship and Civic Activism for Young People program. For example, the implementation of JAA alumni clubs and the use of questionnaires to assess the training provided to both teachers and students are both important features. The former was only begun towards the end of the Economics Education program and was more integrated into its successor. Second, despite the relatively low number of new key features included in the Entrepreneurship and Civic Activism for Young People program (as shown in table above), many of them are significant and generated in-depth changes in the program’s curriculum. For example, the inclusion of the NGO phase and the CSR and business ethics curriculum are integral parts of the program, adding important elements of learning. This integration leads students to study CSR at the same time that they are making decisions about the allocation of their student businesses’ profits, allowing the students to gain exposure to philanthropic endeavors that are designed to build upon entrepreneurial skills learned in class and experientially. Assessment of the JAA Program’s Foundational Components As discussed in the introduction and the summary of the evolution of JAA’s program, the “Entrepreneurship and Civic Activism for Young People” program was built upon programs designed by JA for a high school audience and implemented in over 100 countries over the past several decades. These programs include the JA Company program, through which students organize and operate an actual business after school, and JA Economics, a classroom-based program exploring the structure and actors of a free market economy and how economic principles influence business decisions. In addition, several of the ancillary programs offered by JAA, including JA Titan, are also directly adapted from JA’s offerings. While the existence of these programs over several decades can be viewed as a proxy of their value and effectiveness, additionally, many have also been relatively recently evaluated by third party organizations. One of the studies evaluating the effect of the JA Business program in the United States conducted pre￾and post- surveys and demonstrated with a high level of statistical significance that students scored 10 percent higher on a standardized test after taking part in the program. While JAA did not conduct (or at least did not provide) the statistical significance of its findings, the organization reported in their quarterly reports that its program had an even greater effect on test scores, often resulting in 15–35 percent higher scores. These figures appear to hold stable over time, as similar figures were reported for the school year Activism for Young People Project USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              of 2007-2008, for example.14 This is particularly impressive since JAA surveyed all 1,000 participating students, meaning that the changes recorded were averages across all regions and school types. Additionally, a longitudinal study of students who participated in JA during 5th grade, 8th grade, and 11th grade in the United States found that the program had a significant effect on students in terms of high school graduation, confidence in their ability to complete college, college attendance right after graduation, and summer employment.15 This effect was measured against a comparison group of students who did not participate in the program but were otherwise similar. More details of these evaluations are provided in Appendix 5. Evaluation question 4: How efficient is the project in achieving its goals (e.g., How are resources/inputs [funds, expertise, time, etc.] converted to produce outputs)? What is the balance of administrative vs. program costs? Program Timeline As stated in Section 1 under the description of the Entrepreneurship and Civic Activism for Young People program activities, the two main components of the program occur during the school’s academic year: the business component in the first semester and the NGO component in the second. A detailed timeline for these yearly activities is provided in Appendix 10 of this report. Program Coverage The JAA main office communicates its rationale for selecting the program schools a number of different ways, particularly via its quarterly performance reports and project performance management plan (PMP). To this end, JAA has utilized the following criteria for selecting the program schools: geography, past performance, enthusiasm, and commitment on behalf of the school administration. In the first year of the current program, the JAA regional representatives conducted visits and made phone calls to schools in order to announce the program’s inception, which prompted approximately 200 schools to apply for the program.16 Of those 200 applicants, JAA decided to choose 100 schools to meet its expected output. Ten schools from Yerevan and 9 schools from each of the other 10 regions in Armenia were selected; thus, the first selection criterion was established as geographic regions. 14 Although reports dating back to this period were tied to the previous Economics Education JAA program, the student business component has not evolved significantly across the two programs. However, despite the fact that JAA was at least partly responsible for training teachers and shaping the curriculum for economics during its inclusion into the official social studies curriculum, a comparison of the figures across time may not be appropriate, as the depth at which the subject is covered likely decreased due to sharing the subject with ethics, philosophy, and aesthetics. 15 The Education Group, JA Worldwide Headquarters, The Impact of Student Participation in JA Worldwide: Selected Cumulative and Longitudinal Findings, 2004, available at http://www.myja.org/programs/evaluation/reports/longitudinal_evaluation.pdf. 16JAA Quarterly Report #1 (April – July 2011), p 1. 23 Bjni school principal, Yasha Sahakyan, underwent JAA trainings in the early 2000s. He initiated Bjni school participation in the current JAA program and was coaching students himself. Two years in a row, Bjni school students attracted more funding for an NGO component than many other schools. As a result, their community projects really stood out and contributed to awareness about the project and willingness to further sustain it. USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              Additionally, factors such as the past performance of teachers in JAA activities are considered when selecting schools. JAA documented—and the evaluation team confirmed during site visits—that a substantial portion of teachers had participated in the current or prior JAA programs. According to JAA quarterly reports and key informants, utilizing teachers who had previously participated in JAA programs allowed teachers to build upon their past knowledge and experiences. This approach mitigated potential risks that JAA may have encountered when training and on-boarding new teachers; thus, making the program more efficient.17 While using teachers with prior JAA training and/or experience is important, JAA also aims to attract and train new teachers from all Armenian regions. One way that JAA has gauged whether a school will be a worthwhile investment is by the support and enthusiasm of school administrators. Historically, JAA has observed that there is a strong correlation between a principal’s perceived “ownership” of his or her school’s JAA program in the project implementation phase and project outputs. Site visits suggest that administrators often times contribute school resources and help garner community engagement in the schools’ JAA program. Organization and Roles JAA The JAA administrators develop (or identify) course curriculum; manage regional representatives; raise program money and awareness; form program partnerships; participate in cross-regional student events; evaluate program performance; conduct school site visits to evaluate teachers; and engage in other organizational management activities. JAA regional coordinators work on behalf of JAA to provide support to the program and to participating schools. Most coordinators are part-time employees who oversee a marze in which they select schools to participate in the JAA program, conduct site visits, administer and/or collect JAA surveys, and play other administrative support roles. A detailed diagram in Appendix 11 outlines the current distribution of responsibilities and the associated JAA staff member’s information. Schools For the JAA program to be adopted in schools, the principals must at a minimum provide their authorization, provide financial support, and be supportive of the program. Therefore, principals play a key role in the initiation and performance of the program at their respective schools. The principals may know the total (direct and indirect) cost of operating the program and have insight into how much funding (pecuniary and non-pecuniary) that JAA provides to the school, how much is incurred by the school and its staff, and whether the funds are sufficient to sustain the school’s participation, with and without JAA funding. It is also expected that the principal will have an opinion on the effect the JAA program has on teachers and students in his or her school. Participating teachers are trained by JAA to provide classes in applied economics and civic activism. Teachers are also responsible for guiding the students through major JAA activities, such as the student business and NGO components. Teachers may have been involved in student planning for major JAA activities and the level of involvement of business consultants in classrooms. 17JAA Quarterly Report #1 (April – July 2011), p 1-2. 24 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              Participating students are the ultimate beneficiaries of the program. While being guided by the teacher, JAA staff, and oftentimes a business consultant, the students are responsible for developing a business model and NGO project and then undertaking tasks to complete the program and largely self-determined goals. Achievement of Intermediate Outcomes The JAA project management plan (see Appendix 12), quarterly reports, and annual reports submitted to the USAID/Armenia show that the outcomes and outputs for each of five programs objectives for both 2011–2012 and 2012–2013 program years have been accomplished (where data was readily available). The performance indicators for the JAA objectives were both quantifiably and qualitatively measurable. The quantitative indicators were relatively easy to measure by utilizing the JAA PMP and periodic reports submitted to USAID/Armenia. However, qualitative indicators were inherently more difficult to measure and analyze, given the nature of the indicators, the ongoing status of the JAA program, and the premature assessment of JAA students’ career development. As a result, it was not feasible to assess how participating students’ experience with launching and operating a business, their knowledge of finance, and the lessons learn on CSR have affected their entrepreneurial skills and proclivities. Moreover, apart from isolated cases, there are no data being collection to systematically track JAA alumni’s entrepreneurial activities. Exhibit 3 outlines the five JAA objectives, the associated outcomes and outputs, and whether the evaluation team determined that JAA achieved its intermediate outcomes. Exhibit 3. JAA achievement of intermediate outcomes and outputs Goal 1: Arm young people, namely high school students, with entrepreneurial skills and prepare them for the work force of tomorrow. Component Indicator Target Actual Status Outcome Student companies are established. 100 100 Accomplished Output Teachers are trained to work with students. 100 100 Accomplished Relevant materials for establishing a business are prepared by JAA’s home office. 6 6 Accomplished Market research is conducted by students and business plans are written. 100 100 Accomplished Goal 2: Teach young people to understand and appreciate work ethics and corporate social responsibility (CSR) Component Indicator Target Actual Status Outcome Number of students who understand 1500 1500 Accomplished18 18 JAA conducted tests to measure the conceptual understanding of CSR among students before they participated in the program and after program completion. Assuming that these tests accurately measure the students’ understanding of CSR, the program did increase CSR knowledge, as evidenced by an average increase of 50 percent in correct answers from the pre- to the post-tests. The students’ fulfillment of the second part of the outcome—“use it for the good of their school or community”—can be assessed only if the team assumes that participation in the NGO component, through which they use the profits of their own business to complete a community project, meets this criterion. 25 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                                      what CSR is and use it for the good of their school or community Output Teachers are trained. 100 100 Accomplished Materials on CSR are prepared. 1 1 Accomplished Community problems are identified and one project selected. 0 100 Accomplished Goal 3: Engage corporations in the education of young people and expand young people's horizons with respect to their future careers. Component Indicator Target Actual Status Outcome The number of business representatives who are involved in the teaching process 100 100 Accomplished Output The number of corporations contacted 20 20 Insufficient Data19 The number of corporations working with the children 10 10 Accomplished The number of contacts between businessmen and students 300 300 Accomplished Goal 4: Empower and provide young people with skills that they can utilize to form NGOs to undertake projects benefitting their community. Component Indicator Target Actual Status Outcome The number of NGOs formed by students and the amount of voluntary work undertaken for the betterment of the community N/A 100 Accomplished Output Students form NGOs N/A 100 Accomplished Students select a project N/A 100 Accomplished Students complete a project N/A 100 Accomplished Goal 5: Establish alumni clubs and networks and encourage them to continue their community service. Component Indicator Target Actual Status Output Number of events organized by alumni 33 33 Not accomplished20 Number of projects in which the alumni are involved 200 200 Not accomplished21 Program Efficiency Using the actual cost structure provided by JAA for the first two years of the Entrepreneurship and Civic Activism for Young People program, it is possible to compute several project costs and efficiency metrics. The cost structure used by JAA only incorporated expenditures that were charged to the USAID grant; therefore, the cost structure included summer camps, student awards and prizes, ten NGO grants, 19 Several corporations participated in a seminar held by JAA at the American Chamber of Commerce in May 2011. The exact number of participants is not available. 20 There were total of 69 events organized by alumni during the 3 years of the project 21 A JAA program coordinator clarified that most project alumni are students and temporarily reside in the capital, so many meetings and events were cancelled. 26 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              and the portion of funding for the trade fairs that was charged to USAID. However, it did not include costs related to other programs, including Brain Ring, JA More than Money, or Skills@Work. These metrics are summarized in Exhibit 4 below. The ratio of program costs to total costs is calculated in two ways. The first way only includes specific costs (those that are easily identifiable with and benefit the JAA core program), while the second aims to capture specific costs as well as the share of common costs that can reasonably be attributed to the program.22 Reasonable attribution was based on information in the progress reports submitted by JAA as well as information about the program gathered during interviews.23 These two metrics can be thought of as the lower and upper bounds of the real program costs to total costs ratio, which only JAA can calculate and which might be difficult to compute.24 Exhibit 4 also reports the overall cost of the program, which was defined as the total expenditure under the USAID grant over the first two years per school and per student, to provide a notion of how “expensive” the program is in relation to its site and participant counts. Overall, each student costs approximately $110 a year, and each school costs approximately $1,750 per year. These cost measures are different from the next two in the table, which are costs at the margin (i.e., adding one new school to the program and adding one student to a school already participating). The costs at the margin are different because the cost structure was broken down into two types of costs, fixed and variable. Fixed costs are those that can reasonably be expected to remain the same when another school is added while variable costs are those expected to change due to the addition of a new school or student. For example, the salary of regional coordinators is not expected to rise due to the addition of one new school or a new student in a participating school, since the existing coordinator for a given region could monitor and work with an additional school or student, whereas the inclusion of a new student would affect the cost of required program textbooks and other material. Marginal costs are a useful measure, but it is important to note that they do not scale well, since adding 50 new schools may, for example, require one or several regional coordinators to return to full-time employment, which is not accounted for in the marginal cost calculation. The actual cost of adding a number of schools significantly larger than one is expected, therefore, to fall somewhere between the marginal and the overall cost for an additional school. Exhibit 4. Summary of program efficiency and costs metrics Project Efficiency and Cost Metrics Program Efficiency Measures Ratio of program costs to total costs 46% Ratio of program costs to total costs (incl. common costs) 71% Overall Cost Measures Overall cost per school per year $1,751.80 22 Specific costs include the salary of regional representatives, various aspects of teacher training, transportation for regional representatives, program textbooks and material, summer camps, NGO grants, and various performance awards and prizes for students. 23 Examples of common costs that were at least partially attributed include: salaries of executive director, project coordinator, and driver; fuel and vehicle maintenance; and rent, supplies, and utilities for regional offices. 24 These assumptions include the fact that the executive director spends approximately 40 percent of her time on fundraising and developing other JAA programs like Aflatoun or JA More than Money, that not all travel from the main office will be tied to the core JAA program, and so forth. 27 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              Overall cost per student per year $116.80 Margin Cost Measures Marginal cost of adding a new school $230.00 Marginal cost of adding a student in an existing school $3.80 It is difficult to assess whether JAA is competitive with other programs in Armenia based on the cost estimates provided in Exhibit 4 because no program was identified that had a similar structure and set of objectives. Nonetheless, the overall cost per year per student or school appears to be reasonable given the sustained training provided to teachers and the program’s depth, intensity, duration, and added features like summer camps, the annual trade fair, and school trips to businesses. The ratio of program costs (including common costs) to total costs could be viewed as somewhat easier to compare against established measures, since this is a metric often used by non-profit and charity comparison websites. For example, Charity Navigator, a leading provider of such data, reports this information consistently.25 Exhibit 5 below provides the ratio reported by Charity Navigator for several JA chapters in the United States, as a point of reference. In comparison to these figures, the ratio obtained for JAA is towards the lower end of the spectrum but remains competitive. Exhibit 5. Program expenses ratio for several JA and selected chapters in the United States Organization Name Program Expenses Ratio Junior Achievement USA (Parent Organization) 76.0% Junior Achievement—New York 66.1% Junior Achievement—Rocky Mountain 85.8% Junior Achievement—New Jersey 73.9% Junior Achievement—San Diego & Imperial Counties 80.7% It is important to keep in mind that the method of calculation used by Charity Navigator is likely to differ significantly from the one used for JAA for several reasons. First, the JAA ratios only reflect the expenditures under the USAID grant. If all activities conducted by JAA were included (e.g., JA More than Money, Skills@Work, etc.), the ratios would likely increase significantly.26 Second, Charity Navigator uses IRS filings (forms 990) to compute its ratio, and the limitations of the financial data provided by JAA do not allow for the complex calculation method required by the IRS. 25 For more information about Charity Navigator’s rating scheme, please see http://www.charitynavigator.org/index.cfm?bay=content.view&cpid=1284#.Uul_4hBdUrU. 26 This is because the common costs would be spread over a greater base of programmatic expenditures, all things being equal. However, it is also possible that a full review of JAA’s expenditures would require the inclusion of administrative and common costs not currently counted, at which point the ratios would, at worst, remain the same. 28 Project Results Evaluation question 5: How broad are the scopes and the ideas of the projects implemented by the students? Business Projects To kick off the student businesses, each class is presented with the various responsibilities associated with each of the student business managerial positions before elections are held, and the business registration documents can be completed. The process of elections for managerial positions was nearly the same in all schools: the students conducted closed or open elections (with nearly equal proportions) that allowed each student to vote for the peer he or she thought best suited for each position. Two schools conducted mixed elections through which a president was elected by his or her peers, and he or she appointed a student to each managerial position in the company. Exhibit 6. Types of products selected by student teams by region 0% 10% 20% 30% 40% Cinema Compost Crafts/handmade Electric fly catcher Growing flowers Growing/processing food Lighters Newspaper/magazine Puzzle Tea T‐shirts CDs Percentage of student teams interviewed Aragatsotn Ararat Gegharkunik Kotayq Lori Yerevan Student, teacher, and alumni surveys from all visited sites revealed that the largest share of student businesses produced various types of crafts or handmade products (36 percent, Exhibit 6) including jewellery made of beads, aprons, kitchen towels, wooden picture frames, passports covers, and handmade artificial flowers arrangements. The next most common category of products is foods grown and/or processed (31 percent) including dried fruits and herbal tea (6 percent). This was not an unexpected finding since students tend to use available, easily attainable inputs and use a less scarce resource, their own labor, for the production. Nevertheless, there were significant differences observed in the selection of products among marzes. In the Aragatsotn and Ararat Marzes, the conditions have been most suitable historically for growing fruits and vegetables, while the Gegharkunik Marz is famous for its herbal tea USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project 29 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project and the cultivation of potato. As a result, in addition to traditional crafts and other handmade products, which were observed in every marz, the students in these two regions have produced a significant amount of fruits and vegetables and herbal tea, respectively. This situation was drastically different in the Kotayk Marz and in Yerevan, where no agricultural products were available. There, the range and diversity of products is greater, and each school’s products are more unique. Yerevan students also have access to better tools and technologies, which they could leverage in creative ways (e.g., one school produced electric fly catchers, and a technical school created a neatly packaged computer game CD). Kotayk students exhibited great creativity and used their imagination to devise products like decorated lighters and organic compost, and one school even organized a low-priced cinema for students and the community. Trends were also identified across urban and rural regions. Sixty-four percent of interviewed student groups were located in rural areas, and 36 percent were located in urban areas. Food growing and/or processing was mainly conducted in rural areas (more than 80 percent of cases) and magazine publication was found only in urban areas (e.g., Yerevan and Abovyan). Urban areas exhibited a significantly greater diversity of products. The growing/processing food and craft/handmade products categories combined account for only 47 percent of projects in urban areas, whereas those two categories make up 78 percent of projects in rural areas. Regional coordinators and the JAA regional director mentioned that, although during the last few years there was increased variation in product categories, an important sign for project success in that students used creativity and innovative thinking instead of relying on the JAA booklets and textbooks that provide a list of suggested products. Students made decisions about the product(s) on which their businesses would focus in one of three ways: 1. They identified their product idea(s) as a group, without external assistance and based on their own judgment 2. They relied on a market survey fielded within the school or the greater community. 3. They were advised on a business idea by their business consultants or their teacher. The majority of students used the first or second method above, with only 20% of students using the third method. It is also interesting to mention that 82 percent of student groups located in rural areas used one or several market surveys to select and finalize their business idea, while more than 57 percent students in urban areas received advice that led to their product. Participating students received business advice from their JAA teachers and designated local consultants. However, the advice provided was more general, intended to steer students towards a general direction (62 percent of cases), and in only 8 percent of cases was the advice related to a particular product or service category. Since the interviews conducted covered a number of business consultants that were involved in managing a business, about 30 percent of the advice provided was related to business and CSR policies in place or discussed in their respective companies. Once student teams finalized their business idea, they worked on compiling a detailed business plan. Significant changes to business plans were observed only four times, two of which were due to a discrepancy between the planned and actual amounts raised from shareholders during the sale of stock. Another was due to a change in product distribution during the selling stage, and the reason for the last one was not entirely clear from the interview. When reported, the amount raised from the sale of company shares ranged from 7,500 to 30,000 AMD (the mode was 15,000 AMD). In some cases, student businesses did not return the shareholders’ equity, while in others, the students promised to repay the face value in addition to a guaranteed 10 percent dividend in order to encourage investment. 30 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project With regard to the implementation of the business project, students in each program group demonstrated that they all fully participated in the whole process. They usually divided the work according to the abilities of each individual. For example, in the production of dried fruits or embroidery, the male students were in charge of obtaining supplies while the female students dried the fruit or stitched. Business failures were reported at only two of the visited sites. In one case, the students immediately initiated a second business project. This occurred at a school in Bjni where students had promised to repay their shareholders’ investment plus a 10 percent guaranteed dividend. In the second case, the reason for failure was poor weather during the trade fair, which represents the only opportunity for students to sell their products in large quantity and in a short timespan due to the presence of wealthier individuals and a high density of potential buyers (since it is held in Yerevan, where the standards of living are significantly higher than in the marzes). However, since the type of business selected did not require initial funding as the students produced dry fruit and berries, the students still reported that their business project met their expectations. Before the end of the calendar year, student teams are expected to dissolve their companies and must decide how to distribute the profit obtained after repaying any company shares. Usually, they had two options: to distribute it among themselves or to retain it for the next component of the project, the NGO. It is worth mentioning that in every school visited in Yerevan and Aragatsotn, the students paid a portion of the profits to themselves, while in Lori Marz, all the visited schools retained their profit for the next stage of the program. It was uncovered during interviews that retaining the profit for the NGO component became compulsory as of the second year of the program. Visited schools that provided information about the range of community funding they raised suggested that there is important variation across schools. Certain schools reported obtaining just a few thousand AMD in community funding, which others raised several hundred thousand AMD from a variety of sources (public, corporate, and private). Major factors affected access to community funding including the level of usefulness and ambition of the community project, the level of effort put forth by students (and sometimes teachers and principals) in raising funds, and the relative wealth and diversity of potential funders in each community. However, there was not enough information to draw any conclusions about differences across regions. Student NGOs After the completion of the business component, students learn about corporate social responsibility and the rules governing NGOs in Armenia and then proceed to register an NGO to implement the social project of their choice. The members of the student teams and managerial position positions generally did not change from the business component—in most cases, students continued onto their NGO project without organizing additional elections. 31 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Exhibit 7. Type of NGO implemented by student teams by region 0% 5% 10% 15% 20% 25% Aragatsotn Ararat Gegharkunik Kotayq Lori Yerevan No NGO Community project Psychologist services School project As shown in Exhibit 7, nearly 43 percent of students did not proceed to the NGO stage; some of them were discouraged because they were not awarded one of the $300 USAID grants, while others had not yet reached the NGO stage as of the site visit. Out of those student groups who did proceed with the NGO, 55 percent implemented a school project ranging from planting trees or plants to making improvements and renovations to a school gym, museum, or information center. Forty percent implemented various community projects, including removing the litter and trash around historical monuments, planting trees and flowers around the municipality, or building and installing trash cans around the village. Students in one of the visited schools funded the services of a psychologist for national minority students at their school. There were no major differences among merzes here—aside from the fact that there were more schools that continued to the NGO project in Aragatsotn and Lori. During the interviews, it became clear that the process undertaken to generate project ideas for the NGO component was quite similar to that of the business component. In a majority of cases, students selected a particular project after they identified a need in the community. Students in Aragatsotn Marz accepted teachers’ or business consultants’ suggestion about community needs. The process employed to divide labor was also very similar from the one used during the business project. In NGO projects where students built a library or renovated a room, male students would complete the tasks that involved significant manual labor, such as setting drawers and carrying heavy things, while the female students would clean the room or arrange the books on the shelves. On the contrary, in cases where the community project was related to the restoration of an historical monument or cleaning, the team of students split the work without clear delineations between genders. While no significant changes to business plans were reported during the student business component, the NGO plans had to be revised in many cases. All revisions were due to unexpected setbacks with regard to funding sources and amounts; the NGO plan is typically drafted and sent to JAA before the practical components of the NGO begin, since all student groups apply for the USAID grants. These grants, worth $300, represent a very important sum of money in comparison with the profits typically reported by even the most successful student teams. If a student team’s project does not qualify for the USAID grant, the team has to significantly revise the NGO plan and alter the activities, unless one or several other funding sources can be identified and obtained rapidly. The community funding reported by schools visited (when 32 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project available) ranged from just a few thousands AMD to several hundreds of thousands, depending on the project idea, efforts of the students (and sometimes teachers and principals) to raise funds, and the level of wealth of the community. In general, the teachers rated their students’ NGOs rather highly in terms of their ability to address community needs but somewhat lower in terms of how innovative the project ideas were. Overall, the NGO projects have been graded higher by teachers at schools in the Kotayq Marz. Interestingly, Kotayq Marz also has a great record of accomplishment in terms of participation in various JAA programs. Students from Kotayq have been very active participants in several JAA competitions within previous JAA programs; two students from Nor Geghi school No. 2 were shortlisted to visit the business innovations trade fair in Croatia in 2006, and another student won the business ethics essay competition in Armenia and continued competing at the global level in Europe. In 2012, during the first year of participation, the school managed to raise an additional 1,500,000 AMD from various sources, both public and private, for its NGO project. The school was selected to participate again in 2013 and the students plan to hire a truck and clean the bank of the Hrazdan River, which the evaluation team noted was littered with trash and old cars. Upon completion of the JAA program, NGO projects established by students were generally sustained/or will be sustained by all surveyed schools in the Lori Marz, while they were not continued in nearly all schools in Yerevan. This is most likely explained by the fact that in the Lori Marz, some schools (a vivid example is Vanadzor No. 30) were able to obtain significant funding in addition to the USAID grant. This enabled the design and implementation of projects that could be sustained. 33 A JAA alumnus established a youth club in his village (Nor Geghi) school and together with club members, applied for funding and organized village street lighting. USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project V. CONCLUSIONS The following conclusions can be drawn with regard to the five evaluation questions. Evaluation Question 1: Are the project goals ambitious enough, yet attainable and supported by context analysis (including alignment with the country’s economic, civic, education and youth development policies and strategies)?  The project is aligned with the country’s education and youth development goals, as stated in the State Curriculum of General Education of the Republic of Armenia. The previous format of JAA projects served as a pilot program for the Ministry of Education (MoE), which included economics and civic education within social studies, which is studied in the 10th grade; furthermore, after testing the current JAA programs at schools, the MoE intends to add a new subject called applied economics (for 9th graders) that includes almost all topics and activities covered by the JAA business component. Also, several years ago, MoE adopted a teacher’s training module that encourages teachers to use interactive methods and experiential learning techniques, first introduced in Armenian schools by the JAA.  Within the discussion of the economic context of the country, it was mentioned that the high unemployment and high poverty rates prevailing in Armenia today could be lowered by fostering entrepreneurship in the country. One of the main objectives of JAA Entrepreneurship and Civic Activism for Young People is “arming young people with entrepreneurial skills and preparing them for the work force.” The team cannot yet evaluate whether this objective has been achieved but can suggest that the steps undertaken by JAA may lead to it being achieved (this conclusion is supported by the literature review and similar projects’ evaluation results). Evaluation Question 2: Is the current project design based on sound development hypotheses leading to the achievement of sustainable results?  In worldwide practice, JA programs are mainly sustained by corporate funding, whereas the Armenian project is largely supported by donor organizations. Some steps towards sustainability of the CSR and NGO components are incorporated in the project design (students are encouraged to raise additional funding from local businesses, local municipalities or NGOs). Government adoption of the JAA after-school program into formal curricula would be one step towards sustainability. Evaluation question 3: Has the replication of the past project been a justified approach for the implementation of the current project?  Within the scope of the past project, JAA trained more than 1000 teachers across the country, developed instructor and student manuals, and demonstrated high student test grades and favourable teacher feedback (as stated in JAA surveys and quarterly reports). Replication of the past project contributes to cost saving for trainings and new teaching material development, and it secures project acceptance at schools (given the fact that many school teachers are familiar with it and can introduce it to the school administration). Evaluation question 4: How efficient is the project in achieving its goals (e.g., How are resources/inputs [funds, expertise, time, etc.] converted to produce outputs?) What is the balance of administrative vs. program costs?  The ratio of program costs to total costs (including common costs) is comparable to the ratio in several JAA offices across the United States. But since the team does not know the JAA offices’ method of calculation, the team will report project expenses in relation to site and participant 34 counts. Overall, each student costs approximately $110 a year, and each school costs approximately $1,750 per year. The marginal cost of adding one additional school is $230.00, and the marginal cost of adding one additional student to a participating school is $3.80 (if the scale or fixed costs remain the same). Evaluation question 5: How broad are the scopes and the ideas of the projects implemented by the students?  Student, teacher, and alumni surveys across all visited schools reveal that the largest share of student businesses produced various types of crafts or handmade products and foods grown and/or processed (67 percent). This shows that students tend to use available, easily attainable inputs and use a less scarce resource, their own labor, for the production. Significant differences in the selection of products were observed among marzes. In Aragatsotn and Ararat Marzes, the conditions have been historically most suitable for growing fruits and vegetables, while the Gegharkunik Marz is famous for its herbal tea and the cultivation of potato. As a result, in addition to traditional crafts and other handmade products, which were observed in every marz, the students in these two regions produced a significant amount of fruits and vegetables and herbal tea, respectively. This situation was drastically different in the Kotayk Marz and in Yerevan, where no agricultural products were available. There, the range and diversity of products is greater, and each school’s products are more unique.  Urban areas exhibited significantly greater diversity in terms of products. The growing/processing food and craft/handmade product categories combined account for only 47 percent of projects in urban areas, whereas these two categories make up 78 percent of projects in rural areas. USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project 35 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              VI. RECOMMENDATIONS To conclude the evaluation, the evaluation team has developed recommendations based on the data collected. These recommendations are based on specific findings include both lessons learned and good practices that can be used to inform future decision-making and programming. Project Design Evaluation Question 1. Are the project goals ambitious enough, yet attainable and supported by context analysis (including alignment with the country’s economic, civic, education and youth development policies and strategies)?  Alignment with country development goals. It is widely acknowledged that the JAA program needs to be aligned with a country’s strategic objectives in order to be viable and sustainable. With regards to this, the student businesses (and products) should aim to fit within the main segments of the Armenian economy, such as ICT, tourism, healthcare, education and agriculture.27 The Republic of Armenia (RA) has already adopted functioning strategies for each of above sectors, inherently increasing the student businesses’ and NGOs’ opportunities for guidance and access to funding from respective government bodies and private and public entities.  Organizational collaboration. In January 2014, the European Commission kick started its new Erasmus+28 project for 2014-2020. Erasmus+ aims to boost skills and employability and modernize education, training, and youth work. As the successor of Erasmus Mundus, Erasmus+ places greater emphasis on youth employment and entrepreneurship and incorporates new elements that provide a wide range of opportunities for young people living in both the European Union and its neighboring countries. Under the current Erasmus+ program, Armenia receives substantial benefits in terms of volunteer programs, various short- and long-term training and educational programs, particularly from the Youth in Action program. Armenian young people aged 16-30 (and for some training courses starting from 13 years old) may benefit from participating in these training programs. Several NGOs in Armenia actively applied for “Youth in Action” programs by both hosting and sending participants for various training-courses.29 In this regard, the JAA students and alumni could benefit by participating in these training courses both within and outside of Armenia.30 Based on the data collected, the most active students and alumni would be incentivized by this opportunity to apply for the program. Evaluation question 2: Is the current project design based on sound development hypotheses leading to the achievement of sustainable results?  Enhanced structure of the USAID and RA collaboration. In late 2013, the Ministry of Education (MoE) drafted an order to incorporate the applied economics into formal school curricula for grade 9, which includes an applied component of four or more hours. As a result, 27 Armenia’s Export-led Industrial Policy Strategy, Yerevan 2011 28 http://ec.europa.eu/programmes/erasmus-plus/index_en.htm 29The evaluation team contacted two NGOs actively working within the “Youth in Action” program between January 25-29, 2014, and spoke to their respective vice-chairmen. 30 If training is abroad, the EC covers maintenance costs of participants by 100 percent and travel expenses by 70 percent. The participant needs to cover the remaining 30 percent of travel expenses only. 36 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project MoE plans to cover the school-level project implementation costs (teachers’ training, monthly salaries, study materials costs, etc.). Therefore, in order to sustain the effectiveness of the JAA program, USAID/Armenia should continue funding the trade fairs, summer camps, student visits to various businesses, particularly given the likelihood of increased government support. Project Implementation Evaluation question 3: Has the replication of the past project been a justified approach for the implementation of the current project?  Updates to program curriculum. Currently, grade 10 students (both in private and mainstream schools) take social science courses consisting of a wide range of subjects such as aesthetics, ethics, psychology, and economics. While notable, the evaluation team gathered during its fieldwork that the curriculum is simply an overview of the various subjects, lacking breadth and substance. Additionally, the economics subject has been removed from the current curriculum, although it was previously covered by the JAA materials and manuals. In order to fill this gap, the USAID/JAA should ensure that both the content and practical learning portion of this subject is closely related to and informed by the JAA economic materials and manuals.  Increased principal ownership. Based on the evaluation findings, there is a considerable correlation between increased ownership of school principals in the project implementation phase and the project results (outcome and outputs) in schools. Two factors stand out to support this argument. First, the Armenian context reveals that principals (especially in rural areas) have close relations with community leaders, thereby increasing the likelihood of community involvement and support for the student businesses and NGOs. Second, schools oftentimes supplement USAID/JAA funding for ongoing project activities, so increased buy-in on behalf of the principal is crucial. Therefore, the evaluation team’s recommendation is to organize trainings for school principals similar to that given to teachers to explain the rationale behind the JAA program and showcase the program’s benefits for both the school and local community.  Enhanced training for JAA teachers: There is an evident need for regular teacher trainings beyond the 3-day training currently provided by the JAA during the summer. One of criteria for assessing teacher performance is to identify whether he or she strictly sticks to or goes beyond the program content. The data collected reveal that teachers oftentimes interrupted the program content (taught in the 3-day training) as the ultimate end goal instead of the baseline for involvement. Additionally, in cases where the teacher goes beyond the call of duty, project results for that school increased tremendously. Following this trend analysis, more frequent and clear trainings for the teachers would not only demonstrate added value to the program but also increase the probability of better engaged teachers and students leading to more positive results.  Increasing number of participants for the JAA Summer camps. The JAA program includes sending the best students from participating schools/classes to an annual summer camp. Summer camp is a brilliant opportunity for students to mingle with their peers and gain new information and knowledge about a variety of topics. Moreover, topics are often presented by celebrities representing business, culture, and other fields. This, too, gives students first-hand experience and the opportunity to gain useful information and knowledge. Currently, the number of students who attend the summer camp is one per class. Given the high enthusiasm and performance of students, this number inevitably fosters a difficult decision to select only one student from each class. The 37 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project evaluation team’s recommendation is to increase the ratio of summer camp participants from one per class to two or three per class. Evaluation question 4: How efficient is the project in achieving its goals (e.g., how are resources/inputs [funds, expertise, time, etc.] converted to produce outputs)? What is the balance of administrative vs. program costs?  Setting up a fund/business incubator for JAA alumni projects. The evaluation findings show that JAA alumni resources remain underutilized. The evaluation team thinks that JAA should offer grants for mini business projects on behalf of the JAA alumni in order to encourage sustainability and future buy-in of past JAA students. To this end, the evaluation team suggests setting up a respective fund/business incubator for good/innovative business ideas proposed by JAA alumni.  Better utilize the vast expertise of JA Worldwide. There is a need to foster collaboration with JA Worldwide and offer exchange programs to the best students and/or teachers in order to encourage innovative thinking, share best practices specific to similar youth entrepreneurship projects, and encourage intercultural exchanges. In many countries, big corporations are interested in and ready to fund JA activities. Therefore, the recommendation is for JAA to contact other JA offices around the globe and explore opportunities for raising funds (or joint funding) for student and/or teacher exchanges.  Collaboration with other organizations: Given the scarce resources on the school level, a dire need exists to bring together school-level operating organizations to create synergies. While there are no programs like JAA with similar scale, format, and objectives in Armenia, other school￾based projects can be a good resource and supplement to JAA activities. Collaboration could benefit both entities while also providing teacher trainings, preparing and/or updating the student manuals, enhancing student’s community engagement, and sharing/complementing the grant provision for student NGOs. Evaluation question 5: How broad are the scopes and the ideas of the projects implemented by the students?  Encouraging new business project ideas: The evaluation findings show that a very large share of business projects included the creation of handmade goods or processed food. There is a need to promote innovative thinking and encourage students to come up with substantive business ideas. This could be done via a “most innovative business idea” competition that included a small grant for the winning team. 38 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project VII. APPENDICES Appendix 1: USAID Evaluation Statement of Work Evaluation of USAID/Armenia Entrepreneurship and Civic Activism for Young People Project Purpose and Scope USAID/Armenia requires a mid-term performance evaluation of the Entrepreneurship and Civic Activism for Young People Project. USAID/Armenia has been funding this activity for a number of years, and is currently interested in understanding more about the effectiveness of this investment. Even though the evaluation will focus on the current activity envisages which started in 2011, it is important to consider the history of the project. Many of the implementation approaches have replicated those of the previous project, so the Mission has been faced with the question whether any adjustments in program implementation would have been feasible. Another issue raised by the Mission was whether the approach has any lasting effect on the participating schools, considering sustainability issues, and the breadth and depth of the activities carried out under the project. This evaluation is triggered by the above issues, as well as by the question of whether the project is using the funding efficiently, considering administrative vs. program costs. Findings from the evaluation will help make adjustments in the implementation approach and inform future work plans. The evaluation findings will be used primarily by the USAID/Armenia Economic Growth, Program and the Front Offices, the implementing partner, and by other interested entities. The project AOR will develop a plan for incorporation of relevant recommendations from the evaluation in their future work plans. The evaluation should answer each evaluation question empirically, relying on factual evidence, and each evaluation question will be addressed distinctly in the final report. Background and Summary of the Project USAID/Armenia has been providing assistance to Armenian schools in economics education since 2006 through a stand-alone grant to Junior Achievement (JAA). The main purpose of these efforts is to enhance economic and business awareness among the younger generation of Armenia, acquaint them with current international business practices and open their minds to their potential in business. The current program started in 2011 as a natural continuation of the Economics Education project with one addition, that the school students are also involved in civic activities to address communities’ needs. The project’s main directions focus on: a) equipping young Armenians with entrepreneurial skills and prepare them for tomorrow’s business world; b) enhancing the school student’s understanding of business ethics and corporate social responsibility (CSR); c) establishing the links between the business community and young people studying entrepreneurship and economics, d) assisting young people with their career orientation, and e) encouraging young people to form their organizations, such as NGOs, to undertake projects benefiting their communities. The program targets 4,500 students from 300 schools across Armenia. JAA acquaints students with current international business practices, ethics and corporate social responsibility (CSR) issues, and helps unleash their business potential. The program also teaches young Armenians to become active citizens, team-players and to contribute to the welfare of their communities. 39 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project As part of their business activities, the school students (typically in groups) establish and run “mini￾companies,” imitating real business operations. The students devise management structures, conduct market research, write business plans, draft financial and administrative documents, organize the production of goods or services, and design advertising and sales strategies. The proceeds generated from sales are used by the students to carry out community-oriented projects. Students receive guidance from JAA teachers as well as consultants from businesses. During the first year of project implementation schools involved in the project have carried out 111 community projects. Evaluation Questions 1. Project Design: a. Are the project goals ambitious enough, yet attainable and supported by context analysis (including alignment with the country’s economic, civic, education, and youth development policies and strategies)? b. Is the current project design based on sound development hypotheses leading to the achievement of sustainable results? 2. Project Implementation: a. Has the replication of the past project been a justified approach for the implementation of the current project? b. How efficient is the project in achieving its goals, i.e. how are resources/inputs (funds, expertise, time, etc.) converted to produce outputs? What is the balance of admin vs. program costs? 3. Project Results: a. How broad are the scopes and the ideas of the projects implemented by the students? USAID’ S Role in the Evaluation The USAID Mission in Armenia will: • provide relevant programmatic and budgetary information to the Contractor; • provide project documents and evaluations to the Contractor; • facilitate obtaining USAID/Mission input; and • arrange USAID/Armenia meetings. Methodology The Contractor will: 1. Conduct a comprehensive review of performance reports and other materials and identify data gaps. 2. Identify data collection methodology to provide the best possible evidence to answer the evaluation questions, also considering feasibility issues. 3. Identify informants and stakeholders, samples and/or other relevant data sources. 4. Prepare a field work plan. 5. Conduct field research in Armenia. 6. Analyze data and compile key findings, conclusions and recommendations. 7. Revise the draft report addressing comments by USAID. 8. Address implementing partner comments as necessary after USAID/Armenia shares the draft report with implementing partner, and if the partner chooses to submit “Statement of Differences”. 40 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project 9. Submit final report to USAID/Armenia for acceptance. The proposed methodology should address the need for data collection from qualitative and/or quantitative sources, and provide the best possible combination of methods, given the evaluation questions and the available resources and timeline. All evaluation questions need to be answered empirically; therefore the data collection methods should be tailored to ensure that adequate evidence is collected to answer each of the questions in a definitive manner. There is no preference for any particular method. The ability of particular method(s) to properly answer the evaluation questions is important. Data should come from facts, rather than be based on anecdotal evidence. Conclusions should be based on findings received from multiple sources, and strengths and limitations of the methodology should be explicitly communicated in the proposal as well as in the final report. All people-level data should be disaggregated by sex to allow analysis of findings by sex. Evaluation Question 1 a) To answer this question an analysis of the context is necessary, along with considerations of available funding. It is necessary to understand whether the goals are the maximum that can be realistically strived for, or there is more that can be done, given resources available and the context. Possible data sources: desk research, stakeholder interviews, financial data. Evaluation Question 1 b) To answer this question it is necessary to provide evidence that the project approach is leading toward sustainable results, and whether the project design is based on best practices in this area. The contractor should look into the project data to find out whether desired results are being achieved, and whether there is evidence that the results are likely to be sustainable beyond the project. The contractor should also find out whether international best practices in this area have been considered by the project. Possible data sources are desk research, project PMP data, and stakeholder interviews. Evaluation Question 2 a) To answer this question the contractor should find out whether best practices and lessons learned from previous projects have been taken into consideration during the design of the current project. What other information has been considered during the selection of implementation approaches, including developments in the context and in the country in general? Possible data sources include project records, stakeholder interviews, desk research. Evaluation Question 2b) To answer this question it is necessary to look into financial data, the management structure and the internal processes. Possible data sources are financial data, internal organizational documents, stakeholder and staff interviews. Evaluation Question 3 To answer this question it is necessary to study a sample of the projects implemented by the students and describe how they were developed and implemented. What considerations were taken to come up with those projects? How did the students come up with the ideas, and how were those ideas developed into projects? Possible data sources – project documents, stakeholder interviews. 41 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Deliverables The Contractor’s deliverables must include: 1. Written methodology plan (evaluation design and work plan). The evaluation design will include detailed evaluation design matrices (including the key questions, methods and data sources used to address each question), draft questionnaires and other data collection instruments, known limitations to the evaluation design and a dissemination plan. The final designs require COR approval. The work plans will include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation teams. The methodologies will be shared with the implementing partner for their comments before finalizing. 2. Verbal debriefing The evaluation team will meet with USAID/Armenia upon arrival. The team will also provide oral briefing of findings and recommendations to the USAID/Armenia senior management and relevant technical teams prior to departure. 3. Draft Evaluation Report Prepare draft evaluation report which will analyze data and summarize key findings, conclusions and recommendations. The Findings section will list all facts and evidence received from desk research and from field work in relation to each evaluation question. The Conclusions section will analyze the facts specified in the Findings section, and will discuss what worked, what did not work and why. The Recommendations section will address issues of what can be improved for future programming, based on Findings and the analysis provided in the Conclusions section. More specifically the Recommendations section will address the following: 1. How to better align the project design and goals with the context of economic development and competitiveness? 2. Can the ideas and the scopes of the business projects implemented by students be broadened and how? 3. How to ensure sustainability of interventions in the participating schools? Who are the current stakeholders and who might be potential partners for replication and scaling up of the project interventions to involve new schools and to ensure continuity of interventions in the currently participating schools? What might be the best ways to ensure continuous involvement of the current stakeholders with the schools participating in the project? The Evaluation Report must at a minimum contain: 1) a 3-5 page Executive Summary summarizing key points (purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations); 2) a brief description of the project; 3) a section on the purpose and the methodology of the evaluation, including limitations, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.); 4) a section on clearly defined findings, conclusions and action-oriented recommendations. This section should be organized around the evaluation questions defined for each project. 5) Annexes, including the Scope 42 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project of Work; all evaluation tools; all sources of information properly identified and listed; any “statements of differences” regarding significant unresolved differences of opinion by funders, implementers and/or members of the evaluation team; disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. The draft report must be submitted to USAID/Armenia within three weeks after completing the fieldwork. USAID will be responsible for compiling Mission comments for inclusion and submission to the Contractor. USAID/Armenia will provide the Contractor with a summary of such written comments within three weeks of having received the draft report. 4. Final Report The Contractor must submit the final report to USAID/Armenia within two weeks after USAID's comments are provided. The report shall follow USAID branding procedures. The report must include an executive summary and not exceed 30 pages (excluding appendices). The final report will meet the following quality standards (Please see the USAID Evaluation Policy): a) The report will represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why; b) The report will address all evaluation questions included in the scope of work; c) The report shall include the scope of work as an annex; d) Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report; e) Evaluation findings will assess outcomes and impact on males and females; f) Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology; g) Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, or the compilation of opinions; h) Sources of information need to be properly identified and listed in an annex; i) Recommendations need to be supported by a specific set of findings; j) Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. The Contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within 30 calendar days of final approval of the formatted report. 5. Evaluation Data The Contractor will submit Evaluation data to USAID/Armenia along with the final report for record keeping and future use by the Mission. The data will be in an easily accessible format, such as MS Word documents for qualitative data, and SPSS or Excel files for quantitative data. Schedule The anticipated duration of the evaluation is not to exceed five months. The field work for the evaluation should be conducted around September-October of 2013. Staffing and Level of Effort It is expected that the Contractor will provide a team of consultants to carry out the evaluation. The mix of skills necessary for the team is specified below. The team should consist of two experts, although other team compositions might also be considered. Inclusion of a qualified local Armenian expert in the 43 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project evaluation team is highly encouraged. All evaluation team members will be required to provide a written disclosure of conflict of interest. Required qualifications  Experience with evaluation designs, quantitative and qualitative methods and data analysis  Substantial experience with program evaluations, particularly in economic development or youth project evaluations. Prior experience with USAID or other donor project evaluations is a plus.  Substantial experience in developing and/or managing entrepreneurship development, civic education, youth development or related projects  Excellent English writing skills 44 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Appendix 2: METHODOLOGY Initial Data Gathering The evaluation team employed various methodologies to gather data on the JAA program’s work to inform the mid-term performance evaluation. The first stage was to conduct an initial desk review of civic activism, entrepreneurship and youth development, and then the JAA program documentation. Document Review The evaluation team conducted an extensive document review of the past JAA Economics Education Activity program and current JAA Entrepreneurship and Civic Activism for Young People by translating, reviewing and evaluating the information presented in a wide range of project documentation, including the following (see report reference list in Appendix 2): Economics Education Activity  USAID Action Memo (1) and Quarterly Reports (16);  JAA final report (1) and final report chart (1); Entrepreneurship and Civic Activism for Young People  Performance Management Plan (PMP) (1); Annual Work Plans (3); Annual Work Plan Charts (2); Quarterly Reports (10); Financial Request for Advance and Liquidation Reports (41);  USAID-JAA contract and attachments (2);financial reports that described the distribution of program funds across regions and schools (16); stakeholder contact lists (6); program school lists (5); schools lists by grade level (6); school principal lists (3); student rosters (20); memorandums of understandings for various partnerships (7); program instructional materials such as corporate social responsibility workbook (1), guide for business consultants (1), success stories (2), program curriculum (1), and guides on business company and NGO formation for teachers and students (2); and, program evaluation materials such as economics and NGO pre- and post-intervention questionnaires and associated comparative data sets (8), and program evaluation for school administrators, teachers and students (3). The document review phase was imperative because it enabled the evaluation team to identify information gaps that informed what needed to be included in key informant interview protocols. The evaluation team also reviewed research articles and programmatic reports on entrepreneurship, civic activism and participation and youth development by the Government of Armenia, and other international organizations. Additionally, certain documents necessary for the context analysis were researched and synthesized. The document review was used to build a comprehensive understanding of the intervention and to capture background information on the project, its goals, stakeholders, inputs, outputs, and outcomes. It was also used to assess whether the project activities were implemented as planned and any challenges or problems that delayed or altered their implementation. Lastly, the document review was used to inform the evaluation methodology. 45 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              Research Design The research design of the JAA mid-term performance evaluation involved the collection and analysis of data from a wide set of project materials, focus groups, and in-depth interviews. The design, which was revised through a detailed document-review process, was submitted to and discussed with USAID/ Armenia during the November 2013 in-brief meeting in Yerevan (refer to Appendix 4 for evaluation design matrix). The evaluation team designed a logic model based on the key research questions, entrepreneurship education, and JAA’s implementation approach. Appendix 4 lists the operationalized sub-research questions derived from the key research questions. Sampling Framework The evaluation team constructed a convenience sample of 31 schools that participated in the JAA program between 2011 and 2013. The schools were selected to provide diversity along a number of key dimensions, including cohort year, geographic location, and schools’ perceived performance in implementing the JAA program. Eligible schools were chosen based on respondents’ availability and the team’s travel route and time constraints. Sampling of Districts/Marzes First, the team selected a subset of regions that represent a variety of school environments due to varying population densities, political contexts, and economic histories (see Table 8). The JAA program aims to operate in approximately nine schools per region each year. As a result, an equal number of schools were targeted from the six regions and based on availability and willingness to participate, between four and six schools were ultimately selected to comprise the sample of 31 sites. Table 8: Selection of Schools across Regions Region Environment31 Number of Schools in Sample Yerevan Central, Urban 4 Aragastotn Western Central, Rural 5 Kotayq Central, Urban-Mixed 6 Gegharquniq Eastern Central, Rural 5 Lori Northern Border, Urban-Mixed 6 Ararat Central, Rural 5 This sampling framework strategy implemented by the evaluation team was based on a multisite evaluation32,33 approach. Taking local knowledge, past experience, and time constraints into account, the sample framework was determined by obtaining sufficient data from different geographical locations so that it might provide a good representation of Armenian regions and eligible populations. 31Environment characteristics were defined by the local consultants on the evaluation team. 32 Multisite Evaluations: https://communities.usaidallnet.gov/fa/system/files/Multi-Site+Evaluations.pdf 33Imas, Linda G, and Ray C. Rist “The Road to Results – Designing and Conducting Effective Development Evaluations.” 46 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Methods of Categorizing Cohorts The evaluation team categorized schools into one of four cohorts: 2011, 2012, 2013 or mixed years. Participants in the first year of the program (2011) offer a point of comparison for growth and understanding the program’s early results. Current participants offer the most up-to-date information on how the program is currently operated. The 2012 cohort includes the most recent participants who completed the program; this same cohort may be able to inform how participation influenced participants’ outputs and outcomes and on the sustainability of the program. Lastly, schools that had participated for more than one year were able to inform on lessons learned and conditions for sustained implementation. Table 9: Selection of Schools across Program Years Year of School Participation Business Component NGO Component Number of Schools in Sample 2013 In progress Has not occurred 6 2012 Completed Completed 4 2011 Completed Completed 10 Multiple years Varied Varied 11 Methods of Selecting Schools A final criterion for constructing the sample was the level of success achieved by the participating schools. Each year, a small number of schools did not complete the program according to the JAA quarterly progress reports and JAA main office feedback. Mindful of the richness of readily available data, the evaluation team selected two schools in the sample regions that had failed to fully implement the program in order to better understand conditions needed for success from their perspective. Conversely, the evaluation team also requested that JAA provide five schools that they consider particularly successful in implementing the JAA program. Three of these schools were already in the sample regions and were ultimately included for a site visit schedule. The final distribution of schools across these geographic, cohort year and performance criteria is shown in Table 10 below. Table 10: Sampling Framework and Final Selection Region 2013 2012 2011 Multiple years Total Yerevan 1 1 2 4 Aragastotn 1 1 1 2 5 Kotayq 1 2 3 6 Gegharquniq 1 2 2 5 Lori 3 2 1 6 Ararat 2 2 1 5 Total 6 4 10 11 31 47 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Selected Implementation Districts and Schools The team interviewed the following:  JAA Main Office (2)  JAA Regional Coordinators (6)  Principals (29)  Teachers (31)  Student focus groups (25)  Alumni focus groups (2)  Business Consultants (14)  Ministries (1) The finalized sampling framework charts for all stakeholders that were interviewed can be found in Appendix 5 of this report. Data Collection Site Visit to Armenia The evaluation team began the evaluation with an extensive document review as the preliminary information-gathering phase. Thereafter, the next phase of the data collection process was to perform a variety of other data collection methodologies, including focus groups and semi-structured interviews with stakeholders. Accordingly, the site visit to Armenia was conducted by two local consultants and two U.S.-based evaluators between November 11 and December 19, 2013. The site visit was necessary for the evaluation team to understand the overall work of JAA, and the implementation and results of the current USAID funded program. Protocols and Guides For both the interview and focus group protocols, the evaluation team carefully developed the questions to ensure that the research questions and associated sub-questions could be answered by the respondents’ reactions. Semi-Structured Interviews: The evaluation team conducted semi-structured interviews with a variety of project stakeholders to gather information from various vantage points. In particular, interviews performed were with relevant USAID/Armenia staff; JAA program administrators at the main office and regional offices included in the sample; school principals and teachers in the 31 schools visited; relevant consultants of business associations and organizations; and, government official at the Ministry of Science and Education who collaborate with the JAA program. Please refer to Appendix 5 for the list of stakeholders interviews performed. In preparation for the key-informant interviews, the evaluation team developed interview protocols suitable for each of the above categories of respondents (see Appendix 6). The interviews were semi￾structured in nature, thus allowing for an open framework, conversational communication, and probing questions. Therefore, while the interview protocols allowed the interview to be focused on the key evaluation questions, the interviewers also had the flexibility to pursue new lines of questioning during the interview based on the interviewees’ answers. The evaluation team utilized interview best practices and standard guidelines to ensure that the questions were asked in a neutral manner to elicit valid and reliable responses from the interviewees. In addition, the evaluation team effectively utilized probing 48 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project techniques in asking follow-up questions to obtain accurate, precise and in-depth answers during the interview. The evaluation team documented the interviews by taking detailed notes and recordings once the interviewee provided consent. Majority of the interviews during school site visits were conducted in Armenian and the notes were transcribed into English. Focus Groups: Student focus groups were conducted at each school with students that have not yet graduated, and at a central location in Yerevan for students in the 2011 cohort that have graduated (entitled as “alumni”). Students were invited to participate through a personal request from the principal or program teacher and a standard letter from the evaluation team (see Appendix 7) explaining the request. Focus groups protocols were designed (see Appendix 6) and then carried out by a facilitator and supported by a note-taker. The focus groups was also be recorded, when the interviewee provides consent, to verify key findings and quotes for the report. Coding Approach In order to preserve confidentiality and remove biasness when reviewing the data, the evaluation team utilized quantitative techniques to code in-depth interview and focus group data as promised in the evaluation consent form provided to all interviewed stakeholder and beneficiaries (see Appendix 8). Accordingly, data coding categorizations included: 1. Region (e.g. Yerevan – 1, Lori – 2, etc.) 2. School (e.g. S1, S2, S3, etc.) 3. Respondent Type a. Alumni 1 (A1), Alumni 2 (A2), etc. b. Student 1 (S1), Student 2 (S2), etc. c. Teacher 1 (T1), Teacher 2 (T2), etc. d. Principal 1 (P1), Principal 2 (P2), etc. e. Business Consultant 1 (BC1), Business Consultant 2 (BC2), etc. f. JAA regional coordinator 1 (JAA1), JAA regional coordinator 2 (JAA2), etc. g. JAA main office visit #1 (JAA-M1), JAA main office visit #2 (JAA-M2) While this technique was utilized to answer a range of research questions, the evaluation team also grouped the data by region or school system, if necessary for the context of the issue. The following section discussed this discretion more in depth. Data Analysis The evaluation team, in collaboration with USAID/Armenia, determined the analysis techniques that were most suitable for answering the five research questions. Steps were taken by the evaluation team to ensure that the data analysis produces accurate results and identifies any limitations that may take place during the analysis. Specifically, prior to conducting data analysis, the evaluation team developed a comprehensive data analysis plan which detailed the purpose of the evaluation, questions required by the SOW, data quality assurance procedures, data entry, data cleaning procedures, types of analyses and limitations to the data analysis. Quantitative Data Analysis Components The evaluation team identified a range of quantitative goals and targets in the JAA work plans. Therefore, quantitative analysis of the JAA program was utilized by the team where the data are available and meet a 49 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              minimum standard of completeness. This analysis used program administration records to determine the progress toward target goals over time. The target and actual statistics were reported at the project level and within sub-units (e.g. predominately urban versus rural regions). Additionally, the evaluation team used quantitative analysis to determine the program’s efficiency in achieving its goals. The ability to measure program efficiency was dependent on the availability of data and the strength of logical links between program activities and expected outcome. The data envelopment analysis technique (DEA) was the team’s preferred method due to its use of an index to evaluate efficiency across schools.34 However, detailed information about the per-school disbursements of USAID funding (excluding other sources of funds on which JAA relies) was not available The following levels of efficiency were assessed in this evaluation:  Program expenses to total USAID grant value ratio  Overall annual cost per participating school  Overall annual cost per participating student  Marginal cost of adding a new school to the program  Marginal cost of adding a new student to an existing school Qualitative Data Analysis Components While quantitative analysis was undertaken by the evaluation team, the methods in this evaluation were largely qualitative and comparative. Interview and focus group data collected was cross-referenced with information obtained during document review in order to understand and verify empirical evidence. Specifically, interviews were used to gather information on perceptions of unexplored opportunities and unmet needs in the local areas, conditions for success, perceptions of the programs’ strengths and weaknesses, the barriers they have encountered during program operation, and student learning and perceptions of the program. This type of qualitative data was distilled into themes that underscore the applicable research question(s). For example, sustainability was assessed using a mix method of interview data and documentation. For the purpose of this evaluation, sustainability was defined as the potential for JAA activities to continue after USAID funding expires. The evaluation team evaluated this factor on two levels:  Financial sustainability: Diversity of funds to continue JAA activities within Armenian communities.  Community-driven sustainability: Continued commitment of stakeholders in engaging high school students. Data Verification by Triangulation The evaluation team utilized different data verification methods, including proofreading and triangulation. Proofreading data involved the evaluation team conducting peer reviews across data analysis taken from the various data sources. Triangulation was oftentimes used as a way to verify the accuracy of the data gathered, through the use of the various data collection methods. Specifically, the evaluation team’s triangulation methods consisted of collecting the same type of information, but from different sources and using different methods. Data collected from the document 34“Kansas School District Efficiency Study Part I: Efficiency Analysis” 50    reviews, semi-structured interviews and focus groups of various stakeholders, and observations of the program during the site visit were compared to the extent possible, in order to strengthen the reliability of the analysis results. Throughout the evaluation, data received were carefully collated, reviewed continuously, summarized and synthesized. This systematic approach helped to identify trends, and establish the authenticity, validity and reliability of the findings reported. USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project 51 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              Appendix 3: Evolution of the JAA Program 2006-2007 School Year Under the USAID grant awarded in 2006, JAA reorganized its entire structure to be more efficient, including the relocation of the main office into a smaller space and a change to the work regiment of regional coordinators to part-time status. The Economics Education program, already including a component about theoretical economics knowledge and a hands-on component where each class establish and operates a business, started in 477 schools instead of the 500 targeted because a certain number of teachers lost their jobs as part of the tail-end of an optimization program that primarily aimed to raise the student/teacher ratio to a more manageable level, thereby enabling an increase in teachers’ salary.35 In September 2006, a study conducted by JAA staff found that 464 teachers taught the JAA program to 738 classes for a total of 16,143 students involved. To abide to the terms of its grant in relation to teacher training, JAA obtained a letter from the Education Reform Institute asking all schools to allow a teacher to attend JAA training and instructed regional representatives of the program to select schools based on geography and qualifications. To minimize use of funds, JAA organized teacher training by region and utilized experienced JAA teachers to act as trainers for the new group after they completed a seminar to refresh their knowledge. In parallel, a two month training curriculum for new teachers was developed by a committee headed by a professor of economics at the State University. By March 2007, 289 new teachers completed their training and reported through an evaluation that they found the training helpful. JAA also already offered the opportunity for schools to send their best students to the JAA summer camp if they placed among the top 100 in a centralized exam on applied economics. At that time, JAA already offered or was getting ready to offer ancillary activities and programs, including site visits to various businesses in Armenia, Brain Ring, and the community service program “I am the Master of My Nation”. In addition, JAA was also offering a 12 week, case study-based global business ethics program started in the 2004-2005 school year in just 12 schools and funded through a $25,250 grant from JA of the Commonwealth of Independent States (CIS). By mid-2007, JAA had trained a total of 200 teachers for this program, which was taking place in 250 schools and reached approximately 3,000 students. 2007-2008 School Year This was an important year for JAA on many fronts. First, it transitioned from an international branch of JA to a locally registered and operated organization. Second, a more rigorous evaluation scheme was implemented, where JAA fielded short questionnaires to students all over Armenia prior to their exposure to the program and then again after they completed the program. This allowed the organization to measure that program participation increase the number of questions on economics that students answered correctly by 28 percent. Third, JAA was solicited by the Education Reform Institute to assist in training 8th grade social studies teachers and prepare a teaching guide. The training of 974 teachers took place at 40 sites around Armenia using the 140 page manual created by the two organizations. JAA also reported receiving a $15,000 grant from USAID to assist in this effort. Discussions with the Ministry of Education to include economics as a course in high school continued, and JAA met with the head of curriculum and schools and shortly after, a decree was issued recommending that all schools that had a JAA-trained teacher allot hours for the subject. Fourth, JAA requested a voluntary audit with KPMG, who will be contributing $7,000 in services, leaving JAA to pay only $5,000. JAA also trained 192 new teachers in economics towards the end of the school year, and continued its site visits to JAA-participating schools to evaluate teachers and the content of the program. To limit monitoring costs, JAA also randomly selected 35Kuddo, Arvo; Structural Educational Reform: Evidence from a Teacher’s Displacement Program in Armenia, 2009, available at http://siteresources.worldbank.org/SOCIALPROTECTION/Resources/SP-Discussion￾papers/Labor-Market-DP/0902.pdf 52 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project    students participating in the program and conducted a phone interview to ensure that classes were taking place. JAA alumni members also organized a career day for 30 students from various schools, who visited relatively large and established businesses and spoke to high-level officers. 2008-2009 School Year During this school year, JAA continued to hold its summer camp, and trade fair, which this year included a career day held at HSBC where 40 students from remote villages visited the various departments of the company and spoke to a variety of high-level staff. JAA also continued to work with the Ministry of Education to include economics into the official curriculum. In early 2009, the Ministry of Education decided that social studies would be part of the official curriculum and command in all 10th grade classes in September 2009. The subject of social studies includes economics, ethics, philosophy, and aesthetics. The Ministry of Education also sent a decree to all schools requesting that those without a JAA-trained teacher submit the name of the most appropriate candidate who will receive economics and civic education training. A few months later, JAA conducted the 2 month intensive training of 450 new teachers from all regions from Armenia on the economics and civic education component of social studies. As a result, 95 percent of Armenian schools had one or more JAA-trained teacher ready to teach the subject, a total of more than 1,200 teachers. JAA also reported that a number of businesses in Armenia were increasingly concerned about corporate social responsibilities, opening the door for possible partnerships and inputs in the development of a curriculum. Lastly, JAA began a new project in collaboration with the British Council and Beeline Company called “Skills@Work” that involves writing a short essay addressing a specific high-level business decision. JAA also completed the translation of the ethics manual provided by JA of the CIS and which underpins the ethics program offered since 2005. 5,000 copies of the manual and study guide were printed and distributed, but the 2009-2010 school year was the last year in which JAA received funding for the program from JA. 2009-2010 School Year This school year, JAA initiated several new programs ancillary to Economics Education. First, a relationship with VivaCell-MTS was established, which led to a career day event at the company’s headquarter where students met the General Manager and were exposed to marketing strategies, technologies deployed in cellular phone communications, and a visit to the service center and network management center. Second, JAA kicked off the JA More than Money program in partnership with HSBC. This program, targeting students in middle school, was implemented in six schools in the Yerevan region, and prior to its start, JAA trained teachers, translated the relevant material from JA, and created supporting material. Third, the ongoing Hewlett-Packard Global Business Challenge, an online-based program where students are prompted to make business decisions in the context of a simulation, changed name to the JA Titan Challenge, aligning itself with a change made by JA. The organization also continued offering the Brain Ring competitions in collaboration with the Armenian Central Bank, and prepared for the trade fair. In early 2010, JAA received the results of the KPMG audit, which indicated that the change in JAA’s status as a local organization created a social security tax liability. The organization corrected the problem right away, but this impacted its financial health, leading to some targeted cuts in areas with minimal programmatic impact. 53 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              Appendix 4: The Armenian Context Before addressing other aspects of program design, it is important to briefly review the relevant aspects of the Armenian context, which influenced the program at the design, implementation, and results stage. This section does not aim to provide a comprehensive review of the Armenian context, but instead focuses on aspects that are directly tied to areas of program activity. Macroeconomics and Governance Since the fall of the Soviet Union in late 1991, the Republic of Armenia, a small, mountainous, and landlocked country in the Caucasus region, has been struggling to recover from the resulting economic and political shock, including a complete breakdown of the previous, centrally-planned economy and intra-Soviet Union trade patterns. As the “iron curtain” ceased to exist, it became clear that Armenia possessed mostly outdated industrial and technological capacities as well as a lack of human capacities and capital that seriously threatened its competitiveness in the region and globally. The devastating earthquake in 1988 and the war between Armenia and Azerbaijan over the Armenian-populated enclave of Nagorno-Kharabagh further deteriorated the situation by further damaging the infrastructure and industry as well as undermining already challenging relationships between Armenia and ex-Soviet Union nations. To date, the closed borders with Azerbaijan and Turkey and the resulting poor transportation and communication infrastructure pose a significant challenge for the country’s development both directly through high transportation costs or indirectly through a higher perceived risk premium for foreign direct investment. In 1994, LevonTer-Petrosyan, the first democratically elected President of the Republic of Armenia, and his government launched an International Monetary Fund (IMF)-sponsored, comprehensive measures to stabilize and reform the economy. These reforms have helped transform the country into a liberal market economy and have brought about impressive growth. Between 2001 and 2008, Armenia sustained an annual GDP growth ranging between 6.9 percent and 14 percent annually.36 However, this steady growth came to an abrupt end in 2009, when the country’s GDP declined by more than 14 percent despite loans from multilateral institutions, before stabilizing at a growth rate of approximately 3-4 percent. Data from the USAID Country Development Cooperation Strategy show that during the 2009 economic crisis, poverty in Armenia increased to 35.8 percent while the debt-to-GDP ratio also rose sharply to 56 percent.37 The solid streak of growth between 2001 and 2008 notwithstanding, the Armenian economy is generally considered to be uncompetitive, with relatively low levels of labor productivity.38 It is also limited by a narrow export base, which, along with limited natural resources and heavy reliance on imports to satisfy many basic needs, explains the deepening current balance deficit. Furthermore, powerful monopolies, semi-monopolies, and closed “oligarchic” economic networks continue to maintain their control over the supply of various necessary goods and services, and the country is known for high levels of corruption.39The Armenian economy is also largely dependent on remittances from the approximately nine million Armenians living abroad (compared to the 3.2 million Armenians living in the country).40 Armenia is ranked the 20th country in the world with regard to the share of GDP that consists of 36 The World Bank, GDP growth (annual %), available at http://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG/countries?page=2 37 USAID Country Development Cooperation Strategy, Armenia, FY 2013-2017, August 2013, p.8 http://webcache.googleusercontent.com/search?q=cache:vvFWYmLd8sgJ:www.usaid.gov/documents/1863/armenia -country-development-cooWperation-strategy-fy-2013-2017+&cd=1&hl=en&ct=clnk&gl=am 38Economy and Value Research Center and EV Consulting, National Competitiveness Report of Armenia 2012, available at http://www.ev.am/media/documents/ACR/2011-2012/ACR%202011-2012_eng.pdf. 39 See for example http://www.armenianow.com/special_issues/agbumag/8033/competitive_edge_the_pitfalls_of_m. 40 See http://news.bbc.co.uk/2/hi/europe/6382703.stm 54 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              remittances received from workers abroad.41 Armenia’s infrastructure is also rated significantly lower than that of two of its neighbors, Azerbaijan and Georgia, and the county’s ranking in terms of road quality has slipped significantly between 2005 and 2011, despite assistance from donors such as the Millennium Challenge Corporation.42 However, the crisis that took the country by surprise in 2009 appears to have triggered some in-depth reforms. Since then, the country has implemented a range of initiatives to facilitate business activities, and has gained 18 positions in the World Bank’s Doing Business ranking between 2012 and 2013, where it is now ranked 32nd out of 185 economies.43 Nonetheless, the country remains significantly behind its neighbor Georgia with regard to entrepreneurship, having only a quarter of the new business density in 2011.44The report further argues that the more modern and innovative firms in the Armenian economy become, the lower the probability to identify domestic workers with the required qualifications. Armenia therefore suffers from both a low rate of job creation and a mismatch of employee skills with positions available. Unemployment across all groups in Armenia also remains relatively high, as indicated by the official figure of 18.4 percent in 2011.45 However, this figure has been questioned by certain organizations, such as the International Labor Organization (ILO) which reported a value of 27.5 percent in 2010.46 More worrisome, the official statistics indicate that the important 15-24 years age group is most affected by unemployment. In 2011, that group had an unemployment rate of 39.2 percent, which had risen by 0.3 percent from the 2010 figure.47 Post-Communist Mindset Armenia, along with other ex-Soviet Republics, is affected by a legacy of centrally-planned economy. This is commonly referred to as the post-communist mindset. Any interpretation of post-communist mindset must deal with many political, social and economic stereotypes embedded in people living under previously communist regimes. First, one may legitimately ask what is meant by the term “mindset”. In the Learning Guide on Developing an Entrepreneurship, Dr. Max Sengesdefines what is meant by “mindset” as “a set of assumptions that are expressed in fundamental approaches and standard practices held by one or more people or group of people. Or more simply, the way one’s belief system and experience affects one’s outlook and decisions”.48 Michael D. Kennedy, a sociologist at the University of Michigan who studied an exchange program where experienced and trained American business experts were embedded in Polish enterprises during the period of 1990-1991, explained “mindset” slightly differently, as “narrower than ‘values’ or ‘culture’, implying an orientation towards a particular problem”49. Based on these definitions, 41Economy and Value Research Center and EV Consulting, National Competitiveness Report of Armenia 2012, available at http://www.ev.am/media/documents/ACR/2011-2012/ACR%202011-2012_eng.pdf. 42Economy and Value Research Center and EV Consulting, National Competitiveness Report of Armenia 2012, available at http://www.ev.am/media/documents/ACR/2011-2012/ACR%202011-2012_eng.pdf. 43The World Bank, Doing Business 2013 ranking, available at http://www.doingbusiness.org/data/exploreeconomies/armenia. 44 The World Bank and the Kauffman Foundation, see http://www.doingbusiness.org/data/exploretopics/entrepreneurship 45 The official figure includes all age groups (15-75 years old) and can be obtained in table 7.15 in the following document: http://www.armstat.am/file/article/trud_12_9.pdf. 46 Please see http://www.panarmenian.net/eng/economy/news/52555/. 47 Please see http://www.armstat.am/file/article/trud_12_9.pdf. 48Dr. Max Senges: Developing an Entrepreneurial Mindset, Learning Guide 49Michael D. Kennedy and Pauline Gianopoli, Entrepreneurs and Expertise: Making Post-Communist Capitalism in Poland, February, 1993 55 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              “mindset” can be viewed as a set of assumptions that affect a person or a group in particular areas of their decision making. This information is taken from Nickolas Barr’s seminal book “Labor Markets and Social Policy in Central and Eastern Europe”.50 Starting with the background information and description of these factors provided by Barr, a number of authors have tried to identify features and explanations for the post￾communist mindset. Ruslan Stefanov characterized the symptoms of the post-communist mindset them as a “lack of personal responsibility, and the right NOT to choose, to have choices made by the system”.51 Michael Kennedy describes it as “learned helplessness, suspicion of bureaucracy and government”. The themes that commonly appear in research on the subject therefore include a limited habit and even ability to make choices in a various aspects of one’s life, which can translate is learned helplessness and a lack of personal responsibility. These limitations are related to an almost reflex reliance on the system for opportunities and decisions strangely combined with a degree of distrust towards the very system to which decisions are delegated. Further in his publication, Mr. Kennedy characterizes how the post￾communist mindset constraints skills and notions important to business and entrepreneurship as “shortcomings related to ‘maximizing profitability’, understanding the need to predict things in the future (e.g., cash flows), applying basic accounting concepts to make decisions, delegating responsibilities and listening to good ideas from middle-managers, notions of sunk costs and opportunity costs”.52 These features therefore affect important aspects of business, but it is important to remember that his research dates back over 25 years, and the magnitude of these features is likely to have decreased in the intervening time, as ex-Soviet republics adjusted to their new environment. Education System The education sector in Armenia is comprised of general education establishments, such as preschool and school establishments, and professional education establishments such as technical and vocational schools and colleges, and higher education institutions including universities, institutes, and academies. Traditionally, the Armenian family places a high value on education and considers it crucial in providing a solid background for young people to succeed in future stages of their lives. Unsurprisingly, during the Soviet era, the Soviet Socialist Republic of Armenia was one of the most educated Republics with 100 percent population literacy. The post-soviet phase, marked by many hardships and painful transitions, severely impacted the quality and competitiveness of Armenia’s educational system. Even as of 2008, Armenia was spending 3 percent of its GDP on education, the third lowest figure in the region.53 In May 2005, the Republic of Armenia officially joined the Bologna Process, committing to the implementation of Bologna principles and as a result, becoming a part of the European Higher Education Area. The endorsement of a number of policy documents and the elaboration of few laws followed to carry on these reforms. For example, the MoE instituted a new large scale teacher training program and developed a new curriculum and State Standards for Secondary Education which provides the foundation for the gradual introduction of new subject standards, syllabi, and textbooks. Following the reforms of the Bologna process and starting in the 2006-2007 academic year, Armenia converted to a 3 levels, 12 year general education system with elementary school, middle school, and high school comprising 4, 5, and 3 years, respectively. 50Barr N., Labor Markets and Social Policy in Central and Eastern Europe, London, November 2004, p. 7-10 51Stefanov R., The Challenges of Changing the Old Mindset 52 Michael D. Kennedy and Pauline Gianopoli, Entrepreneurs and Expertise: Making Post-Communist Capitalism in Poland, February, 1993 53 UNICEF, Country Profile - Education in Armenia, available at http://www.unicef.org/ceecis/Armenia.pdf 56 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              The gross secondary and high school enrollment data over the period of 2006 to 2012is provided below in Table 11.54,55These numbers underline the ongoing decline in gross enrollment across the board (-1.9 percent and -5 percent for middle and high school, respectively). However, that decline is not evenly split between genders; gross female enrollment remained relatively stable over time (-0.8 percent and -1.2 percent for middle and high school, respectively, over the period of 2006 to 2012), whereas gross male enrollment appears to be steadily decreasing (-2.9 percent and -8.4 percent for middle and high school, respectively, over the period of 2006 to 2012). Findings in the National Youth Aspirations survey support this point, stating that, although the male rural youth generally believes that both men and women should have equal participation in decision-making as it relates to the family, they retain a conservative and traditional perception concerning career development and providing financially for the family”56 Armenia is also the only country in the Caucasus to have a lower net enrollment ratio for primary education than for secondary education.57 There is also some discrepancy in net attendance ratio between rural (89 percent) and urban regions (91 percent). Table 11: Gross Enrollment in Secondary and High School by Year and Gender (%) Enrollment Metrics 2006 2008 2010 2012 Percentage difference 2006-2012 Middle school Gross total enrollment 90.9 90.8 90.1 89.2 -1.9% Gross female enrollment 92.4 92.1 91.8 91.7 -0.8% Gross male enrollment 89.5 89.6 88.6 86.9 -2.9% High school Gross total enrollment 78 81.9 84.4 74.1 -5.0% Gross female enrollment 82.5 86 90.9 81.5 -1.2% Gross male enrollment 73.6 78.1 78.4 67.4 -8.4% In terms of education quality, Armenia was historically below many of its neighbors, although most of the data obtained does not yet reflect the influence of the various reforms completed or still ongoing. For example, in 2003 Armenia participated in the Trends in International Mathematics and Science Study (TIMSS) and scored 8th of the 13 participating countries in the region in mathematics, and second to last in the region in science.58These relatively poor results have been found to be negatively correlated with absenteeism, a phenomenon that has been noted to be on the rise since 2002. While reforms implemented by the government aim to gradually increase the salary of teachers, a policy considered important to raise education quality, salaries remain below the average national wage 54Table 1, Gross involvement of pupils in secondary schools by indicators and years (2006-2012), data by http://armstatbank.am, 2012. 55Table 2, Gross involvement of pupils in high schools by indicators and years (2006-2012), data by http://armstatbank.am, 2012. 56 National Youth Aspirations Research Report UNDP, 2012., Yerevan p.13 http://www.undp.org/content/dam/armenia/docs/National%20youth%20aspirations%20survey%20report_ENG_edit ed%20final.pdf 57 UNICEF, Country Profile - Education in Armenia, available at http://www.unicef.org/ceecis/Armenia.pdf 58 UNICEF, Country Profile - Education in Armenia, available at http://www.unicef.org/ceecis/Armenia.pdf 57 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              (approximately 68 percent of average national wage in 2005).59 As a result, it is difficult for teachers to support themselves without supplemental income. Low salaries also limit the ability of schools to retain experienced teachers and to attract new and highly qualified individuals to the teaching profession. This, combined with the relatively poor quality of public schools, leads to parents often investing in their schools in through the informal purchase of services from teachers through tutoring services. This is especially true in high school, where students planning to attend tertiary education must prepare for entrance exams but there are also various forms of informal fees levied by schools although it is illegal to make such fees compulsory, and parents are also asked to make payments for textbook rental schemes and for school supplies in some cases. As a result, there is an equity gap in educational access between the richest and poorest income brackets. Capacity of Armenian Management In the 2011-2012 edition of its National Competitiveness Report of Armenia, the Economy and Values Research Center (EV) dedicated two chapters on the role of management in driving competitiveness and on specific recommendations to improve management practices in Armenia. The report found several serious shortcomings in the Armenian labor market as they relate to business management. In terms of overall management practices, Armenia ranked last among its benchmark countries. The study suggests that this is largely due to a poor reliance on professional management in Armenia due to low awareness of their importance for the good functioning of a firm, a general lack of trust, and non-merit based hiring practices. The study further broke down management into 3 meshed pillars operations management60, target management61, and talent management62 (using a method developed by McKinsey and Co.), and compared Armenia to its benchmarks and other nations. Armenia scored worse than its benchmark in each of the three pillars, but the gap was particularly large in operations management and in target management. The study found that founder-owned or second generation family-owned firms had the lowest and second￾lowest management practice score, respectively. However, the self-assessment of Armenian managers was noted to be significantly higher than is reflected from measurements in the study (3.89 versus 2.46 on a scale from 1 to 5), by a margin of 58 percent. To verify this hypothesis EV conducted an assessment of the awareness of company managers about contemporary strategic management systems and practices. While budgeting was found to be the most commonly used management tool, when asked about strategic planning, most Armenian companies claimed to have in place an elaborate system, but in fact lacked the vital components of long term strategic planning, including a lack of long-term goals, a limited or absent competitive analysis, and a rarely formulated market strategy. Yet this pattern of higher-than-reality self-assessment was especially pronounced in family-owned and managed firms, most likely reflecting a lack of knowledge of best practices. This is particularly salient since Armenia was found to have a much greater share of firms owned by their founder(s) (56 percent versus 19.8 percent) than its benchmark. Entrepreneurship in Armenia Business ownership remains lower in Armenia than in Europe and Central Asia. Only 2 percent of Armenians reported owning a (formal) business, compared to 4 percent in Georgia and 15 percent in 59 UNICEF, Country Profile - Education in Armenia, available at http://www.unicef.org/ceecis/Armenia.pdf 60 Operations management is defined as the processes and efforts that enable optimization of production processes and generate maximal value. 61 Target management is defined as the processes and efforts that align material and human potential of the whole company. 62 Talent management is defined as the processes and efforts that enable optimization of human capital use. 58                                                                 developing countries.63 In terms of firm entry density, Armenia lags significantly behind Georgia, despite both starting at approximately 1.25 newly registered companies per 1,000 working-age people in 2004. Armenia’s evolution in this area remained relatively flat between 2004 and 2011, whereas Georgia’s rose to approximately 2.25 by 2008 and stood at approximately 4.25 by 2011. Armenia also lags behind the average for Europe and Central Asia throughout this period. Available data suggests that “only about 12 percent of individuals in the labor force have ever attempted to start a business and less than 6 percent have succeeded”.64 However, this data is contradicted by findings from the 2010 Life Transition Survey, which found that nearly half of those who attempted to start a business succeeded.65 However, it is also important to assess the pool of potential entrepreneurs, known as “latent entrepreneurs” in Armenia, as not everyone can or does start a business. Only 17 percent of the labor force and a similar share of the wage-employed can be considered latent entrepreneurs in Armenia. These values are far below those observed in Europe and Central Asia, where 27 percent of the labor force and 22 of the wage-employed fall into this category. Company founders in Armenia were found to possess a postgraduate degree in the vast majority of cases (75 percent), and this remained true regardless of whether the company was focused on high-technology or low-technology products and services.66 This is in sharp contrast with Georgia, where 44 percent of founders had a bachelor’s degree. Founders reported that the two principal reasons they started a business are that they wanted to be their own boss and that they saw an opportunity to make money. When asked about the principal obstacles they are facing in founding or operating a firm, founders reported the three most important factors were recruiting highly skilled employee, the market risk/uncertainty, and difficulties finding the necessary funding.67 When asked about their principal perceived legal and regulatory constraints, Armenian founders identified continually changing taxation regulations and government official favoring well-connected individuals. The former was also the most often-cited constraint in Georgia, but the other was high tax rates. 63Kuriakose, Smita, ed. 2013. Fostering Entrepreneurship in Armenia.Directions in Development. Washington, DC: World Bank. doi:10.1596/978-1-4648-0064-1. License: Creative Commons Attribution CC BY 3.0 64World Bank Armenia, Accumulation, Competition, and Connectivity, p. 17, http://webcache.googleusercontent.com/search?q=cache:ESSbNsRN￾OYJ:documents.worldbank.org/curated/en/2013/04/18042068/armenia-accumulation-competition￾connectivity+&cd=1&hl=en&ct=clnk&gl=am 65Kuriakose, Smita, ed. 2013. Fostering Entrepreneurship in Armenia.Directions in Development. Washington, DC: World Bank. doi:10.1596/978-1-4648-0064-1. License: Creative Commons Attribution CC BY 3.0 66Kuriakose, Smita, ed. 2013. Fostering Entrepreneurship in Armenia.Directions in Development. Washington, DC: World Bank. doi:10.1596/978-1-4648-0064-1. License: Creative Commons Attribution CC BY 3.0 67Kuriakose, Smita, ed. 2013. Fostering Entrepreneurship in Armenia.Directions in Development. Washington, DC: World Bank. doi:10.1596/978-1-4648-0064-1. License: Creative Commons Attribution CC BY 3.0 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project 59 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              Appendix 5: Comparable Programs in Armenia JA Company Component The JA Company program is very similar to the student business component the JAA program currently offered. A brief comparison of the curriculum reveals that the major topical differences are the inclusion of CSR and a broader discussion of the context in which the student business will be implemented for the JAA program, whereas the JA program might dive somewhat deeper into the technicalities of business operations.68 Volunteers are also involved in providing students with guidance on how to implement a successful business; however teachers do not appear to be involved to the same extent or as consistently. This fact, in combination with the fact that the curriculum was noted to be lengthy and detailed by students and volunteers/teachers, led to the recommendation that JA develops a simpler version of the program for sites where volunteers’ and/or students’ commitment and capabilities are less strong. Pre-post surveys at several sites demonstrated with a high level of statistical significance that students scored 10 percent higher after taking part in the program.69 Another study evaluating the effect of the JA Business program in the United States but relying on a large sample size (275 participants) found a similar gain of 9 percent, which was also statistically significant.70 While JAA did not conduct (or at least did not provide) the statistical significance of its findings, the organization reported in their quarterly reports an even greater effect on test scores, often between 15 percent and 35 percent. These figures appear to hold stable over time, as fairly similar figures were reported for the school year of 2007-2008, for example.71 This is particularly impressive since JAA surveyed all 1,000 students participating, meaning the changes recorded were averages across all regions and school types. Attitudinal changes brought about by participation in the JA Company program were generally limited. For example, it was found that 85 percent of students indicated they were earning at least a “B” average at pre-survey time, while 92 percent reported the same after completing the program (no measure of statistical significance stated). Students studied as part of the JA Business evaluation reported they were highly satisfied with the program (mean score of 9 out of 10). The sample of JA Business students studied also reported that the program was interesting (100 percent agreed or strongly agreed) and worthwhile (100 percent agreed or strongly agreed). 93 percent reported that program participation helped them at least a little in communicating effectively with others, working as a member of a team, being a leader, and making presentations to large groups of people. The vast majority also reported that the program helped them at least a little in thinking creatively, solving problems, budget money, and use computers to manage a company. Similar results were found in a subsequent evaluation conducted in 2011 with a large 68 This assessment is based on a comparison between the JAA program curriculum and the description and objectives of the 12 sessions that compose the JA Business program in the United States. This latter information can be found at https://www.juniorachievement.org/web/ja-usa/ja￾programs?p_p_id=56_INSTANCE_abcd&p_p_lifecycle=0&p_p_state=maximized&p_p_mode=view&p_p_col_id= ja-column-1&p_p_col_count=2&_56_INSTANCE_abcd_groupId=14516&_56_INSTANCE_abcd_articleId=19511. 69Evaluation and Training Institute, JA Company Program Summative Evaluation Final Report, June 2007, available at http://www.myja.org/programs/evaluation/reports/ja_company_program_evaluation_2011.pdf. 70JA Company Program Student Evaluation Results, 2011, available at http://www.myja.org/programs/evaluation/reports/ja_company_program_evaluation_2011.pdf. 71 Although reports dating back to this period were tied to the previous “Economics Education” JAA program, the student business component has not evolved significantly across the two programs. However, despite the fact that JAA was at least partly responsible for training teachers and shaping the curriculum for economics during its inclusion into the official social studies curriculum, a comparison of the figures across time may not be appropriate, as the depth at which the subject is covered likely decreased due to sharing the subject with ethics, philosophy, and aesthetics. 60 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              sample size.72 JA staff and volunteers were also positive about the reaction of students to the program, their enthusiasm, and the quality of the material provided. However, the number of JA staff and volunteers interviewed and surveyed was too small to draw conclusions on any larger scale. In the 2011 study for the JA Business program, where the sample size was much larger, volunteers had to select the three skills they felt were most improved due to program participation. 85 percent identified teamwork, and 46 percent identified leadership skills and entrepreneurial skills, mostly agreeing with the self￾reported information from students. In that same study, students were sent a follow-up survey approximately 6 months after program completion. 22 percent reported having completed a business plan for a future venture, and 63 percent reported talking to people around them about a business idea. However, the majority of students in these two categories (78 percent) reported that their business idea was based off the same idea as their JA Company business. 54 percent of students reported that the program positively influenced their future career plans, and 34 percent reported that the program had actually changed their future education and/or career goals. Additionally, a longitudinal study of JA students who participated in JA during 5th grade, 8th grade, and 11th grade in the United States found that the program had a significant effect on students in terms of high school graduation, confidence in their ability to complete college, attending college right after graduation, and getting a summer job.73 This effect was measured against a comparison group of students who did not participate in the program but were otherwise similar. Applied Economics Components The classroom component of JAA’s Economics Education program where students learned about applied economics, and which has now been incorporated to some degree to the MoE curriculum for the subject of social studies, is also similar to JA programs such as JA Economics and variants including the JA Explore Economics.74 However, a brief review of the curriculum suggested for these JA programs suggest that they are somewhat more in-depth than the program offered by JAA, even prior to its incorporation into the MoE curriculum for the subject of social studies. Regardless, no significant and detailed evaluation report was found for this JA program. The JAA curriculum also included components that align with the pilot program JA Global Business Ethics, which focuses on fostering ethical decision-making as students enter the workforce and global marketplace. However, these components are now mostly covered as part of the ethics and philosophy components of social studies, although they might be briefly discussed as part of the CSR component of the JAA program. Based on the findings of various evaluations outlined above, it should be clear that the student business and applied economics components are grounded in sound development hypothesis that have been at least partly confirmed empirically. However, the JAA program as it currently exists also includes a CSR component and the NGO component, which are not found in JA programs. Similarly, these components 72JA Company Program Student Evaluation Results, 2011, available at http://www.myja.org/programs/evaluation/reports/ja_company_program_evaluation_2011.pdf. 73 The Education Group, JA Worldwide Headquarters, The Impact of Student Participation in JA Worldwide: Selected Cumulative and Longitudinal Findings, 2004, available at http://www.myja.org/programs/evaluation/reports/longitudinal_evaluation.pdf. 74 The adjustments made to the theoretical component of the “Economics Education” curriculum in order for it to fit in the social studies are not known in details, but it is expected that a reduction in depth may have had to take place since the class also covers aesthetics, philosophy, and ethics. 61 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                                                                   were not noted as present in the evaluation of the JA Kyrgyzstan conducted in 2005, nor during a review of other organizations that might be operating in the same field as JAA.75 Corporate Social Responsibilities Component Formal adoption of CSR is a new concept in many nations, and while it has formally existed in more developed parts of the world for some time, its operationalization and execution is highly heterogeneous, even within Western Europe.76 In certain countries, such as China, making safe and high-quality products falls within CSR, while in others, such as Germany, CSR is tied to employment practices.77 In Armenia, the concept is relatively new, and was formally brought to the country by foreign companies, including VivaCell-MTS and HSBC. JAA reported first being made aware of the importance of CSR in 2010, after discussions with several companies, including the two mentioned, about possible collaborations. No project to teach or sensitize students to CSR has been found through a survey of other programs in place in Armenia. However, the mindset and context of Armenia, including high levels of litter, the perception, often justified, that business is corrupt, and the wide range of community needs should suffice to suggest that this is an important concept that will continue to spread and become more popular as time goes by. In addition, since so many Armenians, especially those who have higher education, leave the country, CSR is bound to be a useful concept, since it is more established abroad. This is compounded by the fact that more international companies are looking to establish foothold in Armenia, such as American Express, which has a strong CSR component in many countries in which it operates.78 Nongovernmental Organization Component NGOs are a more established concept in Armenia than CSR, yet their emergence is relatively recent and their capacity is weak on average. International NGOs such as CARE and Oxfam established resident missions in Armenia following independence in 1991 and were joined by some diaspora organizations. However, there were only 44 local NGOs operating in Armenia in 1994.79 This situation evolved rapidly in the mid-1990s when Western governments and international agencies began providing grants to promote civil society and democratization in Armenia. By 2007, over 2,000 NGOs were registered with the Armenian Ministry of Justice. As of April 2003, there were 3,450 officially registered NGOs in the country, and this number grew to 5,700 local NGOs by 2010.80 However, a review of these organizations showed that many were NGOs in name only, and that grant from donor organizations were consistently awarded to the same subset of organizations, while the rest appeared to be inconsistently operating. An author estimated that only approximately 15 percent of these local NGOs can be considered operational, and that most in that group are small outfits that often have vague and obscure missions. It was also noted that most NGOs in Armenia are led by Soviet-era elites, often because these tend to be formally established, professionalized, and to have connections and therefore are the most suitable recipients of donor funds since they have the capacity to comply with grant requirements. As a result, they survive 75 For the evaluation report, please visit http://pdf.usaid.gov/pdf_docs/PDACF533.pdf. 76See for example Habisch, André; Jan Jonker, Martina Wegner, R. Schmidpeter (eds.) (2005). Corporate Social Responsibility across the Europe. Heidelberg: Springer. ISBN 978-3-540-23251-3. 77See for example Knox, Simon, ″Corporate Social Responsibility and Business Decision Making″. In Spiritual Motivation: New Thinking for Business and Management (eds. Ramsden, J.J., Aida, S. and Kakabadse, A.). Basingstoke: Palgrave Macmillan (2007).ISBN 978-0-230-54291-4 78 See for example the American Express page on CSR at http://about.americanexpress.com/csr/, or the #Passion Project at http://about.americanexpress.com/news/pr/2013/american‐express‐launches‐‐passionproject.aspx. 79Ishkanian, Armine; Is the Personal Political? The Development of Armenia’s NGO Sector During the Post-Soviet Period; 2003, available at http://iseees.berkeley.edu/bps/publications/2003_03-ishk.pdf. 80Gharabegian, Areg; Non-Governmental Organizations in Armenia; Armenian Weekly; January 14, 2014; available at http://www.armenianweekly.com/2014/01/15/ngo-armenia/. 62 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project                                                              better over the long term.81 Domestic NGOs in Armenia are mostly clustered in Yerevan, although there has been an increase in marz-based NGOs in the last decade. Out of domestic NGOs considered “operational” but which may or may not be active, many focus on human rights and public policy, followed by youth issues. Education is relatively poorly represented.82 Overall, it was found that domestic NGOs in Armenia have failed to extent their outreach and rally greater support and higher levels of citizen participation in their activities. This is part due to the fact that voluntarism for society was not common practice during the Soviet era, NGOs can have a difficult time involving volunteers or retaining them.83 Furthermore, while most NGOs in Armenia have constitutions that outline their governance mechanisms, they are often not followed, and their existence is primarily justified because they are required to receive required permits and attract funds. Another issue commonly afflicting Armenian NGOs is their lack of experience in fundraising methods, basic marketing, and financial management skills, as well as their over-reliance on a few sources of funding. Given this brief overview of NGOs in Armenia, it is rather clear that JAA’s inclusion of an NGO component is sorely needed. Students are sensitized to volunteering at an age when they are most likely to be passionate about a cause, and learn how to organize an NGO effectively, how to overcome issues with funding, including the search for funds from within the community. 81Ishkanian, Armine; Is the Personal Political? The Development of Armenia’s NGO Sector During the Post-Soviet Period; 2003, available at http://iseees.berkeley.edu/bps/publications/2003_03-ishk.pdf. 82Gharabegian, Areg; Non-Governmental Organizations in Armenia; Armenian Weekly; January 14, 2014; available at http://www.armenianweekly.com/2014/01/15/ngo-armenia/. 83Gharabegian, Areg; Non-Governmental Organizations in Armenia; Armenian Weekly; January 14, 2014; available at http://www.armenianweekly.com/2014/01/15/ngo-armenia/. 63 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Appendix 6: Reference List Economics Education Activity List of Documentation Received from USAID/Armenia USAID Action Memo (1)  United States International Development Agency (USAID) Armenia. “Action Memo: Approval of Exception to Competition for Economics Education Activity.” September 13, 2006. Quarterly Reports (16) 2006  Junior Achievement of Armenia (JAA). “Junior Achievement of Armenia Economics Education Quarterly Report 1: September to November 2006.” 2007  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 2: December 2006 to February 2007.”  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 3: March to May 2007.”  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 4: June to August 2007.”  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 5: September to November 2007.” 2008  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 6: December 2007 to February 2008.”  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 7: March to May 2008.”  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 8: June to August 2008.”  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 9: September to November 2008.” 2009  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 10: December 2008 to February 2009.”  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 11: March to May 2009.”  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 12: June to August 2009.”  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 13: September to November 2009.” 64 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project 2010  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 14: December 2009 to February 2010.”  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 15: March to May 2010.”  JAA. “Junior Achievement of Armenia Economics Education Quarterly Report 16: June to August 2010.” List of Documentation Received from Junior Achievement of Armenia Final Report (2)  Junior Achievement of Armenia (JAA). “Junior Achievement of Armenia Economics Education Final Report (2006–2010).”  JAA. “Junior Achievement of Armenia Economics Education Final Report Chart (2006–2010).” Entrepreneurship and Civic Activism for Young People Project List of Documentation Received from USAID/Armenia Performance Management Plan (1)  Junior Achievement of Armenia (JAA). “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Performance Management Plan (PMP).” 2013. Annual Work Plans (3)  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Revised Workplan for the Period May 2012 to September 2013: Cooperative Agreement AID￾111-G-00-06-00082.” 2012.  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Revised Workplan for the Period September 2013 to August 2014: Cooperative Agreement AID￾111-G-00-06-00082.” 2013.  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Workplan for the Years 2011–2012: Cooperative Agreement AID-111-G-00-06-00082.” 2011. Annual Work Plan Charts (2)  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Work Plan Chart for the Period April 1, 2012, to August 31, 2013: Activity and Milestones.” 2012.  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Work Plan Chart for the Period September 2013 to August 2014: Activity and Milestones.” 2013. Quarterly Reports (10) 65 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project 2011  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Quarterly Report 1: April to July 2011.”  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Quarterly Report 2: July to October 2011.”  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Quarterly Report 3: October 2011 to January 2012.” 2012  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Quarterly Report 4: January to April 2012.”  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Quarterly Report 5: April to June 2012.”  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Quarterly’s Report 6: July to September 2012.”  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Quarterly Report 7: October to December 2012.” 2013  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Quarterly Report 8: January to April 2013.”  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Quarterly Report 9: April to June 2013.”  JAA. “Junior Achievement of Armenia Entrepreneurship and Civic Activism for Young People Quarterly Report 10: July to September 2013.” Financial Reports (41) 2011  USAID Armenia. “Federal Financial Report for Junior Achievement of Armenia: April 1, 2011– June 30, 2011.” June 30.  USAID Armenia. “Federal Financial Report for Junior Achievement of Armenia: April 1, 2011– January 31, 2015.” September 30.  USAID Armenia. “Federal Financial Report for Junior Achievement of Armenia: April 1, 2011– January 31, 2015.” December 31.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for April 1, 2011–April 30, 2011.” May 11.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for May 1, 2011–May 31, 2011.” June 16.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for June 1, 2011–June 30, 2011.” August 9.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for July 1, 2011–July 31, 2011.” August 16. 66 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for August 1, 2011–August 31, 2011.” September 16.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for September 1, 2011–September 30, 2011.” October 14.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for October 1, 2011–October 31, 2011.” November 15.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for November 1, 2011–November 30, 2011.” December 12.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for November 1, 2011–December 31, 2011.” January 30.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for April 1, 2011–April 30, 2011.” April 13.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for May 1, 2011–May 31, 2011.” May 11.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for June 1, 2011–June 30, 2011.” June 16.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for August 1, 2011–August 31, 2011.” August 17.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for September 1, 2011–September 30, 2011.” September 16.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for October 1, 2011–October 31, 2011.” October 14.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for November 1, 2011–November 30, 2011.” November 15.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for December 1, 2011–December 31, 2011.” December 12. 2012  USAID Armenia. “Administrative Approval Form and Checklist.” September 24.  USAID Armenia. “Federal Financial Report for Junior Achievement of Armenia: April 1, 2011– January 31, 2015.” March 31.  USAID Armenia. “Federal Financial Report for Junior Achievement of Armenia: April 1, 2011– January 31, 2015.” June 30.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for January 1, 2012–January 31, 2012.” March 1.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for February 1, 2012–February 29, 2012.” March 12.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for March 1, 2012–March 31, 2012.” April 25.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for April 1, 2012–April 30, 2012.” May 17. 67 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for May 1, 2012–May 31, 2012.” June 18.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for June 1, 2012–June 30, 2012.” July 23.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for July 1, 2012–July 31, 2012.” August 30.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for August 1, 2012–August 31, 2012.” September 23.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for September 1, 2012–September 30, 2012.” December 5.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for January 1, 2012–January 31, 2012.” January 30.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for February 1, 2012–February 29, 2012.” March 12.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for March 1, 2012–March 31, 2012.” March 16.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for April 1, 2012–April 30, 2012.” April 25.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for May 1, 2012–May 31, 2012.” May 17.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for June 1, 2012–June 30, 2012.” June 18.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for July 1, 2012–July 31, 2012.” July 23.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for August 1, 2012–August 31, 2012.” August 30.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Request for Advance for September 1, 2012–September 30, 2012.” September 23. List of Documentation Received from Junior Achievement of Armenia Grant Agreement (2)  USAID-JAA Agreement, signed March 31, 2011, subject: “Award No. AID-111-A-11-00001, Entrepreneurship and Civic Activism for Young People (Cover Page).”  USAID-JAA Agreement, signed March 31, 2011, subject: “Award No. AID-111-A-11-00001, Entrepreneurship and Civic Activism for Young People (Attachments A–C).” Financial Reports (16)  JAA. “NGO Grant Recipient Financial Report: Ararat Village #2.” 2013.  JAA. “NGO Grant Recipient Financial Report: Azatan.” 2013.  JAA. “NGO Grant Recipient Financial Report: Byuarakan.” 2013. 68 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project  JAA. “NGO Grant Recipient Financial Report: Khachik.” 2013.  JAA. “NGO Grant Recipient Financial Report: Koghb #1.” 2013.  JAA. “NGO Grant Recipient Financial Report: Mayisyan.” 2013.  JAA. “NGO Grant Recipient Financial Report: Stepanavan.” 2013.  JAA. “NGO Grant Recipient Financial Report: Vanadzor #30.” 2013.  JAA. “NGO Grant Recipient Financial Report: Voskehat.” 2013.  JAA. “NGO Grant Recipient Financial Report: Yegheg.” 2013.  JAA. “NGO Grant Recipient Financial Report: Yegheg #1.” 2013.  JAA. “Non-USAID Detailed Financial Report: April 2011–March 2012 (First Year).”  JAA. “USAID Detailed Financial Report: April 2011–March 2012 (First Year).”  JAA. “USAID Detailed Financial Report: April 2012–March 2013 (Second Year).”  JAA. “USAID Funds Distribution across Regions and Schools Report.” October 2013.  USAID Armenia. “Public Voucher for Purchases and Services Other Than Personal: Liquidation Report for April 1, 2011–April 30, 2011.” May 11, 2011. JAA Program Stakeholder Contact Lists (6)  JAA. “Alumni Contact List.” 2011.  JAA. “Alumni Contact List.” 2012.  JAA. “Business Consultant Contact List.” 2011.  JAA. “Business Consultant Contact List.” 2012.  JAA. “List of JAA Business Partners: Contact Information.” 2013.  JAA. “Roster of the JAA Regional Representatives.” 2013. Program School Lists (5)  JAA. “Teachers and Schools Enrolled in the Program by Region (First Year: April 1, 2011– March 31, 2012).”  JAA. “Teachers and Schools Enrolled in the Program by Region (Second Year: September 1, 2012–March 31, 2013).”  JAA. “Teachers and Schools Enrolled in the Program by Region (Third Year: September 1, 2013–March 31, 2014).” Armenian Version.  JAA. “Teachers and Schools Enrolled in the Program by Region (Third Year: September 1, 2013–March 31, 2014).” English Version.  JAA. “The Most Successful Projects and the Most Unsuccessful Projects (2011–2013).” Program School Lists by Grade Level (6)  JAA. “Aragatsotn School List by Grade Level.” 2013.  JAA. “Ararat School List by Grade Level.” 2013.  JAA. “Gegharkunik School List by Grade Level.” 2013.  JAA. “Kotayq School List by Grade Level.” 2013.  JAA. “Lori School List by Grade Level.” 2013.  JAA. “Yerevan School List by Grade Level.” 2013. 69 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Program Schools’ Principal Lists (3)  JAA. “Principal Names, Addresses, and Phone Numbers for Schools Enrolled in the Program by Region.” 2011.  JAA. “Principal Names, Addresses, and Phone Numbers for Schools Enrolled in the Program by Region.” 2012.  JAA. “Principal Names, Addresses, and Phone Numbers for Schools Enrolled in the Program by Region.” 2013. Program Schools’ Student Rosters (20)  JAA. “Program Schools’ Student Rosters for Ararat Region by School.” 2011. Armenian.  JAA. “Program Schools’ Student Rosters for Ararat Region by School.” 2012. Armenian.  JAA. “Program Schools’ Student Rosters for Armavir & Aragatsotn Regions by School.” 2011. Armenian.  JAA. “Program Schools’ Student Rosters for Armavir & Aragatsotn Regions by School.” 2012. Armenian.  JAA. “Program Schools’ Student Rosters for Gegharquniq Region by School.” 2011. Armenian.  JAA. “Program Schools’ Student Rosters for Gegharquniq Region by School.” 2012. Armenian.  JAA. “Program Schools’ Student Rosters for Kotayq Region by School.” 2011. Armenian.  JAA. “Program Schools’ Student Rosters for Kotayq Region by School.” 2012. Armenian.  JAA. “Program Schools’ Student Rosters for Lori Region by School.” 2011. Armenian.  JAA. “Program Schools’ Student Rosters for Lori Region by School.” 2012. Armenian.  JAA. “Program Schools’ Student Rosters for Shirak Region by School.” 2011. Armenian.  JAA. “Program Schools’ Student Rosters for Shirak Region by School.” 2012. Armenian.  JAA. “Program Schools’ Student Rosters for Syuniq Region by School.” 2011. Armenian.  JAA. “Program Schools’ Student Rosters for Syuniq Region by School.” 2012. Armenian.  JAA. “Program Schools’ Student Rosters for Tavusih Region by School.” 2011. Armenian.  JAA. “Program Schools’ Student Rosters for Tavusih Region by School.” 2012. Armenian.  JAA. “Program Schools’ Student Rosters for Vayots Dzor Region by School.” 2011. Armenian.  JAA. “Program Schools’ Student Rosters for Vayots Dzor Region by School.” 2012. Armenian.  JAA. “Program Schools’ Student Rosters for Yerevan Region by School.” 2011. Armenian.  JAA. “Program Schools’ Student Rosters for Yerevan Region by School.” 2012. Armenian. Memorandum of Understandings (7)  JAA. “Memorandum of Understanding: Junior Achievement of Armenia & Coca-Cola.” 2011. Armenian.  JAA. “Memorandum of Understanding: Junior Achievement of Armenia & Kanaka.” 2011.  JAA. “Memorandum of Understanding: Junior Achievement of Armenia & Mary Kay.” 2011.  JAA. “Memorandum of Understanding: Junior Achievement of Armenia & Marriott Hotel.” 2012. 70 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project  JAA. “Memorandum of Understanding: Junior Achievement of Armenia & Megerian Carpet.” 2011. Armenian.  JAA. “Memorandum of Understanding: Junior Achievement of Armenia & Ministry of Science and Education in Armenia.” Armenian.  JAA. “Memorandum of Understanding: Junior Achievement of Armenia & Viva-Cell.” 2011. Armenian. Annual Work Plan Charts (1)  JAA. “Junior Achievement of Armenia’s Work Plan Chart for the Period April 1, 2011 to April 1, 2012: Activity and Milestones.” 2012. Program Instructional Materials (11)  JAA. “Entrepreneurship and Civic Activism for Young People Corporate Social Responsibility (CSR) Workbook.” 2012. Armenian.  JAA. “Entrepreneurship and Civic Activism for Young People Guide for Business Consultants.” 2011. Armenian.  JAA. “Entrepreneurship and Civic Activism for Young People Manual Success Stories (2012).” 2012. Armenian.  JAA. “Entrepreneurship and Civic Activism for Young People Manual Success Stories (2013).” 2013. Armenian.  JAA. “Entrepreneurship and Civic Activism for Young People Program Curriculum: Student Entrepreneurship Organizations (September–December).” 2013. Armenian.  JAA. “Entrepreneurship and Civic Activism for Young People Student Business Company– Guide for Teachers and Students.” 2012. Armenian.  JAA. “Entrepreneurship and Civic Activism for Young People Student NGOs–Guide for Teachers and Students.” 2011. Armenian.  JAA. “Old Manual of Business Companies.” 2011. Armenian.  JAA. “Republic of Armenia Legislation on Non-Governmental Organizations.” Armenian.  Seferyan, Nazareth. “Corporate Social Responsibility (CSR) Workbook.” 2010. Armenian.  United Nations Development Programme. Corporate Social Responsibility Concept of Government. 2010. Armenian. Program Evaluation Materials and Data (12)  JAA. “Alumni Questionnaire Statistics.” English.  JAA. “Economics Questionnaire–After the Program.” Armenian.  JAA. “Economics Questionnaire–Before the Program.” Armenian.  JAA. “Economics Questionnaire Comparative Data: April 2011–December 2011 (First Year).” Armenian.  JAA. “Economics Questionnaire Comparative Data: September 2012–December 2012 (Second Year).” Armenian.  JAA. “NGO Questionnaire–After the Program.” Armenian.  JAA. “NGO Questionnaire–Before the Program.” Armenian.  JAA. “NGO Questionnaire Comparative Data: January 2012–March 2012 (First Year).” Armenian.  JAA. “NGO Questionnaire Comparative Data: January 2013–May 2013 (Second Year).” Armenian. 71 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project  JAA. “Program Evaluation for School Administrators.” Armenian.  JAA. “Program Evaluation for Students.” Armenian.  JAA. “Program Evaluation for Teachers.” Armenian. Optimal-JAA Verification Document (1)  Optimal. “Memo to JAA Regarding Reporting Verification Documents Requested and Technical Questions for the Mid-Term Evaluation of USAID/Armenia Entrepreneurship and Civic Activism for Young People Project.” November 4, 2013. 72 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project List of Other Documentation Reviewed by the Evaluation Team for the Evaluation (29) Academy for Educational Development. “Civic Participation Project Youth and Community Action Program (CPP/YCAP) Final Report: September 2004–November 2010.” 2011. Burean, Toma, Daniela Angi, Sorana Constantinescu, and Marius Mosoreanu.“Quantitative Evaluation Regarding Educational Results of IMPACT (2009–2011).”Quantitative Research Briefing. Center for Educational Research and Consulting.“Final Evaluation of ‘ZANG’ Armenian Legal Socialization Program.” March 2013. Cockalo, Dragan, et al. “Quality of Business, Entrepreneurship Education and Business Start-Up Intentions Amongst Students in Serbia: Research Results.” Industrija 41 (3): 125–145. November 6, 2013. European Commission. “How to Support SME Policy from Structural Funds: Building Entrepreneurship Mindsets and Skills in the EU.” Guidebook Series. 2012. EV Consulting. “Market Study on HE Graduate Labor Demand and Employment in Armenia: Final Report.” Yerevan. 2011. EV Consulting. “National Competitiveness Report of Armenia: Agenda for Upgrading Management Practices.” 2011–2012. Evaluation and Training Institute. “Junior Achievement High School Pilot Program Final Report: JA Global Business Ethics.” Los Angeles, California. June 29, 2007. Evaluation and Training Institute. “Junior Achievement Program Summative Evaluation Final Report.” Los Angeles, California. June 2007. Government of the Republic of Armenia. “Armenia Development Strategy (ADS) for 2012–2015: Final Draft.” Yerevan. November 2012. Government of the Republic of Armenia, Ministry of Industry and Trade.“Concept for Small and Medium Entrepreneurship Development Policy and Strategy in Armenia.” Yerevan. 2000. Hutchinson, Jared, Matthew Guttentag, Yvonne Chen and Karen Mo. “Youth Entrepreneurship in Serbia: Constraints and Opportunities.” In Collaboration with George Washington University. May 4, 2012. Israel, Ron, and Sarah Nogueria-Sanca.“Promising Youth Development Strategies. Youth Service Programs: A Study of Promising Models in International Development.” In Collaboration with the Education Development Center, Inc. November 2011. Junior Achievement (JA). “The Impact on Students of Participation in JA Worldwide: Selected Cumulative Longitudinal Findings.” Colorado Springs, Colorado. January 26, 2004. JA. “JA Company Program: Student Evaluation Results.” No date provided. Kennedy, Michael, and Pauline Gianopolis. “Entrepreneurs and Expertise: Making Post-Communist Capitalism in Poland.” In Collaboration with the National Council for Soviet and Eastern European Research. Washington, DC. February 4, 1993. 73 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Millennium Challenge Corporation. “Scorecard for Armenia: Fiscal Year 2013.” National Youth Report of Armenia, analytical part, RA Ministry of Youth and Sport ISBN 978-99941-2-856-3, Yerevan 2012, p. 56-58 PH International. “ZANG Armenian Legal Socialization Program: Developing Knowledge and Promoting Justice.” Brochure. Prince’s Charities Youth Business International. “Global Youth Entrepreneurship Survey 2011: How Non-Financial Report Is Valued by Young People Starting and Growing a Business.” 2011. Prince’s Charities Youth Business International. “Youth Enterprise: Unlocking Ambitions, Creating Growth. How Young Entrepreneurs in the UK Fare Against Leading Enterprise Nations.” In Association with GEM. Senges, Max. “Entrepreneurship I: Developing an Entrepreneurial Mindset Learning Guide.” Soule, Suzanne. “Beyond Communism and War: The Effect of Civic Education on the Democratic Attitudes and Behavior of Bosnian and Herzegovinian Youth.” In Collaboration with the Center for Civic Education. October 2000. Stefanov, Ruslan. “The Challenge of Changing the Old Mindset.” In Collaboration with the Center for International Private Enterprise Development Institute. Tadevosyan, Areg, ZaruhiLavchyan, and AnahitMinassian. “Reviews on Youth Policies and Youth Work in the Countries of South East Europe, Eastern Europe & Caucasus: Armenia.” In Collaboration with European Union, Council of Europe. January 26, 2011. Thorup, Cathryn L., and Shelia Kinkade. “What Works in Youth Engagement in the Balkans.” In Collaboration with the International Youth Foundation. 2005. Tovmasyan, Tigran, and Marcie Taylor Thoma. “Grants to Support Social Science and Policy-Oriented Research Final Analytical Report: The Impact of Civic Education on Schools, Students and Communities.” In Collaboration with the Carnegie Corporation of New York, Caucasus Research Resource Centers and Eurasia Partnership Foundation. Yerevan, Armenia. 2008. Williams, Kate, and Diana Contreras Suarez. “IMPACT Clubs: Reviewing the Evidence.” In Collaboration with Social Entrepreneurship and Economic Development Unit at World Vision, Australia. January 2014. World Bank.“Fostering Entrepreneurship in Armenia.”Edited by SmitaKuriakose. 2013. United Nations Development Programme, Armenia.“National Youth Aspirations Research Report.”In Collaboration with the Republic of Armenia Ministry of Sport and Youth Affairs.Yerevan. 2012. 74 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Appendix 7: Statements of Differences The USAID/Armenia Statement of Work requests that the evaluation team provide any “statements of differences” regarding significant unresolved differences of opinions by funders, implementers and/or members of the evaluation team. Accordingly, please find below the evaluation team’s response to this request. Differences of Opinions by Funders The evaluation team has no statements of differences to report at this time. Differences of Opinions by Implementers The evaluation team has no statements of differences to report at this time. In fact, the conclusions made after completing the field work are nearly identical as those made after the initial days of interviews and focus groups. Differences of Opinions by Evaluation Team Members The evaluation team has no statements of differences to report at this time. 75 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic 76 Appendix 8: Evaluation Design Matrix Core Area Key Evaluation Questions Sub-Questions Potential Data Elements Data Sources/Method Project design 1. Are the project goals ambitious yet attainable and supported by context analysis?  What are the project’s target goals and performance indicators?  Do the goals align with the country’s economic, civic, education, and youth￾development policies and strategies? Project quantitative and qualitative goals Performance indicators from past 2 years (target and actual values) Project inputs and resources Current government policies and initiatives (e.g., economic, youth)  Program administration documents  Public government documents  JAA staff interviews  Program Performance Indicators/project￾management plan 2. Is the current project design based on sound development hypotheses leading to the achievement of sustainable results?  Is the current project design based on sound development hypotheses?  What conditions for successful implementation and sustainability have been identified, and how have they been addressed?  Is the project achieving its intermediate outcomes?  What key elements of the project were specifically designed to contribute to the sustainability of interventions in participating schools after the USAID grant ends? The number and type (e.g., gender) of students involved in each business and community project The number and type of business projects that generate a profit The number of students joining the alumni network Key youth-development theories Best practices (internationally) in youth-entrepreneurship and civic￾activism programs Project logic model Barriers to project implementation  JAA staff interviews  School administrator interviews  Project adviser interviews (e.g., teachers, JAA staff, business consultants)  Program administration documents Activism for Young People Project USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic 77 84The evaluation team has not clearly identified the precursor to the current Entrepreneurship and Civic Activism for Young People program at this stage. JAA has a brief overview of the Economics Program (referenced in the statement of work) on its website. This program is still ongoing and does not appear to include the creation of “mini-businesses” or a community project component, focusing instead on teaching a curriculum that fosters skills that will be highly useful in the business world and global economy. The Economics Program also has significantly greater scope, operating 760 classes and reaching more than 23,000 students annually. Core Area Key Evaluation Questions Sub-Questions Potential Data Elements Data Sources/Method Perceived benefits, strengths, and weaknesses of the programs Project implementation 3. Has replication of the past project been a justified approach for implementation of the current project?  Which aspects of the current program are new, and which were used under previous iterations of the program?84  What criteria or evidence was used to design the current program?  How does the program differ from similar youth programs in the area and internationally? Lessons learned from Economics Education and other JAA projects Pre-implementation analyses (e.g., environmental scan, country asset mapping, best practices)  JAA staff interviews  School administrator interviews 4. How efficient is the project in achieving its goals?  What is the ratio of administrative to program costs?  What is the ratio of school￾level payments (e.g., administration, advisers, student grants) to number of business and community projects completed? Funds distributed to JAA Funds and resources distributed to participating schools Funds distributed to student projects Administrative costs Number of student projects  Program administration documents  JAA staff interviews Activism for Young People Project USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic 78 Core Area Key Evaluation Questions Sub-Questions Potential Data Elements Data Sources/Method  What is the ratio of school￾level volunteer hours to number of business and community projects completed? Project results 5. How broad are the scopes and ideas of the projects implemented by the students?  How did the students identify potential project ideas?  What did students consider in determining a project?  How did the students develop their projects over the course of the program?  What tactics and processes do project advisers use to guide students through the program? Charitable sectors covered (e.g., environment, social welfare, housing/shelter, children) Individuals/organizations recruited for implementation (e.g., government, NGOs, community members, alumni network)  Program administration documents  Project advisor interviews (e.g. teachers, JAA staff, business consultants)  Student interviews  Community-member interviews Activism for Young People Project USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Appendix 9: Sampling Framework and Site Visit Schedule JAA Main Office Interview Schedule Stakeholder Executive Director Program Coordinator and Yerevan Regional Coordinator Role in JAA Intervention JAA administrators develop (or identify) course curriculum, manage regional representatives, raise program money and awareness, form program partnerships, participate in cross-regional student events, evaluate program performance, and complete other organizational management activities. JAA Regional Offices Schedule Stakeholder District Yerevan Aragatsotn Lori Ararat Kotayq Gegharquniq Role in JAA Intervention JAA regional coordinators work on behalf of JAA to provide support to the program and to participating schools. Most coordinators are part-time employees who oversee a given marz in which they select schools participating in the JAA program, conduct site visits, administer and/or collect JAA surveys, and play other administrative support roles. 79 Program School Schedule—Principal, Teachers, Students, and Alumni Region School Name Week of Visit Aragatsotn Aparan High School Week 1 Ujan Week 1 Voskevaz Week 1 Byurakan Week 1 Nor Yedesia Week 1 Ararat Nor Kharberd #1 Week 1 Norashen Week 1 Shahumyan Week 1 Dvin Week 1 Ararat #2 Week 1 Yerevan Yerevan St. College of Information Week 2 Yerevan #127 Week 2 Yerevan #105 Week 2 Yerevan #149 Week 2 Kotayq Bjni Week 2 Armas Week 2 Abovyan #1 High School Week 2 Kamaris Week 2 Nor Geghi #1 Week 2 Yeghvard #1 Week 2 Lori Vanadzor #30 Week 3 Gugarq #1 Week 3 Vanadzor #13 Week 3 Vanadzor #10 Week 3 Gar-Gar Week 3 Kurtan Week 3 Gegharkunik Vardenik Week 4 Gavar High School Week 4 Lichq Week 4 Tsovinar Week 4 Zolaqar #2 Week 4 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project 80 Business Partnership Interview Schedule USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Stakeholder World Vision Mary Kay Hotel Marriott Kanaka Viva-Cell Role in JAA Intervention Business organizations provide funding for the program and/or other nonfinancial forms of support (e.g., student opportunities). Government Officials Schedule Stakeholder Ministry of Science and Education Role in JAA Intervention Armenian ministries provide funding for the economics education curriculum and may provide other nonfinancial forms of support. 81 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Appendix 10: Evaluation Tools Interview Guide for JAA Main Office #1 United States Agency for International Development Learning, Evaluation, and Analysis Project (Bureau for Economic Growth, Education, and the Environment) AID-OAA-C-11-00169 Mid-Term Evaluation of Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Evaluation Interview Guide—JAA Main Office #1 Interview Best Practices The protocol for the interviews will start with a brief introduction of the evaluation team and the purpose of the interview. The interviewer will then ask whether the interviewees have any questions before proceeding with the interview questions. Respondent Role The JAA Executive Director and Project Coordinator assume many responsibilities to ensure that the JAA program is implemented according to its design, such as developing (or identifying) the curriculum for the economics course, CSR component, and NGO guidelines; contacting potential business consultants; co-running Aflatoun and More Than Money programs (matching funds); raising money and the program’s profile through partnerships (e.g., World Bank); choosing student winners of the grant; choosing student participants in the trade fair; and evaluating the regional representative’s performance. Interview Questions General Project Information 1. Organizational chart: Briefly describe your responsibilities and the roles and responsibilities of JAA staff, paid and voluntary. Project Design Based on Sound Hypothesis—Key Project Elements Designed to Contribute to Its Sustainability 2. What approaches has (e.g., collaboration w/ Ministry of Education and Science) and will JAA undertake to institutionalize and sustain its after-school programs? 3. What implication do these approaches have on JAA as an organization? What modifications or changes are necessary to sustain and/or institutionalize its programs? 4. Have you determined ways to maintain the program results despite elections and other conditions that divert attention from the program? 5. What challenges and opportunities for sustainability do you see in your funding strategy? 6. What information sources did you consider in developing the CSR and NGO components of the program? Why were these the best sources for your program? 82 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Efficiency of the Project in Achieving Its Goals—Ratio of Administrative to Program Costs 7. For the paid staff, who is full-time and part-time? 8. What is the average number of hours contributed by volunteers? Efficiency of the Project in Achieving Its Goals—Conditions for Successful Implementation 9. What are the challenges faced by JAA in reaching its strategic goals? 10. What barriers to ongoing implementation have you found given your expectations in the design? 11. What other institutional, regulatory, programmatic, or other changes are necessary to achieve objectives/goals? What about to sustain objectives/goals? 12. What have you learned about the role of regional reps in making the program a success? 13. What have you learned from your evaluations of the program (teachers, students)? Efficiency of the Project in Achieving Its Goals—Inclusion of International Best Practices 14. Do you know of best practices in the area of youth entrepreneurship? What about civic activity? Who do you consider your peer counties? a. We noted [insert any best practice we are considering putting in the final report]. Do you consider that a best practice in your field? Why or why not? Are the Project Goals Ambitious Yet Attainable—Ambitious 15. What are your expectations about what effect that JAA participation would have on students (academically, higher-education, labor-market, business, civic engagement)? 16. Does the business community have comparable expectations for JAA and non-JAA students? Explain 17. Do the school administrators and teachers have comparable expectations? What differentiates the school administrators and teachers at schools that are high performers from those that are poor performers? Explain. 18. What are the students’ expectations, typically? Are there persistent differences between boys and girls? 19. How did you determine the number of target schools and target students? a. Your target for the number of students trained is typically 1,500. What does 1,500 students represent (enrolled in the class, passing the exam, what is passing)? b. What constraints do you face expanding the targets? 20. How do you determine the appropriate type and level of difficulty for your curriculum? 21. Do you face challenges recruiting and engaging local business consultants? International business consultants? If yes, please explain the challenges. How have you addressed these challenges? Are the Project Goals Ambitious Yet Attainable—Target Goals/Performance Indicators & Achievement of Intermediate Outcomes 22. In your opinion, how would you characterize how the JAA program influences students’ aspirations? Are there persistent differences between boys and girls? 23. How does JAA measure/conceptualize student entrepreneurship? How does JAA measure students’ disposition toward entrepreneurship? 83 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project 24. Do you anticipate that the JAA programs will improve the rate of entrepreneurship in Armenia? What are the specific performance measure(s), threshold(s), and benchmark(s)? Explain. 25. How does JAA measure/conceptualize student civic behavior/engagement? 26. Do you anticipate that JAA programs will improve the civic engagement in Armenia? What are the specific performance measure(s), threshold(s), and benchmark(s)? Explain. 27. Do you think the evaluation tools provide useful information about the program? Are they completed on time and completely by students and teachers? What have you learned from the teacher and student evaluations of the program? Are the Project Goals Ambitious Yet Attainable—Alignment with Armenia’s Context 28. How does JAA conceptualize the barriers to entrepreneurship in Armenia? What are the current barriers? 29. What are the barriers to enhanced civic engagement in Armenia? What are the current barriers to changing behavior? Justification of the Replication of Past Project—New or Present under Previous Program 30. Are there specific ways that the previous iteration of the program informed the current program? 31. What has prompted changes in the CSR and NGO curriculum/guidelines (content and delivery methods)? 32. What are the major programmatic aspects of the initiative that are new since the 2011 funding from USAID (e.g., student activities, teacher trainings)? a. Are there aspects of the program that existed previously but have become more formalized since 2011? Justification of the Replication of Past Project—Criteria or Evidence Used for Project Design 33. How do you decide to add new activities to the program, such as the OSCE model game? Are there any that you have decided to discontinue or not pursue? Why is that? Justification of the Replication of Past Project—Comparable Youth Programs 34. Are there comparable programs to JAA’s Entrepreneurship and Civic Action in Armenia? In what ways are they similar (objectives, mission, availability, operations, funding mechanisms)? a. For instance, does YouthBank in Armenia have comparable objectives to the JAA program? 35. How do JAA’s evaluation (pre-post) results compare with JAA evaluation (pre-post) results in the United States and other countries? Explain the differences. 84 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Interview Guide for JAA Main Office #2 United States Agency for International Development Learning, Evaluation, and Analysis Project (Bureau for Economic Growth, Education, and the Environment) AID-OAA-C-11-00169 Mid-Term Evaluation of Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Evaluation Interview Guide—JAA Main Office #2 Interview Best Practices The protocol for the interviews will start with a brief introduction of the evaluation team and the purpose of the interview. The interviewer will then ask whether the interviewees have any questions before proceeding with the interview questions. Respondent Role The JAA Executive Director and Project Coordinator assume many responsibilities to ensure that the JAA program is implemented according to its design, such as developing (or identifying) the curriculum for the economics course, CSR component, and NGO guidelines; contacting potential business consultants; co-running Aflatoun and More Than Money programs (matching funds); raising money and the program’s profile through partnerships (e.g., World Bank); choosing student grant winners; choosing student participants for the trade fair; and evaluating the regional representative’s performance. Specifically, this interview guide aims to fill any gaps left at the end of the data-collection period as well as to obtain a more detailed narrative and clarify certain things with the JAA Executive Director. Interview Questions Introductory Questions 1. What does an average day of work at JAA look like for you? 2. What prompted you to start a chapter of Junior Achievement in Armenia? Conditions for Implementation 3. In your view, what are the most important co-occurring trends (e.g., increased access to the Internet, political reforms, etc.) in Armenia that can benefit the JAA program and its effectiveness? 4. What institutional, regulatory, programmatic, or other changes are necessary to achieve program objectives/goals? What about to sustain these objectives/goals? 5. In your experience, what are the most common and/or significant limiting factors of the program? 6. During our data-collection effort, we spoke to World Vision about its involvement in the JAA program. Was this partnership experience valuable for JAA? 85 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project o o o o o o o o o o [F-Yes] Do you plan to continue to work with World Vision in the future? How do you see your collaboration with World Vision evolving in the future, based on the results of your past collaboration? 7. Do you foresee JAA initiating collaborations with NGOs other than World Vision in the future? [F-Yes] In what areas of JAA project work do you feel there is the greatest need for collaboration with other NGOs? 8. Regarding the JAA alumni club: What are the key functions of this club within the greater scope of the project? What role, if any, does JAA play in organizing and/or sustaining the alumni club? What tools, if any, exist for alumni of the JAA program to find each other, post job openings, network, etc.? Level of Ambition of JAA Program—Armenian Context 9. Do you foresee that the JAA program as it exists today (including the NGO practical component) will be adopted and incorporated in the Ministry-designed curriculum in coming years? Why or why not? 10. How did JAA select which age groups to target for participation in the program? 11. Do you think that the age groups currently targeted by the JAA program are the ones where the maximum impact can be obtained? 12. Over the years of participating in trade fairs, site visits, and discussions with teachers and coordinators, did you notice a tangible change in students’ mind-sets toward: Entrepreneurship? Please explain. Education? Please explain. Their community and students’ role within it? Please explain. 13. How do you anticipate the JAA activities will change civic behavior in the students? [F] What are the current barriers to changing behavior? 14. Is there any intentional or opportunistic interplay between the JAA program and “brain ring” initiatives at the school, regional, or national level? 15. Do you anticipate the JAA activities will improve the rate of entrepreneurship in Armenia? [F] What are the current barriers? 16. What is the traditional process used by JAA to recruit business consultants? Are they generally recruited by the JAA main office, the regional coordinators, or the schools themselves? 17. How do you decide to add new activities to the program, such as the OSCE model game? Are there any that you have decided to discontinue or not pursue? Why is that? Were the Project Goals and Associated Activities Achieved, on Time? 18. In your view, what have been the most successful areas of the JAA program? 19. What about the areas you believe are most in need of improvement? How Does the Program Compare to Similar Youth Programs in the Area? 20. Does your organization benchmark itself against any other youth program implemented by governmental or nongovernmental organizations that have comparable objectives (youth development and entrepreneurship)… 86 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project o o o o o … in Armenia? … in the Caucasus region?  [F-YES to either question] Briefly describe and name these other youth programs.  [F-YES to either question] In your opinion, do these other youth programs have:  effective approaches?  achieved targeted objectives? Is There Evidence to Confirm That the Results Are Likely to Be Sustainable beyond the Project? 21. Do you expect any significant shift in the way the JAA program is currently implemented in the coming years? 22. How do you foresee the evolution of the various sources of funding on which the JAA program relies in the coming years? 23. What is your overall strategy (if any) for making the JAA program: … sustainable in its current form, even with reduced or no JAA funding? …better integrated into the ministry-approved curriculum? …better integrated with communities in which the program operates? 24. What do you see as the next steps in the evolution of the JAA program (e.g., greater coverage of schools, longer duration/permanent offering at each school, change in integration with the ministry, etc.)? Closing Questions  Are there any other comments you would like to make?  Do you have any questions for us about this interview? 87 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Interview Guide for JAA Regional Coordinators United States Agency for International Development Learning, Evaluation, and Analysis Project (Bureau for Economic Growth, Education, and the Environment) AID-OAA-C-11-00169 Mid-Term Evaluation of Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Evaluation Interview Guide – JAA Regional Coordinators Interview Best Practices The protocol for the interviews will start with a brief introduction of the evaluation team and the purpose of the interview. The interviewer will then ask whether the interviewees have any questions before proceeding with the interview questions. Respondent Role JAA regional coordinators work on behalf of JAA to provide support to the program and to participating schools. Most coordinators are part-time employees who oversee a given marz in which they select schools participating in the JAA program, conduct site visits, administer and/or collect JAA surveys, and play other administrative support roles. They are knowledgeable about the JAA program’s workings, success stories and lessons from their region, and processes in place to ensure the program runs smoothly. Interview Questions Introductory Questions 1. What are your role/responsibilities within the JAA program? a. Do you play a role in developing the JAA program’s curriculum? b. Do you play a role in collecting data for JAA main office? 2. How long have you been involved with JAA? 3. Since JAA told us that coordinators are part-time positions, may we ask you what other position(s) you hold? 4. Do you think that the JAA program bridges any gaps that exist in the regular, state￾sponsored school curriculum? Conditions for Implementation 5. What kind of Monitoring & Evaluation (M&E), if any, do you implement on behalf of JAA’s main office? 6. In your experience, what are the most common and/or significant limiting factors of the program? Level of Ambition of JAA Program in Armenian Context 7. Did you notice a change in students’ mind-sets toward: a. Entrepreneurship? 88 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project b. Education? c. Their community and their role within it? 8. How do you anticipate the JAA activities will change civic behavior in the students? a. [F] What are the current barriers to changing behavior? 9. Do you anticipate the JAA activities will improve the rate of entrepreneurship in Armenia? a. [F] What are the current barriers? Were the Project Goals and Associated Activities Achieved, on Time? 10. What makes a “successful” student business project? 11. What makes a “successful” student NGO project? 12. When do you call a project a “success story”? How Does the Program Compare to Similar Youth Programs in the Area? 13. Are you aware of other youth programs implemented by governmental or nongovernmental organizations that have objectives comparable to those of JAA’s program (youth development and entrepreneurship)… a. … in your region? b. … outside your region? c. [F-YES to either question] Briefly describe and name these other youth programs. d. [F-YES to either question] In your opinion, do these other youth programs have: i. effective approaches? ii. achieved targeted objectives? Is There Evidence to Confirm That the Results Are Likely to Be Sustainable beyond the Project? 14. Do you believe that the program could be continued if JAA funding were no longer available? a. [F-YES] How? 15. Are there resources in your region that could be used or leveraged to expand the JAA program to more students and/or more schools? 16. Do you have any suggestions about how the JAA program could be improved to maximize its effect on students, teachers, and the broader community? Closing Questions 17. Are there any other comments you would like to make? 18. Do you have any questions for us about this interview? 89 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Interview Guide for School Principals United States Agency for International Development Learning, Evaluation, and Analysis Project (Bureau for Economic Growth, Education, and the Environment) AID-OAA-C-11-00169 Mid-Term Evaluation of Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Evaluation Interview Guide—School Principals Interview Best Practices The protocol for the interviews will start with a brief introduction of the evaluation team and the purpose of the interview. The interviewer will then ask whether the interviewees have any questions before proceeding with the interview questions. Respondent Role For the JAA program to be adopted in schools, the principals must at a minimum provide their authorization, provide financial support, and be supportive of the program. Therefore, they play a key role in the acceptance and success of the program early in the school year. The principals may know the total cost (direct and indirect) cost of operating the program and have insight into how much funding (pecuniary and nonpecuniary) that JAA provides to the school, how much is incurred by the school and its staff, and whether the funds are sufficient to sustain the school’s participation. It is also expected that the principal will have an opinion of the JAA program on teachers and students in his or her school. Interview Questions Introductory Questions 1. How did your school become involved in the JAA program? 2. What is your involvement in the administration of the JAA program at your school, if any? Conditions for Successful Implementation 3. How did you select the teacher(s) for the JAA program? 4. How are students selected for the JAA program? What are the criteria? 5. In your experience, what factors constrained or enhanced the implementation or effectiveness of the JAA program? Alignment of the Program with Armenian Context 6. Does the JAA program bridge or diminish any gaps in the current education curricula? 7. Does the JAA program have any spillovers, positive and/or negative, on participating teachers, students, or the broader community? Explain. 90 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Level of Ambition of JAA Program in Armenian Context 8. What affect have the JAA teaching techniques had on student performance (including student engagement)? 9. Has the JAA program enhanced community engagement of participating students? 10. In your opinion, have participating teachers been affected by the JAA program’s training and participation? If yes, in what ways? 11. Do you anticipate (and/or have you observed) that the JAA program will affect students’ labor market outcomes (entry, persistence, wages/earnings)? Explain. Is There Evidence to Confirm That the Results Are Likely to Be Sustainable beyond the Project? 12. In your opinion, has the JAA program affected participating students’ economic, business, and/or civic mind-set? 13. [Previous participating school]: Has your school continued to provide the JAA program? If no, why? 14. Are you aware of any resources currently not utilized that could be leveraged to improve the JAA program at your school? 15. [Previous participating school]: Were the resources (pecuniary and nonpecuniary) provided by JAA sufficient to operate the program? 16. Describe the resources that the school provided (pecuniary and nonpecuniary) to operate the program? 17. Could the school sustain the JAA program if JAA’s support decreased or ceased? Closing Question 18. How could the JAA program and approach be improved? 91 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Interview Guide for School Teachers United States Agency for International Development Learning, Evaluation, and Analysis Project (Bureau for Economic Growth, Education, and the Environment) AID-OAA-C-11-00169 Mid-Term Evaluation of Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Evaluation Interview Guide—School Teachers Interview Best Practices The protocol for the interviews will start with a brief introduction of the evaluation team and the purpose of the interview. The interviewer will then ask whether the interviewees have any questions before proceeding with the interview questions. Respondent Role JAA teachers are trained by JAA to provide classes in Applied Economics and Civic Activism. Teachers are also responsible for guiding the students through major JAA activities, such as the student business companies and the NGO project. Teachers may have insight into student planning for major JAA activities and the level of involvement of business consultants in classrooms. Interview Questions Introductory Questions 1. How did you become involved in the JAA program activities (e.g., volunteered, were invited to participate, were required to participate)? 2. What is your core teacher training discipline (e.g., history, social sciences, life sciences, mathematics, librarian, other)? Conditions for Implementation 3. How would you rate the training you received from JAA? 4. Did the JAA training adequately prepare you for your role in the JAA program? 5. How would you modify the JAA training to improve teacher preparation and/or student outcomes? 6. Were there any topics not included in your JAA training that would have enhanced your effectiveness and improved student outcomes? a. [F-YES] If so, which one(s)? Level of Ambition of JAA Program in Armenian Context 7. In your opinion, how did the JAA program influence participating students’ aspirations? 8. Does the JAA program: Question Likert Scale Explain … improve students’ high school 1 2 3 4 92 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project academic performance? 5 … improve students’ university aspirations? 1 2 3 4 5 … affect students’ occupational or career choices? 1 2 3 4 5 … affect students’ labor market outcomes? 1 2 3 4 5 … provide important knowledge for doing business in Armenia? 1 2 3 4 5 … enhance students’ interest in entrepreneurship? 1 2 3 4 5 … improve students’ awareness of civic activism? 1 2 3 4 5 … improve students’ civic activism/engagement? 1 2 3 4 5 Likert scale: 1=strongly disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, 5=strongly agree Were the Project Goals and Associated Activities Achieved, on Time? Student businesses 9. What are your indicators for a successful business project? 10. Did the student businesses meet your expectations? Explain. [Schools having gone through a full program cycle] NGOs 11. What are your indicators for a successful NGO project? 12. Did the NGO projects meet your expectations? Explain. How Did the Students Develop Their Project over the Course of the Program? 13. How did students select team members for the business project? a. Was this process significantly different for the NGO project? 14. Did the students (and/or you) talk with potential customers, community stakeholders, program alumni, or business mentors prior to implementation of the two projects? a. [F-YES] How did feedback from potential customers, community stakeholders, program alumni, and business mentors influence the students’ approach? Business Company 15. [Schools having participated more than 1 year] Was the business project a replication or follow-up from a previous year’s project? 16. Were there any peculiarities about the way in which the business company was implemented by JAA students? [Schools having gone through a full program cycle] NGOs 17. [Schools having participated more than 1 year] Was the NGO project a replication or follow-up from a previous year’s project? 93 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project 18. Were there any peculiarities about the way in which the NGO was implemented by JAA students? 19. What opportunities were/are there for additional funding for the NGO project? a. [F-Opportunity] Did the students receive additional funds for their NGO project? i. [F-NO] What were/are the main barriers to accessing additional funds? 20. What happened to the NGO project when the school year ended? What Did Students Consider When Choosing a Project? Business Company 21. What techniques or steps did the students undertake to reach out to community members, program alumni, and family members in selecting their business companies? 22. Could you describe the process and decisions made by students with regard to distributing funds as the student business wound down? [Schools having gone through a full program cycle] NGOs 23. What techniques or steps did the students undertake to reach out to community members, program alumni, and family members when selecting their business companies? What Else Could the Students Have Considered to Successfully Complete Their Projects? Business Company 24. Were the student businesses: Question Likert Scale Explain … well-planned? 1 2 3 4 5 … innovative? 1 2 3 4 5 … able to address community needs? 1 2 3 4 5 … utilizing knowledge from the classroom and after-school training? 1 2 3 4 5 … able to enhance students’ academic performance? 1 2 3 4 5 Likert scale: 1=strongly disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, 5=strongly agree [Schools having gone through a full program cycle] NGOs 25. Were the NGO projects: Question Likert Scale Explain … well-planned? 1 2 3 4 5 … innovative? 1 2 3 4 5 … able to address community needs? 1 2 3 4 5 94 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project … utilizing knowledge from the after-school training? 1 2 3 4 5 … able to enhance students’ 1 2 3 academic performance? 4 5 Likert scale: 1=strongly disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, 5=strongly agree How Does the Program Compare to Similar Youth Programs in the Area? 26. Are you aware of other youth programs implemented by governmental or nongovernmental organizations that have objectives comparable to those of JAA’s program… a. … in your region? b. … outside your region? c. [F-YES to either question] Briefly describe and name these other youth programs. d. [F-YES to either question] In your opinion, do these other youth programs have: i. effective approaches? ii. achieved targeted objectives? Is There Evidence to Confirm That the Results Are Likely to Be Sustainable beyond the Project? 27. If provided the opportunity, would you be able to train new economics teachers aligned with the JAA curriculum? 28. If provided the opportunity, will you continue to participate in the JAA program in your school? 29. Do you believe that the program could be continued if JAA funding were no longer available? a. [F-YES] How? 30. Have the JAA teaching techniques (including nonformal training techniques) been adopted or applied in other parts of the school curriculum or by teachers in your school who did not directly receive JAA training? 31. What effect, if any, have the JAA teaching techniques had on student performance (including student engagement)? 32. Has the JAA program, including the CSR component, enhanced community engagement of participating students? a. [F-YES] Could you provide specific examples? 33. Were the resources (pecuniary and nonpecuniary) provided by JAA sufficient to operate the program? a. [F-NO] What other resources (pecuniary and nonpecuniary), if any, did you provide or obtain from others to operate the program? 34. Describe the resources that you provided (pecuniary and nonpecuniary) to operate the program? Closing Question 35. Are there any other comments you would like to make? 95 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Focus Group Guide for School Students and Alumni United States Agency for International Development Learning, Evaluation, and Analysis Project (Bureau for Economic Growth, Education, and the Environment) AID-OAA-C-11-00169 Mid-Term Evaluation of Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Evaluation Focus Group Guide—School Students and Alumni Focus Group Best Practices To maintain the group dynamics, the interviewer will conduct focus groups consisting of 5 to 10 students, accompanied by a translator who will be interpreting in real time, taking notes, and asking follow-up questions, if necessary. In the introductory period, the interviewers will provide an overview of expectations for the focus group; thereafter, the interviewers will conduct the focus group, which is estimated to take 60 minutes to complete. Interview Questions Introductory Questions 1. Can someone provide an overview of what you did in the program? How did you first get involved? And what happened next? 2. Is that how it worked for everyone? Did anyone have a different experience? What was different? (In the mixed cohort groups, clarify whether differences are by year.) Students’ Business Company Projects 3. Can someone tell me about the business you started? 4. Was everyone involved in that project? 5. How did you divide the work? Why did you do it that way? 6. How did you decide on that business? 7. Did you have a lot of ideas based on your research? How else did you get ideas (e.g., talk to family member in that business, other classmates, the teacher, conduct interviews)? 8. Was it hard to find a good business idea? Why? What would have helped? 9. How did you narrow your ideas down and pick one? Did you talk to people? 10. If yes, whom did you talk to? 11. Did you receive critical feedback that prompted you to alter your approach? 12. How did you feel when you received critical feedback? 13. While selecting a business idea, did you use anything you learned in your regular classes (outside the JAA program)? a. [F-YES] Please specify? 14. What were the most important challenges you had/have in running your business (e.g., the legal aspects, working as a team, making the product, finding customers)? 15. Why was that a challenge? Did others have that challenge? 16. What did you do to resolve these challenges? 96 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project 17. Did you use your business plan? 18. Whom did you ask for advice? 19. How did you work it out? 20. What made [is making] your business project successful (e.g., certain people, research, etc.)? 21. Was there a necessity to alter your business plan? a. Why? 22. How did you evaluate whether changes in your business plan or approach were effective? 23. Were you able to accomplish what you wanted in the time you had? 24. How did you decide how much to pay yourselves for the work you did? 25. How did you decide how to distribute the profits of your business when it came time to wind it down? 26. Was there disagreement regarding how much to save for the NGO versus how much to pay to yourselves? a. [F-YES] How did you reach a decision? Students’ NGO Projects 27. Can someone tell me about the NGO you started? 28. Was everyone involved in that project? 29. How did you divide the work? Why did you do it that way? 30. How did you decide on that NGO? 31. Did you have a lot of ideas based on your research? How else did you get ideas (e.g., talk to a family member, other classmates, members of the community, the teacher)? 32. Was it hard to find a good idea for the NGO? Why? What would have helped? 33. How did you narrow your ideas down and pick one? Did you talk to people? a. If yes, whom did you talk to? 34. Did you receive critical feedback that prompted you to alter your approach? 35. How did you feel when you received critical feedback? 36. While selecting an idea for the NGO, did you use anything you learned in your regular classes (outside the JAA program)? a. [F-YES] Please specify. 37. What were the most important challenges you had/have in running your NGO (e.g., understanding the legal aspects, working as a team, finding community needs, implementing the project)? 38. Why was that a challenge? Did your peers have that challenge? 39. What did you do to resolve these challenges? 40. Did you use the plan you devised for the NGO? 41. Whom did you ask for advice? 42. How did you work it out? 43. What made [is making] your NGO project successful (e.g., certain people, research, etc.)? 44. Were you able to accomplish what you wanted in the time you had? a. [F-NO] What is happening (or has happened) to the project you started? b. [F-YES] Do you think what you created as part of the project will need future and/or ongoing maintenance/support? Why? c. [F-YES] Did you put in place a system to maintain the project? 97 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project d. [F-YES] Did you receive assistance or commitment from the community? 45. Was it necessary to alter your plan for the NGO? a. [F-YES] Why? 46. How did you evaluate whether changes in your plan or approach were effective? The Program’s Effect on Students 2013 Cohort 47. How are you expecting to benefit from participation in the JAA program? 48. Will you continue on to the NGO project? Why/Why not? 49. Does (or did) the JAA program: Question Likert Scale Explain … improve your high school academic performance? 1 2 3 4 5 …. improve your interest in attending university? 1 2 3 4 5 …. affect your occupational or career choices? 1 2 3 4 5 …. enhance your interest in entrepreneurship? 1 2 3 4 5 Likert scale: 1=strongly disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, 5=strongly agree 2012 Cohort and Mixed Cohorts 50. What advice would you give your classmates who join next year (e.g., about how to pick a business, how to find an NGO project, etc.)? 51. Does (or did) the JAA program: Question Likert Scale Explain … improve your high school academic performance? 1 2 3 4 5 …. improve your interest in attending university? 1 2 3 4 5 …. affect your occupational or career choices? 1 2 3 4 5 …. enhance your interest in entrepreneurship? 1 2 3 4 5 …. improve your awareness of civic activism? 1 2 3 4 5 …. improve your civic activism/engagement 1 2 3 4 5 Likert scale: 1=strongly disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, 5=strongly agree 2012 Cohort 52. Are there ways that the JAA program affects you now? [3.1.7] 53. Has the JAA classroom work helped you since graduation? How so? 98 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project 54. Have the business or NGO projects helped you since graduation? How so? 55. Has the alumni network helped you? How so? 56. What were the important lessons? What would you change? 57. Did the JAA program: Question Likert Scale Explain … improve your high school academic performance? 1 2 3 4 5 … improve your college academic performance? 1 2 3 4 5 … affect your occupational or career choices? 1 2 3 4 5 … affect your actual or anticipated labor market outcomes? 1 2 3 4 5 … provide important knowledge for doing business in Armenia? 1 2 3 4 5 … enhance your interest in entrepreneurship? 1 2 3 4 5 … improve your awareness of civic activism? 1 2 3 4 5 … improve your civic activism/engagement? 1 2 3 4 5 Likert scale: 1=strongly disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, 5=strongly agree Alumni 58. Given what you have heard about the NGO component, do you think it would have helped you if you had participated in this aspect of the JAA program? 99 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project o o o o o Interview Guide for Business Consultants United States Agency for International Development Learning, Evaluation, and Analysis Project (Bureau for Economic Growth, Education, and the Environment) AID-OAA-C-11-00169 Mid-Term Evaluation of Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Evaluation Interview Guide—Business Consultants Interview Best Practices The protocol for the interviews will start with a brief introduction of the evaluation team and the purpose of the interview. The interviewer will then ask whether the interviewees have any questions before proceeding with the interview questions. Respondent Role Business consultants act as mentors to students during the creation of student companies. JAA and specific businesses enter into a memorandum of understanding, which outlines expectations and roles of businesses within JAA programs. Business consultants may have insight into student efforts during the business’s creation and the readiness of JAA students for the Armenian workforce. Interview Questions Introductory Questions 1. How did you become involved in JAA program activities? 2. How are you currently involved in JAA program activities? 3. How long have you (and your company) been involved with the JAA program? 4. What was your motivation in getting involved in JAA program activities? 5. Do you support JAA program activities financially? [F-YES] How much have you donated since 2011? 6. Do you support JAA program activities through student mentoring? [F-YES] How frequently have you participated since 2011? Process through Which Students Identify and Implement Project Ideas 7. Are you aware of, or were you involved in, the process through which students identify business ideas? 8. Did you or your business play a role in: … helping the students select and draft a business plan? … providing critical feedback to the students? [F-YES to either question] Were the students receptive to feedback and guidance provided? Explain. 9. What techniques/tools/business strategies did you provide to students to help them create their business? 100 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project o o o o o Level of Ambition of JAA Program in Armenian Context 10. In your opinion, has the JAA program affected participating students’ economic, business, and/or civic mind-set? How? 11. Does the JAA program, including the corporate social responsibility component, enhance any gaps in the current education curricula? [F-YES] Which one(s) in particular? 12. Has the JAA program, including the corporate social responsibility component, enhanced community engagement of participating students? [F-YES] How? 13. In your view, are the skills and concepts taught within the JAA program enhancing the employability of participating students? 14. Are the skills and concepts taught within the JAA program valuable and applicable to starting and operating a business in Armenia? 15. How could the JAA program modify its approach to better prepare students for the Armenian workforce? 16. As you know, corporate social responsibility is a central component of the JAA training curriculum; do you believe that it has the potential to improve the entrepreneurial ecosystem and business environment in Armenia? [F-YES] How? 17. In your opinion, has this project promoted civic activism among participating students? [F-YES] How? 18. How do JAA students’ activities affect their respective communities? What Else Could Have Been Considered by the Students to Successfully Complete Their Projects? 19. Student business and NGO projects were: Question Likert Scale Explain … well planned. 1 2 3 4 5 … innovative. 1 2 3 4 5 … responsive to feedback/guidance provided. 1 2 3 4 5 … feasible. 1 2 3 4 5 … potentially profitable. 1 2 3 4 5 … potentially socially impactful. 1 2 3 4 5 Likert scale: 1=strongly disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, 5=strongly agree 101 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project o Is There Evidence to Confirm That the Results Are Likely to Be Sustainable beyond the Project? 20. In your opinion, has the JAA program transformed participating students with regard to inspiring a business-oriented mind-set? 21. Are you satisfied with the JAA program and how it is implemented in your area? 22. Based on your experience, do you plan to continue providing support to students involved in the JAA program? [F-YES] Will your degree of involvement in the JAA program change? If so, how? 23. Are you aware of any resources currently not utilized that could help expand the JAA program? 24. Are you aware of any resources currently not utilized that could help sustain the JAA program? Concluding Question 25. What differentiates JAA students from otherwise-similar students who have not participated in JAA? 102 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Appendix 11: Template of Introductory Letter to JAA Program School Administrators Friday, November 1, 2013 Dear Sir/Madam, Optimal Solutions Group, LLC, USA (Optimal), has selected your school to participate in the evaluation of Entrepreneurship and Civic Activism for Young People, a program implemented by Junior Achievement Armenia (JAA) and partially sponsored by the United States Agency for International Development (USAID). USAID has contracted Optimal to evaluate this program, which is designed to arm the young people of Armenia with entrepreneurial skills so they are prepared for the workforce of tomorrow, teach them to understand and appreciate work ethics and corporate social responsibilities, and empower them with skills and notions that can be leveraged to undertake projects benefiting their communities. The program not only is observed within an applied economics class but also encompasses a hands-on component: the creation of a student-run business. Profits from this business are then directed toward the creation of a student-run community project that aims at addressing a range of community needs. Given your school’s recent participation in this program, Optimal has selected your school to participate in a series of short interviews with the school’s headmaster, the teacher(s) who implemented the program, and the students currently enrolled in the program. If your school participated in the program in past years, then Optimal’s evaluation team would ideally like to conduct an interview with these students as well. These interviews will aid Optimal in evaluating various aspects of the program, including how it is implemented and the types of student activities undertaken within the student business company and community-project components. Optimal plans to visit your school between [START DATE] and [END DATE]. Interviews with the headmaster, teacher(s), and students are estimated to last no more than 60 minutes each. Additionally, Optimal plans to interview the currently participating students as one group and, if applicable, students who participated in previous years in another focus group. To move forward with these interviews, Optimal will contact your school by telephone in the coming weeks to identify a day and time that is convenient for its visit. Please note that participation in this evaluation is voluntary; however, your participation will help USAID learn whether the program is meeting its objectives and the extent to which stakeholders like yourself are satisfied with the program’s implementation to date. A consent form for participation is enclosed that explains the purpose, use, and confidentiality of all information collected for this study. If you do not want to participate, then please indicate your position and return it in writing or state it when contacted by telephone. Optimal greatly appreciates your collaboration and looks forward your participation in this important evaluation. If you have any questions about the study, please contact [OPTIMAL PERSON OR CONSULTANT], the study’s [TITLE] at [TEL #] or [EMAIL]. Thank you in advance for your help with this important study. Sincerely, Dr. Mark Turner, LEAP Chief of Party and Evaluation Team Leader 103 USAID Armenia – Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Project Signature Date Printed Name Consent form approved by [OPTIMAL PERSONEL] on [DATE]. Contact: E-mail: [EMAIL ADDRESS] Confidentiality and Usage Storage and Access Appendix 12: Interview and Focus Group Participation Consent Form USAID Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People Program Informed Consent Informed Consent: It is necessary to introduce the stakeholders to the interview and obtain their consent to participate. Make it clear to them that their participation in the interview is voluntary. Thank you for the opportunity to speak with you. We are a research team from Optimal Solutions Group, LLC, USA (Optimal), which has been contracted by the United States Agency for International Development (USAID) to evaluate the Entrepreneurship and Civic Activism for Young People, a program implemented by Junior Achievement Armenia (JAA). As part of the Entrepreneurship and Civic Activism for Young People evaluation, the research team will be conducting interviews with program stakeholders. You have been selected to participate in an interview that includes questions on such topics as your understanding of the program, involvement in program activities, and opinion regarding the effects of the program on the students, schools, community, and republic. The interview includes questions about the program generally and questions about your individual participation in the program. It should take about 60 minutes to complete, and your participation is entirely voluntary. Confidentiality and Usage If you agree to participate, you can choose to stop at any time or skip any question you do not want to answer. During the interview, the evaluation team will be recording and taking written notes of the conversation. The primary purpose of these recordings and notes is to allow the evaluation team to accurately document and report findings. Any information used in the development of the report to USAID will be attributed to the original source, unless that source specifically asks to be taken “offtherecord.” When comments are made “offtherecord,” the answers will be completely confidential, and the Optimal evaluation team will not disclose any information that identifies you. Storage and Access The interview recording and notes will be stored in a password-protected file on a secure server that is accessible only by the evaluation team. The interview recordings and notes will not be provided to USAID staff. Do you have any questions about the interview or what I have said? If in the future you have any questions, concerns, or comments regarding this interview, I welcome you to contact Mark Turner, Chief of Party, Optimal Solutions Group, LLC, at Armenia-JAA@optimalsolutionsgroup.com. I will leave one copy of this form with you for your records. Your signature on this form grants the evaluation team permission to conduct the interview as described above for its study. The evaluation team will not use the interview notes for any reason other than those stated in this consent form without your permission. Do you agree to consent to participation in the full study? Yes No Name: _________________________ Signature: _________________________ I, _________________________, the enumerator, responsible for the interview taking place on __________, 2013, certify that I have read the above statement to the participant. If the participant has consented to the interview, I pledge to conduct this interview as indicated in the instructions. Consent form approved by Mark Turner, Chief of Party, on November 1, 2013. 104