TEEM: Training Entrepreneurs for Excellence at MEET Cooperative Agreement No. AID-294-A-15-00011 (September 24, 2015-September 24, 2017) Final Evaluation Report (Cohort II focus) Submitted by: Rakheli Hever, Compliance Officer Partially based on surveys created by: Roni Kipnis, Information and Evaluation Coordinator To: Nidal Abed El Gafer, USAID AOR Submission date: October 31, 2017 DISCLAIMER The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 2 of 24 TABLE OF CONTENTS 1. Executive Summary ……….……………………………………………………………………………….……..………….……..…2 2. Program Introduction….…………………………………………………………………..……………………..……………………3 3. Evaluation Methodology……..……………………………………………………………………………………….……….……..5 3.1 Evaluation Foci……………………………………………………………………………………..……………………..….5 3.2 Data Sources……………..……………………………………….………………………..…………………….…………..5 3.3 Evaluation Participants……………………………………….………………………….……….………….…………..6 3.4 Data Analysis………………………………………………………………………………………….……………………….7 4. Cohort II Findings………………………………….………………………………………….………………….……………………….8 4.1 Computer Science Skills and Knowledge..………………………………….………….……………………..….8 4.2 Entrepreneurship Skills and Knowledge...…........................................................................9 4.3 “Deeper Understanding”……………………….…………………………………………..……….…………………10 4.4 Program Staff, Structure and Additional Characteristics………………………………..……………….11 4.5 Comparison with Equivalent Cohort I FY16 Results (from Interim Report)……..………………13 5. Cohort I in FY17 ………………………….………………………..………………………………………………..………………..…17 6. Cohort III in FY17………………………………………….……..…….…………………………………………..………………..…19 7. Discussion and Recommendations………………………….……………………………………………..………………..…21 Appendix I: TEEM Project Performance Indicator Table – Targets vs. Actuals………………………(attached) 1. EXECUTIVE SUMMARY This final evaluation report focuses on the outcomes of the second year for the second cohort of the TEEM program (Section 2), whose primary objective is to develop Israeli and Palestinian students’ computer sciences (CS) and entrepreneurship knowledge and skills, while simultaneously cultivating shared values and fostering a deeper understanding of the “other”. Using quantitative and qualitative methods (Section 3), the report builds on online survey results and additional documents to assess the three main aspects of the program and its results, namely CS knowledge and skills (Section 4.1), entrepreneurship knowledge and skills (Section 4.2), and “deeper understanding” of the “other” (Section 4.3), covering both the main indicators outlined in the program’s Monitoring and Evaluation Plan, and covering additional issues related to program staff, structure and general programmatic lessons (Section 4.4). The results are compared and contrasted with the equivalent outcomes for Cohort I in the first year of implementation, derived from the Interim Evaluation Report (Section 4.5). Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 3 of 24 The report also briefly highlights and evaluates the outcomes of Cohort I and Cohort III in FY17 (Sections 5 and 6, respectively), for whom additional activities were added to the original TEEM Implementation Plan with a special, approved revision. The report concludes (Section 7) that the program generally achieved its stated goals regarding Cohort II of TEEM and the program overall, for the most part meeting and sometimes exceeding the indicator targets set in its Monitoring and Evaluation Plan. In the second year of the project, in light of the findings of the Interim Evaluation Report, effort has been dedicated to reducing CS knowledge gaps (via a revised curriculum and the offering of “open sessions” providing support to those needing it). This was reflected in greater achievement (higher rated projects and higher self-ratings) and gender gaps narrowed down in some CS-focused survey questions. However, other CS-related survey items still demonstrate lower results for Israeli girls, and further attention should be paid to gender and nationality differences regarding these and all evaluation indicators. As with CS gaps, there was also improvement regarding the previous report’s findings on TAs, as the need for more TAs wasn’t prevalent in FY17 as it was in FY16. Summer TAs had special training that received positive reviews, and it is recommended to continue along the same lines with future Yearlong Program and Summer Program modules (while also attempting to make the TA cadre as diverse as possible in terms of gender and nationality). An accelerator program for the graduating cohort of students, launched in FY17 with TEEM Cohort I, proved to be successful, and it is recommended to continue to implement it in future years with adjustments and improvements as fitting. Finally, MEET launched a new, integrative “Deeper Understanding” curriculum to address in a gradual and realistic way issues pertaining to the Conflict and to joint bi-national work. Further research in the coming years will be needed to evaluate the implementation and impact of this curriculum. 2. PROGRAM INTRODUCTION “TEEM: Training Entrepreneurs for Excellence at MEET” (TEEM) is a two-year program of the Israeli non-profit “MEET: The Middle East Entrepreneurs of Tomorrow”, in partnership with USAID. The program’s overarching goal was to better position the next generation of Israeli and Palestinian leaders to bring about positive social and political change in the region. The program’s primary objective was originally to develop 150 excelling Israeli and Palestinian students’ (age 15-16) computer sciences (CS) and entrepreneurship knowledge and skills, while simultaneously cultivating shared values and fostering a deeper understanding of the “other”. Additional activities for the original two cohorts as well as for a third cohort of 80 students were added with approved revisions of the Implementation Plan. TEEM’s original main anticipated results(for the project’s entire 2 years) were as follows: 1) increased capacity (knowledge and skills) of a networked cadre of some 150 excelling Israeli and Palestinian youth to create joint bi-national CS and entrepreneurship projects for the benefit of their communities (involving increased CS and entrepreneurship knowledge and skills as well as improved perceptions about the other and about joint work for common goals); 2) increased exposure among program participants’ family members and among stakeholders (e.g., entrepreneurs, hi-tech company representatives) to program activities, values and products (e.g., successful examples of bi￾national CS/entrepreneurship projects and presentations). Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 4 of 24 The previous, interim evaluation report, focused on the first cohort to take part in the program, which commenced its activities on October 24, 2015 and concluded them according to the original Implementation Plan on August 9, 2016. These activities included: a) an Afterschool Yearlong Program comprising of 24 weekly sessions per group focused primarily on CS and entrepreneurship, as well as several special events and conferences, and b) an intensive 15-day Summer Program held at the Israel Arts and Science Academy with volunteer instructors from MIT, focused on CS, entrepreneurship, and “Deeper understanding” sessions, in addition to special outreach events and complementary activities. The main focus of this report is on the second TEEM cohort’s activities in FY17, the second and final year of the program. These included similar, yet improved programming: A. An Afterschool Yearlong Program comprised of: - An opening event in October 2016, where Nazareth and Jerusalem students met each other and the program staff and learned about the schedule, contents and requirements of the afterschool year-long program. - Two semesters of an afterschool yearlong program involving 3-hours afterschool sessions held between October 2016 and mid-May 2017 at the Jerusalem and Nazareth hubs, focused primarily on CS (Python, Terminal, GitHub, Object Oriented Programming) and entrepreneurship (Venture Capital, ideation, customer discovery, use cases, analyzing competition, going to market, creating good presentations, etc.). - Two overnight MEET Conferences as part of the yearlong program component (in December 2016 and April 2017), where sessions were dedicated to “Deeper Understanding” elements, project work and programming. - A visit to the Interdisciplinary Center (IDC) in Herzliya in May 2017, where TEEM students presented their entrepreneurship projects focused on improving education to the staff and students of the Zell Entrepreneurship Program. B. An intensive 17-day Summer Program held at the Israel Arts and Science Academy with volunteer instructors from MIT, including: - A total of 113.5 academic hours of CS (61 hours), entrepreneurship (22 hours) and “Deeper Understanding” (30.5 hours) sessions, in addition to social and fun activities and excursions. - Work in bi-national groups on CS and entrepreneurship projects focused on “reimagining communication and community”, of which the former were presented to a panel of hi-tech judges at Samsung NEXT, and the latter were presented to a panel of entrepreneurs in Jerusalem as part of a “Lean Canvas Fair” event. - A final event with parents and other family members, showcasing the summer program and the projects created by the students. In this second year of the program, the scope of activities was expanded, so that in addition to the equivalent Yearlong Program and Summer Program attended this time by the second Cohort of TEEM, a special accelerator program (also comprised of Yearlong and Summer activities) was held for Cohort I of TEEM (see Section 5), and a special Summer Program was held for a new cohort of 82 students (see Section 6). The purpose of this report is to assessthe program outcomes among the second cohort of TEEM (e.g., increased skills and knowledge, changes in perceptions and attitudes), comparing them with those Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 5 of 24 of the first cohort in FY16, and highlighting lessons learnt and suggestions for improvement for future programming. 3. EVALUATION METHODOLOGY 3.1 Evaluation Foci This final evaluation report will focus primarily on the three main aspects of the program and its results among the second cohort, namely CS knowledge and skills (Section 4.1), entrepreneurship knowledge and skills (Section 4.2), and “deeper understanding” of the “other” (Section 4.3), covering both the main indicators outlined in the program’s Monitoring and Evaluation Program and additional related issues. Special attention will be paid to gender and nationality differences, as relevant. Moreover, the report will touch on questions related to program staff, structure and general programmatic lessons (Section 4.4), and will compare FY17/Cohort II results with the equivalent ones of FY16/Cohort I (Section 4.5). Brief sections will assess the activities and outcomes for the two additional cohorts in FY17, also highlighting aspects related to CS and entrepreneurship knowledge and skills, as well as “deeper understanding” aspects (Sections 5 and 6). 3.2 Data Sources The evaluation uses multiple qualitative and quantitative data sources, derived from both internal records and from specific data gathering activities, including: A. Online surveys administered anonymously via GoogleForms, which included a mix of closed￾ended questions, statements followed by 1-5 Likert-scale agreement ratings, three PPR indicator statements followed by a choice between “agree” and “disagree” (see Figure 1 below), and a few open-ended questions (e.g., what would students change about the program, what specific things they’ve learned from an activity, etc.). In addition to demographic questions (stating nationality, gender, hub, and cohort), the surveys included questions probing for students’ estimations of their acquired CS and entrepreneurship knowledge and skills, and questions focused on team work, attitudes and perceptions towards the “other”, questions about specific program items, events and personnel were also included, primarily for internal MEET purposes. There was some variance in the questions depending on the cohort. The primary data source used for the Cohort II evaluation was the Year-End Survey administered in August 2017, on the final day of the Summer Program, filled out by 47 of the 58 students who completed the program (81.03%). Data from two additional surveys administered to the same cohort at earlier points in time (December 2016 and April 2017) were also used, with caveats (see Table 1 in the next sub-section). Sections 5 and 6, focusing on Cohort I and Cohort III, utilize data from the Year-End Survey administered to those cohorts, also in August 2017 (See Table 2, further below). Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 6 of 24 Figure 1: Samples of questions and answers from survey administered to TEEM Cohort II in Aug. 2017 B. Focus group with Summer instructors – Held with the overseas volunteer instructors who taught each of the cohorts (three groups, by cohort), and led by MEET’s CEO and the Program Manager. The sessions eliciting instructors’ overall impressions of the program as well as responses to specific topics (e.g., was the required material covered, were the goals achieved, what can be improved, etc.). C. Additional documents – including event reports, protocols of staff meetings and debriefs, and data on student products, including online forms used by external stakeholders to evaluate project presentations. 3.3 Evaluation Participants Cohort II of TEEM began its FY17 activities in October 2016 with 74 students overall, ages 15-16, male and female, Israeli and Palestinian, hailing from various cities, towns and villages in the north of Israel, Jerusalem and the area, and the West Bank (Bethlehem and Ramallah). Of these students, 61 completed the Yearlong Program and 58 completed the Summer Program. The numbers (and breakdown) of students who filled out the relevant surveys for this cohort are presented in Table 1 below. Table 1: Cohort II survey respondent numbers and breakdown # of respondents from among Cohort II of TEEM December 2016 survey April 2017 survey August 2017 Year-End Survey Females: 27 263 264 Males: 20 16 21 Israelis: 141 183 195 Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 7 of 24 Palestinians (incl. Pal. Citizens of Israel): 33 24 28 Members of Jerusalem hub: 20 20 23 Members of Nazareth hub: 27 22 24 Total # of respondents: 47 42 47 Overall response rate (respondents/total participants): 95.92%2 64.62%3 81.03% 1 Fewer Israelis due to the higher dropout among them and the conference being held during the Hannukah holiday. 2 Based on 47/49 (95.92%) who attended the Winter MEET Conference, which was the focus of the survey. However, these were just 68.06% of the 72 students who were in the program at the time. 3 The relatively low response rates (due to only 2/3 of students attending the activity at which the survey was administered), are problematic, as the survey relates to the overall Yearlong Program. Moreover, only four Israeli boys filled out the survey (compared with 14 Israeli girls), rendering gender/nationality comparisons pointless. 4 Despite the larger number of female, their response rate is lower than that of the males (there are fewer males, due to higher dropout among Israeli males in particular). 5 Dropout rates among Israeli students are higher, and hence the Israeli respondents are fewer. As mentioned above, FY17 also included activities for two additional cohorts, Cohort I of TEEM, on average a year older than those of Cohort II, and Cohort III, on average a year younger than Cohort II. Students of both these cohorts filled out the Year-End Surveys, which were taken into account in the assessment of their FY17 activities. Their numbers and breakdown per gender, nationality and hub are outlined in Table 2 below. # of Year-End Survey respondents TEEM Cohort I TEEM Cohort III Females: 18 39 Males: 23 39 Israelis: 16 41 Palestinians (incl. Pal. Citizens of Israel): 25* 37 Members of Jerusalem hub: 18 40 Members of Nazareth hub: 23 39 Total # of respondents: 41 78 Overall response rate (respondents/total participants): 78.85% 95.12% * Dropout rates among Israeli students are higher, and hence the Israeli respondents are fewer. 3.4 Data Analysis Data from the multiple sources outlined in Section 3.2 were separately analyzed using qualitative and quantitative methods, respectively. Specifically, we utilized thematic analysis for focus group discussions, documents and to a lesser extent the responses to open survey questions, while 5-point Likert-scale survey questions data were collapsed to agreement/disagreement/neutrality percentages (responses of “agree strongly” or “agree somewhat” were counted as agreement, likewise for “disagree strongly” and “disagree somewhat”, counted as disagreement). A frequency count was used for questions where answers were selected from a qualitative category and/or for recurring answers in open-ended questionnaire items. When possible and relevant, data were further Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 8 of 24 disaggregated by gender, nationality and hub, and large differences are noted and discussed below. Data were then triangulated according to each evaluation focus, with greater emphasis placed on the post-Summer Year-End Survey results over those of the Post-Yearlong Survey results (due to their larger number of respondents, the later date on which they were administered, and their inclusion of end-of-year questions). 4. COHORT II FINDINGS 4.1 Computer Science Skills and Knowledge During the Yearlong Program, the focus of the CS curriculum for this cohort was on “applying object￾oriented programming fundamentals toward each of them creating individual Python programming projects in the form of a complex, modular software system: a chat engine”. The curriculum also “introduced them to thinking about user interfaces, […and] front-end fundamentals - HTML and CSS - toward the construction of web applications.” To help address gaps in CS knowledge among participants (especially, but not only along gender lines), optional “open sessions” were offered. It should be noted that most students attended these “open sessions” at least once, and they were utilized dramatically more by girls compared to boys and by Palestinians compared to Israelis. The Summer Program curriculum augmented this with a focus on building web apps, “which are websites with functionality like collecting and storing user data and changing content. This involved both frontend (HTML and CSS) and backend (functionality hidden from the user) development.” (quote from instructors). The survey administered in December 2016, after the Winter Conference where students engaged in coding apps for Android devices, had 78.72% of respondents reporting that they became “more motivated to study CS”, with the increase in motivation primarily taking place among girls (88.89% compared to 65% of the boys). A somewhat smaller percentage reported that they understand the CS material well (70.21%), with no gender differences (however, Israelis reported lower understanding than Palestinians, and likewise Jerusalemites vs. Nazarenes). The survey administered in April 2017 (and filled out by only 64.62% of participants) showed a grimmer picture, with only 45.24% reporting that they understood the material well, and only 66.67% asserting that they look forward to learning more CS. This was despite the fact that 76.19% noted that their CS instructor explained lectures and topics in a way that was easy to understand and was easy to talk to when they had questions or group problems. In a MEET staff meeting discussing these findings, some possible reasons were mentioned, including the higher level of the curriculum this year, students’ undervaluation of their own coding abilities, and the stress and difficulties involved in creating their CS projects. Only a small number of students provided any explanation in response to a relevant survey questions, including: “We didn’t have enough time for the personal project and it was very hard to come to the open sessions”; “It's hard for me to remember all the new material”; “I think we should move slower and give each lesson more than one session”. A couple of students raised the need for more TAs and more time to learn and to work on their projects. Staff speculated that students perhaps did not feel comfortable enough approaching TAs from the other nationality and gender, and that possibly the level of English in the CS lectures was too high for them. The Year-End Survey results were much better: 91.49% of the respondents indicated that the program helped them gain practical experience and increased their CS knowledge, understanding Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 9 of 24 and skills (no gender difference, and higher among Palestinians at 96.43% compared with 84.21% among Israelis). Most of them (74.47%) recognized that studying in MEET increased their confidence in CS, although this percentage was considerably lower among Israeli girls at 63.64%, compared to 87.50% among Israeli boys and an average of 75% among Palestinian boys and girls (gender/nationality interaction effect). A similar pattern was revealed regarding the survey item “I’m interested in learning more CS” (overall 70.21% of agreement), where there was a small difference between Palestinian girls and boys (76.92% vs. 66.67%) compared to a staggering difference between Israeli girls and boys (54.55% vs. 87.5%). Conversely, the Palestinian girls are those who feel less proud about the personal work they made on the group CS project (66.67%, compared to about 90% among Israelis and 76.92% among Palestinian boys). Regardless, the majority of students were proud both of their group CS project as a whole (80.85%) and the personal work they contributed towards it (78.72%). During the Yearlong Program, 82.43% of the students successfully completed individual projects, and 68.92% did group projects (mobile apps on the theme of movement), meant to exercise and test their coding skills. During the Summer Program, the students created 12 group CS projects, which they presented to a stakeholder judges at Samsung Next. “In addition to providing students with feedback on the spot, judges also rated the projects using an online Google form for the MEET staff, along four dimensions: a) Professionalism, Presentation & Speaking Skills; b) Technical Complexity / Technical Communication & Understanding; c) Thinking Big: Creativity and Ideation; and d) Embracing Teamwork. For each of these dimensions, a rating between 1 (poor) and 5 (excellent) was awarded by each judge. The average rating was well above average, at 3.785. The highest rated dimension was for teamwork (4.07), and the lowest rated dimension was technical, at 3.44. […] Several judges even volunteered to serve as mentors for the groups in the future” (quote from the relevant event report). 4.2 Entrepreneurship Skills and Knowledge Per program documents, the Yearlong Program’s entrepreneurship curriculum aimed to introduce students to Venture Capital, entrepreneurship and entrepreneurs, and to teach them the basics of ideation and preliminary brainstorming, customer discovery and presentation skills, use cases, getting to know the competition, going to market, creating good presentations, forming teams, ideation and working on entrepreneurship projects in bi-national groups. During the final Summer Program event, the MIT volunteer summer entrepreneurship instructors highlighted their triple focus on 1) design thinking (using the Stanford methodology, which emphasizes empathy), creating and testing prototypes; 2) the Lean Canvas, a concise and quick way to think about businesses; and 3) case studies of a "company of the day". Since the focus of the Winter MEET Conference was not on entrepreneurship, the December 2016 survey did not contain any relevant data. The Post-Yearlong Survey findings regarding the entrepreneurship aspect of the curriculum were very positive: 90.48% of the respondents understood the material very well, and 88.1% were looking forward to learning more (with somewhat higher satisfaction among the Nazareth hub students). Overall, the students were pleased with their entrepreneurship instructors, with 88.1% stating that their entrepreneurship instructor “explained lectures and topics in a way that was easy to understand,” and 83.33% that they were “easy to talk to” when students had questions or group problems (here, too, there was somewhat higher ratings among the Nazarene students). Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 10 of 24 Along similar lines, the Year-End Survey saw 85.11% of the respondents agreeing with the statement that MEET helped them gain practical experience and increased their entrepreneurship knowledge, understanding and skills (the average rating among Israeli girls was significantly lower than the general average, at 63.64%). During the Summer Program, the students demonstrated their increased capacity by creating 12 projects in bi-national, gender-mixed groups, which they presented at a special Lean Canvas Fair to stakeholder judges from the business and entrepreneurship sphere, as well as to their parents at a special final event. The judges rated the projects based on the idea and business plan, as well as the clearness and effectiveness of the presentation. In total, the maximum number of points each project could receive from the judges was 60. The average score of the 12 projects was 48.66 points (out of 60), amounting to 81.1%. The highest scores were awarded in the categories of clear and effective presentations (88.06%) and cogent responses to questions (83.7%), with the lowest average score being in the in the innovation category (76.94%). Accordingly, the students felt proud of their group projects (78.72%) and of their own personal work on them (70.21%), with somewhat lower ratings among boys (and especially Israeli boys). They were also overall proud of their group’s teamwork towards the projects (65.96%, with a gender/nationality interaction effect making Israeli girls the least pleased group). 4.3 Deeper Understanding Per program documents, the “Deeper Understanding” element of MEET’s programming aims to help students “better understand one another, MEET’s mission and MEET’s values: strive for excellence, treat everyone with respect and equality, think big, embrace teamwork, act with integrity and lead by example”. The majority of the related dedicated sessions took part during the Summer Program (which will therefore receive greater emphasis in this section), while during the Yearlong Program, they were mostly limited to special events like the MEET Conferences (though Jerusalem students’ sessions each opened with a social activity and Nazareth students’ had brief “Deeper Understanding” activities at the beginning of theirs). In the Post-Yearlong Survey, only 38.10% of the students reported that working with their peers from the "other" side improved their last project's results. However, 61.9% of them indicated that their time so far in MEET had changed the way they view those from the "other" nationality, and only 23.81% indicated that “working on a project is more difficult if you work with people from the other nationality.” The respondents to the Post-Yearlong Survey most frequently described the group work on projects as “challenging” (78.57%) and “difficult” (59.52%), yet “interesting” (47.62%). In their answers to the related open-ended question, none of them mentioned difficulty factors related to the bi-national or gender-mixed nature of the work groups, focusing instead on general interpersonal aspects and personal characteristics. One answer, for example, was that, “when you work in a group with new people, it's always challenging, but it's also fun.” Another example is the following answer: “I find working in groups to be challenging, especially when all the members want to be leaders...” When diversity aspects were mentioned, it was in a positive context: “I think the way we were divided was really good, in which we got to work with people from the other nationality, listen to their ideas and learn more about their lives and compare the differences between their daily lives with mine.” Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 11 of 24 The Year-End Survey revealed that the vast majority of respondents, through their participation in the program, have become better able to understand the perspective of people from the other nationality (89.36%), more positive about people from the other nationality (87.23%), and more willing to work with people from the other nationality on common issues (95.74%). The responses to the first question, about becoming better able to understand the other nationality’s perspective, showed a marked gender/nationality interaction effect, where Palestinian boys’ agreement percentage was only 61.54%, compared with 100% among the other gender/nationality division sub￾groups. For the statement related to a more positive attitude towards people from the other nationality, the pattern was different, with the Israeli girls having the lowest agreement rate (72.73%), considerably lower than that of the Israeli boys (100%), with their Palestinian counterparts’ agreement percentages somewhere in between (92.31% for boys and 86.67% for girls). Similarly, although the vast majority of Israeli girls were expressed their willingness to work with people from the other nationality on common issue (81.82%), their agreement rates were lower than that of the other sub-groups (100%). The Israeli girls' ratings may have been influenced by an incident that took place during one of the "Deeper Understanding" sessions at the Spring Conference (just prior to the time the survey was administered). At the time, three of the Israeli girls expressed their dissatisfaction with one of the activities in the session, which they felt was biased. In the same survey, the vast majority of students (87.23%) also reported that their time in MEET increased their “willingness, ability and motivation to cooperate and work on projects together with students from the other nationality.” This was more prominent among Palestinian students (92.86%) than among Israeli students (78.95%). The students also indicated that they feel part of a unified group (78.72% average agreement rate, with boys’ agreement somewhat lower than that of girls), demonstrating that they have come closer together despite nationality and gender differences. During the Summer Program, students studied and had “deeper understanding” sessions in sub groups of 15-20 students, known as the “small groups” (different than project work groups, which typically included fewer students). The vast majority of students asserted that they felt safe as part of those groups (89.29%, with Palestinian boys higher than the other groups at 100% agreement), and most of them also reported feeling trust and a close connection with their group peers (75%, with boys dramatically less so than girls). The good relations between Israeli and Palestinian students were also noted by the overseas volunteer summer instructors in the post-summer focus group. 4.4 Program Staff, Structure and Additional Characteristics Instructors and TAs - Overall, students’ approval rates of their instructors were fairly high (typically between 85%-90% for the Summer Program, with a wider variation in range at 75%-100% for the Yearlong Program) when responding to items such as “My instructor explained lectures and topics in a way that was easy to understand”, “My instructor was easy to talk to when I had questions or a group problem”, and “My instructor knows a lot of [CS/entrepreneurship] and can teach me even more”. Satisfaction with some of the TAs was lower in the case of the Yearlong Program, where ratings for CS TAs for helping students understand the material and their CS knowledge varied between 60%-67.57%., and ratings for the entrepreneurship TAs were somewhat lower (partially explained by their unavailability for some of the groups). The TAs of the Summer Program received Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 12 of 24 special training prior to the program (as a lesson learned from past evaluations), and were overall satisfied with it (100% agreement among those who responded to a special TA survey). In the words of one of the TAs: "We learned good methods to deal will all different kinds of situations, we got a better picture of MEET's current goals and set our own goals, and we also got to know each other before the summer started." Only a couple of students mentioned the need for more CS TAs and it wasn't mentioned by the instructors in their feedback. Yearlong Program “best” and “worst” aspects per students – The most commonly mentioned “best” feature of the Yearlong Program in the Post-Yearlong Survey was overwhelmingly the students themselves - whether referred to as the people, the other students, the friends made, etc. (57.14%). Other appreciated aspects included the instructors (26.19%), the gained knowledge, academic content and/or material (16.67%), the food (16.67%), the entrepreneurship sessions (16.67%), the fun/social activities (14.29%), and the "open sessions" (11.9%). When asked about the aspects they disliked about the program, students' most frequent answers included the following: not enough interaction/socializing between groups (different cohorts, hubs, or weekdays within the same hub (33.33%), the repetitive or boring nature of studies (30.95%), stress/not enough time (21.43%) and insufficient "Deeper Understanding" activities (21.43%). On an interesting note, the most prevalent complaint of the previous cohort from their first Yearlong Program, involving issues with CS (difficulty of studies, need for more help/TAs), was only mentioned by 4 students (less than 10%), and none of them mentioned the need for more TAs. This was possibly due to improvements introduced to the program this year. Fittingly, the most common suggestions for improvement of the Yearlong Program were to add more interactions between the other groups, especially between the hubs (35.71%), to engage in more or better "Deeper Understanding" activities (21.43%), and to add more creative/fun activities (11.9%). Summer Program “best” and “worst” aspects per students – As with the Yearlong Program, the most prominent "best" aspect mentioned by students about the program was the social, peer to peer interactions (having a unified cohort group, "the people", getting closer to other students, reuniting with friends, as mentioned by 34.04% of the students). Other positive aspects frequently mentioned include the instructors (as a whole, or specific ones, at 19.15%), the projects and how professional they felt (17.02%), the "Deeper Understanding" aspects of the curriculum (14.89%), and the interactions with students from other cohorts, mostly the younger ones (12.76%). Students' most common suggestions for improvement included extending their nightly curfew (36.17%), having more/better "Deeper Understanding" activities (29.78%) and fun/social activities (21.28%). Moreover, some requested better rooms (air conditioning, cleanliness, 19.15%), better food (17.02%, a significant improvement from last year, where food was the primary complaint), and more free time (12.77%). Overall, however, the clearest indicator of students' satisfaction, their willingness to recommend the program to a friend or family member, was overwhelmingly positive at an average of 89.36% (lowest percentages among Israeli girls at 72.73%, compared with 100% among Palestinian girls). Feedback from the overseas summer instructors highlighted the need for: more extensive alignment of expectations and contact between the instructors and the local staff (including re scheduling and timetables), coordinating more extensively the academic aspects of the Yearlong Program and Summer Program, adding more time for CS in the program, addressing the passivity and lack of ability to receive constructive professional feedback among some of the students early in the program, and Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 13 of 24 providing them with more detailed and concrete academic assessment, as well as local examples of successful enterprises (even on a small scale). 4.5 Comparison with Equivalent Cohort I FY16 Results (from Interim Report) Several changes/improvements were introduced to MEET's curriculum that led to differences in curriculum between TEEM Cohort I in FY16 (the focus of the Interim Evaluation Report) and Cohort II in FY17 (the focus of this report). These primarily included changes to the CS curriculum during the Yearlong Program, such as making it more systematic, tracking students' individual progress, a greater highlight on programming for apps, a personal project of creating a chat interface rather than a game, efforts at gender mainstreaming, and the option for students to receive help via voluntary "open sessions". MEET's "Deeper Understanding" curriculum also underwent significant revision, which took effect primarily during the Summer Program. With this in mind, this section will attempt to tentatively compare the results (outputs and outcomes) of the first and second TEEM cohorts, along the indicators outlined in the program's M&E Plan. Where output indicators span more than one cohort, the results for the specific cohort are isolated (the total for all cohorts can be seen in the Performance Indicators’ Table in Appendix I). # Indicator name / description Cohort I FY16 result Cohort II FY17 result Comments and explanations 1 No. of events 9 10 The overall numbers reported under the PPR indicator are higher as they include additional cohorts. 2 No. of trainees 52, of which 28 male and 24 female 58, of which 24 male and 34 female. This is despite the fact that equal numbers of males and females recruited for the program, dropout among Israeli males was high, primarily due to a competing program focused solely on CS. This also led to an imbalance in numbers among Israelis and Palestinians.) The overall numbers reported under the PPR indicator are higher as they include additional cohorts. In the recruitment cycle of Cohort II, MEET increased the capacity of its Nazareth hub by 25%, adding 10 more students than in the previous year, explaining the higher number. Note that 70 students were originally recruited for Cohort I and 80 for Cohort II. The number of trainees here reflects only those who completed the requisite percentage of Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 14 of 24 attendance throughout the year. 3 Standard perceptions indicator 88.98% (90.28% among males and 86.11% among females) * An average of: 1) become better able to understand the perspective of people from the other nationality (84.31%); 2) more positive about people from the other nationality (86.54%); 3) and more willing to work with people from the other nationality on common issues (96.08%). The responses to the first question, about becoming better able to understand the other nationality’s perspective, showed rather large gender and nationality differences, with boys agreeing more than girls (91.67% compared with 75%), and Israelis agreeing less than Palestinians (73.68% compared with 90.63%). 90.78% (90.48% among males and 91.03% among females) * An average of: 1) become better able to understand the perspective of people from the other nationality (89.36%); 2) more positive about people from the other nationality (87.23%); 3) and more willing to work with people from the other nationality on common issues (95.74%). The responses to the first question, about becoming better able to understand the other nationality’s perspective, showed a marked gender/nationality interaction effect, where Palestinian boys’ agreement percentage was only 61.54%, compared with 100% among the other gender/nationality division sub-groups. No real changes in overall average, but slight narrowing down of the gender differences for this average. Nevertheless, note the complex pattern of gender/nationality effects in responses to each of the specific survey items comprising this average. 4 % of program participants reporting increased capacity to act for positive change N/A N/A Abandoned per advice of external evaluator, because the term "positive change" is very broad and somewhat vague, which would increase the likelihood of student responses being affected by their wish to present themselves positively ("social desirability"). 5 # of bi-national CS and/or entrepreneurship summer projects 24 (12 entrepreneurship and 12 CS projects). 24 (12 entrepreneurship and 12 CS projects). The overall numbers reported under the PPR indicator are higher as Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 15 of 24 that will be developed by teams of Israeli and Palestinian (including Palestinian citizens of Israel) students and presented to stakeholders and family members. * The average project rating (out of 5) by stakeholders was 3.65 for the CS projects and 3.8 for the entrepreneurship projects. * The average project rating (out of 5) by stakeholders was 3.875 for the CS projects and 4.055 for the entrepreneurship projects (converted from 48.66/60). they include additional cohorts. Note that Cohort II projects received higher ratings from stakeholders. 6 # of parents and/or family members attending summer outreach events 119 120 The overall numbers reported under the PPR indicator are higher as they include additional cohorts. 7 # of stakeholders attending outreach events, or exposed to participants’ presentations through special events and excursions. 42 43 The overall numbers reported under the PPR indicator are higher as they include additional cohorts. 8 % of targeted participants who report improved CS knowledge and skills 69.23% (% of students who agreed in the Year-End Survey that they understood the CS material very well”) * Only half of the Israeli females in the program thought that they understood the CS material very well, compared with 100% of the Israeli males (and with 61.54% and 60% among Palestinian females and males, respectively). 91.49% (% of students who agreed in the Year-End Survey that the program helped them gain practical experience and increased their CS knowledge, understanding and skills) * No gender differences, somewhat higher percentages among Palestinians compared to Israelis. Higher results for Cohort II, yet note Year-End Survey questions were not identical for both cohorts. However, despite lower result for Cohort II in the Post-Yearlong Survey, it seems their CS learning was better overall as evinced by the Year-End Survey, possibly due to the contribution of the open sessions both during the Yearlong and the Summer Program. Perhaps most importantly, gender differences in CS were narrowed down. 9 % of targeted participants who demonstrate 83.82% 63.51% The structure of Yearlong Program in FY17 was different. Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 16 of 24 improved CS knowledge and skills via completing individual projects using Python and group projects creating small functional computer games. * Cohort I trainees who completed individual CS projects with a grade 6 or above + did group projects during their first Yearlong Program, out of the number of those who began the Yearlong Program * Cohort II trainees who completed individual CS projects (chat interface project) + did group projects during the Winter Conference in their first Yearlong Program, out of the number of those who began the Yearlong Program Rather than doing advanced group CS projects in the second semester as Cohort I did in FY16, Cohort II did their group CS projects during the Winter Conference (end of first semester). About a third of the students could not attend that conference (abroad for Hannukah/Christmas holidays), lowering the total percentage for FY17 (despite achievement as demonstrated in creating the highly rated Summer Program group CS projects, see under Indicator 5). 10 % of targeted participants who report improved entrepreneurship knowledge and skills 96.15% 85.11% A decrease from FY16 results, though it should be noted that those were well above the original target of 80%, possibly largely due to a popular instructor in FY16 who was not part of the FY17 staff. 11 % of targeted students who demonstrate acquired entrepreneurship skills and knowledge via successful presentations at outreach event 77.94% 71.62% The numbers reflect the percentages of the trainees who attended the relevant Zell presentations event during their Yearlong Program. The decrease mean relatively fewer students were able to attend that event in FY17 (which was held at a later date than originally planned due to the late start of the schoolyear at the IDC). Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 17 of 24 12 % of targeted participants who report increased willingness to work together, increased belief in the importance and impact of bi￾national cooperation, and other relevant self￾report measures not covered by the standard custom indicator 1.6.2-01. 72.44% (Average of three existing MEET survey items: 1) Working with people from a different culture improve my last project’s result; 2) I believe that working with people from the other nationality will help my professional skills; 3) I believe that working with people from the other nationality is important). 87.23% (Response to new composite survey item: “MEET increased my willingness, ability and motivation to cooperate and work on projects together with students from the other nationality”.) There is a marked increase between FY16 and FY17, possibly due to the new “Deeper Understanding” curriculum. However, it should be noted that the survey items weren’t identical. 5. COHORT I IN FY17 With an approved revision of the Implementation Plan, Cohort I of TEEM additionally benefitted from two advanced programming modules in FY17, which piloted an accelerator program devised by curriculum expert and MIT graduate Lorenzo Brown (in addition to "open sessions" offered for those requiring additional support). This was done with the aim of helping them launch their own "startups" and create advanced project prototypes by the end of the 2017 Summer Program, for the first time in MEET's history. The Cohort I students prepared for this throughout the 2016-17 Yearlong Program, building up their knowledge of programming (especially full-stack web development), of entrepreneurship and of common pressing issues in the region, where all three elements were interwoven (e.g., through case studies). Students brainstormed in bi-national, mixed-gender groups throughout a process of project ideation and development, which resulted in 11 group project ideas. As part of the new piloted curriculum, four from among these projects were selected for further development before the end of that module. For this purpose, members of each project team prepared pitches and video "commercials", which were presented at two pitch deck events to stakeholder judges and/or featured in a special online campaign with feedback from the general public as well as additional stakeholders and MEET community members (over 3,000 people reached overall), in the form of comments and "virtual donations" (more "donations" = more successful project). Over the second of two MEET Conferences during this Yearlong Program, with the help of special guest mentor Catalin Voss, members of the original winning project teams worked on formulating their vision, solidifying project teams and roles integrating their peers from the other project groups, and making work plans for the summer. During the subsequent three-week Summer Program, students effectively formed startup teams dedicated to developing their projects' concepts and prototypes, contacting stakeholders for field research and outreach purposes, attending relevant workshops and sessions and soliciting advice and professional guidance from overseas volunteers instructors from prestigious academic institutions (such as MIT, Stanford, and Harvard). Three peak outreach events took place during the Summer, Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 18 of 24 including: 1) "Networking Night", where the students travelled to the Google Campus in Tel Aviv to make "elevator pitches" to stakeholders, showcasing their project ideas and prototypes/website, addressing questions and feedback; 2) "Series A Night" a week later, where students made professional "pitch deck" presentations of their updated projects to a select group of judges at the Malha Technological Park in Jerusalem; and, on the final day of the program, 3) the Graduation ceremony, presenting their projects, their personal path within MEET, and their vision of the future to an audience of family members and stakeholders. The following is a concise assessment of this pilot based primarily on a Year-End anonymous online survey administered to the students filled out by 41 students (78.85%), event reports, staff discussions, focus group of the volunteer overseas instructors for this cohort, and oral feedback from stakeholders who attended the "Series A Night". Triangulating data from all these sources, this pilot appears to have been a great success, and not only in the sense of the planned outputs and outcomes that were met. In terms of outreach, 22 professional stakeholders and 229 family members attended key outreach events, with over 3,000 engaged online as part of the online campaign (which took place for the first time in MEET's Student Program). Stakeholders attending presentations expressed their amazement at the young students' achievements over the summer, with comments such as: "This [Series A night] is one of the most impressive events I've ever been in, I can't believe it was only three weeks [of work on the projects]!" or "I didn't hear any bad ideas, like it normally happens at such sessions... I congratulate the teams for coming up with such great ideas!" They typically also gave specific feedback and advice per project team (e.g., "going business to client rather than business to business", "Your business model is amazing, but I don't think it should be a non-profit", "Do it as a webservice, it won't work as a mobile app"), pointing out possible challenges and improvements (e.g., a potential problem with users' motivations and privacy issues in one project, the need to convince business owners of the group's competitive advantage despite their young age in another), as well as strong points (e.g., "a seamless demo", "this is a huge need", "you can bring young audiences", "you can scale up and get investments ready pretty soon"). In terms of practical experience and increased CS knowledge and skills, the vast majority of students reported that their time at MEET helped them gain practical experience and increased their CS knowledge, understanding and skills (92.68%, including 100% of the girls). It increased their confidence in CS (78.05%, with Israeli girls bringing down the average at 50%), and has left them interested in learning more CS (73.13%, with a gender/nationality interaction effect where Israeli boys were the highest at 90% and Israeli girls the lowest at 50%). Likewise for entrepreneurship, a remarkable 100% of the students acknowledged that MEET helped them gain practical experience and increased their entrepreneurship knowledge, understanding and skills (likely due to the focus of this year for them - creating their own startups). In terms of attitudes, perceptions and motivations related to the other nationality and joint work, 85.37% of the students claimed that MEET increased their willingness, ability and motivation to cooperate and work on projects together with students from the other nationality (more so among Palestinians at 92% than among Israelis at 75%). Moreover, following the program, they are more positive about people from the other nationality (73.17%, with boys somewhat higher than girls), can now understand the perspective of people from the other nationality (82.93%, with Israeli girls the Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 19 of 24 lowest at 66.67% and Israeli boys the highest at 90%) and are willing to work with people from the other nationality on common issues (85.37%). The survey examined additional parameters related to the students’ feelings about their work process, their achievements and the program, which were not included in the program indicators. Possibly the most significant findings are that 92.68% of the students felt proud of their group projects (70% among Israeli boys and 100% among all others), 80.49% said they would like to continue working on their startup project after the summer (66.67% among Israeli girls), and an overwhelming 97.56% would recommend MEET to a close friend or family member. The most beneficial activities to the students' projects according to the Year-End Survey were the "Networking Night" (80.49%) and the "Series A Night" (58.54%), and these were also the most positive personal experiences (82.93% and 63.41%, respectively). In the open-ended part of the survey, the students highlighted their work on the projects, including self ownership and responsibility for the work, the real startup aspect of it, and the teamwork and role division as most frequently mentioned program highlight (46.34%), followed by social aspects (being with friends, feeling part of a group, being like a family, etc., at 19.51%). The most common suggestions for improvement were to add more social/fun/complementary activities (26.82%), further improve the quality of the food ("it was better than last year, but still", 24.39%), to provide better accommodations (14.63%), and to push the bedtime curfew until later at night (12.2%). The students' most frequent suggestion for improvement, to add fun/social activities, was echoed in the feedback of the instructors' focus group, who felt that as it was during the summer, the program was more emphasis on "achievement and task performance, and less on enjoyment". This was likely the result of the focus of this cohort on pushing their own startups forward within such a short timeframe. However, the instructors' sentiment that the program was possibly too stressful and overloaded for the students was not echoed in this cohort's survey results. The instructors' also felt that students lacked professional information and tools needed for roles such as project management and design, and these roles became the default for those who were less proficient at coding. A related improvement suggested by the Head of Curriculum, was to add a personal track of technical specializations throughout students' final year in MEET, as part of the accelerator. Suggested tracks include: 1) mobile development + fullstack development, 2) graphic design + front-end development; and 3) media production + social media + UX. Another improvement suggested was the clarification and settling of intellectual property rights in advance, so as to avoid potential problems should one of the projects become a registered company. Some additional points raised by the instructors' involved some instructors' perception of a bias towards the Palestinian side in the program, a minority of girls who did the coding in their project groups, the need among the instructors' for more background information on local realities, the great load on the educational staff, and the need for more involvement of the educational staff in the academic sessions and for more free time for the instructors' team. 6. COHORT III IN FY17 With an approved revision of the Implementation Plan, another cohort of 82 participants (Cohort III) benefitted from TEEM training in a Summer Program module held in July-August 2017. The program Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 20 of 24 offered full board and a mix of academic and fun, social activities. About 60% of the academic hours were spent on coding (primarily teaching the basics of the Python programming and the Turtle graphics environment for Python programming). The remaining academic time was divided between entrepreneurship classes and MEET's "Deeper Understanding" curriculum. The former covered the basic concepts of entrepreneurship, how to bring an idea to become a business, entrepreneurship as a way of living and looking at things, and soft skills, such as effective and respectful negotiation, storytelling, and presentation skills. The latter involved discussions and workshops about identities, narratives, and more, whose goal was to help students have a positive experience of being part of a bi-national group. The CS and entrepreneurship sessions were taught by eight overseas instructors, mostly from MIT (and including one MIT student who's a MEET alumnus). The program also included fun activities such as bowling, a talent night, and more. In addition to the ongoing lessons, sessions, and complementary activities, two special outreach events were held: one with hi-tech stakeholders at Facebook, and a final summer event to which family members were invited. Of the 82 participants, 78 filled out the online anonymous post-Summer survey (95.12%). The survey data was augmented by event reports and a qualitative content analysis of MEET’s staff meeting protocols and feedback from the overseas instructors’ debrief focus groups and summarizing documents. In general, the results aligned with the promised outputs and outcomes for that group or exceeded them. In terms of outputs, the number of participants who completed this module (82) was higher than the projected number (76). Moreover, the students created 40 projects in bi-national, gender￾mixed groups (20 CS projects and 20 entrepreneurship projects), all focused on themes of sustainability and the environment (over the projected minimum). Falling within the planned range of 120 to 200, 187 family members and stakeholders attend one of these the outreach mentioned above. The main outcomes planned for the program concerned the improved knowledge, skills and experience of the students. In terms of practical experience and increased CS knowledge and skills, the overwhelming majority of students indicated that their summer at MEET helped them gain practical experience and increased their CS knowledge, understanding and skills (98.72%), increased their confidence in CS (89.74%), and has left them interested in learning more CS (91.03%). Likewise for entrepreneurship, 96.15% of the students asserted that MEET helped them gain practical experience and increased their entrepreneurship knowledge, understanding and skills. There were no significant gender/nationality difference regarding these parameters. In terms of attitudes, perceptions and motivations related to the other nationality and joint work, 96.15% of the students indicated that, following the program, they are more positive about people from the other nationality, and 79.49% that they can now understand the perspective of people from the other nationality. Moreover, 88.46% reported that MEET increased their willingness, ability and motivation to cooperate and work on projects together with students from the other nationality, and yet more (94.87%) said that they were willing to work with people from the other nationality on common issues. As a whole, the students felt like they were part of one unified Year1 group (85.90%), felt a close connection to students in their activity group (84.62%), and felt like they could trust them and were safe with them (91.03% and 93.59%, respectively). There were no significant gender/nationality differences regarding either of these parameters. Students’ general satisfaction with the program is demonstrated by the fact that almost all of them (93.59%) would recommend MEET to a close friend or family member. Their favourite things about Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 21 of 24 the program include social aspects (improving social skills, meeting new people from different places, making new friends, as indicated by 50% of the students), the staff and instructors (20.51%), the Facebook visit and presentations (16.67%), complementary and fun activities (14.10%), and classes, new knowledge and learning (11.54%). A few students also specified enjoying entrepreneurship and/or CS. The most frequently mentioned aspects for improvement were the dorm rooms (in terms of cleanliness, showers and lack of AC) and the food, both raised by about a third of the students. Smaller numbers of students also requested more breaks/free time, or cancelling/extending the nightly curfew (14.10% and 12.82%, respectively). Others would have appreciated more visits or trips to outside locations (companies, the Old City, etc.), or more/better “Deeper Understanding” discussions (11.54% and 10.25%, respectively). The picture painted by the feedback and documents from MEET’s staff and from the volunteer overseas instructors is likewise very positive, while highlighting some smaller points for improvement. Overall, staff are in agreement that the program achieved MEET’s goal of having the students enjoy a positive experience of working together. Moreover, both the CS and entrepreneurship instructor teams report that the students covered the required material, created individual and group projects and on the whole, successfully completed their tasks. Both staff and instructors agreed that it was a great idea to invite instructors to take part in the complementary activities planned for students, as informal time after the academic hours helped the them bond. Indeed, according to the staff, most of the students felt strongly connected to the instructors and want the same team of instructors to work with them in their next summer at MEET. Feedback from the instructors suggests that there is a need for better alignment of expectations and for more regular and coordinated contact between the instructors and the local staff, including the logistics staff (e.g., holding regular work meetings with a pre-planned schedule between the instructors team and the educational and logistics teams, clarifying who is responsible for what, sharing schedules, etc.). The CS instructors lamented problems with the wi-fi in the classrooms and issues with the computing equipment. They also highlighted the importance of having voluntary "open hours" for additional student questions and recitations. Likewise, the entrepreneurship instructors would have preferred to have more academic hours dedicated to their topic, in order to have enough time for practice, in addition to extra “voluntary” time in which to augment/complement the material. The “Deeper Understanding” facilitators’ main points for improvement were diversifying the DU sessions with more activities (rather than a long conclusion/discussion), planning fun, quick ice￾breaker activities at the beginning of each session, and possibly using music sometimes when the students are engaged in activities such as drawing or working on something during the session, to help the flow of the session. 7. DISCUSSION AND RECOMMENDATIONS In general, it may be said that the program achieved its stated goals regarding both cohorts of TEEM, meeting and sometimes exceeding the indicator targets set in its Monitoring and Evaluation Plan (for the full set of indicators, presenting targets vs. actuals, see Appendix I): Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 22 of 24 1) In terms of numbers of events and trainees for the program's lifetime, 38 events in total were held (compared to 37 in the revised plan and 17 in the original plan), and 192 trainees completed their required training (compared to 182 in the revised plan and 120 in the original plan). Note that for this and for other indicators that do not reflect percentages, the numbers represent the overall achievement of all cohorts who benefited from the program (rather than Cohort I in FY16 and Cohort II in FY17). 2) In terms of increased CS skills and knowledge, 69.23% of the respondents of the Cohort I FY16 Year-End Survey reported that they understood the CS material very well, while 91.49% of their Cohort II counterparts agreed with the relevant revised survey item, testifying that the program helped them gain practical experience and increased their CS knowledge, understanding and skills. Thus, Cohort I closely matched while Cohort II exceeded the target percentage of 70% reporting improved CS knowledge and skills. Perhaps more importantly, students of both cohorts also demonstrated this knowledge to the satisfaction of their instructors and external stakeholders through creating individual and group CS projects, including demos presented at Google or Samsung NEXT during the Summer Program, which received above average grades. The percentage of Cohort II trainees who completed both individual and group CS projects during the Yearlong Program (63.51%) fell short of the equivalent date among Cohort I (83.82%) and the target of 80%. As mentioned in Section 4.5 above, the structure of Yearlong Program in FY17 was different with Cohort II creating their group CS projects during the Winter Conference, which many of them could not attend due to scheduling conflicts in the December holidays period. It is hereby recommended to reconsider the use of this type of indicator (% or number of students who create projects, when their creation is tied to a narrow range of specific dates), as it is too affected by scheduling issues, and to consider using measures of project quality when possible, such as stakeholder ratings. 3) In terms of increased entrepreneurship skills and knowledge, 96.15% of the respondents of the Cohort I FY16 Year-End Survey reported that they understand the entrepreneurship material very well, and 85.11% of their Cohort II counterparts agreed with the relevant revised survey item, that the program helped them gain practical experience and increased their entrepreneurship knowledge, understanding and skills. It should be taken into consideration that the original target of 70% was adjusted to 90% for Cohort II based on Cohort I's achievements, which they did not fully match (though staying within a 10% margin of variation from the target). Both cohorts also demonstrated this knowledge to the satisfaction of their instructors and external stakeholders through creating entrepreneurship projects presented to a variety of stakeholders. 4) In terms of improved perceptions about the other and about joint work for common goals, students' responses to the relevant PPR indicator survey items showed that the vast majority, through their participation in the program, have become better able to understand the perspective of people from the other nationality (84.31% for Cohort I and 89.36% for Cohort II), more positive about people from the other nationality (86.54% and 87.23%, respectively), and more willing to work with people from the other nationality on common issues (96.08% and 95.74%, respectively), all exceeding the target percentage of 80%. MEET’s internal indicators also showed high results 72.44% in FY16 (within 10% range of the target of 80%) and 87.23% in FY17 (exceeding the target). 5) In terms of increased exposure among program participants’ family members and among stakeholders to program activities, values and products throughout the project's lifetime, 527 family members and 120 stakeholders attended outreach events (compared to 350 and 80 in the revised plan, and 185 and 20 in the original plan). Moreover, over 3,000 people were engaged as part of the online campaign of Cohort I in FY17. While the impact of this exposure on stakeholders Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 23 of 24 and family members was not within the scope of this evaluation, several positive indications were revealed, including the positive feedback from stakeholders’ on students’ projects, parents’ positive view of the program and its effect on their children (per the Interim Evaluation Report), and several parents suggesting that they should also meet. Several issues were outlined in the Interim Evaluation Report as requiring further attention and treatment. Perhaps the most prominent of these were the CS knowledge gaps among students and related gender/nationality differences (which also appear for some other parameters, such as perceptions towards the other). Per that report, "[s]tudents join MEET with different levels of pre￾existing knowledge in computer science and often different personal inclinations. [...] Some learn it at school, and admit that without school they wouldn’t have managed the program, whereas others, especially Palestinians, do not." Girls, especially Israeli girls, tend to express less interest and motivation as regards CS, and undervalue their own abilities in this respect, likely due to cultural and societal factors. To address this (and other issues), MEET engaged two MIT graduates who served as volunteer instructors at MEET for several years past, to revise the curriculum. As part of this, in addition to the "mandatory" weekly sessions and Summer Program sessions, students were also given the opportunity to get support from instructors on a more voluntary and personal basis through "open sessions", which many of them took advantage of. Attendance records attest that this option was utilized by girls more than boys, and by Palestinians more than Israelis. Accordingly, Cohort II's FY17 CS achievements show a marked improvement in CS-related indicators (both self-report measures and project ratings). It is also evident that some of the gender differences disappeared or were at least narrowed down (see in particular Sections 4.1 and 4.5 above). Nevertheless, more work is required in order to narrow gender/nationality gaps regarding results for other indicators, and attention should be paid not only to differences between boys and girls or Israelis and Palestinians, but also to gender/nationality effects, especially as regards CS and "Deeper Understanding" measures. The Interim Evaluation Report also highlighted the issue of TAs and the need for a support system for them, as well as for more CS TAs. MEET attempted to address this via a special training program for the Summer TAs, which received great reviews. The need for more CS TAs was seemingly also addressed, the Cohort II students didn't clamor for them (possibly as they could also receive assistance as part of the "open sessions"). It is recommended to continue along the same lines with future Yearlong Program and Summer Program modules. Moreover, MEET should try to make the TA cadre as diverse as possible in terms of gender and nationality, so that each of the students would be able to find a TA they would feel comfortable coming to with questions. A third and important past finding had to do with the Israeli-Palestinian Encounter and the need to "directly address what was referred to by some participants as the 'elephant in the room', that is the Palestinian-Israeli Conflict." This was addressed in FY17 with a revision of MEET's "Deeper Understanding" curriculum and the creation of an integrative model setting appropriate goals for students according to their stage of progress within MEET and gradually helping them move forward. Thus, in their first summer at MEET, MEET aims for the students to have a positive experience of being part of a bi-national group at MEET. In the following Yearlong Program, students will strengthen their relationships with one another and their connection to MEET. In their second summer at MEET, Cooperative Agreement No. AID-294-A-15-00011 Training Entrepreneurs for Excellence at MEET (TEEM) Final Evaluation Report Page 24 of 24 students will explore and question identity and different points of view. As part of their second Yearlong Program, students will gradually become more aware of the status quo, of the reality around them. This awareness will be utilized to create group projects challenging the status quo, which will be the focus of their third and final summer at MEET. For this final summer, the goal will be for the students to have a positive experience of challenging the status quo. Further research in the coming years will be needed to evaluate the implementation and impact of this curriculum. It is important to act sensitively and avoid perceived bias, in addition to noting gender, nationality, and gender/nationality interaction effects in this context. A related issue not previously highlighted is that of dropout from the program. In past years at MEET, there have been no clear patterns of gender-related dropout. However, in FY17, dropout among Israeli males was dramatically higher than dropout among any other gender/nationality subgroup. Furthermore, there does appear to be a trend of Israeli participants dropping out at higher rates than Palestinian participants over the years (possibly due to more available options, though further research is required to analyze the crucial factors in this respect). As a result, although equal numbers of Israelis and Palestinians are recruited to the program, the further a cohort progresses within the program, the ratio of Palestinians to Israelis increases. This could have a snowball effect of some Israelis feeling under-represented and possibly dropping out as a result. The imbalance in numbers also has ramifications for "Deeper Understanding" sessions, which should be carefully considered. Finally, as highlighted and discussed in greater detail in Section 5 above, MEET launched a pilot for a unique accelerator program for the graduating cohort of students in FY17 with TEEM Cohort I. This pilot proved to be a great success, both in terms of students' self reports and the responses of external stakeholders. Students felt proud of their projects and expressed their will to continue them even after their time as students at MEET. Whether or not they will be able to do so is still in question, considering their young age and the fact that the year after they graduate from MEET is their final year in high school, involving the stress of matriculation exams, followed by military service for most Israelis and university studies (often abroad) for most Palestinians. Nevertheless, it is recommended to continue to implement it in future years with adjustments and improvements as fitting (see also Section 5). If possible, MEET should also support these students and provide them advice and resources should any of them decide to pursue the projects further later on. Appendix I: TEEM Project Performance Indicator Table – Targets vs. Actuals No. Level PMP Indicator Type Indicator Name FY16 Targets FY16 Results FY17 Targets FY17 Results Total Over Proj Life target Total Over Proj Life actual 1 IR1 PPR 1.6.2-12-2 # of USG-funded events, trainings, or activities designed to build support for peace or reconciliation on a mass scale (bi-national) 14 15 22 23 37 38 2 IR1 PPR 1.6.2-14 # of people participating in USG￾supported events, trainings or activities designed to build mass support for peace and reconciliation. Disaggregation: 132 130 182 192 182 192 1.6.2-14-1 Male 66 68 91 93 91 93 1.6.2-14-2 Female 66 62 91 99 91 99 1.6.2-14-3 Youth 132 130 182 192 182 192 3 Strategic objective Standard Custom turned PPR indicator Formerly 1.6.2-01-cust, now 1.6.2-C096: % of targeted participants who agree that they have better understanding of the others that helped in changing their attitude positively. Disaggregations: 80% 88.98% 80% 90.78% 80% 89.88% 1.6.2-C096a The above % for Males 80% 90.28% 80% 90.48% 80% 90.38% 1.6.2-C096b The above % for Females 80% 86.11% 80% 91.03% 80% 88.57% 4 Strategic objective MGT/Custom % of program participants reporting increased capacity to act for positive change 80% abandoned 80% NA 80% NA 5 IR 1 MGT/Custom # of bi-national CS and/or entrepreneurship summer projects that will be developed by teams of Israeli and Palestinian (including Palestinian citizens of Israel) students and presented to stakeholders and family members. 25 59 62 68 121 127 6 IR 2 MGT/Custom # of parents and/or family members attending summer outreach events 185 317 350 527 350 527 7 IR 2 MGT/Custom # of stakeholders attending outreach events, or exposed to participants’ presentations through special events and excursions. 55 51 70 73 80 120 8 Sub-IR 1.1 MGT/Custom % of targeted participants who report improved CS knowledge and skills 70% 69.23% 70% 91.49% 70% 80.36% 9 Sub-IR 1.1 MGT/Custom % of targeted participants who demonstrate improved CS knowledge and skills via completing individual projects using Python and group projects creating small functional computer games. 80% 83.82% 80% 63.51% 80% 73.67% 10 Sub-IR 1.2 MGT/Custom % of targeted participants who report improved entrepreneurship knowledge and skills 70% 96.15% 90% 85.11% 90% 90.63% 11 Sub-IR 1.2 MGT/Custom % of targeted students who demonstrate acquired entrepreneurship skills and knowledge via successful presentations at outreach event 70% 77.94% 70% 71.62% 70% 74.78% 12 Sub-IR 1.3 MGT/Custom % of targeted participants who report increased willingness to work together, increased belief in the importance and impact of bi-national cooperation, and other relevant self-report measures not covered by the standard custom indicator 1.6.2-01. 80% 72.44% 80% 87.23% 80% 79.84% * With the exception of indicators #1, #6 and #7, all numbers and percentages refer to solely to our trainees (“program participants”, “targeted participants”) and/or to events/activities focused on them. ** Baseline values for all indicators are 0. *** Some survey items (for indicators 8, 10 and 12) differ between FY16 and FY17 – see full report for details and explanations.