September 2017 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Dr. William Dant, Ms. Charu Vijayakumar and Ms. Amun Nadeem of Dexis Consulting Group, Washington, DC. EDUCATION IN CRISIS AND CONFLICT NETWORK: Mid-term Evaluation EDUCATION IN CRISIS & CONFLICT NETWORK: MID-TERM EVALUATION September 2017 USAID Contract No. AID-OAA-M-14-00007 MOBIS # GS-10F-0406P DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Contents ACRONYMS ..................................................................................................................................................... a EXECUTIVE SUMMARY...................................................................................................................................... I EVALUATION PURPOSE, QUESTIONS & METHODOLOGY................................................................................... 1 Purpose and Objectives ............................................................................................................................... 1 Evaluation Questions................................................................................................................................... 1 ECCN Project Background ............................................................................................................................ 1 Evaluation Methods & Limitations ............................................................................................................... 3 Sampling Strategy........................................................................................................................................ 7 Data Collection ............................................................................................................................................ 8 Data Analysis............................................................................................................................................... 8 Limitations................................................................................................................................................... 9 FINDINGS, CONCLUSIONS &RECOMMENDATIONS.......................................................................................... 10 1. What progress has the ECCN ST made in carrying out the five work streams and advancing towards the objectives outlined in their workplan? ....................................................................................................... 10 2. What successes and challenges has the ECCN ST encountered in carrying out their five work streams and meeting their objectives?........................................................................................................................... 24 3. What is the nature and extent of member engagement and how is it supported by the ECCN ST, Steering Group and Advisory Group?....................................................................................................................... 33 4. To what extent are the priority areas and the themes within the ECCN project relevant to USAID Office of Education and the ECCN member community? .......................................................................................... 39 5. To what extent does the ECCN, as a Community of Practice, meet the evolving needs of USAID’s Office of Education?................................................................................................................................................. 42 ANNEXES.......................................................................................................................................................... i Annex I: Evaluation Statement of Work........................................................................................................ ii Annex II: Data Collection Instruments .........................................................................................................xv Annex III: Sources of Information ............................................................................................................. xxix Annex IV: Survey results............................................................................................................................ xxx Annex V: KII Code List ................................................................................................................................. lx Annex VI: Disclosure of any Conflicts of Interest........................................................................................lxiii a ACRONYMS ADS USAID’s Automated Directives System AEP Accelerated/Alternate Education Programming AG Advisory Group APS Annual Program Statement CO Contract Officer CoP Community of Practice COR Contracting Officer’sRepresentative C&C Conflict and Crisis Dexis Dexis Consulting Group DC Washington, DC CIES Comparative and International Education Society ECCN Education in Crisis and ConflictNetwork E3/ED Office of Education in the Bureau for Economic Growth,Education and Environment EDC Education DevelopmentCenter EiCC Education in Crisis andConflict FCR Findings, Conclusions andRecommendations(matrix) FSO Foreign ServiceOfficer INEE Inter-Agency Network for Education in Emergencies IRB Institutional Review Board IP Implementing Partner KII Key InformantInterview LOE Level of effort PD Professional Development PMP Performance ManagementPlan RERA Rapid Education Risk Analysis RFA Request for Applications RFP Request for Proposals SOW Statement of Work SG Steering Group ST Support Team TOC Theory of Change USAID United States Agency for InternationalDevelopment I EXECUTIVE SUMMARY Purpose and Objectives Dexis Consulting Group (Dexis) was contracted by the United States Agency for International Development (USAID) Office of Education in the Bureau for Economic Growth, Education and the Environment (E3/ED) to perform a mid-term evaluation of the USAID Education in Crisis and Conflict Network (ECCN). The ECCN is a five-year-long E3/ED initiative implemented by the Education Development Center (EDC), focused on five inter-related objectives or work streams: 1. Develop and support a community of practice (CoP); 2. Build and disseminate evidence for the field; 3. Test, disseminate and use guidance, tools, metrics, and theories of change; 4. Apply tools in crisis and conflict; and 5. Develop, implement and assess training. EDC’s contract for implementing this initiative began on July 28, 2014 and is intended to run through July 27, 2019. This purpose of the mid-term evaluation was to assess the progress made towards the ECCN’s stated objectives, as well as to serve as a source of learning for the EDC ECCN Support Team (ST) and E3/ED more broadly. Specifically, the evaluation team sought to answer the following five evaluation questions: 1. What progress has the ECCN ST made in carrying out the five work streams and advancing towards the objectives outlined in their work plan? 2. What successes and challenges has the ECCN ST encountered in carrying out their five work streams and meeting theirobjectives? 3. What is the nature and extent of member engagement and how is it supported by the ECCN ST, Steering Group and Advisory Group? 4. To what extent are the priority areas and the themes within the ECCN project relevant to USAID’s Office of Education and the ECCN member community? 5. To what extent does the ECCN, as a Community of Practice, meet the evolving needsof USAID’s Office of Education? Based on findings and conclusions from the evaluation, this report provides E3/ED with a comprehensive set of lessons learned and actionable recommendations to inform decision-making in ECCN and, more broadly, within USAID’s Office of Education. Evaluation Methods and Limitations This mid-term evaluation used a mixed-methods, utilization-focused approach, incorporating both quantitative and qualitative data collection and analysis methods. This was done through three data collection streams: 1. a desk review of available background and reporting documents; 2. key informant interviews (KIIs) with the ECCN implementation team, USAID staff directly involved with the implementation of the ECCN contract, and a sample of USAID Education Staff based in Washington, DC, and USAID missions globally; and 3. an electronic survey distributed to all registered ECCN members (n=629). The evaluation team conducted 36 KIIs; and the survey produced a total of 93 responses, a 14.79% overall response rate, and 67 (10.65%) complete responses. The evaluation team used multiple sources of data to address each evaluation question and triangulate across the relevant sources to develop comprehensive and evidence￾based findings, conclusions, and recommendation. The primary limitation of this evaluation is that the evaluation team was not able to collect perspectives from a truly representative sample of the ECCN member community. While the key II informants were purposively selected to elicit the most relevant information for the evaluation questions, they were focused on USAID-affiliated stakeholders. In order to mitigate this, the survey was intended to reach a broader sample of the network. However, the survey response rate was low. Summative Findings and Conclusions by Evaluation Question 1. What progress has the ECCN ST made in carrying out the five work streams and advancing towards the objectives outlined in their workplan? The ECCN ST has made steady progress advancing all five objectives defined in their contract. This includes: • Registered ECCN members increased from 162 members at the end of Year 1 to 701 members at the end of Year 3. • The ECCN Resource Repository has steadily grown, with 268 separate items now; approximately 25% is original content and 75% is from othersources. • Survey responses showed the conflict sensitivity checklist and RERA are the most-utilized resources; the least-consulted resource was costing guidance. Outreach, communications, research, and professional development programming are evolving and may benefit from further clarity and consideration. While the process of development and implementation of training/professional development receives staff and funding support, the objective’s progress is less than other areas. • Some informants noted ECCN research resources, notably ECCN-produced reports and articles, were not always practitioner-oriented or easy to draw on for fieldstaff. In terms of the degree of uptake and use of opportunities, concepts, evidence, tools and guidance across each of ECCN’s five work streams, this evaluation assessed ease of access, degree of utilization/application, and changed practice as reported by survey respondents. • Ease of access and utilization of research and resources are positive across respondents. However, changed practice is less so, with a heavier emphasis on resources serving as occasional inputs rather than long-term changed practice inputs. 2. What successes and challenges has the ECCN ST encountered in carrying out their five work streams and meeting their objectives? The ECCN ST has faced both process and product oriented successes in carrying out the five work streams and meeting project objectives. These include adaptive management, member identification and network growth, website launch and participation, tools and resource development, and a highly qualified and responsive ST. The ECCN ST has been flexible and effective in adapting to strategic shifts and incorporating lessons learned into implementing the five work streams. Despite logistical challenges, the ST has been successful in generating a steady increase in the number of members accessing and using ECCN resources and continues to see a steady increase in member growth across all web-based platforms. • Successes: Adaptive management; growing the CoP; improved timeliness; interactive website; tools and resource development; management by ST 44.9% of total respondents and 48% of USAID affiliated respondents reported finding the CoP very well managed. III The ST also faced numerous challenges. • Challenges: Staff turnover and delays early in the project; lack of clarity on ECCN’s mandate and coordination between E3/ED and EDC; outreach and engagement; SG and AG engagement • Lack of time, resources, and awareness/availability were cited as primary reasons for limited active participation of ECCN community members in knowledge sharing and training opportunities offered through ECCN. • Overall, it seems that dual roles are expected of the ST: 1) content/technical drivers and 2) outreach, engagement, and communications managers for the CoP. Less than ideal communications and coordination between EDC and E3/ED, especially in programmatic vision, continues to be a challenge in meeting objectives. 3. What is the nature and extent of member engagement and how isitsupported by the ECCN ST, Steering Group and AdvisoryGroup? Although member engagement is positively evolving into a more multi-directional process that combines both push and pull approaches, it is still primarily unidirectional, with the CoP passively receiving quality and curated information from ECCN. Field representation and engagement continues to be a challenge. • 52 of 78 survey respondents felt that the ECCN offered either adequate or many opportunities for engagement. • 51 out of 78 survey respondents reported regularly engaging through the newsletter, followed by website (30 respondents), webcasts (15 respondents), and face-to-face events (10respondents). • The newsletter has an opening rate of 35-40%, with a 30% click through rate. • 20 of the key informants mentioned that the ECCN is US-centric, while less than five individuals mentioned that Implementing Partner (IP) and/or field involvement is improving. Of the 20 Mission staff contacted for KIIs, three declined to be interviewed citing lack of any awareness of ECCN. Additionally, of the six individuals interviewed, half of them reported only limited familiarity with the ECCN. • Multiple KIIs mentioned the lack of field representation in the SG. 4. To what extent are the priority areas and the themes within the ECCN project relevant to USAID Office of Education and the ECCN member community? While the range of USAID and ECCN member community may interpret relevance in different ways, the vast majority of survey respondents reported that all four priority areas and the themes within them are very relevant. • A majority of overall survey respondents indicated that all priority areas were considered “very relevant” to E3/ED and the ECCN member community. • Priority area 1 (Learning Environments are Safer) received the most “very relevant” ratings and priority area 4 (Local Institutions are Stronger) received the least number of “very relevant” ratings from USAID affiliated respondents. • Priority areas 2 (EiCC Programming has Improved) and 4 (Local Institutions are Stronger) received the most “very relevant” ratings from IP respondents. • Most respondents found all themes within priority areas to be “very relevant.” Two slight exceptions were “Advanced program and activity costing guidance,” and “Knowledge IV related to ICT application in EiCC,” which had a significant proportion of respondents selecting “not relevant” or “somewhat relevant.” • The priority areas developed are very relevant to E3/ED because they were developed the USAID EiCC team and reflect the major outcomes outlined in the USAID Education Strategy for Goal 3. 5. To what extent does a Community of Practice (CoP) in general, and ECCN in particular, meet the evolving needs of USAID’s Office of Education? The ECCN largely meets the evolving needs of E3/ED, however, it does not yet function as a traditional CoP with multidirectional engagement, as envisioned by the educational theorist Étienne Wenger. • Various elements of the ECCN network and its resources were considered by E3/ED informants in leadership positions to be applicable to other areas of the USAID Education Strategy. • The Safe Learning Environment work of ECCN seemed to some respondents to be useful for USAID's Goal 1 of successfully addressing the advancement of early-grade reading. • The evidence gap tool and other research and tools developed by ECCN were also deemed by some as applicable to other programming areas that the E3/ED office was promoting. • ECCN is meeting evolving needs through potentially higher demand for the addressing of education needs in crisis and conflict-affected settings, which themselves are increasing due to growing global unrest, and therefore demanding more emergency education and the potential resources for it. • ECCN was often discussed and considered as a network by interviewees, without a particular assumption by respondents beyond E3/ED and the ECCN ST that it involved robust multidirectional sharing andcollaboration. Recommendations While the report presents a set of recommendations tailored to each evaluation question, below are a list of overarching recommendations that are cross-cutting. ● Clearly define the scope of the ECCN mandate and the priorities within it. This includes agreement between the ECCN implementation team and USAID on the target audience and the subsequent purpose of the content and activities developed and the allocation of effort and resources across work streams. While there has been a strategic shift in focus from process￾oriented to content-oriented work-planning, there remains lack of clarity on what requests, expectations, and activities fall within or outside the scope of the contract, as well as which tasks or areas require prioritization. ● Streamline processes and communications. While key ECCN implementationstakeholders have been flexible and adaptive throughout the first two years of the project, the project would benefit from instituting clearly defined processes that streamline planning and approvals. This includes clear understanding of roles and structured communications that facilitate efficient decision-making and more timely execution of outputs and activities. V ● Revisit LOE allocations across team members. At this point, the focus is shifting from ‘push’ to ‘pull’ strategies that require more active outreach and communications efforts. Consider strategies to create flexibility in staffing levels and expense caps by category and function and assess whether the current staffing for communications is sufficient for anticipated needs. ● Conclusively decide whether ECCN needs to focus on engaging USAID field staff in activities. If the ST and USAID decide that this is the direction they would like to pursue for ECCN, going forward seek ways to emphasize engagement in the field and consider whether outreach should especially target field-based staff (or education teams) through direct interactions with key USAID individuals to broker relationships between DC and missions, as well as within Missions/in￾country. 1 EVALUATION PURPOSE, QUESTIONS & METHODOLOGY Purpose and Objectives Dexis Consulting Group (Dexis) was contracted by the United States Agency for International Development’s (USAID) Office of Education in the Bureau for Economic Growth, Education and the Environment (E3/ED) to perform a mid-term evaluation of the USAID Education in Crisis and Conflict Network (ECCN). The ECCN is a five-year-long E3/ED initiative, implemented by the Education Development Center (EDC), through a Support Team (ST) of nine professionals experienced in education in crisis and conflict (EiCC). EDC’s contract for implementing this initiative began on July 28, 2014 and is intended to run through July 27, 2019. Evaluation Questions To meet the evaluation objectives described above, the evaluation team investigated five evaluation questions identified in the statement of work (SOW),1 with modifications later provided by the Office of Education: 1. What progress has the ECCN ST made in carrying out the five work streams and advancing towards the objectives outlined in their workplan? 2. What successes and challenges has the ECCN ST encountered in carrying out their five work streams and meeting theirobjectives? 3. What is the nature and extent of member engagement and how is it supported by the ECCN ST, Steering Group and AdvisoryGroup? 4. To what extent are the priority areas and the themes within the ECCN project relevantto USAID Office of Education and the ECCN membercommunity? 5. To what extent does the ECCN, as a Community of Practice (CoP), meet the evolving needsof USAID’s Office of Education? The evaluation team developed and implemented this assessment to better understand the overarching goals and assumptions that guide ECCN’s approach and focus areas, to determine the progress the team has made towards its objectives, and to gain feedback to improve engagement and processes in place to carry out the five work streams identified in the work plan. Based on findings and conclusions from the evaluation, this report provides E3/ED with a set of lessons learned and actionable recommendations to inform decision-making in ECCN and more broadly, within E3/ED. ECCN Project Background The five specific objectives of the ECCN project are: 1. Develop and support the EiCC CoP: This objective is intended to nurture and grow an intentional USAID focused sub-community (ECCN) through whichmembers canengage in a rich exchange of knowledge and share it with the larger EiCCCoP. 2. Gather, build, consolidate evidence for the Field: This objective will synthesize existing 1 See Annex I 2 research, engage in new research, and disseminate findings through documents, training, and other forums. 3. Test, disseminate, and use guidance, tools, metrics, and Theories of Change (TOCs): This objective focuses on developing tools and guidance that concentrate on the most effective approaches to increasing equitable access to EiCC environments. Specifically, it will develop operational guidance and tools; develop and test TOCs, indicators, and log frames; and provide costing methodologies and templates related to Goal 3programming. 4. Develop and apply guidance/tools in Crisis and Conflict (including Rapid Education Risk Analysis (RERA)): Activities in this objective will refine, add to, and implement existing EiCC programming tools to promote more conflict sensitive and context appropriate education programming in Crisis and Conflict (C&C) environments. 5. Develop, implement, and assess professional development: The project will develop, deliver and assess a range of professional development opportunities designed to build the capacity of USAID, Goal 3 implementers and local and regional institutions to use evidence-based research and resources in project design, implementation, andmonitoring. Towards the second year of the project, in addition to the five objectives, USAID and ECCN introduced a complementary perspective to plan and track progress through five priority areas: • Make learning safer, • Design, implement, research and evaluate for better EiCCprogramming, • Understand and strive for equity, • Strengthen local institutions, and • Cross-cutting and management activities. 3 Taken together, ECCN goal and objectives comprise the following TOC: Figure 1: ECCN’s Theory of Change Ultimately, by nurturing and convening a CoP/Network2 and spearheading key activities, such as the development of tools and metrics, information-sharing through the development of a website and communication platform, and systematic reviews of evidence, the ECCN Support contract aims to advance E3/ED towards its strategic objectives; namely, to increase access to safer, higher quality learning opportunities in crisis and conflict-affected environments. Evaluation Methods & Limitations This mid-term evaluation used a mixed-methods, utilization-focused approach, incorporating both quantitative and qualitative data collection and analysis methods. This was done through a three￾phase evaluation design: 1) evaluation design and preparation for data collection; 2) data collection and synthesis; and 3) analysis and report writing. During each phase, the evaluation team maintained regular communication with USAID and approached each activity iteratively to ensure feedback was solicited and incorporated appropriately. 2 This report uses both terms when referring to the product of the first objective outlined above. The distinctions between them are discussed in detail under the consideration of Evaluation Question 5 USAID ECCN Goal: Improved policy, planning and practice for USAID Education Goal 3 programming USAID Education Strategy Goal 3: Increased Equitable Access to Education in Crisis and Conflict Affected Environments 4 To address the five key evaluation questions outlined above, the evaluation team implemented three data collection streams: 1. a desk review of available background and reporting documents (please see Annex III for a complete list), 2. key informant interviews (KIIs) with the ECCN implementation team, which we define as including the ST, the Steering Group (SG), and Advisory Group (AG); and other concerned USAID education staff, and 3. an electronic survey distributed to all registered ECCN members. Beginning with a desk review allowed the evaluation team to become well-versed in the context and progress of ECCN, and informed the development of the KII and survey instruments. The interviews and surveys were conducted simultaneously, and information from each data source helped consistently fine-tune the KII instruments to identify and explore emergent themes, as well as divergent views. This methodology conforms to the USAID-provided evaluation SOW, which requested a mixed-methods approach including a document review, quantitative data through surveys, and qualitative information through KIIs.3 The evaluation team mapped each data collection effort to the appropriate evaluation questions to ensure relevance of data, and to serve as identifiers for data triangulation (Table 2 below – design matrix). We used multiple sources of data to address each evaluation question and triangulated across the relevant sources to develop comprehensive and evidence-based findings, conclusions, and recommendations. The mixed-methods approach permitted the evaluation team to examine both the performance of ECCN as well as the processes of implementation and engagement, per the criteria in the E3/ED SOW. The desk review, which included the E3/EDC contract, and quarterly and annual progress reports, provided evidence and discussion about progress, successes, and challenges; the survey and interviews uncovered further findings and perceptions about performance, processes, and engagement and factors influencing each across stakeholder groups including, USAID personnel directly involved with the ECCN contract, the broader USAID education staff, EDC ST staff, and ECCN members. The evaluation team examined performance in reference to set targets and goals, as well as planned and implemented engagement and activities through the document review, and then further explored emergent findings by soliciting feedback across different stakeholder groups. Please refer to Tables 1 and 2 for a comprehensive delineation of data sources, data collection approaches, and methods by evaluation question. This in-depth investigation produced findings, conclusions, and actionable recommendations that the ongoing project can draw upon to further improve its processes and programming through the end of the activity in mid-2019. 3 Original SOW and Mid-Term Evaluation Kick-Off Meeting PowerPoint presentation 5 Table 1: Data collection sources and methods for adjusted evaluation questions Evaluation Questions (adjusted) Desk Review KII (USAID staff directly involved with ECCN contract) KII (ECCN ST, AG, SG) KII (USAID Education Staff – Washington and Mission – based) Survey (ECCN members – USAID and non-USAID) 1 What progress has the ECCN ST made in carrying out the five work streams and advancing towards the objectives outlined in their work plan? X X X X X 2 What success and challenges has the ECCN ST encountered in carrying out their five work streams and meeting their objectives? X X X X X 3 What is the nature and extent of member engagement and how is it supported by the ECCN ST, steering Group and Advisory Group? X X X X X 4 To what extent are the priority areas and the themes within the ECCN project relevant to USAID Office of Education and the ECCN member community? X X X X X 5 To what extent does the ECCN, as a Community of Practice, meet the evolving needs of USAID’s Office of Education? X X X X 6 Table 2: Design Matrix EQs Data Sources Approach/linkages between data sources and EQs 1.. What progress has the ECCN ST made? Desk review: Work Plans; Progress and Quarterly Reports; PMP; all available reports, information, and documents on the ECCN website KIIs with: USAID directly involved with ECCN contract; ECCN ST; ECCN SG and AG; ECCN members Survey of ECCN members 1. Output and outcome mapping: The evaluation team will review ECCN project documents to determine outputs and outcomes at each stage of the ECCN project. Findings from the desk review will inform discussions in KIIs as we will corroborate our findings and seek reasons for any gaps in understanding in interviews. Lastly, data from the survey of ECCN members will also be used to triangulate findings about outcomes and outputs. This method will particularly uncover ongoing contributions of the project, and uncover individuals, groups, and organizations that have been/are affected, in what ways, and through which specific program activities. 2. Process mapping for implementation: The evaluation team will review documents and contribute to implementation of the ECCN. The process map will be discussed in interviews for further understanding and refinement. 2. What success and challenges has the ECCN ST encountered? Desk review: Work Plans; Progress and Quarterly Reports; PMP; all available reports, information, and documents on the ECCN website KIIs with: USAID directly involved with ECCN contract; ST, SG, AG; ECCN members Survey of ECCN members The evaluation team will conduct thematic analysis on documents reviewed to identify themes that arise regarding successes and barriers. We will also take detailed notes during interviews and create an Excel based interview data matrix that, to the extent possible, qualitatively code findings from interviews so that responses can be analyzed for overall trends and generalized findings. This will also allow for identification of range of successes and challenges, including unanticipated ones. Survey data will also be analyzed. 3. What is the nature and extent of member engagement and how is it supported by the ECCN ST, SG, and AG? Desk review: ECCN Website; Progress/Monitoring Reports; Communications strategy; all available reports, information, and documents on the ECCN website KIIS with: USAID directly involved with ECCN contract; ST, SG, AG ECCN members Survey of ECCN members 1.Construct common narratives from members about their experiences with engagement. Identify common and variety of barriers through thematic analysis of interview and survey data. 2.Construct common narratives from individuals in the ECCN ST and the SG about their perception of management strategies and implementation. Specifically, we will identify whether there are systematic and agreed upon processes in place; gauge the level of cohesion and cooperation amongst the ST, AG, and SG. 4. To what extent are the priority areas and the themes within the ECCN project relevant to USAID E3/ED? Desk review: E3/ED Documents relating to the priority areas/themes (Learning Agenda, etc.); all available reports, information, and documents on the ECCN website KIIs with: USAID directly involved with ECCN contract; ST, SG, AG; ECCN members (USAID) Survey of ECCN members (USAID) The evaluation team will review all available E3/ED documents relating to priority areas and themes to develop an understanding of, and to compare pertinent E3/ED and ECCN priority areas. We will conduct KIIs with USAID staff who are involved with the ECCN contract implementation as well as USAID staff who are ECCN members and the ECCN implementation team to construct common narratives about their perceptions about the relevance of the ECCN structure to the work and aims of E3/ED. This will be compared with survey responses from USAID ECCN members. 4a. To what extent are the priority areas and the themes within the ECCN project relevant to the ECCN member community? Desk review: E3/ED Documents relating to the priority areas/themes (Learning Agenda, etc.); all available reports, information, and documents on the ECCN website KIIs with: USAID directly involved with ECCN contract; ECCN members (non-USAID) Survey of ECCN members The evaluation team will analyze findings from the ECCN member survey to identify ECCN member perceptions about the relevance of the ECCN project to them and their work. For this EQ, we will also select (purposive sampling) some non-USAID respondents from the survey for interviews to gain detailed insights into their perceptions of the usefulness of ECCN priority areas and themes. 5. To what extent does the ECCN, as a CoP, meet the evolving needs of USAID E3/ED? Desk review: USAID Education Strategy; all available reports, information, and documents on the ECCN website KIIs with: USAID directly involved with ECCN contract; ST, SG, AG; ECCN members (USAID) First, the evaluation team will review the USAID Education Strategy to familiarize itself with the needs of USAID E3/ED. We will conduct KIIs with USAID stakeholders and the ECCN ST and construct common narratives about their perceptions about ECCN’s contribution to the evolving needs of USAID E3/ED. 7 Sampling Strategy Key Informant Interviews: For the KIIs, the evaluation team used a purposive sampling methodology to capture specific individuals and groups of individuals who are considered key stakeholders of the ECCN and could provide the most relevant data for the targeted evaluation questions. The sampling approach for KII respondents at the first level of stratification included the following three stakeholder groups: 1. the ECCN implementation team, which consists of the ECCN ST, AG and SG, 2. the USAID team involved with the ECCN contract and 3. Sample of USAID Education staff based in Washington DC and Missions globally. While the sample for the ECCN implementation team and USAID team members involved with the ECCN contract included all current members of that stakeholder group, the sample selection of other USAID education staff was done in collaboration with USAID. USAID provided the evaluation team with a comprehensive list of USAID education staff based in DC and USAID missions globally. The evaluation team selected a sample from each of these lists based on the following criteria: 1. For the USAID DC list, the selection was based on the E3/ED team or regional bureau affiliation to capture a diversity of perspectives and 2. For the mission list, the selection was two-tiered: stage 1, country selection and stage 2, individuals within each country selection. The countries selected for interviews reflected countries with EiCC programming, which were further categorized by region, and by countries defined as being affected by crisis versus conflict. The number of countries selected within each region was based on proportionality to the entire list of countries, minus the Europe and Eurasia region, as there were no countries there with EiCC programming. The individual selection within each country was based on whether the individual was a registered ECCN member, their employment type (FSO, LES, etc.), and seniority based on title. The evaluation team developed and used these criteria to ensure that the sample would capture a diverse range of stakeholders and perspectives. We identified first, second, and third-choice lists, and a total of nine USAID DC individuals and 20 USAID mission individuals were contacted. Given the evaluation’s focus on USAID stakeholders and in the interest of limited resources, non-USAID ECCN members were not targeted for KIIs; however, their survey responses provided considerable depth and variation in responses. Additionally, during KIIs the evaluation team took note of viewpoints provided by non￾USAID stakeholders who were members of the SG and AG, and, qualitative responses from the survey provided the evaluation team with non-USAID stakeholder perspectives as well. ECCN Member Survey: The evaluation team administered the electronic survey, through SurveyMonkey, to the entire universe of ECCN members, identified as currently registered members of the network. The ECCN ST provided the evaluation team with a list of all registered members, which included 629 e-mail addresses. This sampling approach allowed for the most comprehensive understanding of ECCN membership perspectives, as all members were equally targeted for input. The evaluation team administered the ECCN member survey with messaging support from USAID and the ECCN ST. Due to the ECCN ST’s experience with low response rates, the evaluation team worked with USAID and the ECCN ST to administer the survey through multiple channels as a mitigation strategy. These multiple channels included: 1. USAID sent introductory e-mails to all USAID ECCN members with the survey link embedded in the email to encourage responses, 2. ECCN ST included the survey link in the monthly newsletter encouraging the member community to respond, and 3. The evaluation team sent direct e-mails to each of the 629 registered ECCN members through SurveyMonkey. The survey was open for two weeks, with one reminder e-mail 8 that was sent to all non- respondents at the end of the first week. The survey produced a total of 93 responses, a 14.79% overall response rate. However, not all responses gathered were complete: 67 of the 93 respondents (10.65%) completed the entire survey, while 26 of the respondents partially completed the survey. As such, the number of respondents per survey question varies. Data Collection The evaluation team approached each data collection effort systematically using standardized templates across all three team members to ensure data quality and allow for seamless data synthesis. Specifically, we developed data collection instruments for each of the three data streams: 1. a document review template; 2. a set of KII protocols – one for each stakeholder group; and 3. a survey questionnaire, and mapped each data collection instrument to the evaluation questions to ensure relevance of all data collected and to facilitate coding during the analysis process. The evaluation team conducted KIIs as semi-structured interviews that covered the main lines of inquiry and allowed the team to expand exploration with respondents having a familiarity with the various aspects of the project. As interviews began to reflect trends and themes, we tested such initial potential findings with later interview participants for verification. The sample included respondents with both current and past engagement with ECCN, to gain a better perspective on evolution and progress over time. Two evaluation team members conducted each, approximately one hour long, interview, with one person leading the interview and the other capturing electronic notes in a pre-formatted template. The evaluation team conducted a total of 36 KIIs over the course of three and a half weeks. This included all nine of the ST members, all but one of the SG members, two of the five AG members, all six of the USAID team members involved with the ECCN contract, five USAID education staff based in Washington, DC, and six USAID staff based in Missions. Some individuals fall within multiple categories, for example, an individual could be both on the SG and the ST. Data Analysis The evaluation team conducted both qualitative and quantitative analysis to produce a set of findings, conclusions, and recommendations that address each of the five evaluation questions. We conducted qualitative analysis, specifically content and thematic analysis, on information gleaned from the desk review, key informant interviews, and qualitative responses from the online survey. In addition to identifying themes, the evaluation team also explored single voices and narratives that provided insights into how ECCN’s development, progress, and implementation is perceived. The evaluation team reviewed and conducted content analysis on all relevant documents to form an initial understanding of the ECCN, and continued to expand on the document review process throughout the evaluation to identify additional information as needed. Following the initial desk review, the evaluation team developed initial qualitative codes that mapped to the evaluation questions that were used to develop the KII notes matrix. Then, the evaluation team systematically coded all qualitative KII data to highlight themes as well as single voices that provided relevant information mapped to each line of inquiry. Finally, the evaluation team also performed content analysis on the qualitative survey responses to identify both convergent and divergent perspectives. 9 We also conducted quantitative analysis, using SurveyMonkey’s built-in analysis functionality and Microsoft Excel, on the survey results. Given the small sample size, the evaluation team focused statistical analysis on descriptive statistics and cross tabulations that specifically explored survey results by stakeholder group to provide comparisons in responses, such as disaggregating by USAID or non- USAID respondent or type of USAID affiliation. The evaluation team triangulated findings from all three data collection streams and utilized a findings, conclusions, and recommendations (FCR) matrix to ensure that all analysis and subsequent conclusions and recommendations are evidence based. This FCR matrix allowed the team to map all data collected to each of the evaluation questions and develop logical links from the data to findings, the conclusions that can be drawn from those findings, and the recommendations that come from those conclusions. This exercise facilitated triangulation of all the data sources, both quantitative and qualitative, to validate findings and identify discrepancies. Limitations The primary limitation of this evaluation is that the evaluation team was not able to collect perspectives from a truly representative sample of the ECCN member community. While the key informants were purposively selected to elicit the most relevant information for the evaluation questions, they were focused on USAID-affiliated stakeholders. To mitigate this, the survey was intended to reach a broader sample of the network. However, the survey response rate was low. The survey responses cannot be interpreted as representative of the entire ECCN member community because it is likely that those who responded are a subset of the larger community who are more aware of, active in, and engaged with the ECCN. Alternatively, since the ECCN network website is largely accessible without a member login, we likely missed some individuals who draw on the ECCN for resources without having registered. Finally, the team received a lower than ideal response rate from mission contacts for KIIs, which limited the ability to conduct complex statistical analysis considering factors such as geography and employment type. As such, the summary statistics provide useful information about emergent trends and divergent findings, but cannot be used as a basis for representative conclusions. In other words, external validity of our data is low, so we recommend caution in using these evaluation findings to inform generalizations. Finally, even though some analysis in this report is broken down by professional affiliation (USAID and non-USAID), the evaluation team collected individual perception data, which may not necessarily reflect the view of the agency/organization that the respondents work(ed) in. Specifically, survey respondents and interviewees were instructed to respond about their individual experience with ECCN, given their current or past employment position(s). Survey and KII questions inquired about individuals’ perceptions on numerous aspects of ECCN characteristics and performance. While it is reasonable to assume that individual perceptions are based on their own experience with /knowledge of ECCN, which is likely to be affected by their position (in USAID HQ, Mission, etc.), the evaluation team is unable to conclusively attribute individual perceptions to organizations/agencies. 10 FINDINGS, CONCLUSIONS &RECOMMENDATIONS This section will present the findings, conclusions and recommendations developed for each of the five research questions separately to allow more in-depth exploration of the data captured and target focus on the individual questions. In some cases, findings emerging from the data applied to more than one of the five questions. To allow each individual question discussion to stand on its own, some of what is reported is minimally duplicative across some of the individual question discussions. 1. What progress has the ECCN ST made in carrying out the five work streams and advancing towards the objectives outlined in their workplan? Progress by Each of the Five Work Streams/Objectives 1. Develop and support a CoP A review of the work plans and quarterly and annual reports identifies the following overarching progress markers or sub tasks for Objective 1:41) Establish a CoP, AG, working groups, and task teams; 2) establish ECCN learning agenda through outreach and information gathering; and 3) engage in outreach and communication with USAID. The ECCN ST has made steady progress on all markers. The AG and SG are up and running, task teams are evolving, and outreach, information gathering, and overall engagement is ongoing. According to the progress summary report for 2015, by the end of Year1, registered members numbered 162; at the time of this report it had reached over the 700-member mark. The original SOW placed in the contract creating ECCN defined the targeted members for ECCN as both concerned USAID staff and Implementing Partners (IPs) working with EiCC programming. However, in KIIs with informants inside E3/ED in Washington, our team heard a clear expectation that now USAID staff working with EiCC programming are the intended primary focus/recipients of ECCN’s products and services. As one key senior E3/ED staff member put it, “We are not going to get away from USAID staff- specific professional development… If USAID staff don’t know how to apply/understand those good practices within USAID and write [them] into new projects--then our 4 These are not an exhaustive list of progress markers; instead these are overarching, summative markers most oft discussed in the reports reviewed. Conclusion: The ECCN ST has made steady progress advancing all five objectives defined in their contract. Outreach, communications, research, and professional development programming are evolving and may benefit from further clarity and consideration. • Registered ECCN members increased from 162 members at the end of Year 1 to 701 members at the end of Year 3. • The ECCN Resource Repository has steadily grown, and with 268 separate items now; approximately 25% original content and 75% from othersources. • Survey responses showed the conflict sensitivity checklist and RERA are the most-utilized resources; the least-consulted resource was costing guidance. • Some informants noted ECCN research resources, notably ECCN-produced reports and articles, were not always practitioner-oriented or easy to draw on for field staff. • While ease of access and utilization of research and resources are positive across respondents, changed practice to-date is less positive. 11 IPs won’t do it… we have to change our own practice in-house before we can ask IPs.” On the other hand, ECCN staff, and to an even greater degree, members of its SG and AG, sometimes see ECCN’s mission and audience as including the broader network of IPs (which several SG members represent), other EiCC-focused agencies, and their networks. These sometimes- differing viewpoints and priorities on whom and what ECCN is targeting still seem to need sorting- out and clarification. This point is further discussed under Evaluation Question 3. 2.Build and disseminate evidence for thefield Sub tasks for this objective, as laid out in progress reports and workplans, included, but were not limited to: 1) develop reference documents and M&E tools related to school safety; 2) execute a field based comparative analysis of alternate education programs; 3) analysis of existing USAID goal 3 programs to inform must have indicators to capture school safety; 4) produce 2 quick win documents or tools of immediate use to USAID and/or other USAID/ECCN stakeholder groups; and 5) define what constitutes good evidence. The ST has made substantial progress on this objective and both program reports and KIIs reflected that outputs to the network have increased steadily in number and variety, going from a monthly newsletter to several new tools, an expanding, cross- referenced, searchable repository of research and models for EiCC,5 webcasts, blogs, face-to- face and online training courses, TOCs, and EiCC program evaluations. One area that appears to be lagging in progress is in the provision of research, and how such research is presented. From the perspective of E3/ED, the expected progress is not currently prioritized highly enough by the ST, noting in informant interviews that output deadlines in this area have sometimes been deferred, also witnessed in progress reports. From a field perspective, we heard from some informants that ECCN research resources, notably ECCN-produced reports and articles, were not practitioner-oriented or easy to draw on for field staff who have limited time (and possibly, limited academic background) to take it on board as presented. For example, in our own review of three recent ECCN-produced research reports6 ranging from 22 to 69 pages in length each, we found no executive summary or other easy guides for practitioners. 3.Test, disseminate and use guidance, tools, metrics, and TOCs; and 4. Apply tools inC&C Progress on these two parallel objectives has advanced in unison, in that much of the original content that ECCN has developed for dissemination was tested in the field as it was developed (RERA, evidence gap maps, for example) before its broader dissemination took place. RERA has been the most successful and it has now been included as a deliverable on recent EiCC activities/projects and is recommended by E3/ED. Website analytics show that hits on the website RERA page have consistently gone up and have remained high. The holistic approach of RERA began creating the conditions for cross-sectoral conversation at least within USAID. Evidence gap maps, the conceptual framework and assessment tool on safe learning environments, and TOC cataloguing of various interventions were also cited as especially useful. The Accelerated Education 5 At the time of our research a 2.0 version of the ECCN website was under development and promises to offer users expanded functionality. We were not able to see it function during our research period, however. It should go live before the end of FY 2017 6Alternative education in the DRC (Final research report)”; “An analysis of theories of change in USAID solicitation for education programs in crisis and conflict affected environments”; “Analysis of indicators used in USAID education projects in crisis and conflict environments.” 12 Working Group has also been successful in creating important resources, including a literature review and case studies. Additionally, webinars/webcasts were cited as being particularly useful and analytics show that attendance for webcasts remains consistently high. Further, the ECCN website was launched in 2015 with just 20 pages of content, and at the time of this evaluation, it was reported to have close to 450 pages for resources, blog posts, events, news, bulletin board, and public discussion boards--a significant overall growth in content.7 The resource repository has also steadily grown, and with more than 250 separate items now, features approximately 25% original content and 75% from other sources overall. Blog, newsletter and webcast content is nearly 100% ECCN-created or organized, whereas other resource areas— vetted research materials for example— represent about 10% ECCN-produced and the rest drawn from other sources.8 Resources are organized along the lines of the five work streams of the project, with cross-cutting categorization available by region/country, resource type and stakeholder type. “Vetting” of such external materials seems to be the process of the expertise on the team reviewing the sources for their usefulness and application to the ECCN TOC--nearly 93% of posted content currently has been vetted by the ST.9 While members who answered the survey question about accessing findings or reports overall was relatively high, only slightly more than half of the respondents chose to respond to this detailed question. This may indicate that the balance skipped the question out of lack of interest or use to date of this specific resource. The findings from the member survey reflect somewhat limited use of several other of these tools--notably costing guidance. This may be a tool with a very narrow application to few users, however. (For graphic illustration of the above discussion, see Figures 3a and 3b below.) 5. Develop, implement and assess training Subtasks for objective 5 include, but are not limited to: 1) develop, implement, and assess Professional Development (PD) for effective education program management in USAID G3 countries; 2) develop, implement, and assess trainings (e.g. On TOC and RERA); and 3) knowledge sharing and roundtable events. While the process of development and implementation of training (also called “professional development”) receives significant support through the levels of LOE and funding provided to the ST10 as outlined in the contract with EDC in section C.6 (three current ST members reported working on various aspects of training support in our interviews with them), this objective’s progress is less than some of the other areas. This was attributed by team members in part to the realization that if field- based, face-to-face training is to be developed, it must be carefully tailored to the context and the audience, rather than adopting the initial concept of a generic training approach to EiCC that could be universally applied. The challenges in this process to date are discussed fully under Evaluation Question 2. 7 As of July 26, 2017, EDC shared the following information: 297 Resource pages, 58 published public events, 36 published blog posts, 37 news pages, 31 published resource pages, 30 bulletin board, 5 public discussion boards and 4 hidden ones 8 Source: https://eccnetwork.net/repository/ and https://eccnetwork.net/blog 9 Source: https://eccnetwork.net/repository 10 The evaluation team did not review contractual LOE. Instead, these findings and conclusions are based on what was reported in KIIs and the survey 13 Overall Progress on TOC ECCN’s primary aim is “to change the practice of designing, managing, implementing and evaluating education programs in the world’s most challenging and volatile environments.”11 To this end, progress is determined by monitoring ECCN members’ degree of uptake and use of opportunities, concepts, evidence, tools and guidance across each of ECCN’s five work streams. In other words, the ECCN TOC is based on three components that build upon each other: 1) ease of access; 2) degree of utilization/application; and 3) changed practice. Ease of Access In KIIs with concerned USAID staff, some field-based informants said they rely on ECCN’s web￾based resources. Others told us that while they were aware of them, they did not seek them out in any systematic fashion. And a few mentioned that they were unaware of ECCN as a resource. The member survey responses,12 which slightly overlap but go beyond the KII field interviewees, reflected that among the 60 members responding (of whom 34, or 57% of those respondents, said they were USAID-affiliated), their access to ECCN’s suite of resources varied by specific type: the checklist for conflict-sensitive programming and RERA--both strongly recommended tools for new EiCC programming--were accessed by over 75% of all respondents (and approximately 80% of USAID- affiliated respondents). Other ECCN resources were accessed somewhat less, with some modest fluctuations between the full group and USAID-affiliated respondents. Further, if the 97 respondents taking the time to respond to our survey (which demanded approximately half-hour of their time for those who completed it) might be considered a proxy indicator of “engaged member” among the member network, it was a noteworthy finding that of the 83 survey respondents offering their experience profiles, 56% indicated that their experience in C&C programming was four years or less (and among USAID-affiliated staff, that number was 58% with four years’ or less experience). KII respondents not regularly accessing ECCN resources13 were clustered in missions with large EiCC programming portfolios and robust programming resources (Afghanistan, Jordan notably; other large-portfolio missions were not in the KII sample). They indicated that they have in￾country resources for support and advice, often collaborating with and relying on other international agencies doing work similar to USAID for such needs. Key informants in the large missions we interviewed, on the other hand, were all individuals well-seasoned in EiCC programming over time, and several had served in more than one Mission stressing such programming. In response to the survey question about ease of opportunities to engage, 87% of respondents found it somewhat or very easy to be able to engage and access ECCN resources. Access is specifically available through international workshops, conferences, webcasts, blogs, webinars, newsletters and other events and trainings. Some KII respondents cited that limiting the login requirement to access a smaller portion of resources on the website as a step that positively impacted the ease of access. Figures 2a and 2b below show survey responses for “Have you accessed (specific) programming 11 ECCN Progress Monitoring Report, 2016 12 Detailed breakdowns of these responses can be found in Figures 2 through 13 below. 13 Or even knowing much about them, as some missions’ access to the ECCN website is blocked by their embassies for security reasons. The ECCN Team is aware of this issue and reported that they were working to address the problem 14 resources?” A majority of respondents had accessed each programming resource, and a minority had not accessed and did not plan to access resources. Most narrative comments from individuals who had not accessed resources showed that lack of relevance and time limitations were the biggest factors curtailing their access. Several survey respondents have not accessed training resources (see Figures 5a and 5b below), and they again cited limitations of available time as the major impediment; also mentioned were resources needed and accessibility. The monthly newsletter and website were selected as the most common source for reaching reports and findings. The following data figures graphically present specific findings from the perspective of the member network survey respondents. As pointed out earlier, these data reflect the opinions of a limited proportion of the network overall (n=97 of the 623 members receiving the survey) and only 67 of the respondents answered all questions asked. Response rate per item is noted in each figure, along with sub-illustrations of responses by USAID-affiliated respondents. Figure 2a: Respondents Access of Programming Resources Figure 2b: USAID-Affiliated Respondents Access of Programming Resources 15 Figure 3a: Access of Research Findings/Reports Figure 3b: USAID-Affiliated Respondents Access of Research Findings/Reports Figure 4a: Respondents’ Methods for Accessing ECCN Materials 16 Figure 4b: USAID-Affiliated Respondents’ Methods of Accessing ECCN Materials Figure 5a: Respondents’ Access to ECCN Training Opportunities Figure 5b: USAID-Affiliated Respondents’ Access to ECCN Training Opportunities 17 Figure 6: Reasons Respondents have not Accessed Trainings Utilization/Application: The figures below show survey responses about respondents plans to apply programming resources, lessons learned from trainings, and research findings and reports. A majority of respondents have either already applied or plan to apply lessons learned from each category of resources and opportunities available through ECCN. The most-utilized resources are standard requirements or tools for new EiCC programming (Conflict sensitivity checklist used by 35% of respondents and RERA, used by 30% of respondents); the least-consulted resource was costing guidance, with 11% of respondents having reported use of this resource. Figure 7a: Application of Programming Resources 18 Figure 7b: Application of Programming Resources by USAID-Affiliated Respondents Figure 8a: Respondents’ Application of Training to work Figure 8b: Application of Training to work by USAID-Affiliated Respondents 19 Figure 9a: Application of Research Findings/Reports to work Figure 9b: Application of Research Findings/Reports to Work by USAID-Affiliated Respondents Changed Practice While ease of access and utilization of research and resources are positive across respondents, our findings on responding members’ changed practice do not reflect so positive an outlook to date. One interviewee summarized the challenge: “We are sharing resources but not necessarily getting to change; changed practice/demonstrable impact will take some time.” Survey responses below show that for the majority of respondents, programming resources, trainings, and research from ECCN have provided “occasional inputs” into their work. A greater number of respondents credited training activities for having a longer-term impact. 20 Figure 10a: Influence of ECCN Resources Figure 10b: Influence of ECCN Resources on USAID-Affiliated Respondents Figure 11a: Influence of ECCN Resources 21 Figure 11b: Influence of ECCN Resources on USAID-Affiliated Respondents Figure 12a: Influence of ECCN research on work Figure 12b: Influence of ECCN Research on Work of USAID-Affiliated Respondents 22 Figure 13a: Areas in which Learning has been Applied Figure 13b: Areas in Which Learning has been Applied by USAID-Affiliated Respondents 23 Recommendations 1) Focus field-based engagement to align with needs. Going forward seek ways to emphasize engagement in the field (if the ST and USAID agree that this is the direction they would like to pursue for ECCN) and consider whether outreach should especially target field-based staff (or education teams) with limited expertise and resources, and not focus on being a resource for all. For example, such new staff could be matched with a network mentor in EiCC programming for follow-up post-arrival at post. This may help increase the level of changed practice affected by ECCN outputs. 2) Prioritize and tailor research outputs for easier access and utilization. Advance progress on research outputs by making them a higher priority and adhere to set deliverable deadlines. Additionally, make research reports more reader-friendly and accessible for the field-based practitioner through executive summaries and/or quick￾reference guides that could assist those who find the full reports too long or too complex for their available time and programming needs. 3) Identify focus areas/work streams and ensure adequate resource allocation. Re-consider whether equal attention is needed for all five original objectives. Should some be de- emphasized? Subsequently, continue to monitor LOE apportioned to different project functions and consider strategies to create flexibility in staffing levels and expense caps by category/function. Continue to assess whether the current increased staffing for communications and outreach is sufficient for anticipated needs, and whether other areas may be de-emphasized to offer the additional support that may be needed to accomplish this. 24 2. What successes and challenges has the ECCN ST encountered in carrying out their five work streams and meeting their objectives? A team of nine experienced professionals in EiCC comprise the ECCN ST, which supports ECCN with the ultimate aim of supporting “acceleration of the impact and scale of core and supplementary education programs in crisis and conflict environments.”14 Objectives (also referenced as work streams), which are output-focused, run across the outcome- focused priority areas. To answer this evaluation question, the Evaluation Team focused on the five objectives outlined previously; we identify and discuss below the successes and challenges encountered by the ECCN ST with reference to these five work streams/objectives. Findings and Conclusions The following findings and conclusions are grouped by “Challenges” and “Successes”, and report more fully on some points presented under Question 1 above. Challenges 1. Staff turnover Early in the contract period of performance, there was some unexpected staff turnover. Finding and onboarding replacements delayed progress, specifically on Objective 1 activities designed to lay the foundation for the CoP. In 2014, the EiCC specialist was not available and USAID did not approve the first proposed alternate; hence the position was not filled until the final quarter of 2014. Similarly, in the first quarter of 2015, the Communications Specialist left and this led to delays in work on the proposed Communication Platform Plan and Marking and Branding Plan. The Senior Researcher also left in the final quarter of 2015, and this negatively impacted progress on establishing the ECCN learning agenda through outreach and information gathering. 14 July-Sept 2016 Work Plan, page 1 Conclusion: The ECCN ST has consistently adapted to overcome challenges. Less than ideal communications and coordination between EDC and E3/ED, especially in programmatic vision, continues to be a challenge in meeting objectives. • Successes: Adaptive management; growing the COP; improved timeliness; interactive website; tools and resource development; management by ST • Challenges: Staff turnover and delays early in the project; lack of clarity on ECCN’s mandate and coordination between E3/ED and EDC; outreach and engagement; SG and AG engagement • Lack of time, resources, and awareness/availability were cited as primary reasons for limited active participation of ECCN community members in knowledge sharing and training opportunities offered through ECCN. • 44.9% of total respondents and 48% of USAID affiliated respondents reported finding the COP very well managed 25 2. Lack of clarity on ECCN’s mandate The relationship between E3/ED and EDC continues with mutual respect, support and collaboration, but differences on perspective and priority continue to be reflected at this midpoint. The relationship between E3/ED and EDC faces two challenges: 1) clarity in terms of approach/role/scope and 2) coordination. KII responses indicate that the initial intent of the contract design has the CoP as secondary, or as one key USAID affiliated interviewee central to the strategy’s conceptualization put it, the CoP was just a “means to obtain the resources needed.” Another USAID interviewee stated “(the) rationale for the contract type is that it was the easiest and fastest thing to do.” ST members predominantly voiced concerns that USAID seemed to envision ECCN, and to an extent still does, as a technical extension of the E3/ED office. Nevertheless, as the CoP mechanism was built into the contract, there is an impetus on the EDC ST to also conduct activities such as outreach, communications, publicity, and management of consultants to make the CoP successful. As such, an oft repeated concern across KIIs was that the contract currently does not budget enough LOE and resources for EDC to be able to successfully carry out both tasks. The desk review and KIIs also revealed that early in the project, some planned approaches and activities suffered from a lack of clarity in terms of what is expected from the ECCN ST. There was confusion about EDC’s proposed Contractor Teaming Agreement with Morgan Borszcz Consulting, which coincided with a change in the USAID Contracting Officer for the project. This issue remained unresolved for at least two quarters. Similarly, it took two quarters for the EDC team to clarify whether work on the Secondary Analysis and Results Tracking (SART) contract mechanism could begin; eventually this work was shelved for the time being as SART was focused on Goal 1, rather than Goal. The planned geo-mapping activity was also eventually dropped, as there was lack of clarity aboutits use, and insufficient resources allocated to implement it. Access to the web design labor category was also delayed due to contractual issues. Finally, there was also substantial uncertainty of the overall purpose of the ECCN online resource repository, who it's targeted audience is, and how this audience would engage with the repository. ECCN was also expected to work with USAID’s education technology specialist to further develop ECCN’s support for the INEE Technology Task Team and related efforts (Objective 4). After one or two miscommunications with the individual and other non- ECCN organizations also working with the specialist, ECCN requested further clarification of their role from USAID, after which USAID developed a plan and shared it with ECCN. The findings above led the evaluation team to conclude that greater clarity is needed with regards to EDC’s mandate. Evidence gathered and reviewed by the evaluation team indicates that dual roles are expected of the ST: 1) content/technical drivers and 2) outreach, engagement, and communications managers for the CoP. However there seems to be a perception, predominantly amongst the ST, that resources, especially labor, are under-budgeted for EDC to be able to satisfactorily complete both these tasks. Progress reports, and survey and interview responses indicate that EDC is performing very well on creating technical resources and tools like the RERA, but they are having less success in communications and engagement. An EDC ST member acknowledged that “we are getting a lot of the technical work done, but not so much on the engagement.” Another ST member mentioned that there were many technical things that EDC spent time on, but no one knows about them because EDC did not do a webcast or event focusing on them. This is related to the issue of limited LOE discussed above, and is also impacted by the 26 lack of clarity in the expectations from EDC- essentially it seems it is not entirely clear whether EDC is supposed to take on both tasks, and if so, then how to successfully do so within the contracted LOE. Contractually, the type of award made to EDC for implementation of ECCN is an acquisition contract which highly pre-defines staffing, funding lines and outputs because of the type of award mechanism chosen. This acquisition contract: • dictates specific staff backgrounds and worklevels; • closely defines all staff positions as “key” (leaving the implementer no options to redefine staffing according to need as it evolves); and ties funding levels to a time-and- materials billingbasis. The contract as written does not seem to square with the intended objectives and outcomes of the project, described by USAID in its contract scope, of working to “convene and collaboratively structure a program of work...and work with the Community of Practice…” The collaborative structuring expected, by its implied flexibility, is difficult to generate when all staffing and funding levels are predetermined. 3. Sustainability/Long term vision Related to lack of clarity regarding EDC’s role and mandate is the issue of the long-term vision for ECCN. The Evaluation Team was unable to find any discussion about long-term plans in the documents provided by USAID beyond the current contract. Interviewees across the ST, SG, and USAID expressed concern about this. 4. Outreach and Engagement As discussed above, clarity regarding EDCs role and ability to conduct outreach, engagement, and communications seems to be lacking. Engagement is an essential component of a successful CoP, but all data collected (documents reviewed, KIIs and survey findings) indicated that there have been many challenges with this (engagement is further discussed in detail under EQ3). One of the problems with below-par engagement is discussed above in that EDC is not always able to satisfactorily push out everything that it is planning and developing. As such, ECCN members and USAID stakeholders may not be obtaining the full benefit of ECCN, as lack of outreach and engagement makes it difficult for stakeholders to have complete knowledge of what all is available for their use. 41% of total survey respondents (n=79 for this survey question), and 43% of USAID affiliated respondents, said they have never shared knowledge through active participation in the ECCN. Reasons cited included “don’t feel sufficiently engaged,” “didn’t know I could,” “would love to, but how?,” and “not sure how to share knowledge, (it) does not feel that users are invited to share knowledge.” 21% of total survey respondents, and 24% of USAID affiliated survey respondents were unaware of research findings and reports available through ECCN. Similarly, 37% of total respondents, and 30% of USAID affiliated respondents were unaware of training opportunities available through ECCN. In terms of Objective 5, professional development and training is measured in terms of the number of people reached/trained and so on. However, interviewees, particularly from USAID Missions, lamented that ECCN is not reaching enough people inside USAID. Even though the main objective is to train USAID staff and most courses are USAID-specific, often very few USAID individuals have attended. 27 Outside the September 2015 field workshop in Abuja, Nigeria, which included 60 participants, the participants in subsequent trainings--which after Nigeria have targeted USAID staff only-- have been relatively low: in Ethiopia in May 2016, ECCN trained 12 USAID staff (10 field-based from 6 missions, 2 Washington-based); the Bangkok event in November 2016 was attended by 11 (9 field￾based from 6 missions and 2 Washington-based), and at the May 2017 workshop for Latin America/Caribbean education staff, hosted in Washington, DC in order to piggy-back on a larger regional meeting, the attendees numbered 31 (26 Washington-based staff among whom 9 were from the US Department of Labor; 5 field based staff from LAC missions).15 Some of the challenges in getting USAID stakeholders to participate is that they do not have the time or their travel for training is not allowed or covered, and so it is difficult for them to take out the time and to justify engagement with ECCN trainings. One ST member stated, “it takes a ton of time to prep for those (workshops); I wonder if the impact that they have is worth the input we give...the Bangkok workshop had 15 people but it took so much effort. And, maybe their time could have been used on other things.” 61% of total respondents (n=44 for this question) and 46% of USAID affiliated respondents said they had never accessed any training opportunities available through the ECCN; the primary reasons cited were: lack of time, lack of resources, and lack of awareness/availability. ECCN was also unable to build a research/knowledge-sharing group due to lack of interest, as members responding who are practitioners said they did not have time to read so much. Comments in KIIs and the survey also indicated that some people do not engage because they are not fully aware of the many modes available for engagement. For instance, numerous individuals did not know that they could directly email others in the CoP. 5. Limited LOE As mentioned above, perceived lack of sufficient LOE was a persistent concern raised by the ST in KIIs, and also identified as a challenge across all objectives in quarterly and annual reports. Insufficient LOE for the Communications Specialist role has been cited as particularly disadvantageous as it affects recruitment of new members, as well as engagement of current members. Recently, however, the Communication Specialist’s LOE has been increased. Numerous ST interviewees suggested that they were working more than their contractual requirement. They mentioned that even though everyone was part-time, most staff were working additional hours as there was so much to be done. Even with an EDC ST willing to take on more work than the contract allocates them, there still seems to be a shortfall in the LOE required to achieve activity goalposts by set deadlines. For instance, in the final quarter of 2015, it was reported that the development on the ECCN online repository was going slower than expected due to low staff bandwidth and uncertainty of its overall purpose. Revisions to the learning agenda were also not completed on time due to internal ST capacity constraints. 6. Coordination between E3 and EDC In quarterly and annual reports and interviews, there were various mentions of delays in terms of obtaining USAID approval for EDC plans and deliverables. For instance, in 2015 the internal 15 ECCN quarterly and annual reports 28 USAID stakeholder baseline report was submitted but COR approval did not come through in time. Additionally, in 2014, due to delays related to the EDC contract, the web development labor category was not activated, negatively impacting outreach and communications (objective 1). Similarly, in the second quarter for 2015, the PMP was not developed, as USAID did not provide formal approval for the work plan. EDC did, however, create a draft PMP anyway. USAID’s approval of the scope and definition of outcomes to be costed was also delayed, affecting the timeline for costing. Furthermore, USAID’s review of the PMP analysis document took longer than expected and subsequently delayed distribution and presentation to the wider ECCN. In the final quarter of 2016, approvals were again delayed on the USAID side. Interviewees and progress reports acknowledged that it is difficult for both the Program Director and the COR to consistently make time/coincide for these calls. They now are considering the possibility of shifting to monthly in-person meetings instead. USAID E3/ED staff, in KIIs, also acknowledged the issue of coordination and they posited that EDC may be taking on too much and USAID would instead prefer them to take on less so that they can manage within the provided LOE. The EDC-USAID coordination challenge has also manifested itself in terms of delayed sharing of data and documentation. Review of goal 3 TOCs (objective 3) has been hampered due to difficulties in obtaining RFA, RFPs, Task Orders and APS documents. The challenges between E3/ED and ECCN mentioned above were acknowledged across interviewees in USAID and ECCN, however there was also a recognition that “the team is starting to coalesce...we didn’t imagine the amount of time that it would take to coalesce.” However, there is still not complete clarity about the identity and goal of ECCN; as one interviewee put “(it is) Not clear to me what people feel success would look like. Being a resource to the world is too broad.” 7. Limited Steering Group and Advisory Group Engagement This challenge was raised in progress reports and mentioned during KIIs. The Quarter 3 progress report mentioned that the previous SG meeting was not well attended because the meeting occurred after very short notice. Although virtual and alternate attendance options were made available forthis and other SG meetings, they were not very interactive. Generally, interviewees mentioned that they had had difficulties in getting SG members’ attention and input on ECCN work. This challenge exists broadly among most SG members and is noted through lackluster participation and feedback to ST requests for feedback regarding the tasks at hand. On the other hand, the SG mentioned that it does not get enough updates on progress and challenges from EDC. Some respondents also attributed their lack of engagement to their own busy schedules, and other commitments. While there are challenges in engaging the SG, KIIs mentioned that the relationship is constantly evolving positively. The AG, however, is still not faring well as its members are minimally engaged and in some cases due to their positions and responsibilities, not as available to ECCN as had been hoped. 8. Initial delays in planned activities/tasks Identification of the AG was initially delayed. Once identified, there were further delays in AG meetings (Objective 1). Similarly, quarterly reports and KIIs highlighted the delays in drafting indicators on equity (Objective 3). Other delays such as in the creation of the learning agenda, 29 costing indicators are discussed above. Delays were more widespread at the beginning of the project, and recently timeliness has improved and delays have been minimized. This is discussed in further detail below. Successes 1. Adaptive Management (All objectives) EDC has been flexible in managing ECCN and has rapidly adapted when there have been delays or other issues that may hamper ECCN’s progress. Some manifestations of EDC’s nimbleness include: • the switch from objectives to themes to allow for more robust programming and monitoring; • in the face of a delay in accessing the web design labor category on the contract, EDC identified a web developer who also met the criteria of the Communications Specialist and was able to bill a limited number of hours to this category, which allowed them to getsome urgently needed web developmentunderway; • when the final PMP was delayed because two rounds of commenting took longerthan anticipated, EDC built in more rigorous internal procedures to ensure quality and they allowed longer turnaround time from submission to USAID to final release for all future products. One EDC ST interviewee aptly summarized EDC’s adaptive management approach: “Just the way EDC works - the proposal is really secondary, what's important to consider is what happened after the project was initiated.” The team is inventing as they go, based on successes and obstacles they are facing on the way. Progress on Timing The ability to plan and execute activities realistically and within accurate timelines has increased each year, as reflected in both ST reports and our interviews with the team. They attributed this progress to lessons learned and trial-and-error experience. Member identification and network growth (Objective 1)/ building a CoP Even with a slow start, unanticipated staff turnover, contract-related delays and engagement challenges, the ECCN ST has been successful in generating a steady increase in the number of members accessing and using ECCN resources. Project quarterly and annual reports reflect consistent increases in the number of individuals signed up for the newsletter and in the number of blog posts. One interviewee mentioned, “one of the major successes of ECCN was its availability as a networking platform.” The website is also more interactive than before and Overall, ECCN is still seeing a steady increase of member growth across all its web-based platforms. Successes in addressing Objective 1 are discussed in further detail under EQ1 above. 2. Tools and Resources Development (Objective 4) Tools development was cited as a resounding success across KIIs and the survey. One interviewee summarized, “They (EDC) have done a really good job at producing tools and research that seem to be pretty demand-driven; which is not always the case.” Further discussion is presented under EQ1. 3. Good, knowledgeable, committed, and responsive ECCN ST (All objectives) 30 Survey and KII responses consistently lauded the EDC team for their availability, responsiveness, and general willingness to listen and help. The EDC team was called a “terrific team,” “very helpful,” and interviewees recounted receiving “immediate assistance from the ECCN ST.” Figure 14a and 14b below shows the survey responses for “How well do you feel the CoP is managed?” 44.9% of total respondents and 48% of USAID affiliated respondents reported finding the CoP very well managed. Figure 14a: How well do you feel the CoP is managed? Figure 14b: How well do you feel the CoP is managed (USAID-Affiliated Respondents) Recommendations 31 1) USAID E3/ED should clearly define and communicate its expectations regarding ECCNand EDC’s role to the EDC ST. USAID needs to clearly define, iterate, and reiterate what they envision as EDC’s role and ECCN objectives and future. As discussed above, EDC and USAID may not always be on the same page regarding the vision and approach for ECCN, and this manifests itself in a myriad of challenges ranging from LOE exhaustion to delays and confusion. As such, the evaluation team recommends that USAID E3/ED and EDC ST have candid conversations16 about expectations and abilities, and emerge with a concrete and specific plan for the future of ECCN and the roles EDC and USAID are to play. If there are differences in this vision, USAID needs to take the initiative on making final decisions as the manager of this contract. Overall, there needs to be more open and regular communications between USAID and EDC, and the chain of command needs to be more clearly defined. USAID should provide more clarity about its role in leading this project or in allowing the project itself to be more nimble and demand-driven. KIIs with USAID and EDC reflect that these issues are already being thought about, so the evaluation team recommends that this issue be further articulated and addressed head-on. This may help avoid confusion, setbacks and delays in the future. 2) Revisit the project budget and LOE allocations. Even if the previous recommendation is heeded, it may still be that LOE will be a problem and for that the budget and LOE allocations will need to be revisited. As discussed, the requirements of the acquisition contract used to award ECCN call for alternative approaches to address some of the unmet needs and priorities of the project. 3) Merge the AG and SG into one entity. Consider whether there is a true need for both the AG and SG. Consolidating purposes and merging the bodies can create efficiencies and ensure increased cohesion of decision￾makers and strategic influencers. 4) Streamline demands/asks of SG members. As discussed, SG members indicated that they weren’t always sufficiently engaged. We recommend that the ST should produce and provide the SG with short, easy-to-read summaries of progress on a regular basis so that they can keep up with the many activities and plans of ECCN. It is, however, also important to note that USAID staff and SG members have many things on their plate, and being on the SG is voluntary. Hence, to optimize SG engagement, the ST should streamline what is sent out or asked of the SG. Rather than ad￾hoc demands, ideas that are fully developed or structured, and have a clearly defined request, should be presented to the SG. Shorter meetings should also be considered. Overall, the SG should be better informed of updates, but their time should be used efficiently. 5) Reconsider the strategy and approach to ProfessionalDevelopment. USAID staff attendance at field-based PD events overall has been sub-par to date; rather than continuing to allocate substantial ST time and resources on developing sparsely attended trainings, EDC might consider instead a training-of-trainers approach. This could help multiply the audience for trainings in an efficient manner whereby direct 16 The evaluation team is aware that previous discussions around these topics have occurred, but the evaluation team observations indicated that clear and targeted conclusions and follow up were lacking to such conversations 32 attendees could take back lessons learned from EDC trainings and be expected to pass them on formally to their EiCC colleagues and IPs. Motivations for individuals to pass on trainings would need to be considered for this, and these can include a fixed-cost subsidization of costs for those who conduct training sessions over a given time period after they attend a training. 33 3. What is the nature and extent of member engagement and how is it supported by the ECCN ST, Steering Group and Advisory Group? The ECCN contract to EDC stipulates that the project shall develop "a CoP that shall include key stakeholders including donors, researchers and practitioners, and will develop a process for collective learning and knowledge dissemination in the domain of Goal 3 of USAID's Education strategy." As a CoP, member engagement is a critical component of the ECCN. For the purposes of this evaluation, engagement is broadly defined to capture information sharing and relationship building. This includes dissemination of resources and knowledge; new knowledge development; joint activities, discussions, and dialogue; collective learning; and building strategic partnerships. As such, engagement is not limited to a unidirectional process from EDC to ECCN members, but is rather intended to include that as well as multi-directional actions. This broad definition of engagement has been distilled from language in the ECCN Activity Contract, the ECCN Progress Monitoring Reports, and the ECCN Communication Strategy documents. Engagement Opportunities The ECCN recognizes that there are both “push” and “pull” aspects to fostering member engagement. Throughout most the first half of the contract, years 1 and 2, the ST has largely focused on the push side of engagement. This refers to the development and dissemination of technical content to the member community. However, starting in year 2, the team began emphasizing the pull side of member engagement through various initiatives including the Partner Initiative, member presentations at the CIES conference, the establishment of the Bulletin Board,17 and development of the Working Groups. To date, the engagement opportunities offered through the ECCN can be categorized in three ways: 1. Electronic/online engagement; 2. In-person engagement; and 3. Combination ofelectronic/online and in-person engagement. The electronic/online engagement opportunities include the website, 17 The Bullet Board was an online discussion forum feature that was introduced to the website in September 2016. Conclusion: Although member engagement is positively evolving into a more multi-directional process that combines both push and pull approaches, it is still primarily unidirectional, with the CoP passively receiving quality and curated information from ECCN. Field representation and engagement continues to be a challenge. • 52 of 78 survey respondents felt that the ECCN offered either adequate or many opportunities for engagement. • 51 out of 78 survey respondents reported regularly engaging through the newsletter, followed by website (30 respondents), webcasts (15 respondents), and face-to-face events (10respondents). • The newsletter has an opening rate of 35-40%, with a 30% click through rate. • 20 of the key informants mentioned that the ECCN is US-centric, while less than five individuals mentioned that IP and/or field involvement is improving. Of the 20 Mission staff contacted for KIIs, three declined to be interviewed citing lack of any awareness of ECCN. Additionally, of the six individuals interviewed, half of them reported only limited familiarity with the ECCN. • Multiple KIIs mentioned the lack of field representation in the SG. 34 monthly newsletter, twitter, blogs, and webcasts. The in-person engagement opportunities include trainings, conferences, and team/Group meetings. The combination opportunities include working groups/task teams that can engage both through online content available and in-person workshops/meetings as necessary. The ECCN member survey found that a majority of survey respondents (52 of 78 who responded to this item) felt that the ECCN offered an adequate amount or many opportunities for engagement. Of the different engagement opportunities available, the monthly newsletter has the most engagement as defined by access. The ECCN web metrics show that the newsletter has an opening rate of 35-40%, with a 30% click through rate. 18 Similarly, of the 78 evaluation survey respondents for this question, the newsletter ranked highest in the medium of engagement category with 51 respondents selecting “regular” for the frequency of engagement. This was followed by website (30 for regularly), webcasts (15 regularly), and face-to-face events (10 regularly). Type of Engagement The ECCN member community has doubled almost every year as reported by EDC, ending 2016 with 701 members and with website users growing steadily to 2,017 identified users (who may or may not be registered members) at the end of 2016.19 However, extent of engagement is difficult to assess given that members do so in two ways: 1. active engagement and 2. passive consultation. During the evaluation’s KIIs, many interviewees reported that they are consumers of information rather than active participants or contributors to the ECCN community. This is further corroborated by the ECCN’s metrics of number of posts on online forums (see Table 3 below). The ECCN is working towards facilitating more active engagement as mentioned in their communication strategy: “For USAID ECCN’s Communications Strategy to move the CoP from passively receiving quality and curated information to truly wanting to engage and participate, USAID ECCN needs to serve an unmet need in the EiCC community.” Table 3: Member Content-sharing with ECCN in 2016 Bulletin Board Postings 16 members (direct posts and requested posts) Event or Materials advertisement via website or newsletter 5 members Posting in Online Discussion Forums 2 members Resource posting on Website Document or tool 6 members PowerPoint Presentation 23 members Our evaluation survey indicated that a slight majority of respondents (42 of 74) shared knowledge through active participation. However, it is worth noting that it is likely that ECCN members who chose to complete the survey are more likely to actively engage with the ECCN. According to the Year 3 workplan, the vast majority of those respondents indicated that their preferred method of sharing 18 USAID ECCN Progress Monitoring Report 2016 19 Ibid. 35 knowledge was through direct interactions with other community members such as face-to-face events (58%), webcast participation or follow up (49%), emailing other members (39%), and participation in working groups (34%). This indicates that while the majority of the ECCN members engage with the community passively as consumers of the push strategy, there is a significant appetite among active members for more active engagement that can be facilitated through the pull strategy. Engagement Audience The target audience for the ECCN community membership has evolved both informally and formally over the course of the first 2.5 years of the project. While the ECCN contract included a wide range of stakeholders, both USAID and non-USAID, the project has evolved to focus more directly on USAID stakeholders, while not excluding non-USAID members. This has been articulated in the revised ECCN Communication Strategy (2016) through the listing of primary and secondary audiences. However, this distinction is not necessarily clear to the wider member community. A review of project documents indicated that USAID ECCN’s primary audience consists of USAID staff in missions and Washington, DC.; USAID IPs (organizational-, program-, and project-level); technical specialists in EiCC; and research institutions such as universities and think tanks. USAID ECCN’s secondary audience consists of: Bi- and Multilateral Donors; EiCC Project Staff Based in the Field; Local Grassroots Organization; and General Public. While this listing indicates a focus on USAID staff as the primary audience, it still includes a broader community. However, interviews indicate that from the USAID perspective, the primary audience is USAID staff, and all other audiences are secondary or tertiary. In addition, the evaluation found that to date, the ECCN audience has been largely US-centric. Twenty of the key informants mentioned that the ECCN is US-centric, while less than five individuals mentioned that implementing partner and/or field involvement is improving. One key informant mentioned that ECCN is a “DC beltway group.” This was further supported by the evaluation team’s scheduling and conducting KIIs with USAID Mission representatives. In addition to a limited response rate, of the 20 Mission staff who were contacted, three declined to be interviewed citing lack of any awareness of ECCN. Additionally, of the six individuals interviewed, half of them reported only limited familiarity with the ECCN. One potential explanation revolves around personal linkages that facilitate engagement. The evaluation survey found that the vast majority of respondents reported that they can engage with ECCN somewhat easily (n=34) or very easily (n=34), with only seven respondents reporting “not easily.” However, the qualitative responses provided for this question indicate that ease of engagement is largely facilitated by direct interaction with individuals, either through participation on a working group or communication with ST members. One respondent specified “somewhat easily because I am familiar with folks on ECCN. But if I wasn’t, it would be a bit of a black box.” While personal linkages can be global, interviews suggest that they are stronger within the US. The US-centric focus of the ECCN is also associated with lack of in-country/field representation on ECCN implementation teams. Multiple KIIs mentioned the lack of field representation in the SG, which advises ECCN strategy. While the logistical challenges of ensuring field representation were noted, multiple KIIs also questioned the utility of including field representation within the context 36 of the SG’s role, to serve as liaisons between the SG and their respective organizations staff and partners, including field staff and partners. Additionally, the lack of regional expertise in some professional training opportunities was also noted. While the trainings were cited as providing useful theoretical information, several interviewees noted the need to incorporate regional expertise that is available, but needs to be engaged. The ECCN ST has faced multiple challenges in engaging field and mission staff that can largely be categorized into three areas: institutional bureaucracy, time and resource limitations, and lack of awareness or need. The majority of interviewees noted the institutional bureaucratic challenges that are inherent to USAID and not specific to ECCN. As noted earlier, the need to navigate the appropriate approvals and channels of communication have hindered the ST’s ability to directly and efficiently engage with Mission and field staff. Cited examples include burdensome approval processes and technology restrictions. Two individuals based in overseas missions mentioned that the ECCN website is blocked on their official computers. Almost all interviewees and multiple survey respondents cited time and resource limitations that affect engagement. All mission representatives mentioned competing priorities for their time, which limits ability to engage with ECCN, unless necessary. Additionally, survey responses indicate that some Mission or field staff have limited engagement, particularly with trainings and in-person events because they cannot secure funding or time to attend. Lastly, in addition to lack of awareness of ECCN, multiple interviewees and survey respondents cited the lack of need to engage with the ECCN. “It seems like a solution in search of a problem” was one comment heard from a USAID overseas staff member currently based in a mission with its own well- resourced capacity for EiCC programming. Some Missions already have a set of C&C or education-specific resources, and therefore do not see a need to use ECCN to access or use those resources. Several individuals mentioned that they have been engaged with INEE, and therefore get the resources they need through that network. Several respondents also indicated that while they have not engaged with the ECCN, they know that the resources are there if they need them. Support from ECCN ST, Steering Group, Advisory Group The ST, SG, and AG support engagement through the provision of both content and operations. According to the 2016 Communication Strategy: Toward this end, the USAID ECCN CoP has a more specific goal: that its members share and jointly develop knowledge and tools for policies, programming and practice that improve equitable access to education for children and youth in crisis-and conflict- affected environments. USAID ECCN’s ST, including its communications team, supports this effort by serving as a conduit for and between the work of USAID and the USAID ECCN CoP. Given the focus on the “push” side of engagement in the first two years of the project – the ST’s output has largely been in developing and disseminating content. However, the evaluation survey found that when the ST was asked for assistance from the member community, they readily delivered it. Twenty of the 78 respondents choosing to comment on this question reported asking for assistance from the ST, and the majority reported being very satisfied with how the ST addressed their request. 37 The SG and AG support is part of the ECCN Year 3 workplan, with specific outcomes associated with their support – primarily Outcome 3: AG and AG actively engage with and guide ECCN.20 The Partner Initiative is one example of direct SG support for member engagement: Partner Initiatives will be led by partners, who assume primary responsibility for organizing and delivering the proposed activity. Partners will be supported by the ECCN ST both through the assignment of at least one ST technical liaison who may provide guidance and ideas, and also through support for communications, or logistics as jointly agreed and budgeted between ECCN and the Partner. (Y3 Workplan pg. 4) Examples cited include a planned webinar series on Social and Emotional Learning, the Equity Initiative/Conflict and Fragility Working Group, and a Field Training Workshop. Recommendations: 1) Intensify efforts to promote “pull” engagement strategies. The newsletter, which has a high open and click through rate, should be leveraged to gain feedback on the needs of the community- this can be achieved through strategies like embedding one to two question surveys about what the community would like to see addressed by ECCN; or by sending out targeted emails to those who opened the newsletter to solicit information about their needs. Additionally, rather than expanding the types of engagement opportunities, focus on active engagement through offering and encouraging participation in direct interaction activities such as face-to- face meetings and targeted and direct follow ups. 2) Facilitate communication and engagement amongst ECCN members. As discussed above, numerous survey and KII respondents were unaware of engagement opportunities, such as means to identify or contact other ECCN members. Even though the member list is available on the ECCN website, members were not aware of this. To achieve the CoP’s potential, and encourage multidirectional engagement, the ST should facilitate communication amongst ECCN members. An immediate step to address this could be to provide a summary of engagement modes available to members in the next newsletter. 3) Ensure that the ECCN implementation team and the USAID team involved with the ECCN contract agree about the details of the primary audience for ECCN. This should also be reflected in messaging with the broader ECCN community, including the SG and AG, for clarity of purpose and to facilitate understanding for activity structures, such as the professional development opportunities’ audience. Clarifying the intent and primary audience of the ECCN will help the broader community understand why certain decisions are made, while not excluding their participation in the community. This recommendation corroborates the earlier mentioned need for greater clarity, communication, and coordination between USAID and the ST. 4) Focus on engaging USAID Mission and other field-based stakeholders through direct interactions with key USAID individuals to broker relationships between DC and missions, as 20 ECCN Year 3 WorkPlan 38 well as within Missions/in-country, if this is the direction for ECCN that USAID and the ST agree on. Given that USAID staff are by nature better connected with other USAID staff and better able to navigate USAID communication channels, the evaluation team encourages the ECCN to leverage the existing relationships of both DC based and mission-based USAID staff to increase awareness of and engagement with ECCN. This could involve pairing Mission-based education staff with other education staff in other missions to facilitate mission-to-mission sharing and learning. These direct interactions will be helpful in increasing mission/in-country awareness of ECCN, as well as identifying and incorporating regional/national USAID-specific expertise in ECCN activities. 39 4. To what extent are the priority areas and the themes within the ECCN project relevant to USAID Office of Education and the ECCN member community? Starting with the Year 2 Work Plan, ECCN began the process to shift focus from structuring the project along the five objectives laid out in the initial contract, to a restricting around a thematic lens. While the Year 2 Workplan lists a number of thematic areas, the Year 3 workplan delineates a revised list of four priority areas with a sub-set of themes within each, that the project is now structured around. These include: • Priority Area 1: Learning Environments areSafer • Priority Area 2: EiCC Programming hasImproved • Priority Area 3: Equity is Enhanced • Priority Area 4: Local Institutions areStronger The reasoning behind the shift from objectives to priority areas were two-fold: 1. The objectives were largely process-oriented, not offering guidance on content and 2. A great degree of connectivity across the objectives meant that structuring by priority areas and themes was operationally more logical. Relevance to USAID The priority areas and themes within each were developed in conjunction with ECCN and the USAID EiCC team. As such, the priority areas developed are very relevant to E3/ED. In particular, the priority areas reflect the major outcomes outlined in the USAID Education Strategy for Goal 3: 1. Safe learning opportunities for children and youth are provided; 2. Crisis prevention efforts are strengthened; and 3. Institutional capacity to provide services are strengthened. As one KII respondent noted, the direct connection between the education strategy Goal 3 outcomes and the ECCN priority areas is reflected in the fact that every question received from mission staff always falls within one of the priority areas/themes. Conclusion: While the range of USAID and ECCN member community may interpret relevance in different ways, the vast majority of survey respondents reported that all four priority areas and the themes within them are very relevant. • A majority of overall survey respondents indicated that all priority areas were considered “very relevant” to E3/ED and the ECCN member community. • Priority area 1 (Learning Environments are Safer) received the most “very relevant” ratings and priority area 4 (Local Institutions are Stronger) received the least number of “very relevant” ratings from USAID affiliated respondents. • Priority areas 2 (EiCC Programming has Improved) and 4 (Local Institutions are Stronger) received the most “very relevant” ratings from IP respondents. • Most respondents found all themes within priority areas to be “very relevant.” Two slight exceptions were “Advanced program and activity costing guidance,” and “Knowledge related to ICT application in EiCC,” which had a significant proportion of respondents selecting “not relevant” or “somewhat relevant.” • The priority areas developed are very relevant to E3/ED because they were developed the USAID EiCC team and reflect the major outcomes outlined in the USAID Education Strategy for Goal 3. 40 Additionally, the relevance of the ECCN priority areas to E3/ED is bolstered by the agency mandates for education programming. For example, as noted in ECCN’s Year 3 Workplan in the justification for priority area 4, “USAID ADS 220 (2013) provides policy guidance to “support partner governments taking the lead in designing and implementing defined development strategies and managing their own development.” This policy implements the goal of USAID Forward to ‘support long- term, sustained progress by partnering with governments to use and strengthen their internal systems, build capacity, strengthen core institutions, maximize the impact of assistance they receive, and provide for their own people.” Similarly, the RERA, which falls under priority area 2, is now included as a deliverable on recent EiCC activities/projects and is recommended by E3/ED. Through the evaluation survey we found that a majority of USAID-affiliated respondents indicated that all priority areas were considered “very relevant.” When broken down further by type of USAID affiliation, we found that all Washington-based staff but one found all the priority areas to be very relevant. Similarly, the majority of mission staff also reported that all priority areas were very relevant (as opposed to somewhat relevant or not relevant), with priority area 1: learning environments are safer obtaining the highest number of “most relevant” indications and priority area 4: local institutions are stronger receiving the least number of “very relevant” ratings. While IP respondents also overwhelmingly indicated all priority areas were “very relevant,” priority area 2: EiCC programming has improved and priority area 4 (local institutions) received slightly more “very relevant” ratings than priority areas 1 and 3, suggesting a slight variation in responses between IPs and Mission staff who responded to this survey item. Relevance to Member Community As outlined in the ECCN Year 3 Workplan, the SG was critical in identifying the priority areas and themes within them as the main, high-level interest areas for the community of practice. Given the diversity of representation of ECCN community members on the SG, the evaluation team assumes a certain degree of relevance of the priority areas for the member community beyond E3/ED stakeholders. For example, the Sustainable Development Goals cite equity as a primary area of concern on the global agenda, as noted in the Year 3 workplan discussing priority area3. Additionally, involvement of the community members in activities structured around the priority areas also provides support for the degree of relevance. ECCN has developed working groups and task teams around the priority areas to facilitate targeted member engagement. One example of this is the Safer Learning Environment Working Group,21 which had 20 members as of December 2016 (Year 3 Workplan). Similarly, ECCN is a member of the Inter-Agency Accelerated Education Working Group and provides research, products, and activities. Accelerated/Alternative Education Programming (AEP) is one of the themes under priority area 3. Other examples of member community wide involvement along the priority areas includes the Equity Imitative Conflict and Fragility Working Group which is co-chaired by FHI 360 and UNICEF co-funding the development of the AEP Guide Principles (Year 3 Workplan). 21 “The AEWG is an inter-agency working group made up of education partners working in Accelerated Education. The AEWG is currently led by UNHCR with representation from UNICEF, USAID, NRC, Plan, IRC, Save the Children, INEE, ECCN and War Child Holland.” Taken from: http://eccnetwork.net/wp- content/uploads/AEWG_brief_September2016.pdf 41 Relevance for both USAID and larger member community It is difficult to assess the extent of relevance in a concrete manner because even within the two stakeholder groups--USAID and larger member community--there is a wide range of sub￾stakeholders with varying needs and types of associations with the priority areas, and thus, they may interpret relevance to mean different things. Therefore, the evaluation team relied on a generic Likert scale to assess degree of relevance, and found that the degree of relevance of the priority areas indicated by survey respondents was similar for both USAID and non-USAID affiliated responses. An overall majority of respondents reported that all four priority areas are “very relevant” (49-55 out of 66-68). Out of the total number of respondents, 35-36 respondents were USAID affiliated, while 30-32 were not. Across all four priority areas, there was a relatively equal split of USAID/non-USAID respondents reporting the degree of relevance for each. For themes within the priority areas, the majority of respondents found all themes to be “very relevant.” The two slight exceptions were “Advanced program and activity costing guidance” under priority area 2, with 37 respondents selecting either “not relevant” or “somewhat relevant” and 31 respondents selecting “very relevant.” The second exception was “Knowledge related to ICT application in EiCC” under priority area 3, with an equal split of respondents: 34 selecting either “not relevant” or “somewhat relevant” versus “very relevant”. Recommendations: 1) As discussed above, the evaluation team concludes that, overall, the priority areas and themes are relevant to E3/ED and the ECCN member community. Hence, we recommend that programming continues along priority areas and the themes within them to reflect alignment with E3/ED priorities and SG members who represent the larger ECCN member community. 2) The slight variations in perceptions of relevance between USAID and IP stakeholders should be further investigated and addressed. 42 5. To what extent does the ECCN, as a Community of Practice, meet the evolving needs of USAID’s Office of Education? ECCN’s Applicability to USAID’s Education Strategy Numerous interviewees considered various elements of the ECCN network and its resources to be applicable to other areas of the USAID Education Strategy. For example, the Safe Learning Environment work of ECCN seemed to some respondents to be useful for USAID's Goal 1 of successfully addressing the advancement of early-grade reading. Children learn better when their environment is safe, and this safety may be an internal issue to the school building, not due to the broader context of village, region or country. The evidence gap tool and other research and tools developed by ECCN were also deemed by some as applicable to other programming areas that the E3/ED office was promoting. KIIs also suggested that internal evolution of ECCN's priorities and objectives over time reflects the possibly shifting needs of that office overall, and as such many of these elements are inherently useful across the that office. However, key E3/ED leadership staff interviewed readily admitted that to date, “there is not much cross-pollination among the various CoPs that exist within the office,” and it is possible that institutional siloing and similar institutional barriers, are preventing the use and adoption of strategies, tools and talent across those three goals. Adapting to Changing Priorities/Evolving Needs Interviewees suggested E3/ED now faces a potentially higher demand for addressing education needs in C&C settings, which themselves are increasing due to growing global unrest, and therefore demanding more emergency education and the potential resources for it. At a policy formulation level, the strategies to respond to these needs under new leadership will need to adjust to that leadership’s priorities, and respond to the rationales that make the case for this work. To what extent do the “priority areas” of E3/ED’s work overall correspond or differ from those so designated with ECCN, and could they benefit the overall work if more visible and more easily accessible? And presumably, other efforts can be made towards the other two goals currently promoted to be more accessible to ECCN. This evolution and the potential needs it will generate are important variables to Conclusion: The ECCN largely meets the evolving needs of E3/ED, however, it does not function as a traditional CoP with multidirectional engagement, as envisioned by the educational theorist Etienne Wenger. • Various elements of the ECCN network and its resources were considered by E3/ED informants in leadership positions to be applicable to other areas of the USAID Education Strategy. • ECCN is meeting evolving needs through potentially higher demand for the addressing of education needs in crisis and conflict-affected settings, which themselves are increasing due to growing global unrest, and therefore demanding more emergency education and the potential resources for it. • ECCN was often discussed and considered as a network by interviewees, without a particular assumption by respondents beyond E3/ED and the ECCN ST that it involved robust multidirectional sharing andcollaboration. 43 consider and track as the leadership and priorities of the Agency overall, and hence E3/ED, emerge in the coming months and years. ECCN as a CoP Educational theorist Étienne Wenger and anthropologist Jean Lave22 observed a spontaneous group mentoring process that developed among traditional apprentice tailors in Africa, noting that they readily came together and learned from each other much more than they did from master tailors. Projecting this age-old practice to the late-20th Century reality of electronic communication and interaction starting up among disparately located communities, the phenomenon as it was emerging across groups now connected online was given a name: communities of practice. However, the CoP, as defined by Wenger & Snyder,23 is not how many informants who are exposed to or involved in ECCN's network said they understood it. At the early stages of ECCN’s development, as described in work plans and reports, the creation of a website and content to populate it for the CoP was a high priority, assuming perhaps that the group targeted would readily engage once this was in place--when looking at initial membership and activity of the network, they did not. Only recently did we see this engagement occurring in a limited way, based at least on the growing number of registered users on the site. However, ECCN was often discussed and considered as a network by informants we interviewed, without a particular assumption by respondents beyond E3/ED and the ECCN ST that it involved multidirectional sharing and collaboration. The CoP concept, according to a key USAID informant who was central to the conceptualization of the program, was used in ECCN’s initial planning as a convenient term to help decision-makers grasp the need for a bundling of integrated services to address Goal 3 of the USAID Education Strategy. In contrast to the usually accepted meaning of the term CoP, however, it did not target an already-existing group of interacting professionals around the area of education in crisis and conflict. Its focus was on relatively unconnected individuals in its network who face similar challenges and have similar needs in their EiCC program planning and implementation. The ECCN membership created is still mostly a "network" that pulls information when it needs it, but is not yet fully committed to be the spontaneously collaborative practitioner group that a CoP traditionally is. Recommendations: 1) Build upon early successes to help ECCN evolve into a true CoP. To date, ECCN has witnessed many successes serving as an integrated repository of information, talent and resources available on-demand to USAID’s EiCC-concerned staff and partners. However, to meet the evolving needs of E3/ED ECCN needs to focus on evolving into a true CoP, with multidirectional and broad engagement that further increase the potential benefit of ECCN in addressing E3/ED priorities and needs. 2) Consider and pursue potential avenues for cross-cutting sharing and collaboration among E3/ED staff, particularly between Goals 3 and 1. It is important to consider and address possible barriers across E3/ED's goals, the result of which respondents described as “siloing”, that are preventing the use and adoption of 22 Wenger, Étienne, “Communities of Practice: A brief Introduction”. Retrieved online on 17 May 2017 at wenger- trayner.com/introduction-to￾communities-of-practice. 23 Wenger, É. & Snyder, W., “Communities of Practice: The Organizational Frontier”. Harvard Business Review, Jan.- Feb. 2000. 44 strategies, tools and talent across E3/ED goals. Given that the current Education Strategy is coming to its close, this may be a good time to broaden staff capacities in anticipation of a different focus or priority in any new strategy that emerges from new leadership. ECCN should seek additional opportunities for cross-cutting sharing and collaboration in programming components and consider if and how staffing can be further focused on types of outputs (e.g., safe learning, research, theories of change) that could help overcome siloing and build bridges of collaboration that could strengthen the overall capacity of E3/ED rather than Goals 1/2/3 focusedteams. i ANNEXES • Annex I: Evaluation Statement of Work • Annex II: Data Collection Instruments • Annex III: Sources of Information • Annex IV: Survey results • Annex V: KII code list • Annex VI: Disclosure of any Conflicts of Interest ii Annex I: Evaluation Statement of Work STATEMENT OF WORK Midterm Evaluation OF USAID Education in Crisis & Conflict Network (ECCN) I.PURPOSE OF THE EVALUATION Over the past five years, USAID has seen an increase in the number of Communities of Practice (COP) and Networks that it supports. This is also true in the Office of Education, which has introduced the Global Reading Network, Workforce Connections, and the Education in Crisis and Conflict Network (ECCN), all of which have both unique as well as overlapping objectives, such as the dissemination of knowledge, the development of tools to enhance program design, and the provision of a platform under which key stakeholders in the sector may coordinate and collaborate efforts to advance as a sector. ECCN, is the newest of the COPs/Networks24 and has reached its midpoint. Therefore, it is an opportune time to undertake a mid-term evaluation of the Network. The evaluation being contracted will play two roles. First, it will serve as a mid-term check in to assess the progress the Implementer has made towards reaching its objectives. Second, it is designed to be a source of learning for the ECCN ST and USAID E3/ED more broadly. More specifically, the evaluation should do the following: ● Assess the progress the ECCN ST has made toward meeting its five primary objectives ● Identify the successes and challenges that the ECCN ST have encountered carrying out their five work streams and meeting theirobjectives. ● Describe the ECCN membership and ECCN memberengagement SUMMARY INFORMATION Table 1: Summary Information Project Name Education in Crisis & Conflict Network Implementer EDC Contract Number GS-10F-0406P/AID-OAA-M-14-00007 24 Note that ECCN is referred to as a Community of Practice (CoP), however it is a broader support contract. As will be demonstrated, the development and support of a CoP is the first of five work streams/objectives for which the implementer is responsible. To keep the evaluation activities clear, we will reference “the Network” when referring to the contract as a whole, “the Community of Practice” when discussing the first objective and work stream, and “the ST” when discussing those individuals who work for the IP iii Total Estimated Ceiling of the Evaluated Project/Activity(TEC) $9,967,539 Life of Strategy, Project, or Activity July 2014 – July 2019 Active Geographic Regions N/A E3/ED Operational Strategy 2016 - 2019 C2. Provide high-quality, demand-driven and value-added technical services USAID Office E3/ED II. BACKGROUND A. Description of the Problem, Development Hypothesis(es), and Theory ofChange ECCN, implemented by EDC, was designed to support technical leadership in education in crisis and conflict environments to USAID by ensuring that education practitioners and those who support them have the latest information, most practical tools, and sufficient resources. EDC’s ECCN Support staff has been tasked to work with USAID and partners to develop an Education in Crisis and Conflict Community of Practice, convene and collaboratively structure a program of work for ECCN Support Staff and the Community of Practice, and support that program through evidence- gathering, documentation, research, tool production and dissemination, and training. USAID ECCN aims to “build evidence and capacity to increase equitable access to education in crisis and conflict affected environments per Goal 3 of USAID’s Education Strategy”. By building evidence and capacity, ECCN is designed to help USAID and its partners improve practice around designing, managing, implementing and evaluating education programs in crisis and conflict affected environments, so that they contribute to USAID Education Strategy, specifically the third goal: Increased equitable access to education in crisis and conflict environments for 15 million learners by 2015.ECCN will achieve this goal by pursuing five, inter-related objectives or work streams, each related to different aspects of professional practice. In abbreviated form, these work streams emphasize the development and spread of: knowledge sharing, evidence, program metrics and theories of change, program tools and guidance, and professional development. Taken together, ECCN goal and objectives comprise the following theory of change: iv Figure 1: ECCN Theory of Change Ultimately, by nurturing and convening a Community of Practice/Network and spearheading key activities, such as the development of tools and metrics, information sharing through the development of a website and communication platform, and executing systematic reviews of evidence, the ECCN Support contract aims to advance the Office of Education towards its strategic objectives. Namely, to increase access to safer higher quality, learning opportunities in crisis and conflict-affected environments. B. Summary Strategy/Project/Activity/Intervention to beevaluated The ECCN ST has been tasked with a number of activities that fall under five primary objectives with related work streams: USAID ECCN Goal: Improved policy, planning and practice for USAID Education Goal 3 programming USAID Education Strategy Goal 3: Increased Equitable Access to Education in Crisis and Conflict Affected Environments v 1. Develop and support the Education in Crisis and Conflict Community of Practice. A thriving informal EiCC community of practice already exists, and is composed of USAID EiCC leadership (DC, Missions); implementing partners; and EiCC networks and working groups. This workstream is intended to nurture and grow an intentional USAID- focused sub-community (the USAID Education in Crisis and Conflict Network) through which members can engage in a rich exchange of knowledge and share it with the larger EiCC community of practice. The community of practice will also draw from the USAID ECCN ST’s work in Objectives 2-5. 2. Gather, build and consolidate evidence for the Field. The ST will synthesize and distill existing research, and undertake or support new research as suggested by USAID and by the USAID ECCN, in the area of education in crisis and conflict. The ST will disseminate research findings through documents, training and other forums (as supported by the other objectives). 3. Test, disseminate and use: guidance, tools, metrics, and Theories of Change. This work stream focuses on developing tools and guidance that concentrate on themost effective approaches to increasing equitable access to education in crisis and conflict environments. Specifically, the ST will develop operational guidance and tools; develop and test theories of change, indicators and log frames; and costing methodologies and templates related to EiCCprogramming. 4. Develop and apply guidance/tools. Over the last number of years, the USAID Office of Education developed tools and guidance that aid in better contextualizing and adapting education programming for crisis and conflict-affected environments. Through activities in this workstream ECCN will refine, add to and implement these tools to promote more conflict sensitive and context appropriate education programming in these challenging environments. 5. Develop, implement and assess professional development. The project will develop, deliver and assess a range of professional development opportunities designed to build the capacity of USAID, Goal 3 implementers and local and regional institutions to use evidence-based research and resources in project design, implementation, andmonitoring. C. Summary of the Project/Activity Monitoring, Evaluation, and Learning (MEL)Plan The ECCN ST developed a Results Framework and Performance Management Plan (PMP) that targets each of the five objectives and works streams. The PMP is designed to serve as a tool for planning, implementing, tracking and documenting progress toward achieving the objectives identified for each work stream and expected outcomes of ECCN. It should be noted that the PMP is also designed to be a flexible tool and provides the ECCN ST along with USAID staff an opportunity to revisit the information being collected and to revise the PMP to reflect any new activities. As it currently stands, the PMP has targeted three high level outcomes. These outcomes include the following: ● Outcome One: COP members share knowledge to inform practice forincreased equitable access to education in conflict and crisis-affectedareas. vi ● Outcome Two: COP members access and apply resources developed and/or shared by ECCN to improve practice for increasing equitable access toeducation in conflict and crises-affectedareas. ● Outcome Three: COP members apply knowledge and skills acquired or improved through participation in ECCN professional developmentopportunities. To assess progress towards these outcomes, the ECCN ST and USAID have agreed to the following eight custom indicators: Outcome One: COP members share knowledge to inform practice for increased equitable access to education in conflict and crisis-affected areas. 1.1 Number of partners engaged in USAID-led Communities of Practicethat advance the goals of the EducationStrategy; 1.2 Number of COP members who access ECCN knowledge sharing content(e.g., blogs, newsletters, and social mediaactivity); 1.3 Number of COP members who contribute to ECCN knowledgesharing content and groups; 1.4 Number of products or resources developed by COP through ECCN knowledge share groups. Outcome Two: COP members access and apply resources developed and/or shared by ECCN to improve practice for increasing equitable access to education in conflict and crises-affected areas. 2.1 Number of COP members who access resources developed and shared by ECCN; 2.2 Number of COP members who apply ECCN developed and or/or shared resources to improve practice for increasing equitable access to education in conflict and crisis-affected areas. Outcome Three: COP members apply knowledge and skills acquired or improved through participation in ECCN professional development opportunities. 3.1 Number of COP members who participate in ECCN professional development opportunities; 3.2 Number of COP members who report applying to G3 programs knowledge and skills acquired or improved through participation in ECCN PD opportunities. Upon award the contractor will be provided access to the PIRS and data gathered for the PMP. In order to increase efficiency and to encourage collaboration between the evaluation contractor and the implementer, the contractor should consider using these data to inform the evaluation design report and to complement the data being gathered as a part of the evaluation. Specifically, the data gathered for the PMP should be examined to see if they can help answer any of the primary evaluation questions or create new evaluation questions. vii III.EVALUATIONQUESTIONS To meet the evaluation objectives described above, the Office of Education asks the Evaluation Team to answer three primary evaluation questions. 1. What progress has the ECCN ST made carrying out the five workstreams and advancing towardsthe objectives outlined in their work plan? 2. What successes and challenges has the ECCN ST encountered carrying out their five work streams and meeting theirobjectives? 3. Who are the members of the ECCN and how are theyengaging? The first two questions focus on the progress of ECCN ST implementing their work streams, the progress they’ve made and successes and challenges encountered. The Education Office wants to better understand the processes that drive the network and successes and challenges that the ST have encountered to guide the work planning process for the upcoming calendar year, and to inform the development of future support networks and community of practice projects in the future. Recognizing the potential effect that ECCN members may have on the ST's ability to successfully meet its objectives, the Education Office has included a third question to gain insight into who chooses to engage, both organizationally and individually, at what level they engage, and how the Office in collaboration with the ECCN ST can encourage greater engagement. We believe that this question will also allow us to understand the barriers individuals and organizations face that may be preventing them to engage. This will require the Evaluation Team to gather information from both those organizations and individuals that have opted to engage as well as those whose interests align with ECCN, but have thus far not participated in in the Network. IV. EVALUATIONDESIGNAND METHODOLOGY To answer the evaluation questions, the Office asks that the contractor undertake a performance and process evaluation that examines both the performance of ECCN as well as the processes of implementation and engagement. We anticipate that in order to answer the evaluation questions, the contractor will need to use a mixed methods approach, combining the analysis of existing PMP data, quantitative data gathered through mini-surveys, as well as qualitative information gathered through telephone or in￾person interviews. For the first evaluation question, we recommend that the contractor determine what data need to be collected to examine progress towards objectives by undertaking a preliminary examination of the PMP data gathered by the implementing partner. We expect that the contractor will need to carry out either interviews or mini-surveys (with closed and open-ended items) with key stakeholders such as USAID E3/ED staff involved with ECCN and ECCN members about progress towards the objectives. During this data viii collection process we also suggest that the contractor gather information regarding the implementation of the work streams as well as the success and challenges to implementation, as this information will be critical in answer the second evaluation question. For the third evaluation question and sub-questions, we suggest that the contractor carry out a mini-survey of organizations and individuals engaged with ECCN, and possibly, with individuals who represent organizations whose interests align with ECCN but who are not members nor participate in the network. The mini-survey should target the level of engagement and the types of activities in which individuals and organizations are engaging. To understand how individuals are engaging as well as enabling factors and barriers to engagement, we suggest selecting a subset of respondents from the survey for follow-on in-depth interviews. In the table below, we have summarized types of data that may be used to answer the evaluation questions. However, we encourage the contractor to offer alternate approaches during the design phase. Furthermore, we ask the contractor to provide information regarding the approach that he or she will take in selecting interview participants. Table 2: Design Matrix Questions Approach Potential Data Sources Sampling 1. What progress has the ECCN ST made carrying out the five work streams and advancing towards the objectives outlined in their work plan? Output and outcome mapping Process mapping for implementation PMP data Project Quarterly and Annual Reports Supplemented with interview data as needed To be determined depending on data needs 2. What successes and challenges has the ECCN ST encountered carrying out their five work streams and meeting their objectives? Identify themes that arise in qualitative data regarding successes and barriers. We are not looking for significant findings but all finding Project Quarterly and Annual Reports; Interviews with ECCN support staff Interviews with USAID staff Mini- surveys of ECCN members about successes and challenges All Support Staff Sampling for mini- survey to be proposed by evaluator 3. Who are the members of the ECCN and how are they engaging? Construct common narratives from members about their experiences with engagement. Identify common barriers through theme analysis. Data from PMP, if applicable Lists of ECCN Members, mini￾survey from participants and non￾participants Sample for mini￾survey to be proposed by evaluator V. DELIVERABLES AND REPORTING REQUIREMENTS 1. Planning meeting with USAID E3/ED and Implementing Partner: Within the first 5 days of award, the contractor/evaluation team will participate in a day longplanning meeting. During this meeting the contractor will work with USAID E3/ED Office ix Representative and the implementing partner to describe the ECCN, discuss the purpose of the evaluation, possible approaches, and draft a preliminary timeline. 2. Preliminary timeline: Prior to the planning meeting, the contractor/evaluation team will submit a list of possible activities and a preliminary timeline draft work plan for the evaluation shall be completed by the lead evaluator and presented to the Agreement Officer’s Representative/Contracting Officer’s Representative (AOR/COR). The work plan will include: (1) the anticipated schedule and logistical arrangements; and (2) a list of the members of the evaluation team, delineated by roles and responsibilities. 3. Evaluation Design Report and Workplan: Within 2 weeks of completing the planning meeting, the contractor will submit a draft evaluation design report and workplan. The Evaluation Design Report should consist of the following sections: a. Purpose of the evaluation and the primary evaluationquestions b. A description of the ECCN project including a discussion of the objectives and the workstreams implemented to meet thoseobjectives. c. Methodology Section i. Approach to the evaluation ii. Justification for theapproach iii. Data Needs iv. Data collection v. Preliminary analysis plan d. Reporting andDissemination USAID offices and relevant stakeholders are asked to take up to 5 business days to review and consolidate comments through the Activity Manager. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within 5 days. 4. Draft Evaluation Report: The draft evaluation report should be consistent with the guidance provided in Section IX: Final Report Format. The report will address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, USAID/E3/Education will have 10 business days in which to review and comment on the initial draft, after which point the Activity Manager will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit a revised final draft report 10 business days hence, and again the USAID/E3/Education office will review and send comments on this final draft report within 5 business days of its submission. x 5. Final Evaluation Report: The evaluation team will be asked to take no more than 5 business days to respond/incorporate the final comments from the USAID/E3/Education office. The evaluation team leader will then submit the final report to the Activity Manager. All project data and records will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. 6. Final Presentation: The evaluation team is expected to hold a finalpresentation in person/by virtual conferencing software to discuss the summary of findings and recommendations to USAID. This presentation will be scheduled as agreed upon during the in-briefing. VI. EVALUATION TEAM COMPOSITION All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s evaluation policies and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. The Team Leader, Education Monitoring and Evaluation Expert will participate on the evaluation team in [describe role]. The Education Monitoring and Evaluation Expert will participate on the evaluation team in [describe role]. The Program Specialist will participate on the evaluation team in [describe role]. The Activity Manager of the Evaluation may observe all of the data collection efforts. Requested qualifications and/or skills may relate to: (1) evaluation design, methods, management, and implementation; (2) specific relevant technical subject matter expertise, (c) experience in USAID’s cross-cutting program priorities, such as, gender equality and women’s empowerment (d) crisis or conflict country experience. xi VII. EVALUATIONSCHEDULE Table 3: Illustrative Schedule Activity Deliverable Timeline LOE/ timeframe Consideration Planning meeting Preliminary timeline and activity list Within five days of award 5 days 12/1-2 This should include a review of relevant materials, tentative timeline and preliminary list of activities that will be submitted prior to the meeting. Evaluation Design Report Evaluation Design Report (see above for required sections), Work plan/Timeline, and draft instruments Within two weeks of completing the planning meeting 10 days per person 12/12- 23 Data Collection Weekly updates of data collection progress Three weeks for data collection and updates weekly 15 days per evaluator, updates included in this LOE 01/9-27 Data collection should be carried over a period of three weeks. The memos are a brief update on data collection completed, outstanding and Data cleaning and analysis 2 weeks 10 days per evaluator 01/30 - 02/10 The data does not need to be delivered until the acceptance of the final report Reporting Evaluation Report 2 weeks 10 days per evaluator 02/13-24 This draft should be as close to final as possible, including editing, final figures, and appropriate formatting USAID review of Evaluation Report 2 weeks 02/27-03/10 Reporting Evaluation report final draft 2 weeks 10 days per evaluator 03/13-24 Run additional analyses requested following USAID reviews of draft report. Make modifications to report as needed Presentation of Findings Slides to be submitted 1 day prior to presenting within 5 days of submitting final draft 5 days 03/27 Submit slide for approval one day prior to presentation. Presentation to include USAID and implementer Prepare and submit data Data sets Within 5 days of completing the report 4 days Prepare data sets following USAID Open data policy xii Table 4: Estimated LOE in days by position Position Preparation Data Collection Finalization of Report Total LOE in days Education Monitoring and Evaluation Expert / GS 14 2.5 25 20 47.5 Education Monitoring and Evaluation Expert / GS 14 2.5 25 20 47.5 Program Specialist / GS 13 10 10 10 30 Totals 15 60 50 125 VIII. FINAL REPORT FORMAT The evaluation final report should include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. The executive summary should be 2–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.). The annexes to the report shall include: ● The Evaluation SOW; ● Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluationteam; ● All data collection and analysistools used in conducting the evaluation,such as questionnaires, checklists, and discussionguides; ● Allsources of information, properly identified and listed;and ● Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. ● Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; ● Summary information about evaluation team members, including qualifications, experience, and role on the team; xiii In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATIONREPORT Per ADS 201mah, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated againstthe following criteria to ensure the quality ofthe evaluation report.22 ● Evaluation reports should represent a thoughtful, well-researched, and well- organized effort to objectively evaluate the strategy, project, or activity. ● Evaluation reports should be readily understood and should identify key points clearly, distinctly, andsuccinctly. ● The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. ● Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. ● Evaluation methodology should be explained in detail and sources of information properly identified. ● Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recallbias, unobservable differences between comparator groups, etc.). ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. ● Findings and conclusionsshould be specific, concise, and supported by strong quantitative or qualitative evidence. ● If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males andfemales. If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. IX.OTHERREQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine￾readable, non- proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether xiv in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the Activity Manager. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. X. LIST OFANNEXES (To be Completed) xv Annex II: Data Collection Instruments Key Informant Interview (ECCN Implementation Team version for Steering Group, EDCs ST, Advisory Team, and USAID E3/ED staff directly involved with ECCN) In your request for the interview (by phone or email) provide the potential informant with this information: My name is and I am working through the Dexis Consulting Group, undercontract with USAID. We are performing an independent mid-contract evaluation of USAID’s Education in Crisis and Conflict Network (ECCN) project. This project is funded by USAID’s Office of Education in the Bureau for Economic Growth, Education and Environment (E3/ED) and has been implemented since 2015 by the Education Development Center(EDC). As a stakeholder in this project we would like to discuss with you, in person or by phone/Skype, your observations of, and experience with the ECCN project in order to better learn about the project’s progress, successes and challenges so far. Through this evaluation, we will provide USAID and its partners findings, conclusions and recommendations to increase the project’s success going forward. Depending on the topics we cover, the interview should last approximately 60 minutes. Introduction: “I appreciate your willingness to spend some time discussing the ECCN project, as outlined in my earlier message. Your participation in this study is voluntary and you will be free to end the conversation at any time. Anything you say may be reported in the study’s final report, but no identifying information will be disclosed or shared and you will not be quoted by name without your express permission. Do you have any further questions about this evaluation or discussion? (Refer back, as needed, to information provided in the invitation.) “If you do not have any questions at this time), let us proceed. “With your permission, I would like to record our conversation, only for our team’s personal reference and review. Is that acceptable to you?” a) If yes: “OK, let me start the recording now.” (Begin each recording by stating the name, title and affiliation of the respondent, the date of the interview and your name.) b) If no: “I will take notes as we speak—please bear with me if I need to pause in order to write down key points. (Use this interview script to organize your notes) Introductory Questions: 1) Tell me about your currentrelationship with ECCN and how it developed. xvi (Length of involvement, evolution of role over time, previous relevant experience, etc.) RQ1: What progress has the ECCN ST made in carrying out the five work streams and advancing towards the objectives outlined in their work plan? 2) Let’s talk about ECCN’s stated objectives and your role in their creation and implementation to date (Objectives are: Develop and support a COP; build and disseminate evidence for the field; test, disseminate, and use guidance, tools, metrics, and TOCs; apply tools in C&C; develop, implement and assesstraining). a) Tell me about your role in the creation of any of these objectives b) Now let’s talk about your personal role in their implementation (What role, how it has evolved over time, where it is going in thefuture) 3) Overall, what is your opinion of how the ECCN, to date, has met the stated objectives? (Look for what interviewees are citing as evidence of objectives met/not met i.e. Do they refer to measurable and specific outcomes, or just general opinions? Guide discussion covering: a. availability, access, and use of knowledge-sharing; b. evidence (research, assessments,evaluations) c. programming metrics/theories of change; tools andguidance; d. professionaldevelopment RQ 2: What success and challenges has the ECCN ST encountered in carrying out their five work streams and meeting their objectives? 4) From your perspective in relation to the ECCN, what successes of the ECCN project have you carried out? (Focus on enabling/inhibiting factors to availability/access/use/ and changed practice of the 5 work streams.) 5) From thatsame perspective, tell me about any challenges you think in yourrole, ECCN project has faced. (Team structure, processes, etc.) RQ 3: What is the nature and extent of member engagement and how is it supported by the ECCN ST, Steering Group and Advisory Group? 6) What is yourrole in interacting with orsupporting the ECCN member network? i. For ST and SG members: How often do you communicate with ECCN members? How? (Are there set schedules/ or is it more freeflowing) For ST and SG members: How do you keep track of communications with ECCN members? ii. For STmembers: Isthere a processfor tracking ECCN member engagement/use ofresources pushed out by ECCN? (looking to see whether there a systematic system to record engagement, track analytics, determine what resources were well received,etc.) iii. Can you describe standard processes used in engaging/supporting? iv. Whatis your perspective/understanding of the ECCN membership community and their use of the network effectively? RQ 4: To what extent are the priority areas and the xvii themes within the ECCN project relevant to USAID Office of Education and the ECCN member community? 7) Are you aware of USAID’s Office of Education’s role and mission? i. If no: skip toQ8 ii. Do you think ECCN contributes to the Office of Education’s overall mission? iii. “If yes: Tell me how you think ECCN contributes to the identified role and mission?” If no: “Why not?” 8) Are you aware of the priority areas and themes of the ECCN project? i. Would you say the ECCN priority areas are relevant to the member community? ii. Relevant to USAID office ofeducation? iii. Why or why not? RQ 5: To what extent does the ECCN, as a Community of Practice, meet the evolving needs of USAID’s Office of Education? 9) Do you know what the needs of USAID's Office of Education (E3/ED) are? i. If: “I don’t know about them”—skip to wrap-up ii. If yes: Have these evolved over time? How? iii. Do you think the ECCN contributes to the current evolving needs of E3/ED? If yes: How so? If no: Do you see a better way for those needs to be met by ECCN (please explain), or Do you think those needs are better met by some other approach, and if so, what kind of approach do you think would be helpful? iv. How is the ECCN as a COP mechanism addressing the needsto the Office of Education as opposed to othermechanisms? Wrap-up (The respondent will never see this page; we will just record the info suggested as preferred) 1. Thank you for your time today, your input will be very helpful to our efforts. 2. If I have any follow up questions for you, what is the best way to be back in contact? a) Phone – get phone number: b) Email – get email address: @ xviii II.I Other – pleasespecify: Key Informant Interview (USAID affiliated ECCN members (exclusive of E3/ED staff directly involved with contract) *For E3/ED staff directly involved with contract refer to KII instrument for implementation team* In your request for the interview (by phone or email) provide the potential informant with this information: My name is and I am working through the Dexis Consulting Group, under contractwith USAID. We are performing an independent mid-contract evaluation of USAID’s Education in Crisis and Conflict Network (ECCN) project. This project is funded by USAID’s Office of Education in the Bureau for Economic Growth, Education and Environment (E3/ED) and has been implemented since 2015 by the Education Development Center(EDC). As a stakeholder in this project we would like to discuss with you, in person or by phone/Skype, your observations of, and experience with the ECCN project in order to better learn about the project’s progress, successes and challenges so far. Through this evaluation, we will provide USAID and its partners findings, conclusions and recommendations to increase the project’s success going forward. Depending on the topics we cover, the interview should last approximately 60 minutes. Introduction: “I appreciate your willingness to spend some time discussing the ECCN project, as outlined in my earlier message. Your participation in this study is voluntary and you will be free to end the conversation at any time. Anything you say may be reported in the study’s final report, but no identifying information will be disclosed or shared and you will not be quoted by name without your express permission. Do you have any further questions about this evaluation or discussion? (Refer back, as needed, to information provided in the invitation.) “If you do not have any questions at this time), let us proceed. “With your permission, I would like to record our conversation, only for our team’s personal reference and review. Is that acceptable to you?” a) If yes: “OK, let me start the recording now.” (Begin each recording by stating the name, title and affiliation of the respondent, the date of the interview and your name.) b) If no: “I will take notes as we speak—please bear with me if I need to pause in order to write down key points. (Use this interview script to organize your notes) xix Introductory Questions: 1) Tell me about your currentrelationship with ECCN and how it developed. (Length of involvement, evolution of role over time, previous relevant experience, etc.) RQ1: What progress has the ECCN ST made in carrying out the five work streams and advancing towards the objectives outlined in their work plan? 2) What do you know about the ECCN’s stated objectives? (Objectives are: Develop and support a COP; build and disseminate evidence for the field; test, disseminate, and use guidance, tools, metrics, and TOCs; apply tools in C&C; develop, implement and assess training). Tell me about your role, if any, in the creation and/or implementation of any of these objectives (What role, how it has evolved over time, where it is going in the future) 3) Overall, what is your opinion of how the ECCN, to date, has met the stated objectives? (Look for what interviewees are citing as evidence of objectives met/not met i.e. Do they refer to measurable and specific outcomes, or just general opinions? Guide discussion covering: Develop and support a COP; build and disseminate evidence for the field; test, disseminate, and use guidance, tools, metrics, and TOCs; apply tools in C&C; develop, implement and assess training). RQ 2: What success and challenges has the ECCN ST encountered in carrying out their five work streams and meeting their objectives? 4) From your perspective, whatsuccesses do you think the ECCN has achieved in each of its workstreams? (Name them as necessary based on respondent’s need for help) (Focus on enabling/inhibiting factors to availability/access/use/ and changed practice of the 5 workstreams.) 5) What challenges, if any, do you think the ECCN hasfaced? (Team structure, processes,etc.) RQ 3: What is the nature and extent of member engagement and how is it supported by the ECCN ST, Steering Group and Advisory Group? 6) What is your understanding of the ECCN member community and their use of the network effectively? RQ 5: To what extent does the ECCN, as a Community of Practice, meet the evolving needs of USAID’s Office of Education? 7) Do you have a clear understanding of the needs of USAID's Office of Education (E3/ED)? i. If: “No”—skip towrap-up xx ii. If yes: Have these evolved over time? How? iii. Do you think that the ECCN as a COP mechanism is the correct technical support design to address the needs to the Office ofEducation? If yes: How so? If no: Do you see a better way for those needs to be met by ECCN (please explain), or Do you think those needs are better met by some other approach, and if so, what kind of approach do youthink would be helpful? RQ 4: To what extent are the priority areas and the themes within the ECCN project relevant to USAID Office of Education and the ECCN member community? 8) Are you aware of the ECCN’s priority areas and themes? i. If “No”—offer them a probe (name a priority area/theme) and see whetherthey recall/volunteer further information other than a simple yes/no); then go to Q10 ii. If “Yes—go toQ10 9) Would you say the ECCN project’s work products are relevant to your work and projects in C&C? Why or why not? Wrap-up (The respondent will never see this page; we will just record the info suggested as preferred) 1. Thank you for your time today, your input will be very helpful to our efforts. 2. If I have any follow up questions for you, what is the best way to be back in contact? a) Phone – get phone number: b) Email – get email address: @ II.II Other – pleasespecify: xxi Survey Questions Introduction Thank you for taking the time to participate in our survey. This survey is part of an independent mid￾contract evaluation of USAID’s Education in Crisis and Conflict Network (ECCN) project conducted by the Dexis Consulting Group. The ECCN is funded by USAID’s Office of Education in the Bureau for Economic Growth, Education and Environment (E3/ED) and has been implemented since 2014 by the Education Development Center (EDC). The purpose of this evaluation is to assess ECCN’s progress to date and to provide USAID and its partners with lessons-learned and recommendations for the project going forward. We would like to learn about your observations of, and experience with the ECCN in order to better understand the progress, successes and challenges of ECCN so far. Please use the comment boxes to provide fuller answers and more insight on your experiences with the ECCN. We appreciate your time and input. Please note that your participation in this survey is voluntary and you will be free to end the survey at any time. Any response you provide may be reported in the final report as part of aggregate quantitative analysis or note-worthy qualitative insights from comments. The only identifying information used will be the demographic information provided in the first section of the survey. Demographics * 1. Which country(ies) are you working in or do you carry out work in? Please select all that apply • Afghanistan • Jordan • Pakistan • Burma/ Myanmar • Kenya • Philippines • Democratic Republic of Congo • Lebanon • Senegal • El Salvador • Liberia • Somalia • Guatemala • Mali • South Sudan • Haiti • Nicaragua • West Bank/Gaza • Honduras • Nigeria • Yemen • Other (please specify) * 2. Are you affiliated with USAID in any way? -Yes -No xxii *3. What is your affiliation with USAID? - Washington-based staff -Mission staff - Implementing Partner -Other (please specify) *4. Since when have you been involved with ECCN? Please specify since which year and month. *5. How long have you been working on issues around Education in Crisis and Conflict? - Less than 1 year -1 - 4 years - 5 - 9 years - 10- 14 years -15+ years - N/A -Other (please specify) * 6. What is your professional affiliation? -Donor staff -NGO -National or regional ministry of education or other public education authority - Professional researcher/academic -Other (please specify) *7. Since when have you been involved with ECCN? Please specify since which year and month. *8. How long have you been working on issues around Education in Crisis and Conflict? • Less than 1 year • 1 - 4 years • 5- 9 years • 10- 14 years • 15+ years • N/A • Other (please specify) Engagement with ECCN For the purpose of this survey, engagement refers to any form of interaction with the ECCN, active and passive. For example, this could include actively contributing to the ECCN through discussions, materials, etc. and/or consuming knowledge provided through the ECCN such as newsletter, attending events, etc. * 9. To what extent do you feel that there are opportunities available to engage with the ECCN? • Somewhat • Inadequate Amount • Adequate Amount • Many Opportunities • Not Applicable Comment (please explain) * 10. The ECCN offers members the opportunity to engage through various mediums, as represented below. For each of the mediums listed, please select the frequency with which you engage with the ECCN. Never Rarely Sometimes Regularly N/A Website Newsletter Face to face events Webcasts Other (please specify) xxiii * 11. Please rate the ease with which you are able to engage with the ECCN. -Not Easily -Somewhat Easily -Very Easily -N/A -Comment (please explain) *12. Are there any mediums of engagement that are currently not available through the ECCN that you believe would be beneficial? -No -Yes - Please specify *13. How well do you feel that the ECCN Community of Practice is managed? • Very Poorly Managed • Somewhat Poorly Managed • Somewhat Well Managed • Very Well Managed • Don't Know Comment (please explain) * 14. Have you ever requested assistance/support directly from the ECCN Support Team? -Yes -No *15. Would you say you have gained knowledge through the ECCN? • Yes • No - please explain what, if anything, is lacking in the ECCN that could help you gain knowledge *16. Have you ever shared knowledge through active participation in the ECCN? • Yes • No (please explain why not in the comment box below) *17. Please provide a brief description of the assistance/support you requested *18. How satisfied were you with the way the ECCN Support Team handled your request? • Very Unsatisfied • Somewhat Unsatisfied • Somewhat Satisfied • Very Satisfied Comment (please explain) *19. Would you say you have gained knowledge through the ECCN? • Yes • No - please explain what, if anything, is lacking in the ECCN that could help you gain knowledge *20. Have you ever shared knowledge through active participation in the ECCN? • Yes • No (please explain why not in the comment box below) * 21. What has been your preferred method of sharing knowledge through the ECCN? Please select all that apply • Email other members • Blog posts • Face-to-face events xxiv • ECCN webcast participation or follow-up • Posting on the bulletin board/message boards • Participation in the working groups • Commenting on online context (e.g. blogs) • Other (please specify) Programming Resources 22. Are you aware of the following programming resources available through the ECCN? Please indicate "yes" by selecting all that apply. If "no," please leave blank. Please note that this survey distinguishes between programming resources and research/evidence which will be addressed later in the survey. • Checklist for Conflict Sensitivity in Education • Rapid Education and Risk Analysis • Theories of Change Resources • Indicators/ PMP Analysis • Accelerated Education Principles/ Research Report Costing Guidance • Evidence Gap Maps • Feedback Loops Resources • Safer Learning Environments Tools Other (please specify) * 23. Have you accessed any of these programming resources? Yes Not yet, but I plan to No Checklist for Conflict Sensitivity in Education Rapid Education and Risk Analysis Theories of Change Resources Indicators/ PMP Analysis Accelerated Education Principles/ Research Report Costing Guidance Evidence Gap Maps Feedback Loops Resources Safer Learning Environments Tools Other (please specify) * 24. Please specify why you have not accessed these programming resources available * 25. Have you applied or do you plan to apply this/these programming resources in your work? Yes No Checklist for Conflict Sensitivity in Education Rapid Education and Risk Analysis Theories of Change Resources Indicators/ PMP Analysis Accelerated Education Principles/ Research Report Costing Guidance Evidence Gap Maps Feedback Loops Resources Safer Learning Environments Tools Other (please specify) xxv * 26. Please indicate why you have not applied or do not plan to apply this/these programming resources accessed * 27. To what extent have the programming resources accessed through the ECCN influenced your work in the education in crisis and conflict community? • Not at all • Increased awareness • One-time input • Occasional inputs • Long-term or sustained changed practice (process, tools, etc.) • Other (please specify) * 28. Are there any programming resources currently unavailable through the ECCN that you believe would be beneficial to the education in crisis and conflict community? • No • Yes, please explain in the comment box below Research/Evidence * 29. Are you aware of research findings and reports available through the ECCN? -Yes -No * 30. Have you accessed any research findings or reports through the ECCN? - Yes - No (please explain why not in the comment box below) * 31. How have you accessed research findings/reports through the ECCN? Please select all that apply • Monthly Newsletter Events • Webcasts Social Media Blogposts Website • Other (please specify) *32. Have you ever applied learnings from the research findings/reports accessed through the ECCN in your work? - Yes -No (please explain why not in the comment box below) *33. To what extent has the research accessed through the ECCN influenced your work in the education in crisis and conflict community? • Not at all • Increased awareness • One-time input • Occasional input • Long-term or sustained changed practice (process, tools, etc.) • Other (please specify) * 34. Please select all the areas in which you have applied learnings • Theory of change development • Program design • Training materials • Publications xxvi • Evaluations • Other (please specify) Professional Development * 35. Are you aware of training opportunities available through ECCN? -Yes -No * 36. Have you accessed any training opportunities available through the ECCN? -Yes -No (please explain why not in the comment box below) * 37. Have you applied what you gained through the ECCN training opportunity to your work? • Yes • Not yet, but I plan to • No (please explain why not in the comment box below) * 38. To what extent have the training activities accessed through the ECCN influenced your work in the education in crisis and conflict community? • Not at all • Increased awareness • One-time input • Occasional inputs • Long-term or sustained changed practice (process, tools, etc.) • Other (please specify) ECCN Priority Areas and Themes The ECCN has identified a set of four priority area objectives and corresponding outcomes within each which guide the ECCN’s implementation efforts. For each priority area objective and set of outcomes listed below, please select the extent to which they are relevant to your work. *39. Priority Areas Not Relevant Somewhat Relevant Very Relevant Learning environments are safer EiCC programming has improved Equity is enhanced Local institutions are stronger *40. Outcomes within Priority Area 1: Learning environments are safer Not Relevant Somewhat Relevant Very Relevant Evidence and tools for understanding, monitoring, and improving safety of learning environments Evidence-based practice in implementing Social Emotional Learning and Psychosocial Support in EiCC programs xxvii *41. Outcomes within Priority Area 2: EiCC programming has improved Not Relevant Somewhat Relevant Very Relevant Validated tools for analysis of Conflict in Crisis Contexts (to inform initial program design and ongoing adaptation) Appropriate theories of change and Adaptive Management guidance to EiCC programs Approved Program and Activity Costing Guidance within EiCC programs Improved EiCC Progress/Outcome Indicators and M&E Tools Cross-cutting Research for EiCC *42. Outcomes within Priority Area 3: Equity is enhanced Not Relevant Somewhat Relevant Very Relevant Guidance for Accelerated/ Alternative Education Programming Research and guidance to inform EiCC program design for Equity Knowledge related to ICT applications in EiCC Effective Teaching and Learning Strategies adapted to EiCC 43. Outcomes within Priority Area 4: Local institutions are stronger Not Relevant Somewhat Relevant Very Relevant Research and tools for Local Education Institutional Capacity Development in Crisis and Conflict Contexts Follow-up Interview If there are any responses that we would like to gain additional insight on, we would appreciate the ability to conduct a follow-up interview by phone or skype. If you are willing to do so, please provide your contact details below. Please note that personally identifying information will not be connected to the survey responses or the follow-up interview. All data collected both through the surveys and interviews will maintain the anonymity and only presented at the demographic aggregate levels. 44. Contact Information Name Phone Number E-mail Address Skype Conclusion Thank you very much for taking the time to complete this survey. Your time and input are very xxviii important and will help us better understand the progress, successes, and challenges the ECCN has had to date. xxix Annex III: Sources of Information Table A3.1: Desk review list Copy of ECCN Mid-Term Evaluation Kick-Off Meeting PPT AID-OAA-M-14-00007 to EDC ECCN Revised Communication Strategy _Mary 31 2016 ECCN Community of Practice Survey-Responses SurveyMonkey USAID ECCN PMEP Feb 23_ED Comments Addressed NO HIGHLIGHTS FINAL USAID ECCN Progress Monitoring Report 2016 NB: Format for this report changed from previous reports so breakdown at right is somewhat arbitrary USAID ECN QR July-Sept 2014 FINAL Oct-Dec 2014 final USAID ECCN USAID ECCN QR Jan-March 2015 FINAL USAID ECCN QR Apr-June 2015 USAID ECCN Quarterly Report July-September 2015 Final 2015 4th Quarter and Annual Progress Report Final 2016 Apr-June QR Final 7 29 16 July-September 2016 QR USAID ECCN Final USAID ECCN Year 1 Work Plan revised 2.6.15 USAID ECCN YEAR 2 Work Plan_Final DRAFT Copy of USAID ECCN Year 3 Work Plan Final ECCN Website RERA Tool http://eccnetwork.net/wp￾content/uploads/USAID_Checklist_Conflict_Sensitivity_14FEB27_cm.pdf Communities of Practice: A Brief Introduction, by Étienne Wenger, 2013. Accessed online on 17 May 2017 at wenger-trayner.com/introduction-to-communities-of-practice/ Wenger, E.C. & Snyder, W.M. Communities of Practice: The Organizational Frontier. Harvard Business Review (Jan.-Feb. 2000), pp. 139-145 USAID ECCN prof dev plan 2016-2019 USAID PD Landscape 2015 Stakeholder baseline survey 2016 August USAID Education Strategy 2011 - 2015 (Extended to September 2017) xxx Annex IV: Survey results Descriptive Statistics from the Survey Table A4.1: Country break-up of survey respondents Question: Which country(ies) are you working in or do you carry out work in? Please select all that apply. Country # of respondents Burundi, Colombia, Egypt, Ethiopia, Israel, Laos, Morocco, Papua New Guinea, Rwanda, Turkey 1 each Nepal, Sierra Leone, Tanzania, Uganda, USA 2 each Iraq 3 Bangladesh and Nicaragua 4 each Senegal and Global 5 each Burma/ Myanmar, Guatemala, Haiti, Somalia 7 each El Salvador 8 West Bank/Gaza and Yemen 9 each Lebanon 10 Honduras and South Sudan 11each Pakistan and Philippines 12 each Afghanistan 13 Mali 16 DRC and Jordan 17 each Kenya and Liberia 18 each xxxi Nigeria 30 Answered question (skipped) 93 (0) Figure A4.1: USAID Affiliation of Respondents Figure A4.2: Type of USAID Affiliations Figure A4.3: Length of USAID-Affiliated Respondents’ Involvement with ECCN xxxii Figure A4.4: Length of USAID-Affiliated Respondents’ Work on Education in Crisis and Conflict Figure A4.5: Professional Affiliations of Respondents Not Affiliated with USAID Figure A4.6: Length of Involvement in ECCN of Respondents Not Affiliated with USAID xxxiii Figure A4.7: Length of Work on Education in Crisis and Conflict (non-USAID Respondents) Figure A4.8: Availability of Opportunities to Engage with ECCN Figure A4.9: USAID-Affiliated Respondents’ Perceptions of Availability of Opportunities to Engage with ECCN xxxiv SELECTION COMMENT MANY I HAVE BEEN ENGAGING WITH THROUGH INTERNATIONAL WORKSHOP, WEBCAST, BLOGS, WEBINAR, NEWSLETTERS ETC ADEQUATE BETWEEN WEBINARS, BLOGS, RESEARCH ALL POSTED ON THE WEBSITE (AND IN REALIZATION OF HOW BUSY EVERYONE IS) THE OPPORTUNITIES ARE PERFECTLY ADEQUATE INADEQUATE INEE IS USUALLY A BETTER SOURCE FOR ME ADEQUATE I'M SURE THERE ARE PLENTY OF OPPORTUNITIES IF I SEEK THEM OUT. MANY ECCN PROVIDES THE PLATFORM FOR INTERACTION, SHARE RESOURCES MATERIAL AND OPPORTUNITY TO GIVE FEEDBACK. SOMEWHAT ADEQUATE THE OPPORTUNITIES INCLUDE THE ABILITY TO WATCH WEBINARS OR ATTEND OCCASIONAL EVENTS. NEITHER OF THESE REGULAR ACTIVITIES PROVIDE AN OPPORTUNITY FOR REAL DIALOGUE (WHICH WOULD BE MY DEFINITION OF ENGAGEMENT), EXCEPT EVENTS, WHICH ARE MORE RARE. WHILE ECCN IS OPEN TO SUPPORTING PEOPLE'S IDEAS, IT SEEMS THE ONUS IS ON THE PARTNER TO COME UP WITH THE IDEA AND DO A LOT OF THE PLANNING. THERE ARE FEWER "LOW-LIFT" ENGAGEMENT OPPORTUNITIES THAT EVERYONE CAN ACCESS. INADEQUATE OFTEN NOT A LOT OF LEAD TIME TO TAKE PART, OR TIMING IS OFF FOR MY REGION (TOTALLY FINE THOUGH!) BUT MISS NOT LISTENING IN TO WEBINARS, ETC. DUE TO OTHER CONFLICTS OR TIMING. IHAVEN'T SEEN AN UPSHOT OF THE WEBINARS AND DISCUSSION FOLLOWING THEM, BUT MAYBE I'M MISSINGSOMETHING. Figure A4.10: Frequency of Respondents’ Engagement with ECCN xxxv Figure A4.11: Frequency of USAID-Affiliated Respondents’ Engagement with ECCN Figure A4.12: Ease of Engagement with ECCN Figure A4.13: USAID-Affiliated Respondents’ Perceptions of Ease of Engagement with ECCN xxxvi Figure A4.14: Potential Mediums for Engagement with ECCN MOBILE PHONES PODCASTS REGIONAL MEETINGS OPEN TO NO USAID AFFILIATED STAFF ONLINE TRAININGS AND WORKING GROUPS PROFESSIONAL DEVELOPMENT IN THE FIELD ESTABLISH SUB GROUPS MORE TRAINING OPPORTUNITIES GENERAL MEETING TWICE A YEAR TWITTER MORE WEBCASTS REGULAR WORKING GROUP MEETINGS SHORT, IN-PERSON EVENT SERIES THAT ARE ALSO CAPTURED FOR WIDER DISSEMINATION, ONLINE DIALOGUES AND CROWDSOURCING ACTIVITIES REGIONAL EVENTS ON THE GROUND PERIODIC TDYERS THAT PLAYS A MENTOR/COACHING/CONSULTANCY ROLE ON AN "AD-LARGE" BASIS. xxxvii Figure A4.15: Quality of ECCN COP Management Figure A4.16: USAID-Affiliated Respondents’ Perceptions of Quality of ECCN COP Management SELECTION COMMENTS DON'T KNOW HARD TO SAY - IT'S A BIT ERRATIC TO JUDGE - SOMETIMES IT IS GREAT, AND SOMETIMES IT IS NOT SO GREAT DON'T KNOW ONE OF THOSE THINGS I ALWAY INTEND TO CHECK, BUT DON'T REALLY GET THERE... SOMEWHAT WELL MANAGED WE COULD DO BETTER REACHING OUT TO OTHER TEAMS IN OTHER COUNTRIES. xxxviii SOMEWHAT WELL MANAGED THIS WAS DIFFICULT TO RATE. I THINK ON THE WHOLE, IT IS WELL-MANAGED. HOWEVER, I GENUINELY BELIEVE THEY TAKE ON TOO MUCH WORK. SO IT IS HARD TO SEE WHAT THE CLEAR FOCUS OF THE ECCN IS. SOMEWHAT WELL MANAGED DOING THEIR BEST SOMEWHAT WELL MANAGED ACTIVITIES ARE EXTREMELY SLOW TO MOVE. THE COHERENCE BEHIND HOW ACTIVITIES CONNECT TO EACH OTHER, WHAT THE THEMATIC FRAMEWORK IS (WHICH COULD HELP PEOPLE NAVIGATE CONTENT), AND THE OVERALL COMMUNICATIONS PLAN BEHIND OF ALL THE WORK IS LACKING. THERE SEEMS TO BE AN EFFORT TO PUT THAT ON THE STEERING GROUP, IN A PARTICIPATORY WAY,BUT THAT SEEMS LIKE A LESS SUCCESSFUL APPROACH. THIS IS THE KIND OF THING THAT IS VERY HARD TO PUT FRONT AND CENTER IN THE BUSY LIVES OF OUR EDUCATION OFFICERS. I STRUGGLE WITH THAT WITH OUR NEWSLETTER, EVEN MAKING IT OPEN FOR MISSIONS TO SUBMIT TO DOESN'T ALWAYS MEAN FOLKS WILL SUBMIT NOR READ IT. I THINK THIS IS SOMETHING THAT COULD BE BRAINSTORMED. GET IT OUT THERE IN POWERFUL WAYS, IT'S HARD. IN SOME WAYS, I THINK THAT VIDEO IS POWERFUL. AFTER WRITING AND SO ON, IT CAN BE REALLY NICE TO WATCH A PERSON WHO'S KNOWLEDGEABLE SPEAK. IS THERE VIDEO OF THE WEBINARS. WHAT ABOUT DOING A TEDX ECCN, AND SOLICITING OUR IMPLEMENTERS AND BENEFICIARIES FROM THE FIELD TO ENGAGE CRITICALLY IN WHAT THEY EXPERIENCE -- ALLOWING ROOM FOR INNOVATION? IT WAS NEVER CLEAR HOW ECCN IS DIFFERENT FROM INEE. WE GET MORE BENEFIT FROM THE INEE SERVICES EDC HAS DONE A GOOD JOB OF ENGAGING MEMBERS AND TRYING TO KEEP THE CONVERSATION GOING. VERY WELL. MEEETING ACROSS DESPERATE TIME ZONES CAN BE DIFFICULT, BUT THERE ISN'T REALLY ANYTHING THAT CAN BE DONE ABOUT THAT. THE ST IS ALWAYS TRYING TO THINK OF NEW WAYS TO ENGAGE THE COMMUNITY. THERE IS NEW INFORMATION POSTED EACH TIME AND RESPONSES FROM USERS ARE ATTENDED TO BY KNOWLEDGEABLE PEOPLE. xxxix Figure A4.17: Past Requests for Support from ECCN ST Figure A4.18: Satisfaction with Response from ECCN ST Figure A4.19: Knowledge Gained through ECCN xl Figure A4.20: Knowledge Gained through ECCN by USAID-Affiliated Respondents LOCAL/ REGIONAL EVENTS- WHILE THE WEB PROVIDES EASIER DISSEMINATION, NOTHING SHOULD STOP LOCAL NETWORK MEMBERS FROM CONVENING AND EXTENDING THE ECCN THROUGH LOCAL ACTIVITIES SEEMS VERY REDUNDANT WITH INEE I HAVE NOT ENGAGED IN A WAY THAT WOULD ALLOW ME TO GAIN KNOWLEDGE. THIS IS A FAILING ON MY PART, NOT ECCN'S PART. I WANT TO SAY YES AND NO. THE QUESTION IS NOT WHAT KNOWLEDGE, BUT RATHER THE DEPTH OF KNOWLEDGE. HAVE A GAINED KNOWLEDGE, SURE, BUT THE EVENTS AVAILABLE FOR NON-USAID MEMBERS ARE LIMITED FOR AN DEEPER UNDERSTANDING OF TOPICS. AT TIMES IT FEELS MORE OF A NETWORKING EVENT - NOT JUST FOR ME, BUT FOR THE FIELD GENERALLY. I HAVEN'T PARTICIPATED ENOUGH TO ANSWER YES ECCN HAS WEAK SOCIAL MEDIA AND INEE TENDS TO HAVE MORE ROBUST RESOURCES ECCN CONTENT IS OFTENTIMES TOO ACADEMIC, AND NOT THAT PRACTICAL / SUITABLE FOR PRACTITIONERS THE ECCN TRAINING I ATTENDED PROVIDED LITTLE - IF ANY - PRACTICAL GUIDANCE ON HOW I COULD BETTER PROGRAM EDUCATION IN CRISIS AND CONFLICT PROGRAMMING. xli Figure A4.21: Knowledge Sharing through ECCN Figure A4.22: USAID-Affiliated Respondents’ Knowledge Sharing through ECCN WHY NOT? THE OPPORTUNITY HASN'T PRESENTED ITSELF YET AT THIS POINT I AM A PASSIVE PARTICIPANT AS I LEARN MORE ABOUT EDUCATION IN CRISES. DON'T FEEL SUFFICIENTLY ENGAGED - INTERESTED IN THE TOPICS, BUT TOO OVERWHELMED BY MY OWN WORKLOAD TO TAKE ON MORE. NONE OF THE NEWSLETTERS HAVE BEEN RELEVANT TO MY WORK. THE TIMING OF THE WEBCASTS IS NOT SUITABLE TO MY COUNTRY. MOST OF THE INTERACTION EVENTS COME IN MORNINGS IN DC, WHICH IS THE EVENING IN MY COUNTRY. BECAUSE OF TIME AND PRIORITY xlii NOT THE PRIMARY ECCN REPRESENTATIVE AT MY ORGANIZATION. DIDN'T KNOW I COULD NOT YET, BUT THIS YEAR SEVERAL OPPORTUNITIES WILL PRESENT THEMSELVES. I HAVE NOT BEEN AWARE OF AN OPPORTUNITY FOR ME TO SHARE OUTWARDS. I HAVE SHARED IN THAT I HAVE BENEFITTED FROM KNOWLEDGE OF OTHERS IN WEBCASTS, NEWSLETTERS, AND EVENTS. I HAVE NOT ENGAGED IN A WAY THAT WOULD ALLOW ME TO SHARE KNOWLEDGE. THIS IS A FAILING ON MY PART, NOT ECCN'S PART. I HAVEN'T PARTICIPATED ENOUGH TO ANSWER YES AS USAID STAFF WE ARE ASKED TO SHARE INFORMATION. BUT IT IS DIFFICULT BECAUSE MUCH OF WHAT WE HANDLE IS PROCUREMENT SENSITIVE. IT SEEMS TO ME THAT THAT IS A BIG CHALLENGE, THAT USAID WORKS WITH PROCUREMENT SENSITIVE INFO AND OUR PARTNERS ARE RELUCTANT TO SHARE DUE TO COMPETITION FOR NEW AWARDS. DON'T SEE OPPORTUNITIES I AM MORE OF A CONSUMER OF THEIR PRODUCT. I TEND TO SHARE KNOWLEDGE THROUGH SCHOLARLY PUBLICATIONS. I WILL START SHARING KNOWLEDGE SOON I POSTED RESPONSE WHICH WERE SUPPORTED BY TWO RESPONDENTS AND I WAS HAPPY WITH THEIR RESPONSE BUT TIME CONSTRAINTS . LACK OF TIME I HAVEN'T HAD AN OPPORTUNITY YET AS I AM RELATIVELY NEW. NO, I HAVE NOT SHARED, BECAUSE I JOINED YOU SHORTLY. GUESS MY SCHEDULE NOT SURE HOW TO SHARE KNOWLEDGE, DOES NOT FEEL THAT USERS ARE INVITED TO SHARE KNOWLEDGE JUST STARTED IN MARCH. HAVEN'T HAD MUCH OPPORTUNITY FOR ENGAGEMENT. WOULD LOVE TO, BUT HOW? xliii Figure A4.23: Respondents’ Preferred Methods for Knowledge Sharing through ECCN Figure A4.24: USAID-Affiliated Respondents’ Preferred Methods for Knowledge Sharing through ECCN Figure A4.25: Respondents’ Awareness of Resources Available through ECCN xliv Figure A4.26: USAID-Affiliated Respondents’ Awareness of Resources Available through ECCN Figure A4.27: Respondents’ Past Access of Programming Resources Figure A4.28: USAID-Affiliated Respondents’ Past Access of Programming Resources xlv WHY HAVEN'T YOU ACCESSED THESE AVAILABLE PROGRAMMING RESOURCES? RESOURCE COMMENTS CHECKLIST FOR CSE NOT NEEDED RERA DID NOT KNOW ABOUT IT WHEN DESIGNING RELEVANT PROGRAM N/A TO MY WORK NO NEED TOC RESOURCES NOT RELEVANT NOT NEEDED INDICATORS/ PMP ANALYSIS I DIDN'T HAVE A NEED COSTING GUIDANCE NOT YET RELEVANT TO MY WORK EVIDENCE GAP MAPS LOOKING FOR MORE SPECIFIC CONTENT FEEDBACK LOOPS RESOURCES WE HAVE BEEN DEEPLY ENGAGED IN MANY OF THEM SO NO NEED TO ACCESS THEM AGAIN SLE TOOLS PROBABLY DIDN'T FEEL REQUIREMENT TIME CONSTRAINTS NOT NEEDED Figure A4.29: Respondents’ Application of Programming Resources to Work Figure A4.30: USAID-Affiliated Respondents’ Application of Programming Resources to Work xlvi WHY HAVEN’T YOU APPLIED OR DO NOT PLAN TO APPLY THIS/THESE PROGRAMMING RESOURCES ACCESSED RESOURCE COMMENTS CHECKLIST FOR CONFLICT SENSITIVITY IN EDUCATION NOT RELEVANT I DON'T DIRECTLY DO PROGRAMMING, BECAUSE I DON'T WORK FOR A MISSION NOT RELEVANT TO MY WORK I DON'T WORK DIRECTLY ON EICC PROGRAMMING RAPID EDUCATION AND RISK ANALYSIS DIDN'T HAVE THE OPPORTUNITY NOT RELEVANT TO IMMEDIATE WORK I DON'T DIRECTLY DO PROGRAMMING, BECAUSE I DON'T WORK FOR A MISSION NOT RELEVANT TO MY WORK THEORIES OF CHANGE RESOURCES I STILL NEED TO ENGAGE MORE WITH THE RESOURCES NOT RELEVANT TO IMMEDIATE WORK I DON'T DIRECTLY DO PROGRAMMING, BECAUSE I DON'T WORK FOR A MISSION OUT OF TOUCH WITH THISGROUP DIDN'T FIND THEM APPLICABLE INDICATORS/ PMP ANALYSIS N/A TO MY WORK I DON'T DIRECTLY DO PROGRAMMING, BECAUSE I DON'T WORK FOR A MISSION Figure A4.31: Influence of Programming Resources on Work xlvii Figure A4.32: Influence of Programming Resources on Work of USAID-Affiliated Respondents Figure A4.33: Potentially Beneficial Programming Resources for Inclusion in ECCN Figure A4.34: Potentially Beneficial Programming Resources for Inclusion in ECCN (USAID-Affiliated Respondents) xlviii IF YES, PLEASE EXPLAIN ADVOCACY TOOLS THAT CAN ENABLE "PLANNING BEYOND THE CRISIS" SAMPLE PROGRAMS IN CONFLICT AND CRISIS COUNTRIES, LESSONS LEARNT WORKSHOPS AND PRESENTATIONS SHARED THROUGH THIS NETWORK ALL THE TOOLKITS AND GUIDES. JOB SITE, MORE COMMUNITY COMMUNICATION (MULTI-WAY) FREQUENCY OF FACE-TO-FACE EXPERIENCE SHARING AND LEARNING OF BEST PRACTICES AMONG MEMBERS TO BUILD CAPACITY FOR IMPROVED EFFICIENCY GREATER EMPHASIS TO BRING ECCN AND EDUCATORS FROM THE REPRESENTED COUNTRIES TOGETHER TO WORK ON PLANS AND PROGRAMS. THIS WOULD ALSO REQUIRE FUNDING. MANY - BUT THERE IS ONLY SO MUCH THAT THE ST CAN DO AND WE ARE PRIORITIZING THE PRODUCTS BASED ON WHAT WE SEE AS USER DEMAND. IN PARTICULAR, THE INTERACTIVE DATABASE THAT SHOWS INFORMATION ABOUT EACH USAID-FUNDED EICC PROGRAM WILL BE HUGELY USEFUL, ESPECIALLY IF WE ARE ABLE TO GET DATA FROM EACH OF THOSE PROJECTS (WHICH IS VERY DIFFICULT AND UNLIKELY, BUT WE THINK BIG)! INTEGRATED TOOLS - INTEGRATING SAFER LEARNING WITH RERA SO IT'S ONE TOOL WITH MULTIPLE COMPONENTS RATHER THAN TWO OR MORE TOOLS WITH SOME OVERLAP. RATHER HAVE ONE AND TRAIN ON ONE. TRAINING MODULES IT WOULD BE INTERESTING TO SEE AN EXPLICIT EFFORT LOOKING AT INTERSECTIONS OF OTHER FIELDS TO PROMOTE MORE RESPONSIVE REPROGRAMMING (E.G., CHILD PROTECTION, MENTAL HEALTH AND PSYCHOSOCIAL SUPPORT, SGBV, ETC.). THIS WOULD NEED TO DRAW ON EXPERTS FROM OUTSIDE THE ECC COMMUNITY. PEACE. TEACHING STAFF, BASIC NEEDS FOR CLASS WORK AND SECURITY COMFORT ARE MOSTLY LACKING. SENSITIZATION AND AWARENESS ARE THE BEST WAY TO MOBILIZE SUPPORT IS STAKEHOLDERS AS GOVERNMENT ALONE CANNOT ADDRESS THE PROBLEMS. BEST DONE BY THE MEDIA SUCH VOA EXACTLY WHAT WE PLAN TO EXECUTE IN THE CRISIS AND CONFLICT AREA OF NORTH EASTERN NIGERIA AS FROM MAY 15 TO 21ST MORE ON CONFLICT SENSITIVITY SELF ASSESSMENT AND STRATEGIES AND MORE ON PEACE BUILDING STRATEGIES AND TOOLS MORE RESOURCES ON POST-DISASTER PROGRAMMING. xlix INEE TOOLS -- MANY OF THEM ARE AVAILABLE AND ARE GREAT RESOURCES INEE ALSO HAS A TON OF RESOURCES SO NO NEED TO REPLICATE RESEARCH ON YOUTH ENGAGEMENT IN POST/CONFLICT CONTEXTS. NOT REALLY SURE. Figure A4.35: Awareness of Research Resources available through ECCN Figure A4.36: USAID-Affiliated Respondents’ Awareness of Research Resources Available through ECCN Figure A4.37: Past Access of ECCN Research Findings and Reports l Figure A4.38: Past Access of ECCN Research Findings and Reports by USAID-Affiliated Respondents WHY NOT? NOT HAD THE CHANCE FOR A DETAILED DIVE INTO EXISTING STUDIES I GUESS I HAVE NOT EXPLORED THE WEBSITE ENOUGH. LACK OF TIME TO REALLY EXPLORE - ALTHOUGH PROBABLY MORE LIKELY TO AS A RESULT OF THIS SURVEY! OK, YOU GUYS ARE NOT GOING TO BELIEVE THIS, BUT WHEN I CLICKED ON ONE OF THE LINKS IN THE NEWSLETTER (THE ONE ABOUT EDUCATION IN SYRIA BY NINA ETYEMEZIAN FROM RTI), I COULD NOT ACCESS IT FROM MY DESK AT USAID. SERIOUSLY. SO, I THINK THAT SHOULD GET SORTED TOUT DE SUITE. THE IRONY IS NOT LOST ON ME THAT ACCESS TO A USAID FUNDED PROJECT WEBSITE IS BLOCKED TO USAID STAFF. RERA PILOTING AND TESTING. CONFLICT RESEARCH TIME CONSTRAINTS BEEN AWAY FOR SIX MONTHS ON A RESEARCH ASSIGNMENT NOT SURE WHAT THEY ARE, IN PARTICULAR NOT COUNTRY RELEVANT NO, BECAUSE I JOINED SHORTLY. I PLAN TO DO THAT SOONLY. I DID MAKE ATTEMPT TO ACCESS li Figure A4.39: Means of Accessing ECCN Research Findings and Reports Figure A4.40: Means of Accessing ECCN Research Findings and Reports (USAID-Affiliated Respondents) Figure A4.41: Respondents’ Application of ECCN Research Findings and Reports to Work lii Figure A4.42: USAID-Affiliated Respondents’ Application of ECCN Research Findings and Reports to Work Figure A4.43: Influence of Research Accessed through ECCN on Work in EiCC Figure A4.44: Influence of Research Accessed through ECCN on USAID-Affiliated Respondents’ Work in EiCC liii Figure A4.45: Areas in which Respondents have Applied Learning Figure A4.46: Areas in which USAID-Affiliated Respondents Have Applied Learning Figure A4.47: Respondents’ Awareness of ECCN Training Opportunities liv Figure A4.48: USAID-Affiliated Respondents’ Awareness of ECCN Training Opportunities Figure A4.49: Respondents’ Past Access of ECCN Training Opportunities Figure A4.50: USAID-Affiliated Respondents’ Past Access of ECCN Training Opportunities lv Figure A4.51: Reasons Respondents Have Not Accessed ECCN Training Opportunities Figure A4.52: Respondents’ Application of ECCN Trainings to Work Figure A4.53: USAID-Affiliated Respondents’ Application of ECCN Trainings to Work lvi Figure A4.54: Influence of ECCN Training Activities on Respondents’ Work in EiCC Figure A4.55: Influence of ECCN Training Activities on USAID-Affiliated Respondents’ Work in EiCC Figure A4.56: Relevance of ECCN Priority Areas on Respondents’ Work lvii Figure A4.57: Relevance of ECCN Priority Areas on USAID-Affiliated Respondents’ Work Figure A4.58: Outcomes – Safer Learning Environments Figure A5.59: USAID-Affiliated Respondents’ Perceptions of Relevance of ECCN Priority Area 1 lviii Figure A5.60: Outcomes – Improved EiCC Programming Figure A5.61: USAID-Affiliated Respondents’ Perceptions of Relevance of ECCN Priority Area 2 Figure A5.62: Respondents’ Perceptions of Relevance of ECCN Priority Area 3 lix Figure A5.63: USAID-Affiliated Respondents’ Perceptions of Relevance of ECCN Priority Area 3 Figure A5.64: Respondents’ Perceptions of Relevance of ECCN Priority Area 4 Figure A5.65: USAID-Affiliated Respondents’ Perceptions of Relevance of ECCN Priority Area 4 lx Annex V: KII Code List Codes were used to organize and sort through the interview notes the team compiled. The following codes were created after going through the textual data (KII notes), to fit emerging ideas, concepts, and themes into categories: Question Code list Relationship with ECCN - M&E team - Research Manager - Communications/website - PMU - Curricula/ course development - Co-managing partner programs - Another government network - Advisor - Organizational development/learning - INEE - C&C background - Strategy and TOC - Education Office Familiarity with ECCN objectives - Not aware - Generally aware/familiar - Very familiar Role in creation of ECCN objectives - COP design/ framework (TOC) - Learning agenda - ICT (website) - Engaging local institutions - Collaboration with INEE - Ongoing dialogue/evaluation Role in implementation of ECCN objectives - Objectives vs. themes - Accelerated and alternative education programming - Task teams - Indicator development - Monitoring ECCN communication and engagement/ coordination - Field testing - Coordination (general) - Work planning Opinion of how the ECCN has met its objectives - Functions as COP - Vehicle/platform to bring people together/ networking - Doing quite well/ seems to be moving along - Challenging questions- things are changing/evolving - Iterative process so hard to gauge overall level - Slow start ECCN Successes - Accelerated education - Presentations, webcasts well attended - RERA (specifically mentioned) - Pilot process to adapt and draft lxi - Good, committed team - CIES training in Nigeria - IP involvement/leading/driving activitiesimproving - Communications - Collaborative/joint approach with INEE in many areas - Adaptive management - Removing password protection on website ECCN challenges - Changed practice/ demonstrable impact will take some time - Tracking/analytics/evidence - Indicators (especially equity indicators) lacking - Friction between USAID and ECCN team - USAID control (top-down) - Goals/objectives/vision lacking clarity – especially in the beginning - very U.S. centric - Field/community engagement lacking - Time/resources (manpower) - Clarity in staff roles islacking - Engagement and communications - Competition with COP - Lack of relationships with other COPs in E3/EDoffice Engagement with ECCN members (frequency) - lack of system/follow up - frequent - not actively involved Engagement with ECCN members (mode) - Not aware of all modes available - Unidirectional (through USAID) - Specific modes (like newsletter etc.) Engagement with ECCN members (tracking) - Website and newsletter clicks/opening - Personal notes from conversations - Notsystematic Engagement: standard processes - TORs - Pre-and post-course evaluations - Online-blogs, webcasts etc. - PD survey Engagement: your understanding of membership and network use - Not great in Mission and Field - no/little grassroots involvement/ outside of USAID - Providing resources - Beginning lateral collaborations with members - Website USAID E3/ED: role and mission - aware - not aware USAID E3/ED: ECCN contribution - ECCN greatly contributes to E3/ED's overall mission - ECCN (somewhat) contributes to E3/ED's overallmission - ECCN does not contribute to E3/ED's overallmission USAID E3/ED: ECCN contribution (how/why) - Provide research, information, tools - Publicity/ knowledge sharing - Accelerated alternative programming - (some) work on equity lxii - Help influence decision makers to invest in education - Goal 3 - Not much/ doesn’t seem very bidirectional ECCN priority areas and themes - aware - not aware ECCN priority areas and themes: relevance to members - very relevant - somewhat relevant - irrelevant COMMENTS: - difference between USAID and IP demands - FSO turnover inhibits relationship building with IPs - Security concerns in the field ECCN priority areas and themes: relevance to E3/ED - very relevant - somewhat relevant - irrelevant E3/ED needs - aware - somewhat aware - unaware E3/ED needs: evolved? - sustainability/ scaling - Goal 3 targets are dysfunctional - Not a coherent view - Learning agenda - USAID push for evolving - a lot more rapid info asks E3/ED needs: ECCN contributes? -Yes -maybe -no - don’t know COMMENTS - To the degree they reach out and ask - Tech outputs and assistance - Finding ways to leverage community forsupport - regular/meaningful engagement with EiCC and other USAIDoffices - co-develop products and trainings - focused on USAID (internal staff) capacity building - Don’t know ECCN as a COP to address needs of E3/ED: pros/cons - Positive - Networking - doesn’t work with USAID funding restrictions/accountability mechanisms - is not a COP in practice - unsure of usefulness of COP model - INEE vs ECCN - need to give it time lxiii Annex VI: Disclosure of any Conflicts of Interest Name William P. Dant Title Senior Monitoring and Evaluation Specialist Organization Dexis Consulting Group Evaluation Position? X Team Leader Team member Evaluation Award Number (contract or other instrument) AID-OAA-M-15-00016 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Education in Crisis and Conflict Network Implementer: Education Development Center USAID Contract No. AID-OAA-M-14- 00007 MOBIS # GS-10F-0406P I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or theimplementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or issignificant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience orseeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 07/11/2017 lxiv Name Charu Vijayakumar Title Monitoring and Evaluation Specialist Organization Dexis Consulting Group Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) AID-OAA-M-15-00016 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Education in Crisis and Conflict Network Implementer: Education Development Center USAID Contract No. AID-OAA-M-14- 00007 MOBIS # GS-10F-0406P I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct orsignificant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 07/11/2017 lxv Name Amun Nadeem Title Analyst Organization Dexis Consulting Group Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) AID-OAA-M-15-00016 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Education in Crisis and Conflict Network Implementer: Education Development Center USAID Contract No. AID-OAA-M-14-00007 MOBIS # GS-10F-0406P I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous workexperience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 07/11/2017 lxvi For further information concerning this publication, please contact the Office of Education (E3/ED) at USAID (add contact information here) U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523