This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EDUCATION IN CRISIS AND CONFLICT NETWORK (ECCN) AND THE GLOBAL READING NETWORK (GRN) FINAL EVALUATION REPORT USAID Reading and Access Prepared under Contract No.: AID-OAA-M-13-00010 KATIE PHILLIPS USAID READING AND ACCESS FINAL EVALUATION EDUCATION IN CRISIS & CONFLICT NETWORK (ECCN) AND THE GLOBAL READING NETWORK (GRN) Evaluation Report April 24, 2020 Prepared under Contract No.: AID-OAA-M-13-00010 Submitted to: Ben Sylla, USAID E3/ED Nathaniel Haight, USAID Graciela Briceno, USAID Prepared by: Katharine Mark Lori Reid Zoe Grotophorst With: Alexander Rigaux, Michelle Davis, Mithila Iyer Submitted by: NORC at the University of Chicago DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID Education In Crisis & Conflict Network (ECCN) And The Global Reading Network (GRN) |I TABLE OF CONTENTS EXECUTIVE SUMMARY 1 PURPOSE 1 BACKGROUND 1 EVALUATION QUESTIONS 1 NETWORK RESULTS/OUTCOMES 1 STAKEHOLDER USE OF NETWORK PRODUCTS AND SERVICES 2 MISSION ENGAGEMENT 2 NETWORK MANAGEMENT 2 METHODS 2 FINDINGS AND CONCLUSIONS 2 NETWORK RESULTS/OUTCOMES 2 STAKEHOLDER USE OF NETWORK PRODUCTS AND SERVICES 3 MISSION ENGAGEMENT 3 NETWORK MANAGEMENT 4 LESSONS LEARNED 4 RECOMMENDATIONS 5 1. BACKGROUND 6 2. EVALUATION QUESTIONS 7 NETWORK RESULTS/OUTCOMES 7 STAKEHOLDER USE OF NETWORK PRODUCTS AND SERVICES 7 MISSION ENGAGEMENT 7 NETWORK MANAGEMENT 7 3. EVALUATION METHODOLOGY AND LIMITATIONS 7 4. FINDINGS AND CONCLUSIONS 10 A. ECCN 10 NETWORK RESULTS AND OUTCOMES 10 STAKEHOLDER USE OF NETWORK PRODUCTS AND SERVICES 14 MISSION/REMOTE ENGAGEMENT 22 NETWORK MANAGEMENT 27 B. GRN 28 NETWORK RESULTS AND OUTCOMES 28 STAKEHOLDER USE OF NETWORK PRODUCTS AND SERVICES 34 MISSION/REMOTE ENGAGEMENT 40 NETWORK MANAGEMENT 44 C. SYNTHESIS 46 NETWORK RESULTS AND OUTCOMES 46 MISSION/REMOTE ENGAGEMENT 50 NETWORK MANAGEMENT 51 5. LESSONS LEARNED 52 6. RECOMMENDATIONS 55 II | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID ANNEX A. STATEMENT OF WORK OF THE FINAL PERFORMANCE EVALUATION OF USAID ECCN/GRN 58 I. PURPOSE OF THE EVALUATION 58 II. SUMMARY INFORMATION 58 III. BACKGROUND 58 IV. EVALUATION QUESTIONS 61 V. EVALUATION DESIGN AND METHODOLOGY 62 VI. DELIVERABLES AND REPORTING REQUIREMENTS 63 VII. EVALUATION TEAM COMPOSITION 64 VIII. EVALUATION SCHEDULE 64 IX. FINAL REPORT FORMAT 64 X. OTHER REQUIREMENTS 65 ANNEX B: DATA COLLECTION INSTRUMENTS 66 SECTION 1: WEB SURVEY INSTRUMENT 66 SECTION 2: QUALITATIVE PROTOCOLS 77 ANNEX C: SOURCES OF INFORMATION 97 TABLE 1: KEY INFORMANT INTERVIEW PARTICIPANTS – USAID AND IPS 97 TABLE 2: KEY INFORMANT INTERVIEW/DISCUSSION PARTICIPANTS – ECCN MEMBERS 97 TABLE 3: KEY INFORMANT INTERVIEW/DISCUSSION PARTICIPANTS – GRN MEMBERS 98 TABLE 4: LIST OF DOCUMENTS REVIEWED 98 ANNEX D: DISCLOSURE OF ANY CONFLICTS OF INTEREST 101 ANNEX E: ECCN AND GRN SURVERY DATA REPOTS 107 USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |III TABLES Table 1: Survey respondents broken out by affiliation and location ...................................................... 9 Table 2: Utilization Rate of ECCN Products ............................................................................................ 17 Table 3: ECCN Resource Downloads ........................................................................................................ 18 Table 4: Value of ECCN CoP Activities as Reported by Members .................................................... 24 Table 5: GRN Participants (Workshops and Courses) .......................................................................... 30 Table 6: Utilization Rate of GRN Products ............................................................................................... 38 Table 7: Types of Network Members ........................................................................................................ 51 FIGURES Figure 1 : How Members Applied ECCN Resources or Knowledge ................................................. 15 Figure 2: Reasons for Not Applying Accessed ECCN Resources ....................................................... 16 Figure 3: Expectations Met on Aspects of the ECCN CoP .................................................................. 24 Figure 4: How Members Applied GRN Resources or Knowledge ..................................................... 35 Figure 5: Reasons for Not Applying Accessed GRN Resources ......................................................... 37 Figure 6: Value of GRN CoP Activities as Reported by Members ...................................................... 42 Figure 7: Expectations Met on Aspects of CoP - GRN .......................................................................... 42 Figure 8: Comparison of Areas in Which Network Activities Have Been Useful .......................... 48 Figure 9: Why Resources Were Not Applied .......................................................................................... 49 Figure 10: Most Frequent Avenues for Engagement with Network ................................................... 50 IV | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID ACRONYMS AEWG Accelerated Education Working Group BEC Basic Education Coalition CoP Community of Practice CO Contracting Officer COR Contracting Officer’s Representative (EiCC) Education in Crisis and Conflict ECCN Education in Crisis and Conflict Network EDC Education Development Center E3/ED Bureau for Economic Growth/Education/Environment | Office of Education FGD Focus Group Discussion GPE Global Partnership for Education GRN Global Reading Network INEE Inter-agency Network for Education in Emergencies IP Implementing Partner KII Key Informant Interviews LOE Level of Effort NGO Non-Governmental Organization RERA Rapid Education and Risk Assessment Toolkit SLE Safe Learning Environment Assessment Toolkit ST Support Team URC University Research Co. USAID United States Agency for International Development USAID Education In Crisis & Conflict Network (ECCN) And The Global Reading Network (GRN) |1 EXECUTIVE SUMMARY PURPOSE The purpose of this final evaluation of the Education in Crisis and Conflict Network (ECCN) and the Global Reading Network (GRN) is to document key outcomes achieved, capture critical lessons learned and good practices, and provide details of identified effective management and budget oversight methods for use by USAID AORs/CORs; USAID Activity Managers; and implementing partners who will manage future USAID Office of Education￾funded networks, USAID and implementing partner capacity building efforts, and knowledge management activities. BACKGROUND The ECCN contract was designed to support technical leadership in education in crisis and conflict environments to USAID by ensuring that education practitioners and those who support them have the latest information, most practical tools, and sufficient resources. ECCN had the following objectives: develop and support the Education in Crisis and Conflict Community of Practice; build and disseminate evidence for the field; test, disseminate and use guidance, tools, metrics, and theories of change; apply tools in crisis and conflict; and develop, implement, and assess training. The GRN contract, also known as Reading within Reach (REACH), was designed to support a rapid increase in the impact and scale of primary grade reading programs. GRN had the following objectives: Primary Grade Reading Community of Practice established and supported; evidence for the field consolidated, built, and disseminated; innovations trialed, refined, and promoted to the field; evidence-based implementation protocols, metrics, and tools produced and used; training for use of protocols, metrics, and tools developed and rolled out; and effective communications in support of high impact, scaled reading programs. EVALUATION QUESTIONS This final evaluation focuses on eight evaluation questions (EQs), which have been organized into four broad categories. They are as follows: NETWORK RESULTS/OUTCOMES EQ1. What activities were most successful, and why? What activities were ultimately unfeasible, and why? EQ2. What were the unexpected or unanticipated outcomes of ECCN/GRN? What added value or benefit did they provide to USAID and its implementing partners? 2 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID STAKEHOLDER USE OF NETWORK PRODUCTS AND SERVICES EQ3. How did stakeholders use information they received through the ECCN/GRN? To the extent possible, what specific design, implementation, monitoring or evaluation approaches or adaptations did the information inform, and what was the impact of this uptake? EQ4. What specific knowledge products did stakeholders find most valuable? Why? EQ5. What modes of delivery made activities successful/unsuccessful? MISSION ENGAGEMENT EQ6. How were missions and implementing partners engaged? What aspects of the communities of practice did they find most valuable? EQ7. What would they suggest for the future? NETWORK MANAGEMENT EQ8. What worked well in terms of management of ECCN/GRN? What did not? What might have made management of the activities easier and more effective? METHODS NORC completed the performance evaluation using data collection and analysis methods tailored to each of the evaluation questions, and implemented a consecutive mixed method design. This evaluation included document review; qualitative analysis of data gathered from key informant interviews (KIIs) and focus group discussions (FGDs) with members of the networks,1 USAID staff, and network management; and a web-based survey of network members. FINDINGS AND CONCLUSIONS NETWORK RESULTS/OUTCOMES ● EQ1: The most successful activities for ECCN and GRN were very similar. Both networks developed high-quality knowledge products by leveraging the strength of experts and organizations within their respective Communities of Practice (CoPs). Both networks produced and delivered knowledge-sharing events and capacity building opportunities that were considered valuable by both USAID and implementing partners (IPs), and were commended for their convening power. ECCN was able to complete most of their activities with the exception of the knowledge share groups within the CoP. GRN struggled to fulfill 1 “Member” refers to anyone who subscribed to the network’s mailing list. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |3 many of the objectives outlined in its contract, primarily due to budget and staffing constraints. ● EQ2: Both networks provided additional value to USAID by helping to amplify the work of USAID in the field. ECCN and GRN also provided opportunities for IPs to connect with USAID and better understand USAID’s priorities. ECCN had successes in unexpected areas, such as facilitating cross-agency collaboration and overcoming competition among IPs. This speaks to the network’s strength in convening. GRN at times faced difficultly in responding to USAID direction and feedback because of unexpected involvement from so many individuals at USAID. STAKEHOLDER USE OF NETWORK PRODUCTS AND SERVICES ● EQ3: The main ways that stakeholders used information from ECCN and GRN were in program/activity design, improving implementation, and improving their understanding. Neither ECCN nor GRN’s stakeholders applied network resources or information very often in metrics or monitoring and evaluation (M&E). When stakeholders did not apply resources in their work, the main reason given was that the opportunity had not yet arisen. ● EQ4: The average utilization rate of accessed resources from each network was 42-48%. The most valuable products across both networks were toolkits, which members leveraged for program/activity design and in improving implementation. For ECCN, the most valuable products were the RERA Toolkit and AEWG’s Guide to Accelerated Education Principles. For GRN, the most valuable products were the EGRA 2.0 Toolkit, UDL Toolkit, and Coaching in EGR Programs document. ● EQ5: Stakeholders primarily received information from both networks through email and they were very satisfied with the frequency of communications. However, they noted that most of the resources produced were Anglo-centric and available only in English, potentially limiting the reach of network messaging and approaches. GRN stakeholders found live webcasts to be the most valuable, while ECCN stakeholders preferred in-person events, according to the evaluation survey. Despite their preference for webcasts, GRN stakeholders reported many more problems with webcasts than ECCN stakeholders, including the inability to attend because of the time of broadcast and connectivity challenges. Also unique to GRN stakeholders were complaints about the network website, especially the search functionality. MISSION ENGAGEMENT ● EQ6: For both ECCN and GRN, the most common means of engagement was remote members learning new approaches and knowledge from the network. More than 90% of members in each network were interested in continuing membership after project funding ended. Members in both networks ranked access to tools and resources as the most valuable part of being in the CoP. For both networks, the newsletter was the most frequent means of engagement, but websites and webcasts/webinars had robust participation as well. A greater percentage of GRN members felt they had sufficient opportunities to engage 4 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID meaningfully with their network. ECCN was better able to meet its members’ expectations regarding the CoP, especially in the areas of tools/resources, personal capacity building, partner advocacy with USAID, and the opportunity to interact with USAID and other partners around education. ● EQ7: General members of both ECCN and GRN felt that the networks were too focused on the needs and interests of USAID rather than members. These members recommended the networks shift away from producing USAID deliverables and promoting USAID policy and toward developing a better dialogue with remote members about what they need to be successful. General and Mission members in both networks advised involving individuals in the field more in product development, especially early in the process. There was the perception that this would increase the relevance and practicality of resources. General and Mission members of both ECCN and GRN also felt that the networks could increase their recruitment of and engagement with field-based members, including host country governments and local NGOs. GRN’s members recommended serious improvements for remote participants during GRN events. The audio quality and timing of webcasts, as well as the feeling of being ignored when joining in-person events remotely, and the lack of recordings which could be watched later, all contributed to a poor user experience for some GRN members. NETWORK MANAGEMENT ● EQ8: In both networks, USAID CORs have consistently been very engaged in managing the contracts. Both ECCN and GRN had rigid contracts, which affected staffing and activities. There are numerous differences in management-related features between GRN and ECCN which affected implementation of the contracts. ECCN had a consistent Chief of Party from start to finish, while GRN had four different Chiefs of Party. The ECCN contractor, EDC, is an education-focused company, facilitating access to education expertise, while the GRN contractor, URC, did not possess a deep bench of education experts. GRN had a highly prescriptive contract (unlike ECCN) requiring specific numbers of specific types of products and events each year. This became a problem for both COR and contractor. GRN had no subcontractors, while ECCN benefited significantly from its subcontractors, especially the University of Massachusetts. ECCN was able to provide added value to USAID by leveraging low-cost technical support through its university partner. This relationship enabled ECCN to accomplish more than they would have been able to with only the project team. LESSONS LEARNED  General member expectations for the CoP did not align with USAID’s expectations.  Sustainability was not considered at the outset of these contracts.  Both networks missed the opportunity to learn from and share knowledge with people in the field.  Inconsistent and/or incomplete staffing can have a significant negative impact on ability to meet objectives. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |5  The contract structure was too rigid for the type of work requested.  It is best to have a focused technical content area for a CoP.  It is better to focus on quality rather than quantity.  A small business was not ideal for accomplishing some of the objectives of the GRN contract.  It is important to have a clear technical point of contact at USAID with whom the contractor can work. RECOMMENDATIONS The following recommendations were drawn from input received from respondents in our data collection, as well as derived from analysis of all the data and lessons learned.  Distinguish between competing objectives of the networks: USAID-focus or a broader CoP.  Decide on the scope of the network (i.e., audience).  Design materials with the appropriate audience in mind.  Facilitate and broaden access to network materials.  Strengthen outreach and relevance to broader groups in the field.  Consider targeting the needs of USAID Missions and seeking their input in the development of resources.  Consider measures to enhance sustainability at the outset.  Design resources to enhance utilization.  Ensure that the contracting mechanism and the contract itself do not create rigidities for an activity that needs to be nimble and able to pivot quickly.  Improve ease of use for network resources.  Ensure network leadership can focus on partnership, engagement, and planning, rather than just managing a large number of activities.  When creating “knowledge share groups” or similar sub-groups within a larger CoP or learning network, establish clear goals at the outset to ensure better engagement with group members. 6 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID 1. BACKGROUND Education in Crisis and Conflict Network (ECCN) The ECCN contract was designed to support technical leadership in education in crisis and conflict environments to USAID by ensuring that education practitioners and those who support them have the latest information, most practical tools, and sufficient resources. In July 2014, the five-year, $9,967,539 time and materials contract was awarded to the Education Development Center (EDC) through a GSA MOBIS Schedule contract. The contractor was expected to work with USAID and partners to develop a community of practice (CoP) to cover topics relevant to Goal Three of the 2011-2015 USAID Education Strategy (eventually extended until 2018): “Increased equitable access to education in crisis and conflict environments for 15 million learners by 2015,” which was eventually extended until 2018. The purpose of the contract was to procure EDC’s services to assist USAID/E3/ED in accomplishing the following objectives: 1. Develop and Support the Education in Crisis and Conflict Community of Practice 2. Build and Disseminate Evidence for the Field 3. Test, Disseminate and Use Guidance, Tools, Metrics, and Theories of Change 4. Apply tools in Crisis and Conflict 5. Develop, Implement and Assess Training Global Reading Network (GRN) The GRN contract, also known as Reading within Reach (REACH), was designed to support a rapid increase in the impact and scale of primary grade reading programs as per Goal One of the 2011-2015 USAID Education Strategy (eventually extended until 2018): "Improved reading skills for 100 million children in primary grades by 2015.” The five-year time and materials contract was awarded to University Research Co (URC) through a GSA MOBIS Schedule contract in January 2014 at a value of $9,984,594. The purpose of the contract was to ensure that policy makers, reading practitioners, and those who support them - from global donors to school principals - had the latest information and most practical tools to design evidence-based activities focused on helping students learn to read in the early grades and read to learn in the later grades. Under GRN, URC was tasked with consolidating existing evidence on key reading interventions, supporting a number of operational research activities, and field-testing tools for materials development/storage, materials distribution, and data collection. They were also contracted to build and support a USAID-led global research and practitioner Primary Grade Children Reading CoP focused on improving reading outcomes in the primary grades. The contract was expected to accomplish the following objectives: 1. Primary Grade Reading Community of Practice established and supported 2. Evidence for the field consolidated, built, and disseminated 3. Innovations trialed, refined, and promoted to the field 4. Evidence-based implementation protocols, metrics, and tools produced and used USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |7 5. Training for use of protocols, metrics, and tools developed and rolled out 6. Effective communications in support of high impact, scaled reading programs 2. EVALUATION QUESTIONS USAID has requested a final evaluation of both ECCN and GRN to identify and document outcomes, lessons learned, and describe effective management practices for use primarily by USAID and implementing partner staff who will manage future USAID Office of Education￾funded networks, capacity building efforts, and knowledge management activities. After reviewing the evaluation questions proposed by E3/ED, the NORC evaluation team regrouped them as follows to more clearly guide analysis: NETWORK RESULTS/OUTCOMES EQ1. What activities were most successful, and why? What activities were ultimately unfeasible, and why? EQ2. What were the unexpected or unanticipated outcomes of ECCN/GRN? What added value or benefit did they provide to USAID and its implementing partners? STAKEHOLDER USE OF NETWORK PRODUCTS AND SERVICES EQ3. How did stakeholders use information they received through the ECCN/GRN? To the extent possible, what specific design, implementation, monitoring or evaluation approaches or adaptations did the information inform, and what was the impact of this uptake? EQ4. What specific knowledge products did stakeholders find most valuable? Why? EQ5. What modes of delivery made activities successful/unsuccessful? MISSION ENGAGEMENT EQ6. How were missions and implementing partners engaged? What aspects of the communities of practice did they find most valuable? EQ7. What would they suggest for the future? NETWORK MANAGEMENT EQ8. What worked well in terms of management of ECCN/GRN? What did not? What might have made management of the activities easier and more effective? 3. EVALUATION METHODOLOGY AND LIMITATIONS Our evaluation approach to ECCN and GRN activities involved a careful exploration and understanding of their processes of design, implementation and management; a thorough 8 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID review of the activities followed to engage their stakeholders; and identifying intended and unintended outcomes, along with the factors that facilitated such results from multiple sources. NORC completed the performance evaluation using data collection and analysis methods tailored to each of the evaluation questions, and implemented a consecutive mixed method design. This evaluation included document review; qualitative analysis of data gathered from key informant interviews (KIIs) and focus group discussions (FGDs) with members of the network, USAID staff, and network management; and a web-based survey of network members. In this evaluation report, NORC addresses the evaluation questions using the data analyzed from all strands, using triangulation to increase the validity of findings. DOCUMENT REVIEW NORC reviewed the ECCN and GRN contracts and work plans, as well as yearly and quarterly reports; the ECCN mid-term evaluation and the GRN stocktaking report; key products and publications related to each project; and academic and practitioner publications on networks and communities of practice. This review was used in refining the qualitative instruments and completing initial consultations with key program staff, as well as providing essential input on network activities and results. KEY INFORMANT INTERVIEWS AND FOCUS GROUPS The evaluation team conducted a total of 19 KIIs in order to obtain a wide array of insights regarding outcomes, factors of success and failure, network implementation and management, and initial notions of knowledge and product use. In addition, five FGDs were conducted with members (two in-person with Washington-based members, and three virtually with remote members), to explore their experiences engaging with the communities of practice and using network materials and activities. Types of network members represented in KIIs and FGDs included general members (non￾USAID affiliated members of the networks), Mission members (members of the networks that currently work in USAID Missions), and USAID DC members (members of the networks who work at USAID/Washington). KII participants included the ECCN Support Team (ECCN ST), made up of the ECCN Project Director and key personnel; GRN Support Team (GRN ST), made up of the GRN Project Director; and USAID staff, made up of CORs/ACORs and other key stakeholders. As a first step, NORC randomly selected members to participate from the lists provided by USAID and the two networks. Due to the limited initial lists, low response rate, and little lead time, NORC followed up quickly with all invitees and worked with USAID to expand the list as much as possible. In some cases there were not sufficient people available for focus groups, but in order to ensure a variety of perspectives, the evaluation team rescheduled the majority of these for individual KIIs. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |9 EVALUATION SURVEY NORC also conducted a web-based survey of all ECCN and GRN members. The survey instrument consisted of 38 questions tailored to be relevant to members of the individual networks. The survey was launched by ECCN on November 8th, 2019, and GRN on November 12th , with unique links sent to each respondent. A total of 1,471 ECCN members and 3,257 GRN members received the survey. Reminder messages were sent twice, on November 15th and November 18th , by ECCN and GRN respectively, and on November 25th by both. In total, there were 395 respondents to the GRN survey and 105 to the ECCN survey, yielding response rates of 12% and 7% respectively. The following table summarizes the response to each survey. Table 1: Survey respondents broken out by affiliation and location ECCN GRN Affiliation of Respondent Field2 Home￾based Field Home￾based USAID 11 4 31 8 USAID implementer 9 23 55 40 Other international donor 3 3 33 8 Government 3 0 20 1 University/Academia 4 9 33 13 Independent (Consultant/ Researcher) 8 11 36 24 Local NGO/International NGO 3 4 28 16 Other 5 3 25 4 TOTAL 46 57 261 114 Source: Evaluation Survey LIMITATIONS ACCELERATED TIMELINE There was an unusually short period of time allotted for this evaluation, especially given the multiplicity of stakeholders and the mixed methods approach. The challenges in obtaining a sufficient number of candidates for the FGDs and the multiple rounds of discussions concerning the process for implementing the evaluation survey created additional time pressures. Together these limited the amount of time for full analysis of the data collected. REDUCED NUMBERS OF QUALITATIVE RESPONDENTS NORC was not able to include as many FGD participants as desired due to the short lists of candidates identified by USAID and the two networks, and the short timeline of the evaluation. 2 “Field” respondents are those who identified themselves as living in a developing country. 10 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Despite steps taken to reach as wide a group as possible, the range of views included in the qualitative research is not as comprehensive as was hoped. LIMITED CONTROL OVER SURVEY ADMINISTRATION Because USAID decided to have survey dissemination handled by each network, NORC lost the ability to meaningfully track and diagnose response rate issues and take corrective action as needed. In addition, the uncontrolled dissemination process led to survey launch and follow-up emails being sent at inopportune times rather than at the times suggested by NORC based on existing literature about the optimum time to send a web survey (such as Monday morning). Finally, having the survey come from the network itself may have resulted in actual bias in who received the survey or perceived bias about the survey's independence. 4. FINDINGS AND CONCLUSIONS A. ECCN NETWORK RESULTS AND OUTCOMES EQ1: WHAT ACTIVITIES WERE THE MOST SUCCESSFUL AND WHY? WHAT ACTIVITIES WERE ULTIMATELY UNFEASIBLE AND WHY? MOST SUCCESSFUL ACTIVITIES The ECCN ST, USAID CORs, USAID DC and Mission members, and general members all mentioned ECCN’s ability to produce high quality technical products and tools for the Education in Crisis and Conflict (EiCC) community as one of its greatest successes. ECCN produced many resources through its network of experts and partner organizations. A few that were noted as particular successes by key informants were the Rapid Education and Risk Assessment Analysis (RERA) Toolkit, the Safe Learning Environment (SLE) Assessment Toolkit, the indicators developed for conflict sensitivity and safe learning, the Resilience and Education White Paper, and the Guide to Accelerated Education Principles. Utilization of tools is discussed in more detail in the responses to EQs 3-5. “I’ve seen the feedback from members – tools are probably the most important things that are recognized. What has the membership at large created – good quality of tools for which there is a need – to me, that is a really important aspect of this.” (ECCN ST) “The sheer amount of technical products that have been produced, I think with pretty high quality and relatively low cost, I think is notable in both cases… I think in terms of technical depth and … the amount that was done, I think that’s definitely notable.” (USAID staff) The convening of a wide range of experts, organizations, and other networks through the CoP was a success of ECCN. Mission members, USAID staff, and general members in the FGDs and KIIs noted that despite the competition often found between IPs, ECCN was able to bring together stakeholders from different organizations to successfully USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |11 collaborate to develop tools and guidance on EiCC. ECCN also convened advisory groups, which pulled in experts from across the different stakeholder groups in USAID to advice on the development of tools or resources in their topic areas. ECCN ST, general members, and Mission members also noted ECCN’s success in partnering with other networks, such as the Accelerated Education Working Group (AEWG), the Inter-agency Network for Education in Emergencies (INEE), and the Basic Education Coalition (BEC). For example, ECCN worked with AEWG on the Guide to Accelerated Education Principles, which is discussed in more detail in the response to EQ 4. “The ability just to convene a wide variety of experts in and of itself has really contributed to advancing work in this sector that I think is a global good…” (USAID staff) “The way it brought people from all different organizations together was good…” (General member) ECCN’s knowledge-sharing events have also been well received by members; training in particular was considered ‘extremely successful’ by USAID. Four-fifths of respondents in the evaluation survey said they participated in at least one ECCN-sponsored knowledge-sharing event or training. Overall, attendance at knowledge-sharing events increased in the later years of the project. In Year 3, ECCN trainings had 173 participants, while 792 people attended knowledge-sharing events and webcasts. In Year 4, 106 participated in training, while 907 attended knowledge-sharing events, either in person or virtually.3 The ECCN ST noted that it is possible that, overall, knowledge-sharing events garnered a higher attendance in Year 3 onward due to the targeted and purposeful outreach completed by ECCN and key partners. Furthermore, access to knowledge-sharing events on YouTube (webcast) meant that interested parties living in time zones that did not permit virtual attendance during the event itself were able to access learning by watching the event after the fact.4 In ECCN’s Year 4 annual member survey, 57.5% of respondents reported having applied learning from at least one webcast and 41.3% reported “changed practice as a result of at least one [webcast]." In the evaluation survey, 23 of the 29 people who attended in-person knowledge-sharing events rated them as “very useful.” In terms of training, conflict sensitivity trainings (online and in person) were most successful in terms of the amount of interest and number of attendees. The conflict sensitivity training, for example, garnered over 100 applications for 35 spots.5 “I think the trainings have also been extremely successful. What’s made them different than the way we’ve done professional development training? Our most successful trainings were a mix of USAID staff and implementing partners. I think that it elevated both the content and technical expertise that we were able to deliver to participants, but we also had more meaningful discussion and training participation that I don’t see as much when it’s just USAID.” (USAID staff) 3 These counts include webcasts produced earlier on in the project. 4 ECCN Y3 annual report and Y4 report 5 ECCN Y3 annual report 12 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID UNFEASIBLE ACTIVITIES The idea of creating knowledge share groups within the CoP around certain topics, such as metrics, peace building, or research, was not successful early on, despite a lot of time and energy going into this activity. ECCN did find success in this type of smaller working group once they had an actual project to work towards. “What did work was anchoring it around the project we were working on – for me it was the SLE toolkit. It got people interested in a specific topic they could feed into...” (ECCN ST) Some technology-related requests made of the network were not feasible or were unsuccessfully executed. This may have been due to requests from USAID that were outside the capability of the contractor. For example, a lot of effort was put into building a website for the network, which has now been transitioned to another platform so that the resources and materials may still be accessible after the project ends. One particular USAID staff questioned the value of spending so much time and money to develop such a resource when the contractor did not have a strong technology capability. In addition, according to a review of quarterly reports, ECCN encountered a number of challenges during the process of developing the website.6 In 2017, a new, redesigned website was launched because the previous website was experiencing declining traffic, due to its "flat structure" and difficulty in browsing for content.7 “Anything related to technology, in both [networks], I did not think was particularly successful. …I don’t think there is expertise in either case that really warranted the kinds of work that was being asked of them [such as websites or digital tools for online engagement].” (USAID staff) However, despite these initial challenges, the ECCN ST and members ultimately found the website to be a useful resource. Participants from three KIIs agreed that they found the website easy to navigate. EQ2: WHAT WERE THE UNEXPECTED OR UNANTICIPATED OUTCOMES OF ECCN/GRN? WHAT ADDED VALUE OR BENEFIT DID THEY PROVIDE TO USAID AND ITS IMPLEMENTING PARTNERS? UNEXPECTED POSITIVE OUTCOMES The ability of contractors, who typically see each other as competitors, to work together was noted as an unexpected positive outcome of ECCN. USAID did not expect that there would be competition among the various contractors in the network. On the other hand, the implementing partners were anticipating some competition among their peers. However, the network was able to successfully overcome this challenge, at least in part through finding “the right group of people to engage with,” according to a USAID respondent. “We had 6 Initially there was a delay in hiring a qualified web developer due to contracting issues. Later it was noted that the website was not 508 compliant and needed to be updated to rectify this. 7 ECCN Y2Q1 Report, ECCN Y3Q1 Report, ECCN Y3Q4 Report. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |13 to work through it…We landed on a really good group for advising and steering.” The ECCN ST also listed this as a surprise: “Five years ago the biggest challenge we were facing was how can you get competitors to cooperate?....This has impressed me the most – in all of our work… it’s built on social capital. Collaboration and cooperation between competitors is appreciable.” (ECCN ST) ECCN’s website and resource repository became a reliable source of information and knowledge for other, more mature and larger networks, such as INEE. “INEE had done some survey of their membership and asked where they got reliable technical resources. ECCN was the second most popular source. INEE has 14,000 members, and ECCN has 2,000. I was amazed that another network seeking to understand its own reach, came second. Our ambition was smaller, but here we are.” (USAID staff) ECCN also had success in facilitating cross-agency collaboration at USAID, which is atypical according to a USAID staff member. ECCN accomplished this by pulling in USAID staff from across the agency to participate in advisory groups and contribute to knowledge products. “I think that’s not to be dismissed in terms of both identifying them, but keeping them involved and interested through these processes. And that’s something very unique – you don’t see it that much in the education sector.” (USAID staff) ECCN tools and resources were seen very favorably by USAID colleagues; one unexpected outcome was the desire to emulate tools, resources and infographics developed by the project. According to USAID staff, many people [USAID staff] requested to ‘buy in’ to the contract in order to leverage what they saw as high-quality resources from the IP to help develop knowledge products and outputs for their own needs. ADDED VALUE TO USAID ECCN amplified USAID’s work in the field. One added benefit that USAID identified was the ability of ECCN to identify opportunities (above and beyond their contract deliverables) around relevant USAID evaluations, studies, program milestones or events taking place in the field, and then providing support to those activities. ECCN also helped to promote the work that USAID was doing by organizing and presenting at conferences such as CIES. “Both networks have been particularly good at supporting additional activities that might come out of USAID, like for example a training or supporting a workshop at CIES…” (USAID staff) ECCN contributed clear conceptual models for different types of programming in crisis and conflict. According to one USAID staff member, when the project started, there wasn't a clear conceptual model for programming components that make up a program in an EiCC situation. The evidence behind this was slim. There is now a more advanced understanding of this sector and ability to program around it thanks to ECCN’s contributions. 14 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID ECCN was able to respond to a greater number of technical assistance requests in the field than the USAID could have done alone. One Mission member stated their appreciation that ECCN was able to send technical experts to the field to work directly with Missions on technical issues which they had struggled with and for which they had requested support. The ability of ECCN to send experts allowed USAID to be responsive to more Mission requests than they otherwise would have been able to. This was particularly successful when it came to highlighting and ground testing approaches, such as pilot testing of the RERA Toolkit by Mission staff in DRC. ECCN leveraged LOE and technical support from universities and experts at little or no cost due to the partnership with University of Massachusetts and volunteering of CoP members’ time. The ability to tap into low-cost, graduate student support, as well as the contribution of experts’ time and resources, was of “immeasurable value” to USAID, especially as it enabled the contract to accomplish more with limited financial resources. “I believe there was some really good creative use of university partnerships that I think we should continue.” (USAID staff) “I would say the consultative nature and the willingness of many different experts in the field wanting to contribute their time in many cases with no payment was a particular success and contributed to just the quality of the work. And I would say, that based on what I know about what is most likely to happen in terms of evidence uptake and use involving those who are end users throughout the process, I definitely saw that happen and likely contributed to higher uptake at the end of the day.” (USAID staff) ADDED VALUE TO IPS USAID DC members observed that participation in ECCN helped IPs to better understand USAID’s priorities and framing. Aside from whatever other benefits IPs received from participating in ECCN, there is obviously a clear business incentive to understand USAID’s priorities in the sector. “I think just understanding what USAID’s priorities are and how they’re framing the priorities was just there’s clear business incentives for that.” (USAID DC member) General members noted that IP participation in ECCN also provided greater visibility and involvement in the EiCC sector. For those IPs who committed personnel to serve on the steering committee, they realized the added benefit of being able to demonstrate their commitment to the CoP. It showed that they were a serious player in the EiCC sector in a way that was visible to USAID (General member). STAKEHOLDER USE OF NETWORK PRODUCTS AND SERVICES EQ3: HOW DID STAKEHOLDERS USE INFORMATION THEY RECEIVED THROUGH THE ECCN? TO THE EXTENT POSSIBLE, WHAT SPECIFIC DESIGN, IMPLEMENTATION, USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |15 MONITORING OR EVALUATION APPROACHES OR ADAPTATIONS DID THE INFORMATION INFORM, AND WHAT WAS THE IMPACT OF THIS UPTAKE? ECCN stakeholders have used information received through the network in a variety of ways, especially to improve understanding and in program/activity design. In the evaluation survey, 47% of respondents reported changing their practice due to at least one ECCN output.8 Figure 1 : How Members Applied ECCN Resources or Knowledge Source: Evaluation Survey A substantial number of stakeholders have applied ECCN information in program/activity design, implementation, or program monitoring. Over half (56%) of evaluation survey respondents reported they had applied ECCN information in program/activity design. One Mission member reported that the network led a helpful training on effective program design and management in crisis and conflict settings. He said he constantly refers back to those documents from the training when doing program design. A smaller number applied ECCN information to develop theories of change; 29% of evaluation survey respondents reported doing so. In an FGD with USAID members in Washington, respondents reported using ECCN resources to think through theories of change. In a KII, the ECCN ST observed that USAID solicitations now include theories of change more often (at least partly due to changes in ADS 201 in 2016).9 8 According to the ECCN Y4 Annual Report, 61.8% of ECCN’s survey respondents reported having changed practice from at least one ECCN output (resource, knowledge event, or training). 9 See also “An Analysis of Theories of Change in USAID Solicitations for Education Programs in Crisis and Conflict-Affected Environments,” ECCN, December 2016. 25% 29% 32% 36% 39% 46% 54% 56% 0% 10% 20% 30% 40% 50% 60% Developed publications Theory of change development Integrated into metrics/M&E plans Developed training material Informed policies for promoting new… Improved implementation of an existing program Improved my personal and/or my team's… Integrated into future program/activity design Q. In what ways have you applied resources or knowledge from ECCN in your work? 16 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID ECCN information was used to improve implementation by nearly half (46%) of survey respondents. In qualitative interviews, general members noted comparing their organization’s implementation against the best practices provided in ECCN resources: “[The alternative education] resources came out a little bit late for our programming, but it was kind of a bellwether. It said: here are the effective approaches, here is a checklist, so then we matched that against what we’re doing.” (General member) About 32% of evaluation survey respondents reported using ECCN resources information in metrics or monitoring and evaluation; although the qualitative data reveals limited awareness of ECCN’s resources in this area. While two interviewees were aware of the indicator bank, only one had used it to create indicators for her program. Capacity building was an important use of ECCN. More than half (54%) of evaluation respondents reported using ECCN resources or knowledge in their work to improve their personal and/or team's understanding for future efforts. ECCN members also reported a few other ways they’ve used information gained from the network. One USAID DC member mentioned drawing upon resources when developing talking points for use on Capitol Hill, while another USAID DC member used resources to identify leaders/specialists in certain areas that could be called upon. CHALLENGES TO USE When asked why they did not apply ECCN resources or knowledge in their work, more than two-thirds of evaluation survey respondents reported that the opportunity had not yet arisen. Other reasons cited for non-application included the resources not being relevant (26%) and not lending themselves to practical application (12%). Figure 2: Reasons for Not Applying Accessed ECCN Resources Source: Evaluation Survey 2% 5% 6% 12% 26% 70% 0% 10% 20% 30% 40% 50% 60% 70% 80% Resources did not offer clear, succinct presentation of key actionable findings Resource is in draft form and we prefer to wait for the final Resources were duplicative of what I already do Some resources do not lend themselves to practical application Resources were not relevant for my purposes May use in the future, but have not yet had an opportunity to use Reasons for not applying ECCN resources USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |17 The qualitative data provides more insight on this topic. Mission members noted that some products from ECCN, especially the RERA, were not user friendly or practical. They noted that the RERA is “very big and not easy to use” and that it “requires a lot of expertise to apply.” A USAID respondent noted that ECCN was stronger – and put more resources into – knowledge advancement than into ensuring new approaches got used: “They’re good on the knowledge advancement building side, really not great on the other end of the spectrum…And I think they just didn’t invest in the right staff or resources to take that into consideration.” This may have been partially due to the under-budgeting of the communications function and other staffing limitations, such as the lack of a knowledge management position (see EQ8 for additional detail). Other stakeholders reported that information from ECCN was too broad and resources needed to be more context specific to be useful. As one USAID DC member noted: “It’s hard to use that stuff in design. It only goes so far. It’s the first top level layer, and then you’ve got to get very country contextually specific.” EQ4: WHAT SPECIFIC KNOWLEDGE PRODUCTS DID STAKEHOLDERS FIND MOST VALUABLE? WHY? Evaluation survey respondents were asked whether they had accessed any of a list of products from ECCN, as well as whether they had applied those products. All but one respondent had accessed at least one product. NORC then calculated a utilization rate for each product. The average utilization rate for the resources listed was 42%. Table 2 below shows the products with average and above average utilization rates. Table 2: Utilization Rate of ECCN Products Resource Accessed (Count) Applied (Count) Utilization Rate Guide to Accelerated Education Principles (AEWG) 31 20 64% Rapid Education Risk Assessment (RERA) Toolkit 62 34 55% Accelerated Education in DRC Report 20 11 55% Safer Learning Environments Evidence Gap Maps 20 10 50% Safer Learning Environments Assessment Toolkit 35 16 46% Humanitarian Development Coherence White Paper 17 8 47% Analysis of Theories of Change in USAID Solicitations for EiCC 9 4 44% Education Equity Indicators for Access: Guidance for Practitioners in Crisis and Conflict Affected Contexts 25 11 44% Summary brief of Rapid Education and Risk Analysis (RERA) Field Pilot Research 23 10 43% Evidence Pathways online tool 7 3 43% CSE Indicators 18 7 42% Total respondents 85 83 Source: Evaluation Survey 18 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID The high performance of the Guide to Accelerated Education Principles and the RERA Toolkit was likely assisted by the effort put into design, launch events, piloting (in the case of RERA), revisions and updates, and promotion through another organization’s wide membership (in the case of the Guide to Accelerated Education Principles). A look at downloads from website data for previous years10 provides both interesting substantiation and contrast. While the same resources are among the top downloads, the Safer Learning Environments (SLE) Evidence Gap Maps showed substantially more views than they appeared to in the evaluation survey. This may be because the maps were viewed online but not actually downloaded or used the same way as other products. Table 3: ECCN Resource Downloads Resource Views/Down￾loads11 2017 Views/Down￾loads 2018 Guide to Accelerated Education Principles (AEWG) 42 68 Rapid Education Risk Assessment (RERA) Toolkit 154 350 Accelerated Education in DRC Report 106 Safer Learning Environments Evidence Gap Maps 492 704 Safer Learning Environments Assessment Toolkit 100 Source: ECCN Annual Reports – Years 3 and 4 MOST VALUABLE PRODUCT: RERA TOOLKIT The RERA Toolkit was one of the most utilized products available through ECCN: 55% of evaluation survey respondents that accessed the RERA applied it to their work.12 Qualitative respondents indicated that the RERA was especially helpful for program/activity design: “The [RERA Toolkit] actually informed a current flagship project that we’re implementing. We learned that the project needed to be flexible so instead of limiting it to a few areas in the country we flew it to the entire country, but used a limited scope RERA to actually decide where to expand into through cutting the myths and risks of school children and youth in different areas.” (Mission member) “Let me just underscore the RERA Toolkit.’ In [my Mission], we used it twice to inform our programming, you know, in a more conflict sensitive manner and to adjust along the way.” (Mission member) 10 Annual Reports for Years 3 and 4 of ECCN. Note that not all data was available for all products. 10This can include either downloads or landing page views. 12 According to the ECCN Y4 Annual Report, only 36% of respondents indicated applying the RERA in their work; 25% indicated it had changed their practice. What is the RERA Toolkit? The purpose of the RERA Toolkit was to guide: 1) USAID education staff on procuring and overseeing RERA implementation; and 2) Implementing partners on implementing a RERA. The toolkit includes a large number of tools, case studies, examples, and specific guidance on how to plan, procure, and complete a RERA. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |19 “We’ve used [the RERA toolkit] to inform designs, but we have also embedded limited scale toolkits into the project to be able to continue doing rapid education risk analysis to inform decisions within the project going forward.” (Mission member) In a KII with the ECCN ST, respondents reported that USAID solicitations are now requiring RERAs more often. Demand for conducting RERAs has also originated from IPs themselves, according to an interview with the ECCN COR. The success and high utilization rates of the RERA Toolkit may be due to the high level of effort in both preparation and launch, with an initial pilot version developed (RERA 1.0), followed up by a second adapted version based on feedback from several pilots. The redesign and rollout were also guided by a Toolkit Reference Group that included key specialists from USAID and partners. ECCN made several updates, including integration of an updated Safe Learning Environment (SLE) Qualitative Assessment Tool in the RERA 2.0 annexes. In addition to a typical launch presentation, RERA 2.0 also included an online layout for the tool, which consisted of a one-page toolkit summary, a short description video, a 508 compliant graphic design of the toolkit, and a special webpage on ECCN’s website dedicated to RERA resources. In addition, ECCN offered a training course on RERA 1.0 and RERA 2.0. ECCN also presented RERA at the CIES pre-conference workshop. MOST VALUABLE PRODUCT: GUIDE TO ACCELERATED EDUCATION PRINCIPLES The Guide to Accelerated Education Principles, a product developed by the AEWG to which ECCN belonged, had the highest utilization rate: 64% of the evaluation survey respondents that accessed the document indicated that they applied it in their work.13 This high rate of access may be because AEWG has a large membership base that extends beyond the ECCN listserv, bringing this product to a much larger audience. Furthermore, the large launch event that AEWG sponsored meant that the product was highly promoted across a wide range of global stakeholders. One general member reported in an interview that her organization had compared their programming to the AEWG product: “With the AEWG toolkits and Guide to Accelerated Education Principles, our organization has reviewed some of our accelerated education programming against those. I’m not sure if that’s directly attributed to ECCN. ECCN is a member of AEWG and really made a very large contribution to the development of those products, so from that perspective, yes it informed implementation. 13 According to the ECCN Y4 Annual Report, 34% of respondents indicated applying the Guide to Accelerated Education Principles (AEWG) in their work, and 24% indicated it had changed their practice. 20 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID MOST VALUABLE PRODUCT: SLE ASSESSMENT TOOLKIT The impetus for the SLE Assessment Toolkit and the SLE Evidence Gap Maps came from within the CoP. Two Mission members noted using the SLE Assessment Toolkit in their work, while one specified he used it as part of his Mission’s M&E plan. It was said to be easier to apply than the RERA Toolkit. General members also noted that they used the SLE toolkit when developing proposals and implemented SLE assessments as part of their implementation. Two US-based general members noted that they attended SLE training in Washington and found it useful. The ECCN ST also mentioned the SLE Assessment Toolkit as one of their most high quality tools. “We used the Safe Learning Environment Assessment Toolkit as part of our M&E plan. You know, to inform progress towards safer living environments, so the safer tool that’s out there and has been very helpful.” (Mission member) “I definitely think the SLE Toolkit will continue to be in use for a very long time.” (Mission member) MOST VALUABLE PRODUCT: SLE EVIDENCE GAP MAPS With the highest number of views in earlier years, the SLE Evidence Gap Maps seems to also be among ECCN’s most important products. The Maps identify resources that contribute to evidence of interventions and outcomes. The size of the bubble at the each intersection reflects the quantity of evaluations and other studies available, and provides a listing of – with links to – those studies, along with ratings on the rigor of each study. The Maps cover internal, external, and environmental threats. OTHER VALUABLE PRODUCTS Qualitative respondents identified several other products as particularly valuable. These included the Humanitarian Development Coherence White Paper, which was viewed as useful for framing the issue and concerns; and the Resilience and Education White Paper, which provided language for linking resilience and education together. What is the Guide to Accelerated Education Principles? This AEWG-produced guide highlights 10 evidence-based Accelerated Education Principles for stakeholders involved in accelerated education programs (AEP). It is intended to be used to design, develop, review and evaluate individual programs with the aim of ensuring programs are flexible, inclusive and well-integrated with the education contexts in which they operate. Stakeholders can also use the guide to ensure good quality, well￾resourced AEPs are promoted in all settings, and to advocate for inclusion of AEPs in the strategies, policies and budget lines of key government partners. ECCN played a support role in launching and publicizing this product, including hosting a webcast. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |21 EQ5: WHAT MODES OF DELIVERY MADE ACTIVITIES SUCCESSFUL/UNSUCCESSFUL? In 2018, ECCN provided 63 access points to resources, learning/knowledge sharing events, and professional development/training developed by ECCN, exceeding the target of 50.14 Of all access points, email was the dominant mode of delivery for ECCN members; 74% of evaluation survey respondents indicated that email was one of the top three ways they receive information from ECCN, while 41% noted the emailed monthly newsletter. In interviews and focus groups USAID and general members reported email was their preferred mode of receiving information, noting their limited time and competing priorities. Several noted that they liked receiving notification via email of new products and events so they could skim them and save them for future use. The frequency of email from ECCN was seen as appropriate; 93% of evaluation survey respondents reported that the number of emails they get from ECCN was “the right amount.” With regard to training modalities, 85% of ECCN members found in-person trainings to be most valuable, followed by 9% who chose live webcasts, and 3% who selected recorded webcasts. ECCN hosted recordings of its knowledge-sharing events on YouTube so interested parties living in different time zones could access learning after the fact.15 There is qualitative evidence that members appreciated having access to events on this platform. There is evidence that ECCN’s contact with members increased over time. In Year 3, ECCN had 1,489 Twitter Followers, 332 average monthly newsletter openings, and 4,015 blog post views. A total of 462 people attended live webcasts and 216 viewed recorded YouTube webcasts, 114 people attended in-person events other than training, and 173 people attended in-person trainings. In Year 4, ECCN had 2,085 Twitter followers (+40% from Year 3), 460 average monthly newsletter openings (+39.4%), and 5,783 blog post views (+19.1%). A total of 528 people attended live webcasts (+14.3%), 331 viewed recorded YouTube webcasts (+53.2%), and 240 attended in-person events (+110.5%). The only category where contact decreased was in-person trainings; only 106 people attended in-person trainings in Year 4 (- 61.3%). Despite member satisfaction with the various modes of delivery overall, the Anglo￾centric nature of ECCN’s resources may have been a barrier to delivery for some stakeholders, especially FSNs and non-English practitioners and implementing partners. A review of the ECCN documents tested in the evaluation survey shows only one was available in 14 ECCN Y4 Annual Report. This information is derived from one of ECCN’s indicators. The PIRS in ECCN’s Y4 PMEP explains: The concept of “access points” was introduced in 2017 as part of the reinforced engagement strategy. Targets and actuals for access points were thus not recorded for 2015-2016. Initial target (2017) is calculated as an average of 1.5 access points per output in the annual work plan. Outputs are always featured with a link in the newsletter and are expected to most often have additional access points (e.g., blog posts, home page, webcasts). 15 ECCN Y3 Annual Report and Y4 Report 22 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID a language other than English.16 A number of survey respondents requested more materials in Spanish and French in their write-in recommendations. MISSION/REMOTE ENGAGEMENT EQ6: HOW WERE MISSIONS AND IMPLEMENTING PARTNERS ENGAGED? WHAT ASPECTS OF THE COMMUNITIES OF PRACTICE DID THEY FIND MOST VALUABLE? MEMBER ENGAGEMENT Respondents reported engaging with the ECCN CoP in a variety of ways, including learning from each other and networking with other members. The most common means of engagement reported by evaluation survey respondents was learning new approaches and knowledge (42% participated actively). 31% of respondents indicated they actively participated in networking opportunities, while 25% of respondents reported actively sharing resources and/or knowledge with the community. 40% of respondents reported not engaging in any CoP activities. Most of the network’s members were implementing partners and ECCN succeeded in engaging these organizations. ECCN surpassed its goals in engaging organizations in the development of ECCN resources and events, but fell short on engaging members. Thirty organizations actively contributed to the development of ECCN resources and learning/knowledge sharing events (workplan outputs), greatly exceeding the 2018 target of 10 organizations. However, only 50 of ECCN’s members, or 4%, contributed to such products. This was short of the target goal of 85.17 Engagement with Missions was weaker than engagement with IPs, according to interviews with network members, USAID, and the ECCN ST. Mission members explained feeling that ECCN didn’t understand what it was like in the Missions or how to create products that would be useful for education staff there. Some felt they could contribute to products to make them more applicable, but also noted they were extremely busy. They primarily interacted with ECCN by receiving emails and, occasionally, attending a webcast. Several requested help navigating the website and the great amount of information available. USAID also acknowledged that Mission engagement did not work as well as engagement of US-based IPs. The ECCN ST agreed, noting that engagement was limited in part by USAID’s strategic choice to channel ECCN’s Mission engagement activities through Washington. 16 Only the Accelerated Education in DRC Report (Available in French) resource was available in a language other than English. The RERA Toolkit contained sample final RERA reports in French and Spanish, but the toolkit itself was only available in English. The Summary brief of Rapid Education and Risk Analysis (RERA) Field Pilot Research was not available online for the ET to review. The Resource Repository was excluded from this analysis as it is not a standalone resource. 17 ECCN Y4 annual report. Both figures come from ECCN indicators. An organization is counted as actively contributing by either undertaking a “Partner Initiative” (PI) in which they had the primary responsibility, or making a significant contribution to the development or production of another ECCN output. A member counts as contributing by composing or responding to a blog post, contributing newsletter content, or posting to discussion board or Bulletin Board. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |23 “From where I sat, I didn’t feel like [Mission engagement] was very successful. The most common kind of feedback I’ve heard is that like, yeah, like from Mission staff, ‘Yeah I receive the ECCN newsletter sometimes and the GRN news. I see that, you know, come across my desk every once in a while, but like I’m confused about where to go to get something. It’s all over the place and I can’t find things.’ Or like ‘I’m too busy. I’m too overwhelmed. It’s just too much.’” (USAID staff) “I found that most of the time, I would do a search and a lot would come up. So I would go to my Washington colleagues to ask them to narrow down what I should read from the big list of things that would come up when I did a search on the website. The ECCN website needs to be really concise, transparent, and user friendly. When you’re challenged for time, it can be hard to go through the website and search endlessly.” (Mission member) USAID interviewees reported that the audience for ECCN was not clearly defined, which may have complicated Mission engagement. The prevailing belief among general members was that ECCN was designed mostly to push out USAID’s policies to implementing partners, while Mission members and other general members comprised a secondary audience. Within USAID, there was some confusion about who ECCN was for. “One of the things we could do in general is worry less about volume and more about identifying users more clearly. This is the biggest mistake that people make - this document is for USAID mission staff, implementers, everyone under the sun. People have written articles about this. Be really clear about who your audience is, what you’re trying to achieve…In general, we could be more clear on who the users are, and then tailor those documents to the users. If you’re at the mission, your user needs something clear…does the language match with the user? Sophistication/technical stuff that’s very academic versus something for someone with four minutes to read the thing...” (USAID staff) “I think both [USAID missions and IPs] should be the audience.” (USAID DC member) “I wasn’t clear on who the audience would be [when speaking on an ECCN panel] and maybe that’s part of the issue here because we don’t know who ECCN is for. Is it for our Missions, you know, resources for our missions, and then what? So if it’s external facing, what’s the, well, how are we messaging all of this?” (USAID DC member) However, the vast majority of members that participated in the evaluation survey (90%, including both Mission and general members) reported that they would like to continue being part of ECCN (55% yes, very much; 35% yes). A majority (53%) are also satisfied with the number of engagement opportunities. One USAID DC member was surprised at the enthusiasm and desire among IPs and many USAID colleagues to maintain the networks after the contracts had ended. “I’ve been partly surprised by the level of enthusiasm for these networks to continue, you know, even in an organic fashion or without guaranteed resources. It sounds like, from what I’ve heard, in every case there’s an extremely strong desire to continue this in some way. And I think that speaks to incentives. And folks have found some, you know, some form of incentive for them to continue to participate.” (USAID DC member) 24 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID MOST VALUABLE ASPECT OF COP In reporting their active participation in the CoP, respondents also reported on the value of each activity. The most widely valued CoP activities included forming new relationships, sharing resources and/or knowledge with the community, and learning new approaches/knowledge (Table 4). Table 4: Value of ECCN CoP Activities as Reported by Members Source: Evaluation Survey It is notable that more than twice as many people found activities “very valuable” as “somewhat valuable,” suggesting a high degree of interest. Nearly two thirds of evaluation survey respondents (64%) reported that ECCN fully met their expectations in terms of providing access to tools and resources. Expectations were also fully met for more than 40% of respondents in the following areas: personal capacity building (49%), partner advocacy with USAID (45%), and knowledge sharing (44%), and the opportunity to interact with USAID and other partners around education (42%). Figure 3: Expectations Met on Aspects of the ECCN CoP Source: Evaluation Survey 64% 49% 45% 44% 42% Access to tools and other resources Personal capacity building Partner advocacy with USAID Knowledge sharing Opportunity to interact with USAID and other partners around education Q. To what extent were your expectations met in each of the following areas? (those who answered "fully met") Activities Very valuable (count) Somewhat valuable (count) Active participants (count) Formed new relationships 15 5 20 Shared resources and/or knowledge with the community 16 5 22 Learned new approaches and knowledge 26 9 37 Used networking opportunities 17 10 27 Participated in a working group 11 8 19 Have shared my challenges with the community for help 6 7 13 TOTAL 91 44 138 USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |25 EQ7: WHAT WOULD THEY SUGGEST FOR THE FUTURE? REDUCE THE EMPHASIS ON USAID POLICY AND FOCUS ON MEMBER NEEDS In qualitative interviews and the evaluation survey, general members voiced frustration that ECCN could not operate as a true CoP because the contractor often acted as an extension of USAID, responding more to USAID needs and requests than those of the members. Members recommended making ECCN more network-focused, with increased opportunities to interact with each other and decreased influence by USAID. “The problem I think they both [ECCN and GRN] had was that USAID had limited staff and they became extensions of USAID rather than communities of practice. So they really were responding to what USAID needed done and less to what the ‘members’ [wanted or needed].” (General member) “So there were quite a few of us that would just sort of stand around [at ECCN events] and go, ‘Right. This is old stuff, this is party line, I’m not sure this is improving the quality of the work.’” (General member) Several interviewees contrasted ECCN with other networks that they viewed as more relevant globally because they were not run by a single donor: “I think it’s an error of judgement to start up a very USAID-based network. Time would have been much better spent more broadly, globally, doing different things. I think time would have been better spent doing INEE.” (General member) “I think ECCN is very, very US-based in its focus. I can imagine that a lot of these tools are really helpful if you’re USAID or you receive funding from USAID. They don’t have necessarily a global appeal or use. Even all of the ECCN members are all USAID connections.” (General member) ECCN’s USAID-driven agenda also led implementing partners to infer where USAID was headed based upon the knowledge products and events the group was producing. While implementing partners saw this as an added value of being in ECCN, a USAID DC member reported that it created a competitive environment where USAID participants sometimes felt uncomfortable at events and it took away from the openness of dialogue. One USAID staff member saw this as a fundamental inconsistency that had not yet been addressed: “What is the ultimate vision for what we want to do with communities? Is there an expectation that we foster communities to eventually stand on their own or be absorbed into something else? … Or are we always just wanting a kind of basically a listserv that we can tap into as needed when we’re going about producing the work that we know we need to match the priorities that we have so that we’re being consultative and not so insular in the way that we go about our work. To me, those things are very different.… These are questions that I don’t think 26 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID were fully grappled with at the beginning of these [contracts] and I think it’s coming to fruition more so now than we all realized mid-stream.” (USAID staff) DIVERSIFY THE STEERING COMMITTEE In the evaluation survey and qualitative interviews, both general and Mission members reported feeling that the ECCN steering committee was dominated by education players in the United States and did not represent the types of people and places where crisis and conflict programming is taking place. The dominance of these ‘usual suspects’ was said to be especially visible in presentations and knowledge product development which at times felt out of touch or dated. Several members recommended diversifying the steering committee to be more representative. “What I found was that the team was largely made up of Washington-based colleagues and implementing partners. I tried to bring the voice of the Missions to the group. It wanted to be a really helpful community of practice, but it was difficult for other members of the group to understand what it’s actually like in the field and in the Missions.” (Mission member) INVOLVE REMOTE MEMBERS IN PRODUCT DEVELOPMENT In the evaluation survey and qualitative interviews, both Mission and general members requested to see voices from the field more effectively and intentionally incorporated in the design of ECCN tools and resources. As one Mission member explained in an interview, ECCN should move away from viewing the field/Missions as users of content and see them more as co-producers. Another member noted that it was a missed opportunity that ECCN did not meaningfully engage with the field more. “Are we just on the consumer side or on the co-producer side? You know, where is the cursor being located? We could get more engaged to be collaborative in the development of products. Maybe if there was a call for interest to participate in the development of xyz-tool, you know, that’s one area, one way you could engage folks in the field more, so they don’t see it like they are consumers, but also participate in the development.” (Mission member) “Make resources development more collaborative. Collaboration is sought only for piloting tools, after experts designed them. Local teams should be involved earlier.” (Evaluation survey respondent) Still another evaluation survey respondent referred to “a missed opportunity – less engagement with the ‘field’ where the rubber meets the road.” RECRUIT HOST COUNTRY GOVERNMENTS & LOCAL NGOS Stakeholders from all categories (USAID DC, Mission, and general members) recommended greater inclusion of local NGO and Ministry members in the CoP. A Mission member reported that ECCN would need to change its approach in order to attract Ministry of Education members: USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |27 “I remember mentioning that if they ever wanted someone in a Ministry of Education to show up to a webinar, they had to make them shorter, not an hour long webinar. If someone were even able to get a dedicated internet connection, which is hard, you wouldn’t get them to show up for an hour. And how would they even find out about it? I think we’d have to tell them. So I didn’t see engagement of non-USAID remote users or stakeholders like that.” (Mission member) NETWORK MANAGEMENT EQ8: WHAT WORKED IN TERMS OF MANAGEMENT OF ECCN? WHAT DID NOT? WHAT MIGHT HAVE MADE MANAGEMENT OF THE ACTIVITIES EASIER AND MORE EFFECTIVE? USAID found the ECCN ST to be effective and responsive: “The management in my opinion was excellent. Very open communication.” USAID also commended ECCN’s adaptive management, their responsiveness, community demands, and openness to receiving feedback: “Sometimes we didn’t know what we were asking for and they gave us what we needed…Cornelia was excellent in receiving feedback.” Both USAID and the ECCN ST spoke highly of the productive relationship and the level of trust between them. In the view of the ECCN ST, both the COR and the ACOR were highly engaged, with regular meetings and consistent feedback: “We worked very well as a larger team. I considered us partners. USAID played their role in a good structured way.” “A lot of credit to USAID and team for their collegiality, openness to explore adaptations, explore new thematic frontiers… The quality of their technical inputs was strong.” USAID noted that its management approach was designed to foster close collaboration and USAID-ECCN integration: “I think that the way we structured our teams was very effective. Every time we had a discrete product that we were developing or project we were doing with ECCN, someone from the EiCC team was an activity manager. We had a very similar approach in terms of [bi-weekly] check ins, milestones, activities, deliverables, and feedback. I think that helped us to meet deadlines and made work very predictable.” External members also had positive things to say about working with the ECCN team: “I can say personally it was such a great experience to … work so closely with some incredible people, not just at USAID but from the other organizations and I just was thrilled to be able to do it, and I’m sad it’s over. Yeah, we enjoyed it a lot. We felt we got a lot done and we learned so much from each other in a really safe space and that again, I would attribute to Cornelia and the leadership she offered.” (General member) Both USAID and the ECCN ST believe the rigidity of ECCN’s MOBIS time-and￾materials contract hindered effective implementation. They point to the original miscalibration of funds between labor and other direct costs (ODCs)—with too great a share allotted for ODCs—and inability to shift funds between the two line items, which resulted in understaffing. Labor categories were also cited as limitations by the ECCN ST: “The labor was very specifically defined to have certain qualifications and was very rigid,” and many of the contractual 28 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID specifications did not match the actual needs of the network. The initial allocations for the communications function, for example, were severely under-budgeted, and in other cases ECCN was not be able to bring in the most qualified people to do the work, according to USAID staff. The ECCN ST observed that this got better over time as ECCN and USAID worked pragmatically to resolve some of the issues. It is not clear why this impediment could not have been remedied by the Contracting Officer (CO), but that is beyond the scope of this evaluation. USAID also mentioned the inability to do grants under contract as a constraint, as they would have liked to support local education organizations. Most interviewees had high opinions of the ECCN COP and technical staff. A general member commended the team: “I think that they had really dedicated staff. I think EDC was a really good place for them to be housed, they have a lot of technical expertise and were able to convene a lot of leading experts in the field to exchange ideas and to kind of move the needle on the work.” Research and professional development staff were cited by USAID as being especially strong. They had good technical expertise in most areas, and given EDC’s education expertise, had experts easily available in-house. However, when there were gaps, ECCN would hire outside people; this became problematic, partly because of the rigidity of the contract. “Whenever there was something that couldn’t be done by core staff, ECCN would hire from the outside people who would be very expensive to do things that really should never have cost as much as they did. So the use of resources and I think quality issues were uneven because of it.” (USAID staff) “Maybe we were overly ambitious with the amount of funding that we had, but to actually do quality work in all of the areas that were asked of them to do, they didn’t have the staff to do that well across the board.” (USAID staff) ECCN has a robust Performance Monitoring Plan (PMP). The ECCN PMP includes 13 indicators, which cover network products, statistics drawn from metrics on the website, and data from yearly surveys of network members. According to USAID staff, the PMP shifted considerably after Year 3 and became more realistic, focusing more on developing quality resources, making them accessible, and facilitating application, and shifting away from the earlier emphasis on indicators measuring direct impact on education programming. B. GRN NETWORK RESULTS AND OUTCOMES EQ1: WHAT ACTIVITIES WERE MOST SUCCESSFUL, AND WHY? WHAT ACTIVITIES WERE ULTIMATELY UNFEASIBLE, AND WHY? MOST SUCCESSFUL ACTIVITIES USAID staff, GRN ST, and network members all considered GRN’s production of a large volume of high-quality knowledge products, in particular toolkits, to be a USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |29 major success. Some of the specific knowledge products that were highlighted in KIIs and FGDs included the Landscape Report on Early Grade Literacy, the Early Grade Reading Assessment (EGRA) 2.0 Toolkit, Universal Design for Learning Toolkit (teaching reading to children with disabilities), Critical Topic Series briefs, Classroom Observation toolkit, and Policy Linking Toolkit. USAID staff felt that GRN had made “huge strides” in the areas of disabilities and in getting directly involved in the production and wide distribution of books to promote literacy. These were emerging areas in the education space and significant advancement in knowledge and resources has occurred thanks to resource contributions from GRN. “I definitely think that knowledge product creation was one of the biggest successes of this contract.” (USAID staff) “The knowledge product creation will stand as the greatest achievement.” (USAID staff) GRN’s convening ability through the CoP was also praised. GRN was able to engage a wide variety of experts to contribute their expertise and time in the development of resources for the reading community. GRN was especially successful in leveraging the CoP to develop meaningful tools, such as the EGRA 2.0 Toolkit, the Universal Design for Learning Toolkit, the Classroom Observation Toolkit, and the Literacy Landscape Toolkit. As explained by the GRN ST, these tools were developed through a consultative process with input and feedback from major implementers and other donors, such as the World Bank and UNICEF, who were part of the CoP. These community members were highly engaged, and eager to share their learnings and what they’ve been working on to contribute to these major tools. Members felt that GRN also gave some exposure to new and diverse actors with skills and expertise in the education sector, not just the more well-known institutions and experts. High CoP membership numbers for GRN were notable. 4,444 people18—the majority of whom are not based in Washington, DC, according to the GRN ST—have been engaged with GRN in some way, be it downloading resources, participating in webinars, or attending trainings and events. According to the evaluation survey, over 81% of respondents have accessed resources or knowledge products from GRN, 60% participated in at least one webinar, and 21% attended in-person knowledge-sharing events. “The ability to convene a wide variety of experts in and of itself has really contributed to advancing work in this sector that I think is a global good in both cases [both ECCN and GRN].” (USAID staff) “What I like about the network is that it encourages organizations to share – to share their resources, to share their ideas, to share their best practices. It provides that collaborative… feel. Often… a lot of these organizations are in direct competition with one another on a daily basis and for USAID funding. … It's an opportunity to really push this education field forward and to 18 GRN Quarterly Report #22 (April-June 2019). 30 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID be sharing ideas outside of this competitive world that is education development.” (General member) “I thought they did a really good job of elevating people from all sorts of different backgrounds because they had the skillset to do the work.” (General member) Table 5: GRN Participants (Workshops and Courses) Period Count Oct-Dec 2018 328 Jan-Mar 2019 61 Apr-June 2019 183 July-Sept 2019 375 Total 947 GRN produced a large volume of helpful training and capacity building opportunities in the reading space. USAID staff felt that the training provided by GRN under domain/objective 5 was very successful and most closely met the expectations of the contract (compared to the other domains). GRN produced nearly two dozen unique training courses (in-person and online)19 to support toolkits and resources developed by the network. According to the evaluation survey, 16% of respondents attended at least one in-person training and 9% participated in an online training. GRN reports event, course, and workshop participation on a quarterly basis, but does not present aggregate statistics on participation. However, as an example of reach, GRN courses and workshops reached 947 people in the last full year of the project. (Table 5) 20 Many of GRN’s training and capacity building resources were made available online through the GRN website and were widely accessed. Toolkits and training packages represented the most downloaded knowledge products from the GRN website.21 19 According to the list shared by GRN support team for evaluation survey response options. 20 Though participants were counted carefully to ensure non-duplication during the same quarterly reporting period, it is possible that the same person is counted more than once over the course of multiple quarterly reporting periods. 21 GRN Stocktaking Report provides additional detail on the contents of GRN’s repository and the number of downloads. As examples, there were 1,822 downloads of the EGR training package’s zip files within one years, and 370 downloads of Comprehensive Approaches to Learning (Literacy and Numeracy) within four months. LACReads seeks to increase the impact, scale, and sustainability of early grade literacy interventions in LAC by developing and disseminating of state-of-the-art knowledge resources and providing technical assistance to host country. The program is active in Dominican Republic, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, and Peru, as well as Eastern Caribbean states. LACReads also oversees an evidence database on Early Grade Reading and supported the development of RedLEI, a network of Central American universities that looks to build capacity within the region to produce, disseminate, and use knowledge derived from high quality research to improve early literacy. Though RedLEI initially existed as part of LACReads, the project became independent in late 2019. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |31 One particularly successful training program was around the use of Bloom software,22 which is used to produce decodable and leveled books. GRN activities significantly pushed forward the use of Bloom software through training and guidance for national experts, and the software has now been used by NGOs and university-based teams to design 3,000 books in 15 languages and six countries including Bangladesh, Nepal, Philippines, Indonesia, Haiti, and Nigeria.23 Concomitantly, the Enabling Writers Workshop Guides and Toolkits, which teach local professionals to use Bloom software, are some of the most downloaded GRN resources. For example, there were 396 downloads of Enabling Writers Workshop Guides and Toolkits within eleven months.24 In addition to these training activities, GRN held at least one large in-person professional development event in Washington, D.C., each quarter and hosted monthly webinars to facilitate information exchange and generate broad awareness of evidence-based best practice on a global level.25 These activities provided in-depth training, released new technical guidance, and featured evidence-based best practices from USAID and other donors’ early grade reading programs. “For the live training, Universal Design for Learning to Help All Children Read, the training was exceptional and the resources were very good. I did not expect the training to be so useful.” (General member) UNFEASIBLE ACTIVITIES AND WHY There were quite a few unfeasible activities under GRN, which are described in this section. Potential reasons for this include the unrealistic expectations of objectives from the contract given the size, knowledge and capabilities of the contractor (which was a small business with limited education experience); the nature of the contract;26 and the level of funding available. In most cases, the objectives and expectations were not aligned with the budget, which made it difficult for the contractor to successfully achieve all the objectives under this contract (GRN ST and USAID staff). GRN was unable to fund much primary research (domain/objective 2 of the contract). One reason for this was that the funding allocated to this stream of work was not sufficient to design and implement a research program, according to USAID staff. At least one small research activity was funded in Guatemala with LACReads and RedLEI,27 but it was anticipated that GRN would consistently fund research that would contribute to an evidence￾base, and inform and advance the sector. In retrospect, USAID staff acknowledged that their expectations for this objective were a little unrealistic. 22 Bloom software was not developed by GRN, but GRN developed training and guidance through the Enabling Writers Workshops to help increase the use of Bloom software. 23 GRN Stocktaking Report. 24 GRN Stocktaking Report. 25 GRN Stocktaking Report. 26 See discussion on EQ8 for further information. 27 GRN Stocktaking Report. 32 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID “[GRN was] supposed to do more implemented research in the field, but there was never a budget to match it. Not even close.” (USAID staff) GRN was only able to trial innovations in one situation (domain/objective 3). The contract itself did not have the flexibility or staffing to implement innovations in the field, and it had difficulty identifying a mechanism or project with whom GRN could partner to test innovations. There was only one instance in which GRN was able to meet this objective— through the Enabling Writers Workshop Program, a six country workshop series that demonstrated how to use Bloom software to develop decodable and leveled books at reduced cost. USAID staff acknowledged that this was also an ambitious objective for the contract. Funding and staffing limitations limited GRN’s impact in metrics and protocols (domain/objective 4). Although this objective was extremely important to USAID and the sector at large, it was difficult for the contractor to make strides in this area without high-level experts. The fixed labor rates of the contract did not support the salaries that this level of expertise would require.28 Another reason mentioned was GRN’s inability to partner effectively with the Global Partnership for Education (GPE) and World Bank, as discussed further below. As a result, USAID staff ended up playing a much more involved and active role in activities under this objective (USAID staff). Despite these challenges, almost a quarter of web survey respondents mentioned integrating GRN resources into their metrics or monitoring and evaluation plans. Another required activity that was not achieved was the organization of a series of regional events. These events would have made it possible for people in the field, including Missions, to be more engaged in the network. According to USAID staff, regional events never happened partly because of the difficulty filling the training position and then later, once the position was successfully filled in Year 4, insufficient funding available in ODCs could not support such events. The absence of regional events also meant that GRN was not able to enroll host country governments effectively into the CoP. Of all the stakeholder groups, the one from which GRN had the least participation was host country governments, according to the GRN ST. This was also hampered by language, outreach and diplomacy constraints. It should be noted that when the contract was written, GPE was very active in this space. There was an assumption that they would continue to act as a convening body for host country governments, so GRN did not initially plan explicit activities to engage government stakeholders. In fact, GPE’s strategy shifted and they stopped playing the role of convener in this space, according to USAID. “I think it was incredibly ambitious from the start, but so was everything we were doing.” (USAID staff) Some technology related activities, such as websites and tools for digital engagement, were not executed efficiently or to the expectations of some USAID 28 See discussion under EQ8 regarding all contract issues. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |33 staff. One USAID staff speculated that this may have been due to requests from USAID that were outside the capability of the contractor. The contractor was not an expert in technology design and had not been selected with the expectation that they could provide these types of services. This was further hampered by restrictions preventing subcontractors under GRN, according to GRN ST. “I think they didn’t have the infrastructure needed to pull off some of the more sophisticated items that were being asked of them. So anything related to technology… I did not think was particularly successful.” (USAID staff) EQ2: WHAT WERE THE UNEXPECTED OR UNANTICIPATED OUTCOMES OF GRN? WHAT ADDED VALUE OR BENEFIT DID THEY PROVIDE TO USAID AND ITS IMPLEMENTING PARTNERS? UNEXPECTED POSITIVE OUTCOMES One unexpected positive outcome that GRN noted was the interest in and awareness of GRN from abroad, despite GRN not having strong outreach in the field. One member of the GRN ST speculated that this could have been because GRN filled a need and a void for professional support in the field: “I only know anecdotally from being at CIES that some of the folks from abroad come up and said that people have heard about the GRN’s activities and webinars, and that’s been unexpected and incredibly positive...” This is borne out by the make-up of GRN evaluation survey respondents; 261 (70%) respondents were from the field (developing countries), most of whom were neither USAID nor USAID implementer staff. UNEXPECTED NEGATIVE OUTCOMES It was difficult for GRN to gain traction on activities related to indicators and measurement practices. In the early years of the contract, GRN employed a working group approach that involved other donors (World Bank, GPE, DfID/UK AID, etc.) to help lead the network and contribute expertise in areas that the contractor lacked. However, GRN found it hard to keep these stakeholders engaged. After the first Reading Team Lead left around the middle of the project, the working groups were dissolved.29 One unexpected negative result was that GRN was not able to interact with GPE and the World Bank very successfully because GRN was not managed by a relatively well-known actor and not seen as having that much influence in this area. This made it difficult to gain traction on some activities related to indicators and measurement practices. “We’ve had several rocky times in protocols and metrics in domain 4, where what we wanted to see in terms of measurement practices and tools, and we really couldn’t get the traction we needed.” (USAID staff) 29 As described in KII with GRN CORs. 34 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID GRN experienced conflicting direction and feedback from USAID. GRN was often asked to produce technical resources that were only championed by two to three people in the agency and not widely requested or desired. In addition, the GRN ST recalled receiving really great feedback from the COR, but enormous feedback from other parts of USAID that what the network was producing was not what they were looking for. GRN struggled to be responsive to the various requests. ADDED VALUE TO USAID GRN contributed to amplification and increased awareness of USAID’s work in the field. USAID staff found GRN to be helpful at identifying and supporting additional USAID activities outside the network’s specific deliverables. The ability to identify and support these additional value-added opportunities around relevant evaluations, studies, and program milestones or events, helped to amplify the work of USAID. Multiple Mission members felt that GRN had led to more communication among USAID staff about USAID’s work at the country level, which has helped certain USAID offices have a clearer picture of what’s going on in the field, what’s working, and successes and lessons learned from other Missions. ADDED VALUE TO IPS Networking was a positive outcome that many IPs noted from their participation in the CoP. One general member mentioned that they had met a lot of different people from different organizations through their participation in GRN, and expanded their professional network of practitioners, which they found to be invaluable. One of the added benefits to IPs’ participation in GRN was additional exposure and opportunities to connect with USAID through direct dialogue. This also led to a better understanding of USAID’s priorities, helped IPs to be on the same page with USAID thinking, and provided equal footing among IPs to some extent. One general member felt that GRN provided a forum for smaller organizations to make a meaningful contribution and participate in conversations that would normally be dominated by the big IPs. “Those of us who are USAID implementers are always looking to see how to best go about delivering our programs in light of USAID's new trends and interests. I find that GRN gives good guidance on that and the most up-to-date guidance on how to go about doing that, whether it's on teacher professional development or language and education or setting standards or things like that.” (General member) “I really think it’s helpful for us all to be on the same grounds as far as understanding what USAID’s thoughts are and what their directions are, because I find sometimes certain implementers have more access to USAID and what they’re thinking than others and that they then, you know, have an in here, in there.” (General member) STAKEHOLDER USE OF NETWORK PRODUCTS AND SERVICES EQ3: HOW DID STAKEHOLDERS USE INFORMATION THEY RECEIVED THROUGH THE GRN? TO THE EXTENT POSSIBLE, WHAT SPECIFIC DESIGN, IMPLEMENTATION, USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |35 MONITORING OR EVALUATION APPROACHES OR ADAPTATIONS DID THE INFORMATION INFORM, AND WHAT WAS THE IMPACT OF THIS UPTAKE? GRN stakeholders have used information received through the network in a variety of ways, most commonly to improve understanding, throughout implementation, and when designing programs/activities. In the evaluation survey, 44% of respondents reported changing their practice as a result of GRN’s work. Figure 4: How Members Applied GRN Resources or Knowledge Source: Evaluation Survey The top use of GRN resources or knowledge by stakeholders is to improve their personal and/or team's understanding for future efforts, with 66% of evaluation survey respondents reporting they had applied GRN information to that end. In qualitative interviews, general members reported using GRN information for their own personal development, as well as sharing GRN resources with others to develop their capacities. Stakeholders also applied GRN resources or knowledge in program/activity design. 44% of evaluation survey respondents reported using GRN resources or knowledge in program or activity design. In qualitative interviews, only one member was able to give a concrete example of using GRN information for program design, but others expressed that the increased knowledge from GRN products had improved their design abilities: “While I was working for [an implementing partner], I had to design a new early grade reading program in Bangladesh. At that time, I was very much influenced by the global learnings of reading programs and the resources are posted in Global Reading Network.” (General member). 14% 24% 24% 38% 42% 44% 58% 66% 0% 10% 20% 30% 40% 50% 60% 70% Developed publications Theory of change development Integrated into metrics/M&E plans Informed policies for promoting new concepts/approaches in our organization or community Developed training material Integrated into future program/activity design Improved implementation of an existing program Improved my personal and/or my team's understanding for future efforts Q. In what ways have you applied resources or knowledge from GRN in your work? 36 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID “I would say when I'm actually sitting down to do my work, having read the resources, it makes me think about, you know, I might do a workplan a bit differently or I might design a research study to include something that I have learned from one of their guidance documents. So, I would say, yes, it has because it's definitely contributed to increasing my knowledge and my understanding of how to better do my job, technically speaking.” (General member) Over half the evaluation survey respondents (58%) have applied GRN resources or knowledge during implementation of an existing program. In a qualitative interview, a GRN member provided examples of using network information for implementation: “[I’ve used GRN information] probably in activity management, some of those guidance notes about open source licensing and inclusive education and things like that. It helps to better understand, because Global Reading Network activities help as activity manager.” (Mission member) About a quarter of members used GRN resources or knowledge in metrics or M&E. A significant number of evaluation survey respondents (24%) reported integrating GRN information into metrics or M&E plans. Two interviewees mentioned applying the formative assessment document to their work, but there was no other mention of metrics or M&E: “Yeah, particularly now the assessment, continuous assessment. When we had to formulate a formative assessment for my program, I got a lot of help from the GRN website.” (General member) “As a subcontractor, we get more involved once a project has been awarded and the prime has already done some broad strokes, just redefining of the project as needed. So our work really comes in when we're trying to get things done. And at that point, you know, someone says, we’re going to do a formative assessment and we're all trying to, you know, figure out what the prime thinks that means. What does the COR think that means? What do we think that means and how much budget is there? And GRN, I think, helped in that moment to figure out what everyone thinks we’re talking about and the scope of what they’re looking for.” (General member) CHALLENGES TO USE When asked why they did not apply all GRN resources that they’d accessed, 59% of evaluation survey respondents reported that the opportunity to apply certain products had not arisen. Other reasons cited for non-application included the resources not being relevant (24%) and being duplicative (21%). In qualitative interviews, members reported that sometimes GRN materials were things that practitioners were already doing or things that were already available elsewhere, such as on the USAID DEC. Members noted that some of the larger toolkits, such as UDL, were simply pieces of other work that had been published previously, pulled together, and packaged as a new toolkit. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |37 Figure 5: Reasons for Not Applying Accessed GRN Resources Source: Evaluation Survey Qualitative research with members also revealed concerns about the complexity of GRN resources and the need for simplification. One general member noted she had given feedback about the complexity of a document, concerned it would be hard to apply in the field, but GRN did not take her comments into account. Others offered recommendations on how GRN documents could be made more digestible, including creating executive summaries, one pagers of highlights, and PowerPoint summary decks. “I think if some of the information could be further summarized and the figures compounded by visual illustrations, they will be more eye catching and easier to not only locate, but to also to access.” (Mission member) EQ4: WHAT SPECIFIC KNOWLEDGE PRODUCTS DID STAKEHOLDERS FIND MOST VALUABLE? WHY? Evaluation survey respondents were asked whether they had accessed any of a list of products from GRN, as well as whether they had applied those products. Using these two pieces of data, NORC calculated a utilization rate for each product. The average utilization rate for the resources listed was 48%. Table 6 below shows the products with average and above average utilization rates. 1% 4% 12% 21% 24% 59% 0% 10% 20% 30% 40% 50% 60% 70% Resource is in draft form and we prefer to wait for the final Resources did not offer clear, succinct presentation of key actionable findings Some resources do not lend themselves to practical application Resources were duplicative of what I already do Resources were not relevant for my purposes May use in the future, but have not yet had an opportunity to use Q. For GRN resources you have accessed but never applied in your work, what are the main reasons why you did not apply the resource(s)? 38 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Table 6: Utilization Rate of GRN Products Resource Accessed (Count) Applied (Count) Utilization Rate EGRA 2.0 Toolkit 142 102 72% UDL: Universal Design for Learning (UDL) to Help All Children Read: Promoting Literacy for Learners with Disabilities Toolkit 79 50 63% Coaching in EGR Programs: Evidence, Experiences and Recommendations 87 49 56% Enabling Writers Workshop Program Guides and Toolkits 108 53 49% Latin America/Caribbean Reading Network Policy Papers 22 11 50% Total Respondents 285 277 58% Source: Evaluation Survey MOST VALUABLE PRODUCT: EGRA 2.0 TOOLKIT The EGRA 2.0 Toolkit was among the most accessed resources, as well as the most utilized. 72% of evaluation survey respondents that accessed it applied it to their work.30 Qualitative data from members confirms that this toolkit was the most valuable resource. Additionally, the USAID COR named the toolkit among her list of most useful products that the network produced. MOST VALUABLE PRODUCT: UDL TOOLKIT The UDL toolkit was highly utilized by those who accessed it. 63% of evaluation survey respondents that accessed it applied it to their work. In qualitative interviews, stakeholders often mentioned the value of GRN’s resources on disability. MOST VALUABLE PRODUCT: COACHING IN EGR PROGRAMS The Coaching in EGR Programs document was also highly utilized by those who accessed it. 56% of evaluation survey respondents that accessed it applied it to their work. In an interview, one general member in a remote location noted that the resource helped him understand how to integrate the role of literacy coach into a program he was designing. 30 GRN does not do an annual survey of members, so application and change of practice cannot be contrasted with the evaluation survey. What is the EGRA 2.0 Toolkit? The EGRA 2.0 Toolkit looks to advance suggests a standard approach to EGRA and serves as a guide for countries beginning to work with EGRA in regards to local adaptation of the instrument, fieldwork, and analysis of results. What is the UDL Toolkit? This toolkit provides evidence￾based research and information on how to teach early literacy skills to students with different categories of disabilities, including how to apply these theoretical approaches in practice in low and middle income countries. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |39 OTHER VALUABLE PRODUCTS In qualitative interviews and focus groups, respondents reported several other products as particularly valuable. These included the Open Licensing of Early Grade Reading Materials: Information and Guidance, which was one of the few tools developed with Mission members in mind; the Guide for Promoting Gender Equality and Inclusiveness in Teaching and Learning Materials, which was helpful because it taught members how to review their program materials for inclusivity and included detailed illustrations; and the Assessment to Inform Instruction: Formative Assessment, which helped clarify the concept of formative assessment for members and instructed them on how to conduct one. EQ5: WHAT MODES OF DELIVERY MADE ACTIVITIES SUCCESSFUL/UNSUCCESSFUL? MODES OF DELIVERY Email was the dominant mode of delivery for GRN members; 72% of evaluation survey respondents indicated that email was one of the top three ways they receive information from GRN, while 35% noted the emailed monthly newsletter. Qualitative respondents confirmed that email was their preferred mode of receiving information, noting their limited time and competing priorities. Several noted that they liked receiving notification via email of new products and events so they could skim them and save for future use. The number of emails was seen as appropriate; 83% of evaluation survey respondents reported that the number of emails they got from GRN was “the right amount.” Some stakeholders experienced difficulty navigating the GRN website to find resources. In the evaluation survey, respondents noted that the search functionality on the resources page could be improved. One USAID stakeholder also reported that the website was not executed efficiently or to their expectations. “The website is a platform for sharing. However, finding documents is not easy and sometimes starting outside of the GRN - with Google as the search engine, is the better way to locate files.” (Evaluation survey respondent) “It would be great if EGRA instruments, evaluation reports, and program reports were easier to locate on the website (have broken down categories for each of the above). The search options on the website are not easy to navigate to find such resources.” (Evaluation survey respondent) “I have often had issues accessing full PDFs of resources through the site. Sometimes when you click on a resource, there is no intuitive way to access the full resource. It would be nice to make this a little easier to navigate.” (Evaluation survey respondent) With regard to training modalities, GRN members found live webinars to be most valuable. 40% of evaluation survey respondents indicated that live webinars were the most valuable, followed by the 26% who chose in person trainings. In qualitative interviews and focus groups, GRN members reported a variety of challenges related to webcasts. GRN members and the GRN ST reported that the timing of GRN events (webinars and in person events with 40 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID remote options) made it difficult or impossible for members to attend from the field. These events were seen as being scheduled for the convenience of those in Eastern Standard Time. Additionally, general members reported that connectivity and technology limitations, especially in remote areas, posed problems for them in accessing resources and events from the field. USAID Mission members did not report these problems. Members reported that GRN resources and webinars were typically only available in English and were Anglo-centric in their focus. Document review confirms this, as only two of the 13 resources mentioned in the evaluation survey were available in any language other than English. Both Mission and general members felt that GRN should translate more resources into local languages, or at least widely spoken languages like Spanish, French, and Arabic. Members said this would make resources more accessible to FSNs and implementing partners.31 MISSION/REMOTE ENGAGEMENT EQ6: HOW WERE MISSIONS AND IMPLEMENTING PARTNERS ENGAGED? WHAT ASPECTS OF THE COMMUNITIES OF PRACTICE DID THEY FIND MOST VALUABLE? MEMBER ENGAGEMENT Respondents reported engaging with the CoP in a variety of ways, including learning from each other and networking with other members. The most common means of engagement reported by evaluation survey respondents was learning new approaches and knowledge (41% participated actively); 28% of respondents reported actively sharing resources and/or knowledge with the community, while 22% indicated they actively participated in networking opportunities. More than a third of members (36%) reported not participating actively in any CoP activities. Although many GRN members are located in Missions, engagement with them was limited, according to interviews with network members, USAID staff, and the GRN ST. Missions reported receiving emails from GRN but very few participated in other ways. The lack of time to engage was commonly stated and several participants requested help navigating the large amount of information available from GRN. Many had not even visited the GRN website, and a few who did found it overwhelming. “For me, it’s really a function of time, and I really do wish that I had more of that particular resource to really delve into all that’s being shared out there…We’re way understaffed. I guess every AID Mission says that, but we truly are. And so the volume of it, of the information, sometimes feels overwhelming, like I’ll never know all of this, I’ll never be able to catch up with all of this.” (Mission member) 31 Only the EGRA 2.0 Toolkit (available in French and Arabic) and UDL (available in French) were available in any language other than English. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |41 “I think if we could have someone who could help us go through the network and familiarize ourselves with everything that was in there, perhaps I could go in there and use [resources] to better inform education in this country.” (Mission member) From USAID’s perspective, Mission engagement was challenged by the timing of webcasts and GRN’s inability to carry out the regional events originally in its contract. Additionally, USAID noted that Missions were not a primary target at the beginning of the project and noted there were only a couple of GRN resources designed for a Mission audience. The GRN ST added another reason for limited Mission engagement: they were not allowed to connect directly with Missions, but rather had to go through the COR who then asked regional offices to connect with missions. The GRN ST also noted that no needs assessment was ever done for Missions (as it was for implementers), so they were not aware of what Missions needed. The GRN ST viewed this as a missed opportunity: "Missions have actually never contacted us themselves. We’ve made contract through the regional office – I think there’s a bit of a barrier there. I would’ve loved to connect with missions, I think there’s so much work to be done especially with early grade reading training could’ve been done.” However, the vast majority of members that participated in the evaluation survey (92%, including both Mission and general members) reported that they would like to continue being part of GRN (57% yes, very much; 35% yes). Two thirds of respondents (66%) reported being satisfied with the number of engagement opportunities. MOST VALUABLE ASPECT OF COP Evaluation survey respondents reported actively participating in a variety of CoP activities. For each activity in which they participated, respondents also indicated the value received. Learning new approaches or knowledge was seen as very valuable by the most participants (73%), followed by forming new relationships (69%). Less than half of respondents who shared challenges with the community for help found it to be very valuable. 42 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Figure 6: Value of GRN CoP Activities as Reported by Members Source: Evaluation Survey Nearly half of evaluation survey respondents (46%) reported that GRN fully met their expectations in in terms of providing access to tools and resources. Expectations regarding the opportunity to interact with USAID and other partners around education and partner advocacy with USAID were less fulfilled. Member expectations in these areas were fully met for only 20% and 24% of respondents, respectively. Figure 7: Expectations Met on Aspects of CoP - GRN Source: Evaluation Survey 73% 69% 61% 58% 54% 44% 25% 31% 37% 38% 45% 53% 0% 20% 40% 60% 80% 100% Learned new approaches & knowledge Formed new relationships Participated in a working group Used networking opportunities Shared resources and/or knowledge with the community Have shared my challenges with the community for help Q. For the activities in which you participated actively, to what extent was the activity valuable? (those who answered "very" or "somewhat" valuable) Very Valuable Somewhat Valuable 45.9% 38.4% 36.8% 22.4% 19.9% Access to tools and other resources Personal capacity building Knowledge sharing Partner advocacy with USAID Opportunity to interact with USAID and other partners around education Q. To what extent were your expectations met in each of the following areas? (those who answered "fully met") USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |43 EQ7: WHAT WOULD THEY SUGGEST FOR THE FUTURE? REDUCE THE EMPHASIS ON DELIVERABLES AND FOCUS ON MEMBER NEEDS In the evaluation survey and qualitative interviews, general members noted that the pressure to produce a large number of deliverables appeared to have created a group more focused on producing products for USAID than on building connections between members. Some members felt that they were constantly asked for inputs for deliverables, but their contributions were not always used or acknowledged. This seemed to be a difference between GRN and other networks operating in the same space. The GRN ST also noted that the Chief of Party had to spend too much time managing different tasks to produce deliverables, which came at the expense of partnerships and furthering engagement. “With some other [networks], they’re not just driven by the products that they are trying to create, but they’re also gathering and filtering what might be meaningful to the group and sharing and disseminating that kind of thing. Like there’s a really great inclusive education community of practice, you know, if there are events that are happening, you know, around that might be of interest to members of the group, they’re sharing that information. And they’re sharing something that is not necessarily USAID driven or funder driven, but they truly are sharing resources.” (General member) “The network seemed deliverables/product driven, especially in its last year. It seemed like they were more interested in getting products done to meet their deadlines which seemed to affect the quality of the products. Not sure if the advocacy side worked well. It seemed that the network was eager to please USAID so when calls for advocacy came in (i.e., inputs on new Education Strategy), I'm not sure our inputs were carried forward. There was no transparency on what was shared with USAID after the collective call for inputs. Other organizations/groups do share what was submitted.” (Web survey respondent) IMPROVE THE REMOTE EXPERIENCE DURING EVENTS In the evaluation survey and qualitative interviews, respondents noted many ways that GRN could improve the experience of remote members during events. The audio quality and timing of webcasts were consistent barriers to participation identified by respondents in both the evaluation survey and qualitative data. Other respondents reported feeling like they were forgotten when they remotely joined in-person events. They recommended several improvements in this area, including having GRN host multiple regionally-based versions of webcasts with local participants, as well as archiving recordings on YouTube for later viewing. “The webinars are frustrating as staff want to engage, but the quality is extremely poor and they cannot usually be heard. They often hang up and stop. In addition, the timing is often at odd times that are difficult for staff to listen in at… Short YouTube videos of the webinars or useful topics that can be shared through the local network would be a great help.” (Evaluation survey respondent) 44 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID “When there is virtual participation for in-person events, it has always felt like the people online are an afterthought. This has made the GRN feel like it is mainly for people in D.C.” (Evaluation survey respondent) INVOLVE REMOTE MEMBERS IN PRODUCT DEVELOPMENT In the evaluation survey and qualitative interviews, both Mission and general members requested to see voices from the field more effectively and intentionally incorporated in the design of GRN tools and resources. There is a perception that GRN relies on a group of ‘usual suspects’ as their go-to experts when speakers are needed or deliverables need to be produced. Interviewees noted the need for new voices and some said they had expertise to offer. “[Include] more people who aren’t the usual suspects - more field based people who’ve been in the field for quite some time.” (General member) “The network has a limited conception of who the experts are. Should cast the net more widely to include more than the ‘usual suspects.” (Evaluation survey respondent) “GRN resources highly rely on the context of a specific region, specifically researches, references of education philosophies. It should be more inclusive and have better understanding of the context and suitable pedagogy.” (Evaluation survey respondent) RECRUIT MORE REMOTE MEMBERS In the evaluation survey and qualitative interviews, respondents felt strongly that GRN’s membership should be expanded to include more field-based perspectives, including host country government officials, NGOs, academics, researchers, and practitioners. Some suggested that regional in-person events, which were originally part of GRN’s contract but were unfeasible, could be leveraged to improve engagement and recruit new members. NETWORK MANAGEMENT EQ8: WHAT WORKED IN TERMS OF MANAGEMENT OF GRN? WHAT DID NOT? WHAT MIGHT HAVE MADE MANAGEMENT OF THE ACTIVITIES EASIER AND MORE EFFECTIVE? The current USAID COR and GRN COP have a constructive and productive relationship. After management difficulties, a stoppage of work, and shared frustrations about the nature of the contract, the GRN ST and USAID reported working closely during the final year of the contract to resolve problems and complete a high volume of quality work. The relationship is collegial, and there is a lot of communication. The USAID COR reported being very engaged in GRN implementation, sometimes even working closely on technical products, due to the lack of sufficient technical staff on GRN. USAID described that the selection of the contractor for GRN required a trade-off between the resources one of the “education powerhouses” would bring and the need to establish a space where different stakeholders (often competitors) could USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |45 share openly. “We had to have somebody who’d be seen as an objective broker in this case.” (USAID staff) Accordingly, they decided to contract through MOBIS rather than bidding it through an IDIQ which would have been limited to the major education contractors. Agency priorities further dictated that the bid be issued as a small business set-aside, and URC was selected. One USAID DC member thought that a vehicle other than a contract might have been more apt “if we’re really wanting to have at least a portion of what we’re doing be community driven…[This] could contribute to sustainability at some point if you’ve got a larger stake from other entities that are partners with us.” But according to USAID, the selection of URC presented a number of challenges, including a steep learning curve, given that the company had not done things “at this speed and scale before,” didn’t have offices in many countries, and didn’t have a big roster of international experts in education. This also meant some limitations for GRN, especially in its convening power and its ability to build a sector agenda. USAID acknowledged these limitations in interview with the evaluation team, “That’s not something a small business contractor in D.C. could do – and yet we had this idea that this would be the group that pulls the world together and keeps them focused on an agenda.” Second, URC did not have the in-house education technical expertise. Accordingly, USAID had to spend more time than anticipated managing the contract and providing technical inputs, often at a micro-level. “[We had to] go through several rounds of feedback to help tweak, update, and refine. I think it was unrealistic, unfeasible - just on the management side, it became a lot more than what we had expected.” (USAID staff) “From where we started, we knew it was going to be choppy and it was going to be a challenge and it was. And they didn’t have the right people and probably maybe we were too tough on expectations and having them do too many different things, and they still didn’t entirely staff up.” (USAID staff) Both USAID staff and the GRN ST mentioned the prescriptive, rigid contract as significantly affecting the performance of GRN. “A lot of this contract was built rigidly,” according to USAID. They highlighted the difficulties of the highly prescriptive requirement of deliverables, such as specific numbers of major training events, global events, country-level events, and webinars, which became too difficult to accomplish. “It leads to a lot of acrobatics to try to make the terminology and expectations of the award something the people can do…[it also] was opening the door for a lot of friction between the agency and the contractor.” (USAID staff) There was limited funding for ODCs, the contract did not allow shifting funds between labor and ODC line items, and the contract had no allowance for third party contracting under ODCs. As a result, the network had insufficient funding for holding events and publishing materials, according to USAID and the GRN ST. The prohibition of all subcontracts also led to a major problem in 2015 that brought the network to a standstill, when a major subcontract was issued and then disallowed. The stoppage significantly reduced the activities of GRN for more than a year, reported USAID. 46 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID While the GRN contract had more funding available for labor, roles were pre-defined and often didn’t include categories that were needed, according to the COP, who also believed the labor rates were too low to get appropriate staff for certain positions. It is the evaluation team’s understanding that the GSA MOBIS/PSS contract vehicle does not preclude subcontracting, does not require fixing the labor and ODC contract line items in a task order, and does not preclude revising/adding labor categories at the task order level as long as they can be mapped to the prime contractor’s labor categories in their award. The scope of this evaluation has not allowed further inquiry regarding to whether and the extent to which these issues were raised with the CO and whether they could have been rectified given that both the COR and GRN ST agreed they were hindering effective implementation of the contract. Staffing has been a critical issue for GRN. As suggested earlier, staffing was problematic for GRN, both because of the nature of URC but also the rigidity of the contract, according to both USAID staff and GRN ST. The lack of in-house technical expertise resulted in the need for a lot of hands-on involvement by USAID. USAID staff also observed that “Even at full stop, I don’t think there are more than five people with seniority, senior decision making power, [on the contract].” The lead trainer slot remained unstaffed from the end of Year 2 through the middle of Year 4. As USAID staff explained, “They were cycling through short-term people that they would get approved to consult to keep it going.” The GRN ST corroborated the staffing problem, describing several functions that were needed but could not be staffed, such as a strong communications expert and a full-time designer for publications given the large amount of material being produced. According to USAID staff, “GRN struggled with staff for the whole time from what I could tell in terms of getting the right people in the right positions…. I’ve never seen a good communications or training person come out of that contract ever.” GRN ST felt that having an operations manager was an important gap, and eventually URC funded that position themselves after USAID denied their request for a new labor category. USAID believes the network was weakened by changing leadership throughout its first three years. A USAID staff member noted that for this kind of activity, “I definitely think the quality of your staffing and leadership is something you’ve got to get right – this work suffered terribly from four changes of Chief of Party.” The GRN ST also expressed their view that GRN would have benefited from an additional leadership slot so that someone could manage the high volume of discrete tasks for resource development and leave the Chief of Party more time to focus on partnership, planning, and engagement. C. SYNTHESIS NETWORK RESULTS AND OUTCOMES EQ1: WHAT ACTIVITIES WERE MOST SUCCESSFUL, AND WHY? WHAT ACTIVITIES WERE ULTIMATELY UNFEASIBLE, AND WHY? Commonalities: The most successful activities for ECCN and GRN were very similar. Both networks developed high-quality knowledge products by leveraging the strength of experts and USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |47 organizations within their respective CoPs. Both networks produced and delivered knowledge￾sharing events and capacity building opportunities that were considered valuable by both USAID and IPs, and were commended for their convening power. Both networks also struggled a bit with the technology aspects of the networks, such as developing websites and digital tools. Differences: GRN struggled to fulfill many of the objectives outlined in their contract, although USAID staff admitted that the contract was very ambitious and somewhat unrealistic, especially given the unique budget constraints under their contract and the size and experience of the contractor. ECCN was able to complete most of their activities with the exception of the knowledge share groups within the CoP. EQ2: WHAT WERE THE UNEXPECTED OR UNANTICIPATED OUTCOMES OF ECCN/GRN? WHAT ADDED VALUE OR BENEFIT DID THEY PROVIDE TO USAID AND ITS IMPLEMENTING PARTNERS? Commonalities: Both networks provided additional value to USAID by helping to amplify the work of USAID in the field. ECCN and GRN also provided opportunities for IPs to connect with USAID and better understand USAID’s priorities. Differences: ECCN had successes in unexpected areas, such as facilitating cross-agency collaboration and overcoming competition among IPs. This speaks to the network’s strength in convening. Conversely, GRN faced unexpected difficultly in responding to USAID direction and feedback that was sometimes conflicting because of the involvement of so many individuals at USAID. STAKEHOLDER USE OF NETWORK PRODUCTS AND SERVICES EQ3: HOW DID STAKEHOLDERS USE INFORMATION THEY RECEIVED THROUGH THE ECCN/GRN? TO THE EXTENT POSSIBLE, WHAT SPECIFIC DESIGN, IMPLEMENTATION, MONITORING OR EVALUATION APPROACHES OR ADAPTATIONS DID THE INFORMATION INFORM, AND WHAT WAS THE IMPACT OF THIS UPTAKE? Commonalities: The main ways that stakeholders used information were in program/activity design, improving implementation, and improving their understanding. Substantial shares of both GRN and ECCN members (23% and 33% respectively) applied network resources and information in metrics and M&E. When stakeholders did not apply resources in their work, the main reason given was that the opportunity had not yet arisen. 48 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Figure 8: Comparison of Areas in Which Network Activities Have Been Useful Source: Evaluation Survey Differences: Duplication of existing resources was mentioned as one of the top reasons for not applying GRN resources, but was a lesser concern for ECCN stakeholders. 0% 20% 40% 60% 80% 100% Future program/activity design Theory of change development Improved implementation of an existing program Metrics/monitoring and evaluation plans Developing publications Developing training material Improved my personal and/or my team's understanding for future efforts Informed policies and/or strategies for promoting the new concepts/approaches in our organization or program Other (please specify) None Percentage of Respondent Area Q. In what areas have these activities been useful in your work? ECCN GRN USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |49 Figure 9: Why Resources Were Not Applied Source: Evaluation Survey EQ4: WHAT SPECIFIC KNOWLEDGE PRODUCTS DID STAKEHOLDERS FIND MOST VALUABLE? WHY? Commonalities: The average utilization rate for listed resources from each network was 42- 48%. The most valuable products across both networks were toolkits that members leveraged for program/activity design and in improving implementation. Differences: For ECCN, the most valuable products were the RERA Toolkit and AEWG’s Guide to Accelerated Education Principles. For GRN, the most valuable products were the EGRA 2.0 Toolkit, UDL Toolkit, and Coaching in EGR Programs document. EQ5: WHAT MODES OF DELIVERY MADE ACTIVITIES SUCCESSFUL/UNSUCCESSFUL? Commonalities: Email was the predominant way that stakeholders received information from both networks and they were very satisfied with the frequency of communications. Email alerts provided busy stakeholders with quick snapshots of new products and knowledge events and allowed them to save the emails for later use. Most of the resources produced by each network were Anglo-centric and available only in English, potentially limiting the reach of network messaging and approaches. 0% 10% 20% 30% 40% 50% 60% 70% Some resources do not lend themselves to practical application Resources did not offer clear, succinct presentation of key actionable findings and lessons Resources were not relevant for my purposes Resources were duplicative of what I already do May use in the future, but have not yet had an opportunity to use Resource is in draft form and we prefer to wait for the final Percentage of Respondents Selecting the Reason Reason Q. For ECCN/GRN resources you have accessed but never applied in your work, what are the main reasons why you did not apply the resource(s)? ECCN GRN 50 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Differences: There were differences between the networks in terms of modalities for knowledge events, per the evaluation survey. GRN stakeholders found live webcasts to be the most valuable, while ECCN stakeholders (who had fewer events available overall) preferred in￾person events. However, GRN stakeholders also reported many more problems with webcasts than ECCN stakeholders, including the inability to attend because of timing and connectivity challenges. Also unique to GRN stakeholders were complaints about the network website, especially the search functionality. MISSION/REMOTE ENGAGEMENT EQ6: HOW WERE MISSIONS AND IMPLEMENTING PARTNERS ENGAGED? WHAT ASPECTS OF THE COMMUNITIES OF PRACTICE DID THEY FIND MOST VALUABLE? Commonalities: In both networks, the most common means of engagement was remote members learning new approaches and knowledge from the network. More than 90% of members in each network were interested in continuing membership after project funding ended. Members in both networks ranked access to tools and resources as the most valuable part of being in the CoP. For both networks, the newsletter was the most frequent means of engagement, but websites and webcasts/webinars had robust participation as well. Figure 10: Most Frequent Avenues for Engagement with Network Source: Evaluation Survey 38% 62% 80% 75% 26% 54% 78% 66% In person events (including trainings) Webcasts/webinars Newsletter Website Q. How frequently do you engage with the network in each of the following ways ("often" or "sometimes") ECCN GRN USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |51 Differences: A greater percentage of GRN members felt they had sufficient opportunities to engage meaningfully with their network. ECCN was better able to meet its members’ expectations regarding the CoP, especially in the areas of tools/resources, personal capacity building, partner advocacy with USAID, and the opportunity to interact with USAID and other partners around education. EQ7: WHAT WOULD THEY SUGGEST FOR THE FUTURE? Commonalities: General members of both ECCN and GRN felt that the networks were too focused on the needs and interests of USAID and not enough on the network members. While the evaluation survey cannot be considered representative, it is clear from the make-up of respondents that general members are a significant segment (well over half) for both networks (Error! Reference source not found.). These members recommended shifting away from roducing USAID deliverables and promoting USAID policy and toward developing a better dialogue with remote members about what they need to be successful. General and Mission members in both networks advised involving individuals in the field more in product development, especially early in the process. There was the perception that this would increase the relevance and practicality of resources. General and Mission members of both ECCN and GRN felt that the networks could increase recruitment of and engagement with field-based members, including host country governments and local NGOs. Table 7: Types of Network Members Member Group ECCN GRN Field Home-based Field Home-based N % N % N % N % USAID 11 24% 4 7% 31 12% 8 7% USAID implementer 9 20% 23 40% 55 21% 40 35% Other 26 57% 30 53% 175 67% 66 58% TOTAL 46 57 261 114 Source: Evaluation Survey Differences: GRN’s members recommended serious improvements for remote participants during GRN events. The audio quality and timing of webcasts, feeling of being ignored when joining in-person events remotely, and lack of recordings which could be watched later all contributed to a poor user experience. These problems were not mentioned by ECCN members. NETWORK MANAGEMENT EQ8: WHAT WORKED IN TERMS OF MANAGEMENT OF GRN? WHAT DID NOT? WHAT MIGHT HAVE MADE MANAGEMENT OF THE ACTIVITIES EASIER AND MORE EFFECTIVE? Commonalities: In both networks, USAID CORs have consistently been very engaged in managing the contracts. Both ECCN and GRN had rigid contracts, which affected staffing and activities. 52 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Differences: There are numerous differences in management-related features between GRN and ECCN which affected implementation of the contracts. ECCN had a consistent Chief of Party from start to finish, while GRN had four different Chiefs of Party. The ECCN contractor, EDC, is an education-focused company, facilitating access to education expertise, while the GRN contractor, URC, did not possess a deep bench of education experts. GRN had a highly prescriptive contract (unlike ECCN) requiring specific numbers of specific types of products and events for each year. This became a problem for both the USAID COR and Chief of Party. GRN had no subcontractors, while ECCN benefited significantly from its subcontractors, especially the University of Massachusetts. ECCN was able to provide added value to USAID by leveraging low-cost technical support through its university partner. This relationship enabled ECCN to accomplish more than they would have been able to with only the project team. 5. LESSONS LEARNED General member expectations for the CoP did not align with USAID expectations. Many of the member critiques of ECCN and GRN revolved around the tension between IP expectations of an informal, member-driven CoP structure versus the more formalized and deliverable-driven structure requested by USAID. A frequent critique of GRN was that it did not operate like a true CoP where the focus should be on dialogue, sharing, and learning among the community members to improve practice. Rather, their perception was that GRN was too driven by USAID and producing deliverables. Some members felt that this took away from the opportunity to share about how a project or challenge was addressed and did not allow members struggling with similar challenges to learn through open dialogue. This type of social learning, which is typical of CoPs, did not happen as often as general members would have liked. Similarly, members voiced frustration that ECCN wasn’t able to operate as a true CoP because the contractor was often acting as an extension of USAID and responding to USAID needs and requests more than those of the CoP members. “The problem I think they both [ECCN and GRN] had was that USAID had limited staff and they became extensions of USAID rather than communities of practice. So they really were responding to what USAID needed done and less to what the members [wanted or needed].” (General member The literature on CoPs and Networks generally supports this critique. A CoP is defined as a group of people who share a concern or a passion for something they do and learn how to do it better as they interact regularly. The learning that takes place through the CoP is not necessarily intentional. 32 A learning network is defined as a facilitated, peer-to-peer learning approach that can be highly effective at documenting and sharing knowledge between donors and implementing partners to help strengthen a particular technical area.33 An article published in the KM4D Journal by Sarah Cummings and Arin van Zee looked at the differences and 32 Wenger-Trayner. [Cited 20 Nov 2019]. Available from: https://wenger-trayner.com/introduction-to-communities-of-practice/ 33 USAID Learning Lab. https://usaidlearninglab.org/learning-networks/resources USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |53 similarities between CoPs and networks.34 The article came about because of the authors’ frustration at seeing the terms CoP and network used almost interchangeably. It came to the ultimate conclusion that CoPs and Networks are not so much different types of entities, but rather on different ends of a spectrum or continuum of social learning mechanisms. One end of that spectrum is the more informal and organic CoP and at the other end is the more organized and typically facilitated network. Based on the evaluation team’s understanding of the expectations and ultimate activities of both ECCN and GRN, neither would fit with the traditional CoP structure described in the literature.35,36 The structures and activities of both ECCN and GRN were more in sync with those of a learning network. Unfortunately, calling them CoPs created certain expectations among the members that the contractors were not able to meet due to competing priorities from USAID and the members. Sustainability was not considered at the outset of these contracts. Sustainability was not carefully considered in the planning of either network. For example, there was no long-term plan for what would happen to the ECCN or GRN websites and assets when the contracts ended. Ultimately a decision was made to merge the ECCN website into the EducationLinks platform (the GRN website is also scheduled to eventually migrate to EducationLinks, but would have benefited from an earlier decision to allow for a smooth transition). USAID also does not seem to have anticipated that the members of the CoPs would be so eager to continue the communities after the contracts ended. “Understanding people’s motivations and intent to continuing to participate in communities like these would be a useful lesson to understand going forward when we’re thinking about, what does it mean to be a community? What are the ingredients that you’d want to have in place to be successful?” (USAID DC member) Both networks missed the opportunity to learn from and share knowledge with people in the field. Neither network made it a priority to reach out to practitioners in the field, missions, or host country governments, which was seen as a missed opportunity by USAID CORs, Mission members, ECCN and GRN STs, and general members. GRN held very few events outside of the US and most webinars were only convenient to US members. ECCN did a better job at scheduling events that were convenient for non-US-based members, but also had room for improvement when it came to outreach. Most learning takes place in the field, and both networks missed out on the opportunity to harness that learning. Contracts were not clear on the extent to which the networks should have engaged USAID Missions directly, but they were 34 Cummings, S. and A. van Zee, 2005. Communities of practice and networks: reviewing two perspectives on social learning. KM4D Journal 1(1): 8-22 www.km4dev.org/journal 35 Cummings, S. and A. van Zee, 2005. Communities of practice and networks: reviewing two perspectives on social learning. KM4D Journal 1(1): 8-22 www.km4dev.org/journal 36 Wenger-Trayner. [Cited 20 Nov 2019]. Available from: https://wenger-trayner.com/introduction-to-communities-of-practice/ 54 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID not major targets for either ECCN or GRN. Nevertheless, in the few opportunities made available (such as pilot projects), it was clear there was considerable potential value. Inconsistent and/or incomplete staffing can have a significant negative impact on the ability to meet objectives. It is important to get the right staff in place at the outset and then maintain appropriate staffing throughout the contract. To some extent, both networks suffered from staffing shortages, but GRN dealt with significant challenges due to staff turnover or inability to fill critical positions. This was partly due to contractual issues, but also should be a priority for IPs and USAID. The contract structure was too rigid for the type of work requested. Finding the right contracting mechanism is critical to achieving a project’s objectives. There were many instances in which both networks struggled to complete work within their rigid contract mechanisms, as well as evidence of a lack of action on the part of the COs to rectify it. In addition to a very inflexible labor rate schedule, GRN also was not permitted to subcontract. This made it very difficult for the small organization to deliver on some activities where they did not have expertise, such as in technology development. It is best to have a focused technical content area for a CoP. The EiCC community is small, which helped ECCN to be nimble, responsive, and have a very engaged community. One of the reasons that GRN might have faced more challenges was that they were trying to cover a number of sub-topic areas and regional contexts under reading, as well as a larger membership, which led to more diversity in USAID requests and members’ desires (e.g., the desire for the network to expand into math). It is better to focus on quality rather than quantity. (GRN) There were a lot of specific deliverables in the GRN contract, which led to a real push, especially in the final year to produce deliverables to meet the contract terms. In some cases, USAID staff felt like quality suffered because of the volume of content being produced. A small business was not ideal for accomplishing some of the objectives of the GRN contract. (GRN) It was difficult for a small business to set up a big, international CoP in reading. There was a steep learning curve. URC was not a company that had done things “at this speed and scale before,” it didn’t have offices in many countries, and didn’t have a big roster of international experts in education. This also meant some limitations for GRN, especially in its convening power and its ability to build a sector agenda with other donors. Have clear a technical point of contact at USAID for the contractor to work with. (GRN) The USAID team overseeing GRN wasn’t able to be thoughtful about who would be giving direction to the contractor and how it would be done. This led to confusion for the contractor USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |55 when many different people from USAID were asking for different things. The contractor struggled to be responsive to the varying requests and feedback. 6. RECOMMENDATIONS The following recommendations were drawn from input received from respondents in our data collection as well as derived from analysis of all the data. 1. Distinguish between competing objectives of the networks: USAID-focus or a broader CoP. A major subject discussed by both qualitative and evaluation survey respondents was related to the dual—and sometimes conflicting—roles of the networks, to both serve as a network centered on advancing USAID’s priorities and objectives, and to create a broader, member-centric CoP. (See the first “Lesson Learned” in the previous section). This duality created confusion at times, and may have reduced effectiveness. For future work in this area, USAID may want to put further deliberation into the ultimate goals of the community they want to form. If the goal is to be more consultative with IPs and other donors when producing knowledge products to support the work of USAID in a particular technical area, then a network could be sufficient to accomplish that goal. If the goal is to foster a diverse community where practitioners advance their understanding and improve practice in a technical area, and which could eventually stand on its own or be absorbed into another entity, then a CoP would be more appropriate. 2. Decide on the audience. Concomitant to the decision of whether to serve as a network or a CoP should be a decision on the scope of the effort, namely whether the aim is to (a) engage USAID Missions actively, (b) serve host country governments, and/or (c) serve organizations not directly affiliated with USAID, and whether those efforts should extend to developing materials relevant to each of those audiences. These questions have all been raised by current ECCN and GRN stakeholders, and each have implications for future networks/CoPs, in both design and resourcing. These objectives should be explicit in any future scope of work. 3. Design materials with the appropriate audience in mind. This may require developing different versions or formats of a resource in order to optimize use and application. For example, a shorter how-to guide for Mission staff might be more appropriate than a long guidelines document. Online webcasts that can be viewed on demand would accommodate more field-based staff than a live webinar or in-person event held in Washington, D.C. 4. Facilitate and broaden access to network materials. A significant share of members are field-based (almost half for ECCN, and two-thirds for GRN according to the evaluation survey) and are eager to access and use network resources. The evaluation team recommends that measures be taken to enhance ease of access for international members, including recorded events that can be accessed after live events, alternate means of accessing resources in countries where Internet coverage is not adequate, and timing live events to enable remote attendance by members in different time zones. The addition of regional events would also serve as a way to introduce field-based practitioners to the networks and their materials. 56 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID 5. Strengthen outreach and relevance to broader groups in the field. A large number of respondents asked for more outreach to the field and “localization” of the network, and both USAID staff and the ECCN and GRN STs remarked on the unexpected interest received from the field. These networks have clearly attracted a significant number of members who are not connected directly to USAID and are eager for resources to address their needs. They are joined by USAID stakeholders who also see a need for more “localized” resources, in particular those that are relevant to host country governments and institutions. We recommend that a future network include a specific objective with sufficient funding to focus on outreach and localization (beyond the needs of Missions). If USAID is interested in designing materials relevant for these groups, consider the following steps:  Identify local and/or regional partner organization(s) to serve as “branches” of the network  Create outreach events, products, or tool kits specifically targeting integration of host country government organizations such as the Ministry of Education  Engage local experts and organizations during the development phase – not just the dissemination stage – of new resources  Review materials for terminology, avoiding excessive USAID-centric or USA-specific language or references  Provide Spanish, French, and Arabic versions of some of the most sought-after resources 6. Consider targeting the needs of USAID Missions and seeking their input in the development of resources. It appears that USAID Missions did not benefit as much as they might have. There were different views about whether this was or should be an objective of the networks, but if it is, it should be a clear objective with a distinct strategy, ideally based on a needs assessment of Missions. The strategy might include more customized tools for the specific tasks required of Mission-based staff, opportunities for piloting new approaches, providing professional resources to the field as needed, and providing regional training and networking events. Since the majority of learning takes place in the field, seek input from Mission staff when developing and piloting new tools and resources. 7. Consider measures to enhance sustainability at the outset. Over 90% of members interviewed expressed the wish that the networks continue in some form. Some provided ideas for doing this in a way that would not require as much direct funding from USAID, but this is an area where more focused inquiry would be needed. Ideas offered included the following:  Rotating leadership by major stakeholders  Focus more effort on leveraging partnerships such as those with universities  Provide funding for hosting events  Make specific agreements for inputs with leadership of major actors in the education arena USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |57 8. Design resources to enhance utilization. Both networks produced a plethora of resources that are appreciated, and there is evidence that many of them are being applied. But it is also apparent that the networks could benefit from higher levels of application, as reflected in input from qualitative and evaluation survey respondents. This could be improved by early consideration of how resources should be developed and disseminated for optimal uptake. Involving potential users and/or field staff during the development, as well as adding a phase for piloting and feedback, would both be helpful measures to ensure that products are relevant and “field-ready.” The addition of a position or role focused on knowledge management, in addition to communications, could help improve resource utilization and uptake. 9. Ensure that the contracting mechanism and the contract itself do not create rigidities for an activity that needs to be nimble and able to pivot quickly. Consideration should be given to ensuring that there is room for subcontractors, and that there is sufficient funding for key staff, including senior managers, trainers, and communications and knowledge management specialists. If such rigidities appear to be impeding effective implementation, addressing this should be made a priority for the COR. Depending on objectives, it may also be worth considering mechanisms other than contracts, if primary aims are community buy-in and collaboration. 10. Improve ease of use of network resources. Measures to enhance usability of the networks – especially by professionals who are often very busy and may have connectivity challenges – might include more succinct posts, user-friendly communications and website, and including executive summaries in all documents. Additional recommendations from the evaluation surveys would also be worthwhile to consider. 11. Ensure network leadership can focus on partnership, engagement, and planning rather than just managing a large number of activities. This might include adding another senior management position and ensuring there is a balance in responsibilities between product development and engagement. In addition to including engagement and building partnerships as explicit objectives, scopes of work should not include large numbers of pre-specified products as they may come at the expense of other priorities. 12. When creating “knowledge share groups” or similar sub-groups within a larger CoP or learning network, ensure that there is a clear goal in mind at the outset to ensure better engagement with group members. Having clear goals and expectations for any sub-groups will help to align participants and provide direction to group activities or interactions. Without a clear purpose, it may be difficult to successfully engage members who likely have competing priorities and demands on their time. 58 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID ANNEX A. STATEMENT OF WORK OF THE FINAL PERFORMANCE EVALUATION OF USAID ECCN/GRN I. PURPOSE OF THE EVALUATION The purpose of the evaluation is to conduct a final evaluation of the Education in Crisis and Conflict Network (ECCN) and the Global Reading Network (GRN) to document key outcomes achieved, capture critical lessons and good practices, and provide details of identified effective management and budget oversight methods for use by USAID AORs/CORs; USAID Activity Managers; and the implementing partner(s) who will manage: future USAID Office of Education-funded networks, USAID and implementing partner capacity building efforts, and knowledge management activities. The results may also be useful for members of the current and future networks as well as to both USAID and implementing partner staff who manage USAID Office of Education research activities. II. SUMMARY INFORMATION Strategy/Project/Activity Name Final Evaluation of the Global Reading Network & Education in Crisis & Conflict Network Implementer ECCN - EDC GRN - URC Cooperative Agreement/Contract # ECCN - [GS-10F-0406P/AID-OAA-M-14-00007 GRN - [GS-10F-0182T/AID-OAA-M-14-00001 Total Estimated Ceiling of the Evaluated Project/Activity(TEC) ECCN - $9,967,539 GRN - $9,984,594 Life of Strategy, Project, or Activity ECCN - July 2014 – October 31, 2019 GRN - January 1, 2014 - October 31, 2019 USAID Office E3/ED III. BACKGROUND The ECCN and GRN objectives and results are described below. Contractual documents, such as workplans, quarterly reports, other evaluations, etc. will be provided to the evaluation team. Education in Conflict and Crisis Network The ECCN contract was designed to support technical leadership in education in crisis and conflict environments to USAID by ensuring that education practitioners and those who support them have the latest information, most practical tools, and sufficient resources. Specifically, the Education Development Center (EDC), the ECCN implementing partner, was contracted in July 2014 to work with USAID and partners to develop a community of practice (COP) to cover topics relevant to Goal Three of the 2011-2015 USAID Education Strategy: “Increased equitable access to education in crisis and conflict environments for 15 million learners by 2015,” which was eventually extended until 2018. The purpose of the contract was to procure EDC services to assist USAID/E3/ED in developing the following: USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |59 1. Education in crisis and conflict COP 2. Evidence building and dissemination 3. Guidance, theories of change, metrics, and tools 4. Tools testing and application 5. Training and capacity building of education staff and partners More details about the objectives follow: 1. Develop and Support the Education in Crisis and Conflict Community of Practice: A thriving informal EiCC community of practice existed prior to the procurement of the ECCN contract and was composed of USAID Education Goal Three leadership (DC, Missions); Goal Three implementing partners; and EiCC networks and working groups. Under this objective, EDC was to nurture and grow an intentional USAID-focused sub-community (the USAID Education in Conflict Network) through which members could engage in a rich exchange of knowledge and share it with the larger EiCC community of practice. The community of practice was also intended to draw from the USAID ECCN Support Team’s work in the following objectives 2-5. 2. Build and Disseminate Evidence for the Field: Under this objective, EDC was tasked with synthesizing existing research and engaging in new research, as suggested by USAID, and by the USAID ECCN, in the area of education in conflict and crisis. Research findings were supposed to be disseminated through documents, training and other forums (as supported by the other objectives). 3. Test, Disseminate and Use Guidance, Tools, Metrics, and Theories of Change: Under this objective, EDC was tasked with developing tools and guidance that concentrate on the most effective approaches to increasing equitable access to education in crisis and conflict environments. Specifically, it worked to develop operational guidance and tools; develop and test theories of change, indicators, and log frames; and develop costing methodologies and templates related to Goal Three programming. 4. Apply tools in Crisis and Conflict: Over the last several years, the USAID Office of Education developed tools that aid in better contextualizing and adapting education programming for crisis and conflict-affected environments. Activities under this objective were designed to refine, add to, and implement these tools to promote more conflict-sensitive and context-appropriate education programming in these challenging environments. 5. Develop, Implement and Assess Training: The final objective of the ECCN was to develop, deliver, and assess a range of professional development opportunities designed to build the capacity of USAID, Goal Three implementers, and local and regional institutions to use evidence-based research and resources in project design, implementation, and monitoring. Global Reading Network The GRN contract, also known as Reading within Reach (REACH), was designed to support a rapid increase in the impact and scale of primary grade reading programs as per Goal One of the 2011-2015 USAID Education Strategy (which was eventually extended until 2018): "Improved reading skills for 100 million children in primary grades by 2015.” The contract was awarded to University Research Co (URC) in January 2014 to ensure that policy makers, 60 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID reading practitioners, and those who support them - from global donors to school principals - had the latest information and most practical tools to design evidence-based activities focused on helping students to learn to read in the early grades and to read to learn in the later grades. Under the GRN, URC was tasked with consolidating existing evidence on key reading interventions, supporting a number of operational research activities, and field testing tools for materials development/storage, materials distribution, and data collection. They were also contracted to build and support a USAID-led global research and practitioner Primary Grade Children Reading COP focused on improving reading outcomes in the primary grades. The purpose of this contract was to procure the services of the Contractor to assist USAID/E3/ED with the following: 1. Establishing and supporting a Primary Grade Reading Community of Practice 2. Evidence for the field consolidated, built, and disseminated 3. Innovations trialed, refined, and promoted to the field 4. Evidence-based implementation protocols, metrics, and tools produced and used 5. Training for use of protocols, metrics, and tools developed and rolled out 6. Effective communications in support of high impact, scaled reading programs By the end of the contract, it was expected that standard metrics and operational guidelines would exist for high-impact and scalable core and supplementary interventions and that this knowledge and these innovations would be widely disseminated and used, resulting in the acceleration of the impact, sustainability, and scale of reading programs worldwide. More details about each of the objectives follow: Establish and support a Primary Grade Reading Community of Practice: A globally inclusive Primary Grade Children Reading COP, including practitioners, researchers, funders and others, is essential to build and discuss evidence, collaboratively undertake operational research, and share tools and knowledge about what works to improve reading at a feasible cost and at scale. The goal behind the GRN contract was for USAID to take the lead on developing such a COP, with support from the Global Partnership for Education (GPE), other donors, and partners. The COP was meant to build and share approaches to using evidence for program design and implementation and also provide outreach tools and strategies to keep reading front and center on the global education agenda. Although the COP initially focused on primary grade reading, it was envisioned that USAID or other partners might expand the scope to improving instruction in other subjects, such as mathematics, in later years. Consolidate, build, and disseminate evidence for the field: The GRN contract was also designed to support USAID and the Primary Grade Children Reading COP in consolidating, building, and disseminating evidence for practice. Under the contract, URC was to provide services such as meta-analyses, evidence briefs, needs assessments, research, surveys, monitoring, measurement, and dissemination. USAID envisioned the latter would occur through convening virtual and on-site workshops, conferences, webinars, and training worldwide and at regional or country-specific levels. Studies were meant to address the research-practice gap through identifying barriers to introducing and implementing key USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |61 interventions, developing new approaches to scale and evaluating the cost-effectiveness of interventions and approaches to scale. Under this objective, URC was meant to work with the COR, COP, and other partners to collect and analyze cost data for different approaches and provide that information as part of research findings. Finally, USAID asked URC to set aside a small amount of resources for “just in time” analyses and/or reports in response to USAID or COP queries under this objective. Trial, innovate, refine, and promote innovations to the field: Under this objective, URC was to trial, refine, and promote the use of technologies provided by USAID to achieve Goal 1 of the 2011-2018 USAID Education Strategy. The innovative solutions proposed and implemented should have had the potential to improve the reading of millions of young learners in multiple countries. Produce and use evidence-based implementation protocols, metrics, and tools: Through this objective, URC was to develop, define, and standardize (to the extent possible) metrics and measurement and intervention protocols to guide the core activities of primary grade reading programs, make them more transparent, and make uptake easier for a broad range of partners. The GRN was to develop decision trees (for interventions that may have several evidence-based options, depending on context), easy-to-use operational guidance (based on evidence reviews, academic and operational research, and cost analysis), and metrics for the topics described above (core package, promising practices, options that may emerge over time). This guidance was to be disseminated in hard copy and electronically to the COP and made available to the public on an open-source basis. Provide training for use of protocols, metrics, and tools: This task involved providing professional development to the COP, USAID, and its partners in designing, implementing, monitoring, and evaluating primary grade reading programs. The emphasis was on training and capacity building for organizations that are focused on primary grade reading and could have important country-level, regional, and/or global impact on developing evidence and improving outcomes, thus directly contributing to the achievement of Goal 1 of the 2011-2018 USAID Education Strategy. Provide effective communications in support of high impact, scaled reading programs: Under the GRN, URC was to integrate internal and external communication into every activity as appropriate. USAID expected that all products from the GRN would be open source and disseminated as widely as needed. One aspect of this was to include the development and updating of a project website, with internal and external links and links to other important sites in the reading field. IV. EVALUATION QUESTIONS 1. What activities were most successful, and why? What activities were ultimately unfeasible, and why? (from USAID DC, ECCN/GRN, Mission, and IP perspectives) What modes of delivery made activities successful/unsuccessful? 62 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID 2. What were the unexpected or unanticipated outcomes of ECCN/GRN? What added value or benefit did they provide to USAID and its implementing partners? (from USAID/DC, ECCN/GRN, Mission, and IP perspectives) 3. How did stakeholders use information they received through the ECCN/GRN? What specific design, implementation, monitoring or evaluation approaches or adaptations did the information inform, and what was the impact of this uptake? (from USAID DC, ECCN/GRN, Mission, and IP perspectives) 4. What specific knowledge products did stakeholders find most valuable? Why? (from USAID DC, ECCN/GRN, Mission, and IP perspectives) 5. How were missions and implementing partners engaged? What aspects of the communities of practice did they find most valuable? What would they suggest for the future? (Missions, IPs, and USAID, ECCN/GRN staff) 6. What worked well in terms of management of ECCN/GRN? What did not? What might have made management of the activities easier and more effective? (from USAID and from ECCN/GRN perspectives) 7. What was the cost per product? What in-kind time and effort was required? (e.g. webinar, knowledge product, training) (ECCN/GRN) V. EVALUATION DESIGN AND METHODOLOGY The Contractor should conduct a performance evaluation using data-collection and analysis methods tailored to each of the seven evaluation questions listed above. The Contractor is responsible for proposing the methods that will produce the most rigorous results possible within the timeframe of this activity. Illustrative data collection methods include, but are not limited to:  Key informant interviews with CORs/ACORs, activity or project managers (USAID and ECCN/GRN), ECCN/GRN key personnel, and/or key consultants  A document review of network engagement data; documented stakeholder feedback received through webinars, trainings, engagement surveys, newsletter feedback, website downloads, clicks, etc.; budgets/financial reports; past reports; evaluations; etc.  An electronic survey for members of ECCN/GRN and USAID Missions The Contractor may also consider using methods such as Most Significant Change to identify the most notable contributions of the ECCN and GRN to the global education development community and/or process mapping or tracing to track use of information generated by the networks and the impact of that use. The Evaluation Team should ensure that all evaluation results are triangulated through multiple sources and methods to provide for the most rigorous and unbiased evaluation possible. The evaluation design report should clearly explain how key informants and other sources of information will be sampled in order to assure balanced participation. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |63 VI. DELIVERABLES AND REPORTING REQUIREMENTS 1. Evaluation Design & Workplan: Within 3 weeks of the award of the contract, the evaluation team must submit to the Agreement Officer’s Representative/Contracting Officer’s Representative (AOR/COR) an evaluation design (which will become an annex to the Evaluation report) & draft workplan. The evaluation design will include: (1) a matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (2) draft questionnaires and other data collection instruments or their main features; (3) the proposed selection criteria and sampling approach for document selection, interviewees, surveys, etc.; (4) known limitations to the evaluation design; and (5) a dissemination plan. The evaluation workplan will include: (a) the anticipated schedule and logistical arrangements; and (b) a list of the members of the evaluation team, delineated by roles and responsibilities. USAID offices and relevant stakeholders will be given a minimum of 5 business days to review and consolidate comments through the AOR/COR. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and workplan within 5 days. 2. Two In-briefings: Within 5 days of award, the evaluation team will have an in-briefing with USAID to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, proposed methodologies, the workplan, and/or to adjust the Statement of Work (SOW), if necessary. At the in-briefing, the evaluation team and activity manager will agree on the frequency of check-ins. Prior to the start of the evaluation (late August/early September), the evaluation team will have an in-briefing with relevant USAID, ECCN, and GRN staff for introductions and an explanation of the evaluation. 3. Preliminary findings briefing: The evaluation team is expected to brief relevant ECCN, GRN, and USAID Office of Education staff on preliminary findings, in order to validate findings or validate/collaboratively develop final recommendations that are specific, measurable, achievable, relevant, and time-bound (SMART), likely related to any potential follow-on work. At a minimum, this preliminary findings briefing will be held with USAID. ECCN/GRN participation will depend on the timing of the findings. 4. Final Presentation and PowerPoint: The evaluation team is expected to hold a final presentation in person/by virtual conferencing software to discuss the summary of findings and recommendations to USAID. This presentation will be scheduled as agreed upon during the in-briefing. The PowerPoint deck used to present the final results is also a deliverable. 5. Draft Evaluation Report: The draft evaluation report format should align with USAID evaluation requirements and templates. The report should be no longer than 15 pages, and it should be formatted in such a way that it clearly responds to the purpose of this evaluation, which is to inform the Office of Education’s design and management of the best way to design and manage follow-on activities that may focus on research and learning, communities of practice, knowledge products, and professional development. The report will address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, the Office of Education will have 10 business days in which to review and comment on the initial draft, after which point the AOR/COR will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit a revised final draft report 64 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID ten business days hence, and again the Office of Education will review and send comments on this final draft report within 3 business days of its submission. 6. Final Evaluation Report: The evaluation team will be asked to take no more than 5 business days to respond/incorporate the final comments from the Office of Education. The evaluation team leader will then submit the final report to the AOR/COR. All project data and records will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. VII. EVALUATION TEAM COMPOSITION The evaluator should propose an appropriate evaluation team composition in terms of size and expertise to best address the purpose of this evaluation, the activities being evaluated, and the evaluation questions. USAID encourages evaluation teams to be inclusive and diverse, in terms of gender, race, and disability, among other factors. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s evaluation policies and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. The COR of the Evaluation may observe all or some of the data collection efforts. VIII. EVALUATION SCHEDULE ECCN and GRN are in their final phases of implementation. The evaluation can extend beyond the close of each network’s activities. The evaluation team should consider both the time and availability of staff in these activities in their final quarters as well as in the close-out and post￾close out periods. The evaluation team should propose a schedule as part of the first deliverable (evaluation design and workplan). The evaluation final report should be delivered no later than December 2, 2019. IX. FINAL REPORT FORMAT The evaluation final report should include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. The report should be no longer than 15 pages, and it should be formatted in such a way clearly responds to the purpose of this evaluation, which is to inform the Office of Education’s design and management of follow on activities that may focus on research and learning, communities of practice, knowledge products, and professional development. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |65 The annexes to the report shall include:  The Evaluation SOW;  Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team;  All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides;  All sources of information, properly identified and listed; and  Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of interest. In accordance with ADS 201, the Contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. Draft and final evaluation reports will be evaluated against the ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report. 37 X. OTHER REQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine-readable, non-proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. 66 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID ANNEX B: DATA COLLECTION INSTRUMENTS SECTION 1: WEB SURVEY INSTRUMENT Web Survey Questionnaire Draft - NORC Version (November 5, 2019) You have been identified as an individual who is registered to receive information from Education in Crisis and Conflict Network (ECCN) / the Global Reading Network (GRN). NORC at the University of Chicago, an independent research firm located in the United States, is conducting an evaluation of the ECCN/GRN, and invites you to participate in this Web survey to help us to better understand how the ECCN/GRN supports your work. We encourage you to respond even if you have not used ECCN extensively. Your input will be valuable. The survey is expected to take less than 20 minutes of your time to complete. Your participation is voluntary and is designed to help USAID support work in education. There are no foreseeable risks involved in participating in this survey. Your identity will be kept private and responses will be in summary form only. The results of this survey will be presented to USAID for use in planning their future work. You do not need to respond to all questions, although more complete answers will be more helpful to us. If you have any questions about this survey, please contact Alexander Rigaux at Rigaux￾Alexander@norc.org or Katie Mark at mark-katie@norc.org. [consent] Do you agree to participate in this survey? 1 Yes PROG: SKIP TO SECTION A 0 No [reason_refusal] Would you please explain why you would not like to participate in the survey? 1 __________________________________________________ PROG: UPON ENTRY OF REASON_REFUSAL, CONTINUE TO DISPOSITION PAGE PROG: ALLOW RESPONDENTS TO SKIP QUESTIONS, BUT ADD SOFT VALIDATION MESSAGE ON ALL QUESTIONS: “You have not answered one or more questions on this page. Are you sure you want to proceed without answering?” 1. Which of the following best describes your institutional affiliation: o USAID USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |67 o USAID implementer o Other international donor o Government o University/Academia o Independent (consultant/researcher) o Other 2. In which country are you based ? ______________ 3. Please indicate the regions in which you work most frequently on education projects. (please select all that apply): Afghanistan and Pakistan Africa Asia Europe and Eurasia Latin America and the Caribbean Middle East 4. For how long have you been a member of ECCN/GRN? o Less than one year o 1-2 years o 2-3 years o More than 3 years 5. How often would you say you looked at information from ECCN/GRN in the last year? o Never o 1-4 times in the year o 5-10 times in the year o More than 10 times in the year o Don’t know 68 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID 6. Has your organization’s professional practice in Education in Conflict and Crisis / Education Programming changed as a result of ECCN/GRN’s Work in a way that is clearly documented (e.g., in reports, MEL plans, proposals, work plans, special studies, etc.)? o Yes o No o Don’t know Area 2: Access to and Use of Network Products 7. Have you ever accessed resources or knowledge products from ECCN/GRN? (Y/N) If N skip to Q10. 8. Have you accessed any of the following ECCN/GRN products? (Please select all that apply) ECCN GRN  ECCN Resource Repository  Rapid Education Risk Assessment (RERA) Toolkit  Safer Learning Environments Evidence Gap Maps  Safer Learning Environments Rapid Qualitative Assessment Tool  Safe Learning Environment Indicators (Shortlist w/ PIRS)  Safe Learning Environment Indicators (Extended Spreadsheet)  CSE Indicators  CSE Checklist for ICT in Education  Education Equity Indicators for Access: Guidance for Practitioners in Crisis and Conflict Affected Contexts  Accelerated Education in DRC Report  Alternative Non-State Schools Research and Case Studies  Guide to Accelerated Education Principles (AEWG)  Analysis of indicators used in USAID education projects in crisis and conflict environments  Analysis of Theories of Change in USAID Solicitations for EiCC  Summary brief of Rapid Education and Risk Analysis (RERA) Field Pilot Research  Humanitarian Development Coherence White Paper  Resilience and Education White Paper  Evidence Pathways online tool  Global Book Fund Study  Landscape Report on Early Grade Literacy  EGRA 2.0 Toolkit  Early Grade Reading Program Design and Implementation: Best Practices and Resources for Success (Comprehensive EGR Training Package)  UDL: Universal Design for Learning (UDL) to Help All Children Read: Promoting Literacy for Learners with Disabilities Toolkit  Enabling Writers Workshop Program Guides and Toolkits  Towards the Design and Implementation of Comprehensive Primary Grade Literacy and Numeracy Programs  Coaching in EGR Programs: Evidence, Experiences and Recommendations  Promoting Successful Literacy Acquisition through Structured Pedagogy  Getting Early Grade Reading Right: A case for investing in quality early childhood education programs  Assessment to Inform Instruction: Formative Assessment  Open Licensing of Early Grade Reading Materials: Information and Guidance  Latin America/Caribbean Reading Network Policy Papers USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |69 9. Have you applied any of the following ECCN/GRN products or resources? “Applied” could include instances where concepts, guidance, or research findings have informed programming, actions, or decisions. Check resources that you have applied. [Only those items identified in the previous question appear for this question] The final option is:  I have not applied any of these resources. [if this is checked, skip the next question] 10. In what way or ways have you applied resources or knowledge from ECCN/GRN in your work? (Please select all that apply.) o Integrated into future program/activity design o Theory of change development o Improved implementation of an existing program o Integrated into metrics/monitoring and evaluation plans o Developed publications o Developed training material o Improved my personal and/or my team's understanding for future efforts o Informed policies and/or strategies for promoting the new concepts/ideas/approaches in our organization, program, or community. o other (please specify) o None 11. For ECCN/GRN resources you have accessed but never applied in your work, what are the main reasons why you did not apply the resource or resources? (Please select all that apply.) o Some resources do not lend themselves to practical application o Resources did not offer clear, succinct presentation of key actionable findings and lessons. o Resources were not relevant for my purposes o Resources were duplicative of what I already do o May use in the future, but have not yet had an opportunity to use o Resource is in draft form and we prefer to wait for the final o Other: 12. Filtered for USAID respondents only: To what extent do you think ECCN/GRN contributed to the effectiveness of implementation of: [GRN] Goal 1 of the 2011-2018 USAID Education Strategy: “Improved reading skills for children in primary grades”? [ECCN] Goal 3 of the 2011-2018 USAID Education Strategy “Increased equitable access to education in crisis and conflict environments”? o Contributed a lot o Contributed some o Contributed a little o Doesn’t contribute o Don’t know 13. Have you ever looked at the ECCN/GRN website? 70 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID o Never o One or twice o Several times a year o Often If Never skip to Q15. 14. What do you look at most on the ECCN/GRN website? (Select up to three) o Research findings/reports o Webinars and webinar resources o Reading program resources (GRN only) o EdData datasets/Assessment data (GRN only) o Event information o News and/or newsletter o Blog posts o I don’t consult the website o Other (please specify) 15. How do you most frequently obtain information from ECCN/GRN? (Please select up to three) o I go directly to the website o Emails from ECCN/GRN o Monthly newsletter o Social media posts o Blog posts o Attend in-person events and training o Attend online events/webinars/webcasts o Other (please specify) 16. Do you receive emails from ECCN/GRN? (Y/N) If no, skip to Q20. 17. If Yes, which of the following do you recall seeing in an ECCN / a GRN email? o Feature articles o Spotlight o Resource Repository Updates o Upcoming Events o Past Events o Other: please specify o No specific items. 18. For each item seen, how useful was it? (only show the items selected in previous question – selection matrix) USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |71 o Very useful o Somewhat useful o Not useful o Don’t know. 19. If yes to Q16, How would you describe the number of emails you get? o Too many o The right amount o Too few Events 20. Have you participated in any of the following? (Please select all that apply) GRN:  In-person knowledge sharing events  In-person training  Collaborative research  Webinars (GRN)  Webcasts (ECCN/GRN)  Interest groups on specific topics  Joint development of open source toolkits or guides  On-line training courses  Regional/sub-regional professional networks managed virtually  Large yearly face-to-face network meetings  Other (please specify) ECCN:  In-Person knowledge-sharing events  CIES workshops  CIES panels/presentations  Webcasts  Trainings  Consultations (In person or virtual)  Steering Group Meetings [Items show up with a box to be checked off] 21. Are there any specific events you recall participating in? Y/N 22. If yes, Please list them: 72 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID __________________________________ 23. How useful were these activities? [New list shows only the items checked in previous question, one at a time]  Very useful  Somewhat useful  A little useful  Not useful 24. In what areas have these activities been useful in your work? (mark all that apply)  future program/activity design  Theory of change development  Improved implementation of an existing program  metrics/monitoring and evaluation plans  Developing publications  Developing training material  Improved my personal and/or my team's understanding for future efforts  Informed policies and/or strategies for promoting the new concepts/ideas/approaches in our organization, program, or community.  other (please specify:_____________________________) 25. This question is for only those who checked off Training under Question 20: For training, what modality did you find most valuable?  in-person training  dual in-person and online training (only GRN)  live webinar  recorded webcast (YouTube)  other Engagement 26. Would you consider yourself a member of the ECCN/GRN Community of Practice? o Yes o No o Don’t know [If No/Don’t know, skip to Q22] 27a. Have you participated actively in any of the following Community of Practice activities? Indicate Yes or No for each item; 27b. Where Yes, please identify to what extent it was valuable. (Please select all that apply) [The “valuable” options appear after the respondent ticks off “Yes.”] USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |73 Participated If yes, to what extent was it valuable? Yes No Very valuable Somewhat valuable Not valuable Don’t know Formed new relationships Have shared my challenges with the community for help Shared resources and/or knowledge with the community Learned new approaches & knowledge Used networking opportunities Participated in a working group Other (please specify): 28. How frequently do you engage with ECCN/GRN in each of the following ways? Never Rarely Sometimes Often In person events (including trainings) Webcasts 74 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Never Rarely Sometimes Often Newsletter Website 29. In your view, are there sufficient opportunities available to engage meaningfully with ECCN/GRN? o More than enough o Enough o Not enough o Don’t know 30. How frequently have you yourself actively shared knowledge through ECCN/GRN in the following ways? Often Sometimes Rarely Never Contributed to development of knowledge products (including review, piloting) Participated in working groups or task teams Contributed to a webcast Contributed to a roundtable event Provided training or other capacity building Participated in consultations USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |75 Often Sometimes Rarely Never Other (please specify) 31. Have you experienced or noticed any positive result of ECCN/GRN that you were not expecting? Y/N 32a. Please describe any unexpected positive result(s): ________________________________________________________________ 33. Have you experienced or noticed any negative results of ECCN that you were not expecting? Y/N 33a Please describe any unexpected negative result(s): ________________________________________________________________ 34. Which of the following describe why you wanted to be a member of ECCN/GRN? (Please select all that apply) o Opportunity to interact with USAID and other partners around education o Partner advocacy with USAID o Knowledge sharing o Access to tools and other resources o Capacity building o Other, please specify:___________________ 35. To what extent were your expectations met in each of those areas?  Fully met  Somewhat met  Not met  No answer. 36. For each of the following aspects [only show the items selected in Q34, and which were marked “fully” or “somewhat met” in Q35], to what extent would you say that the discontinuation of a network like this one would negatively affect your work? o Significantly o Somewhat o A little o Not at all o No answer 37. Do you have any recommendations on how the Network might be improved? (open ended) 76 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID ____________________________________________________________________________ ____________________________________________________________________________ 38. Would you like to continue to be part of a network like ECCN/GRN? o Yes, very much o Yes o Maybe o No o Don’t know **End of Survey ------------------------------------------------------------------------------------------------------------------------------- [Closure] Thank you very much! If you have any questions about this survey, please contact Alexander Rigaux at Rigaux-Alexander@norc.org. USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |77 SECTION 2: QUALITATIVE PROTOCOLS Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s 1. What activities were most successful, and why? In your opinion, which ECCN/GRN activities have been most successful? a. What made them successful? b. How do you measure success? x x x x 2. What activities were ultimately unfeasible, and why? In your opinion, which ECCN/GRN activities were unfeasible to implement? a. What were the main challenges to implementatio n? b. What would have helped to overcome these barriers? x x x 3. What were the unexpected or unanticipated outcomes of ECCN/GRN? Have you noticed any positive results of ECCN/GRN that you were not expecting? If so, what were those effects? x x x x x 78 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s Have you noticed any negative results of ECCN/GRN that you were not expecting? If so, what were those effects? x x x x x 4. What added value or benefit, if any, did ECCN/GRN provide to USAID and its implementing partners? In your opinion, what added value or specific benefits has USAID realized from ECCN/GRN? a. More effective implementatio n of goal 1? b. More effective implementatio n of goal 3? x x x What added value or specifc benefits has your organization realized from ECCN/GRN? x x USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |79 Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s What added value or specifc benefits do you think USAID implementers realized from ECCN/GRN? x x 80 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID 5. How did stakeholders use information they received through the ECCN/GRN? To the extent possible, what specific design, implementation, monitoring or evaluation approaches or adaptations did the information inform, and what was the impact of this uptake? Have you ever received information from ECCN/GRN that helped you do your job better or more effectively? a. How has information received through ECCN/GRN helped you with program/activit y design or changed your practice in regards to program/activit y design? What was your task and how did ECCN/GRN help? b. How has information received through ECCN/GRN helped you with implementatio n or changed your practice in regards to implementatio n? What was your task and how did ECCN/GRN help? c. How has x x x USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |81 information received through ECCN/GRN helped you with new tools or changed your practice in regards to tools? What was your task and how did ECCN/GRN help? d. How has information received through ECCN/GRN helped you with metrics, or otherwise assisted you in your M&E? What difference did the help make? e. In what other ways has information received through ECCN/GRN helped you? f. (If did not emerge from earlier questions) Would you say that ECCN/GRN helped improve education outcomes? 82 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s If you have not found information helpful, what made it unhelpful? USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |83 Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s 6. What specific knowledge products did stakeholders find most valuable? Why? Are ECCN/GRN knowledge products valuable to you or not really? a. In what ways are they valuable? (be specific about which products) b. Which specific products have been particularly valuable to you? Give examples. c. Which knowledge products are less valuable? Why? d. How could ECCN/GRN knowledge products be more valuable to you? x x x 84 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s Would you describe ECCN/GRN knowledge products as easy to use or not really? a. In what ways are they easy to use? (identify specific products) b. In what ways are they not? c. How could ECCN/GRN knowledge products be more easy to use for you? x x x USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |85 Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s Would you describe ECCN/GRN knowledge products as relevant to you or not really? a. In what ways are they relevant? (be specific about which products) b. In what ways are they not? c. How could ECCN/GRN knowledge products be more relevant to you? x x x 86 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s 7. What modes of delivery made activities successful/unsuccessf ul? Would you describe ECCN/GRN knowledge products as easy to access or not really? a. In what ways are they easy to access? b. What challenges accessibility? [Probe if not mentioned: Has connectivity or access to technology ever challenged your ability to receive information from ECCN/GRN?] c. What could improve access for you? x x x USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |87 Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s In regards to ECCN/GRN training and capacity building, did you find any particular modality best? [Probe if needed: in￾person training, dual in-person or online training, webinar, youtube, etc.] a. Were there pros and cons to attending in person or online? x x x 88 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s How do you typically receive information from ECCN/GRN? [Probe if needed: Do you go directly to the website, receive email/e￾newsletters, see social media or blogs, go to live or online events, etc.?] a. What is your preferred way to receive information from ECCN/GRN? b. Are there any modes of delivery that you do you not like or do not work for you? Explain. c. What could be improved about the way ECCN/GRN delivers information to you? x x x USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |89 Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s 8. How were missions and implementing partners engaged? What aspects of the communities of practice did they find most valuable? What would they suggest for the future? Would you describe the ECCN/GRN CoP as valuable to you or not really? Why? a. Which aspects have been the most valuable? Be specific. b. What would you change about the CoP to make it more valuable to you? x X X 90 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s How well does ECCN/GRN engage missions and remote members? a. Which activities have been particularly useful in engaging missions and remote members? b. What challenges continue to exist in engaging missions and remote members? c. What could be done to improve engagement? X X X X USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |91 Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s 9. What worked well in terms of management of ECCN/GRN? What did not? What might have made management of the activities easier and more effective? What do you think about project staffing and management? a. Did you have enough staff? At the right levels and in the right positions? b. Were there any major challenges related to program management? c. Did most activities have enough time and budget (generally) to implement as was foreseen? d. Is there anything that would have made management easier or more effective? x x 92 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s Tell us about the relationship between the donor (USAID) and the project staff. a. Did the donor expectations, contract, and real-time implementatio n needs align? b. If no, what aspects were challenging? x x What is your opinion on the effectiveness of the implementing partners in undertaking their activities? a. What were the main challenges that implementing partners had to meet, in your view? x x x USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |93 Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s How does the CoP measure its performance? a. Does the CoP solicit member feedback? If so, how well is it incorporated? x 10. What was the cost per product (e.g, webinar, knowledge product, training), including the costs of labor? What in-kind time and effort was required? In your opinion, which knowledge products are the most cost￾effective to create? x x Has any in￾kind time and effort been required to create any of the network's knowledge products? Who donated their time and effort? How much? Did this work well, or were there challanges? x x Do members incur any costs for participating in the network? What kind of costs? x x 94 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s What are the main lessons learned from implementing ECCN/GRN? x x USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |95 Has the CoP transitioned from “manager owned” to “community owned” or do you plan on facilitating this kind of transition? a. [If already transitioned:] How did you determine that the time was right to do so? How did you make the transition? b. [If not transitioned yet:] How will you know when is the right time? How would you facilitate this kind of transition? c. In what ways do you think the ECCN/GRN will continue beyond the project funding? What concerns do you have about its sustainability? x x x x Which aspects of the project do you think are the most sustainable? x x x x 96 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID Evaluation Questions Protocol Questions Data Sources USAID/ E3/ED (networ k mgmt.) USAID/E3/E D (other) ECCN/GR N ST (IP) General Member s Mission Member s Least sustainable? Reflecting on ECCN/GRN and its impact -- are there any key ways you think a future activity could be improved in order to move forward? x x x x Is there anything else you would like to add to what we discussed? Before we close, do you have any questions for me? x x x x USAID Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) |97 ANNEX C: SOURCES OF INFORMATION TABLE 1: KEY INFORMANT INTERVIEW PARTICIPANTS – USAID AND IPS Name Position/Title Organization/Agency Gender Cornelia Janke Project Director ECCN/EDC F Daniel Levan Protocol and Metrics Specialist M Ash Hartwell Principal Investigator and M&E Specialist M Jim Rogan Research Manager M Gwen Heaner Research Manager F Nina Papadopoulos Contracting Officer’s Representative (COR) - ECCN USAID F Nina Weisenhorn Assistant Contracting Officer’s Representative (ACOR) - ECCN F Jennifer Gerst Project Director GRN F Rebecca Rhodes Contracting Officer's Representative (COR) – GRN & Team Lead for Education and Literacy USAID F Patrick Collins Division Chief for Education Policy Planning and Budget M Nathaniel Haight Learning and Program Specialist M Heather Risley Knowledge Management and Learning Advisor F TABLE 2: KEY INFORMANT INTERVIEW/DISCUSSION PARTICIPANTS – ECCN MEMBERS FGD/KII Description Gender Total Number of Participants M F ECCN, Mission Members (FGD + KII) 1 2 3 ECCN, General Members (KIIs) 1 5 6 ECCN/GRN, USAID DC Members (FGD) 1 3 4 Total number of participants 3 10 13 Breakdown by gender 23% 77% - 98 | Education In Crisis & Conflict Network (ECCN) and The Global Reading Network (GRN) USAID TABLE 3: KEY INFORMANT INTERVIEW/DISCUSSION PARTICIPANTS – GRN MEMBERS FGD/KII Description Gender Total Number of Participants M F GRN, Mission Members (FGD + KII) 1 6 7 GRN, General Members (FGD + KIIs) 1 7 8 ECCN/GRN, USAID DC Members (FGD) 1 3 4 Total number of participants 3 16 19 Breakdown by gender 16% 84% - TABLE 4: LIST OF DOCUMENTS REVIEWED ECCN Documents by type ECCN Evaluation Final Report Evaluation ECCN SOW Project background ECCN Y4 PMEP report final 3_8_2019 Reporting ECCN Year 5 Q1 Quarterly Report - Submitted 4-30-19 Reporting ECCN Year 5 Q2 Quarterly Report - SUBMITTED 7-31-19 Reporting Y3PMEP report FINAL Reporting Year 4_5 Work Plan Project background ECCN Quarterly Y2Q1 Reporting ECCN Quarterly Y2Q2 Reporting ECCN Quarterly Y2Q3 Reporting ECCN Quarterly Y2Q4 Reporting ECCN Quarterly Y3Q1 Reporting ECCN Quarterly Y3Q2 Reporting ECCN Quarterly Y3Q3 Reporting ECCN Quarterly Y3Q4 Reporting ECCN Quarterly Y4Q1 Reporting Guide to Accelerated Education Principle Publication Rapid Education and Risk Analysis Publication Safer Learning Environment Gaps Map Publication 83 GRN Documents by Type Global Reading Network Quarterly report 1 Reporting Global Reading Network Quarterly report 2 Reporting Global Reading Network Quarterly report 3 Reporting Global Reading Network Quarterly report 4 Reporting Global Reading Network Quarterly report 5 Reporting Global Reading Network Quarterly report 6 Reporting Global Reading Network Quarterly report 7 Reporting Global Reading Network Quarterly report 8 Reporting Global Reading Network Quarterly report 9 Reporting Global Reading Network Quarterly report 10 Reporting Global Reading Network Quarterly report 11 Reporting Global Reading Network Quarterly report 12 Reporting Global Reading Network Quarterly report 13 Reporting Global Reading Network Quarterly report 14 Reporting Global Reading Network Quarterly report 15 Reporting Global Reading Network Quarterly report 16 Reporting Global Reading Network Quarterly report 17 Reporting Global Reading Network Quarterly report 18 Reporting Global Reading Network Quarterly report 19 Reporting Global Reading Network Quarterly report 20 Reporting Global Reading Network Quarterly report 21 Reporting Global Reading Network Quarterly report 22 Reporting Building Evidence to Improve Primary Grade Reading Outcomes - AID-OAA-M-14-00001 Executed Project Background Global Reading Network Stocktaking 6_2019 Project Background Website stats – word Metrics Website stats – excel Metrics Website stats - updated disaggregated from project Metrics The Early Grade Reading Assessment Toolkit Publication Landscape Report on Early Grade Literacy Publication Early Grade Reading Program Design and Implementation - Best Practices and Resources for Success (Comprehensive EGR Training Package) Publication Comprehensive Approaches to Learning (Literacy and Numeracy) Publication 100 GRN Documents by Type Universal Design for Learning Publication Enabling Writers Workshop Guides and Toolkits Publication Background reading and academic publications Wenger E, Trayner B, de Laat M (2011) Promoting and assessing value creation in communities and networks: a conceptual framework. http://wenger-trayner.com/wp￾content/uploads/2011/12/11-04- Wenger_Trayner_DeLaat_Value_creation.pdf. Background reading/ Academic publication Flynn-Dapaah K (2003) Ecosystem approaches to human health global community of practice: Report on the design phase consultations, Ottawa: International Development Research Centre. Background reading/ Academic publication Wenger E, McDermott R, and Snyder WM (2002). Cultivating Communities of Practice: A Guide to Managing Knowledge, Boston: Harvard Business School Press. Background reading/ Academic publication McKellar, K.A., et al. (2014). Evaluating Communities of Practice and Knowledge Networks: A Systematic Scoping Review of Evaluation Frameworks. EcoHealth 11, pp. 383–399. Background reading/ Academic publication USAID Learning Lab. https://usaidlearninglab.org/learning￾networks/resources USAID website Cummings, S. and A. van Zee, 2005. Communities of practice and networks: reviewing two perspectives on social learning. KM4D Journal 1(1): 8-22 www.km4dev.org/journal Background reading/ Academic publication If I Build It, Will They Come? Sustaining Active Communities of Practice for Global Health. Online Discussion Forum, Summary Report. February 2011. www.knowledge￾gateway.org/KMWG/communitiesofpractice Background reading/ Academic publication Ghoshal, S. and J. BonTempo. Modern Communities of Practice: Recommendations for Building, Maintaining and Measuring Impact. Health Communication Capacity Collaborative. January 2014. Background reading/ Academic publication Communities of Practice: Knowledge Management Briefs. Produced by the USAID Knowledge-Driven Microenterprise Development (KDMD) project. Sept. 2013. USAID briefs Supporting Communities of Practice: A TOPS Quick Guide to Linking Development Practitioners. Washington, DC. Background reading/ Academic publication 83 ANNEX D: DISCLOSURE OF ANY CONFLICTS OF INTEREST Date 12/06/2019 102 Date 12/06/2019 83 Date 12/06/2019 104 Date 12/06/2019 83 Date 12/06/2019 106 Date 12/06/2019 83 ANNEX E: ECCN AND GRN SURVEY DATA REPORTS ECCN Survey Data Report Produced by NORC at the University of Chicago – 2/5/2020 Source: Evaluation Survey - ECCN 1 – Which of the following best describes your institutional affiliation? # Answer % Count 1 USAID 15% 15 2 USAID Implementer 31% 32 3 Other international donor 6% 6 4 Government 3% 3 5 University/Academia 13% 13 6 Independent (consultant/researcher) 19% 19 7 NGO/International NGO 7% 7 8 Other (specify) 8% 8 Total 100% 103 1_8_TEXT - Other (specify)38 – Refer to Annex 2 – In which country are you currently based? # Answer % Count 1 Home-based 55% 57 2 Field 45% 46 Total 100% 103 3 - Please indicate the regions in which you work most frequently on education projects. (Please select all that apply) # Answer % Count 1 Afghanistan and Pakistan 12% 12 2 Asia 22 % 22 3 Europe and Eurasia 9% 9 4 Latin America and the Caribbean 24% 24 5 Middle East 27% 27 6 Not Applicable 5% 5 7 Africa 62% 62 Total 100% 99 4 - For how long have you been a member of ECCN? 38 All textual responses are provided in the Annex. # Answer % Count 1 Less than one year 13% 13 2 1-2 years 27% 27 3 2-3 years 28% 28 4 More than 3 years 31% 31 Total 100% 99 108 5 - How often would you say you looked at information from ECCN in the last year? # Answer % Count 1 More than 10 times in the year 38% 38 2 5-10 times in the year 36% 36 3 1-4 times in the year 1% 21 4 Never 3% 1 5 Don’t know 21% 3 Total 100% 99 6 - Has your organization’s professional practice in Education in Conflict and Crisis changed as a result of ECCN's work in a way that is clearly documented? (e.g. In reports, MEL plans, proposals, work plans, special studies, etc.) # Answer % Count 1 Yes 46% 44 2 No 30% 29 3 Don't know 23% 22 Total 100% 95 83 7 - Have you ever accessed resources or knowledge products from ECCN? # Answer % Count 1 Yes 91% 86 2 No 9% 9 Total 100% 95 8 - Have you accessed any of the following ECCN products? (Please select all that apply.) # Answer % Count 1 Rapid Education Risk Assessment (RERA) Toolkit 73% 62 2 Resilience and Education White Paper 45% 38 3 ECCN Resource Repository 44% 37 4 Safer Learning Environments Assessment Toolkit 41% 35 5 Analysis of indicators used in USAID education projects in crisis and conflict environments 40% 34 6 Guide to Accelerated Education Principles (AEWG) 36% 31 7 Education Equity Indicators for Access: Guidance for Practitioners in Crisis and Conflict Affected Contexts 29% 25 8 Summary brief of Rapid Education and Risk Analysis (RERA) Field Pilot Research 27% 23 9 Safer Learning Environments Evidence Gap Maps 24% 20 10 Accelerated Education in DRC Report 24% 20 11 CSE Indicators 21% 18 12 Humanitarian Development Coherence White Paper 20% 17 13 Safe Learning Environment Indicators (Shortlist w/ PIRS) 20% 17 14 CSE Checklist for ICT in Education 15% 13 15 Safe Learning Environment Indicators (Extended Spreadsheet) 15% 13 16 Analysis of Theories of Change in USAID Solicitations for EiCC 11% 9 17 Alternative Non-State Schools Research and Case Studies 11% 9 18 Evidence Pathways online tool 8% 7 19 None of these 1% 1 Total 100% 85 11 - Have you applied any of the ECCN products or resources that you accessed? “Applied” could include instances where concepts, guidance, or research findings have informed programming, actions, or decisions. (Please select all that apply.) # Answer % Count 1 Rapid Education Risk Assessment (RERA) Toolkit 41% 34 2 I have not applied any of these resources 28% 23 3 Guide to Accelerated Education Principles (AEWG) 24% 20 4 Safer Learning Environments Assessment Toolkit 19% 16 5 Resilience and Education White Paper 16% 13 6 Education Equity Indicators for Access: Guidance for Practitioners in Crisis and Conflict Affected Contexts 13% 11 7 Accelerated Education in DRC Report 13% 11 8 Safer Learning Environments Evidence Gap Maps 12% 10 9 Summary brief of Rapid Education and Risk Analysis (RERA) Field Pilot Research 12% 10 10 Analysis of indicators used in USAID education projects in crisis and conflict environments 11% 9 11 ECCN Resource Repository 10% 8 12 Humanitarian Development Coherence White Paper 10% 8 110 # Answer % Count 13 CSE Indicators 8% 7 14 Safe Learning Environment Indicators (Shortlist w/ PIRS) 6% 5 15 CSE Checklist for ICT in Education 5% 4 16 Analysis of Theories of Change in USAID Solicitations for EiCC 5% 4 17 Alternative Non-State Schools Research and Case Studies 4% 3 18 Safe Learning Environment Indicators (Extended Spreadsheet) 4% 3 19 Evidence Pathways online tool 4% 3 Total 100% 83 12 - In what ways have you applied resources or knowledge from ECCN in your work? (Please select all that apply.) # Answer % Count 1 Integrated into future program/activity design 56% 33 2 Theory of change development 29% 17 3 Improved implementation of an existing program 46% 27 4 Integrated into metrics/monitoring and evaluation plans 32% 19 5 Developed publications 25% 15 6 Developed training material 36% 21 7 Improved my personal and/or my team's understanding for future efforts 54% 32 8 Informed policies and/or strategies for promoting the new concepts/ideas/approaches in our organization, program, or community. 39% 23 9 Other (please specify) 8% 5 10 None of these 2% 1 Total 100% 59 83 12_7_TEXT - Other (please specify) – See Annex 13 - For ECCN resources you have accessed but never applied in your work, what are the main reasons why you did not apply the resource or resources? (Please select all that apply.) # Answer % Count 1 Some resources do not lend themselves to practical application 12% 10 2 Resources did not offer clear, succinct presentation of key actionable findings and lessons 2% 2 3 Resources were not relevant for my purposes 26% 21 4 Resources were duplicative of what I already do 6% 5 5 May use in the future, but have not yet had an opportunity to use 70% 57 6 Resource is in draft form and we prefer to wait for the final 5% 4 7 Other (please specify) 6% 5 8 None of these 7% 6 9 Not applicable - I applied all ECCN resources I accessed 5% 4 Total 100% 82 13_7_TEXT - Other (please specify) – Refer to Annex 14 - Goal 3 of the 2011-2018 USAID Education Strategy is: “Increased equitable access to education in crisis & conflict environments." To what extent do you think ECCN contributed to the effective implementation of this goal? 15 - Have you ever looked at the ECCN website? # Answer % Count 1 Contributed a lot 54% 7 2 Contributed some 23% 3 3 Contributed a little 8% 1 4 Did not contribute 0% 0 5 Don’t know 15% 2 Total 100% 13 # Answer % Count 1 Often 26% 24 2 Several times a year 46% 42 3 Once or twice 25% 23 112 16 - What do you look at most on the ECCN website? (Please select all that apply.) # Answer % Count 1 Research findings/reports 30% 56 2 Webinars and webinar resources 24% 45 3 Event information 19% 35 4 News and/or newsletter 15% 28 5 Blog posts 7% 13 6 Other (please specify) 3% 5 7 I do not consult the website 1% 2 Total 100% 184 16_6_TEXT - Other (please specify) – Refer to Annex 17 - How do you most frequently obtain information from ECCN? (Please select up to three.) 4 Never 3% 3 Total 92 92 83 # Answer % Count 1 I go directly to the website 34% 31 2 Emails from ECCN 74% 67 3 Monthly newsletter 41% 37 4 Social media posts 3% 3 5 Blog posts 4% 4 6 Attend in-person events and training 12% 11 7 Attend online events/webinars/webcasts 22% 20 8 Other (please specify) 2% 2 9 I do not obtain information from ECCN 2% 2 Total 100% 91 17_8_TEXT - Other (please specify) – Refer to Annex 18 - Do you receive emails from ECCN? # Answer % Count 1 Yes 95% 86 2 No 5% 5 Total 100% 91 19 - Which of the following do you recall seeing in an ECCN email? (Please select all that apply). # Answer % Count 1 Feature articles 51% 44 2 Spotlight 33% 28 3 Resource Repository Updates 51% 44 4 Upcoming Events 86% 74 5 Past Events 15% 13 6 Other (please specify) 3% 3 7 No specific items 3% 3 Total 100% 86 19_6_TEXT - Other (please specify) 20 - For each item you recall seeing in an ECCN email, how useful was it? # Question Very useful Somewhat useful Not useful Don't know Total 1 Feature articles 45% 20 45% 20 5% 2 5% 2 44 2 Spotlight 54% 15 46% 13 0% 0 0% 0 28 3 Resource Repository updates 55% 24 40% 18 2% 1 2% 1 44 4 Upcoming events 58.% 43 34% 25 4% 3 4% 3 74 5 Past events 23% 3 54% 7 8% 1 15% 2 13 6 Text entry 100% 2 0% 0 0% 0 0% 0 2 21 - How would you describe the number of emails you get from ECCN? # Answer % Count 1 Too many 3% 3 2 The right amount 93% 79 3 Too few 3% 3 Total 100% 85 114 22 - Have you participated in any of the following ECCN activities? (Please select all that apply.) # Answer % Count 1 In-Person knowledge-sharing events 32% 29 2 CIES workshops 30% 27 3 CIES panels/presentations 37% 34 4 Webcasts 54% 49 5 Trainings 36% 33 6 Consultations (In person or Virtual) 18% 16 7 Steering Group Meetings 16% 15 8 None of the above 20% 18 9 Other (please specify) 2% 2 Total 100% 91 22_9_TEXT - Other (please specify) – Refer to Annex 23 - Are there any specific ECCN events you recall participating in that were useful in your work? # Answer % Count 1 Yes 48% 43 2 No 52% 47 Total 100% 90 24 - Please list the events - Refer to Annex 25 - How useful were these activities? # Question Very useful Somewhat useful A little useful Not useful Total 1 In-Person knowledge-sharing events 79% 23 17% 5 3% 1 0% 0 29 2 CIES workshops 48% 13 44% 12 7% 2 0% 0 27 3 CIES panels/presentations 50% 17 44% 15 6% 2 0% 0 34 4 Webcasts 53% 26 43% 21 4% 2 0% 0 49 5 Trainings 70% 23 27% 9 3% 1 0% 0 33 6 Consultations (In person or virtual) 56% 9 31% 5 13% 2 0% 0 16 7 Steering group meetings 87% 13 13% 2 0% 0 0% 0 15 8 Text entry 0% 0 100% 1 0% 0 0% 0 1 26 - In what areas have these activities been useful in your work? (Please select all that apply.) # Answer % Count 1 Future program/activity design 59% 43 2 Theory of change development 30% 22 3 Improved implementation of an existing program 48% 35 4 Metrics/monitoring and evaluation plans 33% 24 5 Developing publications 21% 15 6 Developing training material 29% 21 7 Improved my personal and/or my team's understanding for future efforts 70% 51 8 Informed policies and/or strategies for promoting the new concepts/ideas/approaches in our organization, program, or community 47% 34 9 Other (please specify) 3% 2 83 # Answer % Count 1 0 None 0% 0 Total 100% 73 26_9_TEXT - Other (please specify) – Refer to Annex 27 - For training, what modality did you find most valuable? # Answer % Count 1 In-person training 85% 28 2 Live webinar 9% 3 3 Recorded webcast (YouTube) 3% 1 4 Other (please specify) 3% 1 Total 100% 33 27_4_TEXT - Other (please specify) – Refer to Annex 29a - Have you participated actively in any of the following Community of Practice activities? (Please select all that apply.) # Answer % Count 1 Formed new relationships 23% 20 2 Have shared my challenges with the community for help 15% 13 3 Shared resources and/or knowledge with the community 25% 22 4 Learned new approaches & knowledge 42% 37 5 Used networking opportunities 31% 27 6 Participated in a working group 22% 19 7 Other (please specify): 2% 2 8 None of these 40% 35 Total 100% 88 29a_7_TEXT - Other (please specify) – Refer to Annex 29b - For the activities in which you participated actively, to what extent was the activity valuable? # Question Very valuable Somewhat valuable Not valuable Don't know Total 1 Formed new relationships 75% 15 25% 5 0% 0 0% 0 20 2 Have shared my challenges with the community for help 46% 6 54% 7 0% 0 0% 0 13 116 # Question Very valuable Somewhat valuable Not valuable Don't know Total 3 Shared resources and/or knowledge with the community 73% 16 23% 5 0% 0 5% 1 22 4 Learned new approaches & knowledge 70% 26 24% 9 3% 1 3% 1 37 5 Used networking opportunities 63% 17 37% 10 0% 0 0% 0 27 6 Participated in a working group 58% 11 42% 8 0% 0 0% 0 19 7 Text entry 50% 1 50% 1 0% 0 0% 0 2 30 - How frequently do you engage with ECCN in each of the following ways? # Question Often Sometimes Rarely Never Total 1 In person events (including trainings) 8% 7 30 % 24 33% 27 30% 25 83 2 Webcasts/webinars 8% 7 53% 45 26% 22 13% 11 85 3 Newsletter 39% 33 41% 35 11 % 9 9% 8 85 4 Website 27% 23 48% 41 17% 15 8% 7 86 31 - In your view, are there sufficient opportunities available to engage meaningfully with ECCN? 32 - How frequently have you yourself actively shared knowledge through ECCN in the following ways? # Question Often Sometimes Rarely Never Total 1 Contributed to development of knowledge products (including review, piloting) 12% 10 22% 18 20% 16 46% 37 81 2 Participated in working groups or task teams 12% 10 15% 12 20% 16 53% 43 81 3 Contributed to a webcast 6% 5 23% 19 15% 12 56% 45 81 4 Contributed to a roundtable event 4% 3 15% 12 19% 15 63% 51 81 5 Provided training or other capacity building 9% 7 12% 10 15% 12 64% 52 81 # Answer % Count 1 Not enough 26% 23 2 More than enough 12% 11 3 Enough 53% 47 4 Don’t know 9% 8 Total 100% 89 83 # Question Often Sometimes Rarely Never Total 6 Participated in consultations 5% 4 22% 18 16% 13 57% 47 82 7 Other (please specify) 2% 1 0% 0 6% 3 92% 46 50 33 - Have you experienced or noticed any positive result of ECCN that you were not expecting? # Answer % Count 1 Yes 42% 36 2 No 58% 49 Total 100% 85 33a - Please describe any unexpected positive result(s): - Refer to Annex 34 - Have you experienced or noticed any negative results of ECCN that you were not expecting? # Answer % Count 1 Yes 5% 4 2 No 95% 81 Total 100% 85 34a - Please describe any unexpected negative result(s): - Refer to Annex 35 - Which of the following describe why you wanted to be a member of ECCN? (Please select all that apply.) # Answer % Count 1 Opportunity to interact with USAID and other partners around education 74% 64 2 Partner advocacy with USAID 23% 20 3 Knowledge sharing 80% 69 4 Access to tools and other resources 86% 74 5 Personal capacity building 70% 60 6 Other (please specify) 7% 6 7 None of the above 1% 1 Total 100% 86 35_6_TEXT - Other (please specify) – Refer to Annex 36 - To what extent were your expectations met in each of those areas? # Question Fully met Somewhat met Not met No answer Total 1 Opportunity to interact with USAID and other partners around education 42% 27 38% 24 13% 8 8% 5 64 2 Partner advocacy with USAID 45% 9 45% 9 5% 1 5% 1 20 3 Knowledge sharing 43% 30 51% 35 4% 3 1% 1 69 4 Access to tools and other resources 64% 47 34% 25 1% 1 1% 1 74 5 Personal capacity building 50% 30 42% 25 8% 5 0% 0 60 6 Text entry 50% 3 0% 0 50% 3 0% 0 6 37 - For the following aspects, to what extent would you say that the discontinuation of a network like this one would negatively affect your work? # Question Significantly Somewhat A little Not at all No answer Total 1 Opportunity to interact with USAID and other 43% 22 40% 20 4% 2 6% 3 8% 4 51 118 # Question Significantly Somewhat A little Not at all No answer Total partners around education 2 Partner advocacy with USAID 39% 7 39% 7 11% 2 6% 1 6% 1 18 3 Knowledge sharing 42% 27 38% 24 14% 9 5% 3 2% 1 64 4 Access to tools and other resources 41% 29 35% 25 13% 9 7% 5 4% 3 71 5 Personal capacity building 33% 18 45% 25 15% 8 4% 2 4% 2 55 6 Text entry 100% 3 0% 0 0% 0 0% 0 0% 0 3 38 - Do you have any recommendations on how the Network might be improved? – Refer to Annex 39 - Would you like to continue to be part of a network like ECCN? # Answer % Count 1 Yes, very much 54% 46 2 Yes 35% 30 3 No 0% 0 4 Maybe 8% 7 5 Don't know 2% 2 Total 100% 85 83 ANNEX. Text Entry Responses Q 1_8: Which of the following best describes your institutional affiliation? Other (specify) - Text USAID Evaluator DFID Implementer ECCN Member Strategic Convening Organization Member OMEP Nigeria National Secretary & Education Consultant Q12_9: In what ways have you applied resources or knowledge from ECCN in your work? (Please select all that apply.) Other (please specify) - Text used to convey the status of accelerated education programming to research clients. Conducted a RERA conducted a RERA Informed new design Used the evidence map in a synthesis study about evidence maps on adolescents Q 13_7: For ECCN resources you have accessed but never applied in your work, what are the main reasons why you did not apply the resource or resources? (Please select all that apply.) Other (please specify) - Text ECCN could be little more user friendly Resources are variations of what have already been developed elsewhere In my case, I am reviewing various literature and approaches on the issue of crisis education. I am in the phase of documenting and supporting my research and action work on education in crisis in Venezuela. That is why there is still no evidence as to how to use and how to use the information collected from their repositories. But soon the references to these repositories will be made. Note that this survey lists the SLE Qualitative Assessment toolkit - this was an older version of what is now the final consolidated toolkit (SLE Assessment Toolkit) that has both qualitative and quantitative methods. It is this toolkit to which I am referring in answering these questions. Not practical enough Q 16_6: What do you look at most on the ECCN website? (Please select all that apply.) Other (please specify) - Text Tools Technical Resources I would like more information on the role of higher education in crisis care contexts. I would like to get information about the social and educational crisis that is being experienced now in Venezuela. I am investigating these issues and it is very important to start communicating on various platforms not only the diagnosis of the problem in education but also the programs that are already being implemented to address the educational emergency in Venezuela. funding opportunities Evidence maps for my study on maps Q 17_8: How do you most frequently obtain information from ECCN? (Please select up to three.) Other (please specify) - Text via emails forwarded by a colleague 120 Other (please specify) - Text ask ECCN team members Q 19_6: Which of the following do you recall seeing in an ECCN email? (Please select all that apply). Other (please specify) - Text New Resources webinars Q 22_9: Have you participated in any of the following ECCN activities? (Please select all that apply.) Other (please specify) - Text Not directly yet but a colleague does (I started in this position a few months ago) Q 24: Please list the events that were useful in your work Of 39 responses listed: - 11 listed a webcast or webinar as one of the events attended - 4 listed a workshop as one of the events - 10 mentioned being involved in a training, webcast, or webinar related to the RERA toolkit The remaining responses are listed below Please list the events CIES MEXICO WORK SHOP 2018 CIES presentations and panels on research practitioners conducted or evaluation of tools used CSE Training Perspectives on Education and Resilience webcast; Advancing SEL in Policy and Practice webcast Conflict Sensitive Education A 3-day training workshop at speke Uganda November 2018 AEWG launch in DC Oct 2017 RERA & SLE Trainings Divisive Stereotypes webcast, SEL webcast Closing Event, Resilience webcast, Measurement of SEL webcast, RF Merl/What's Good Enough, Refugee/IDP Equity Issues Accelerated education working group meetings, Training Steering Group Meetings, CSE training Safer Learning Environments Training, conflict sensitive education training, many others (CIES, etc) webcast on accelerated education in Mali; AEWG/ECCN training event; other webcasts Accelerated Education Working Group Workshop in Kampala in November 2018 Safe Learning Environment workshops in DC 2017 and Honduras 2018 Training on Education in Conflict and Crisis Accelerated Education Urban Refugee Education Round Table, SEL Policy Round Table, Resilience and Education Webcast, Humanitarian Development Coherence Webcast, Affordable NS schools webcast, DRC Accelerated Education webcast, SEL measurement webcast RERA Training 2019 CSE training and RERA training EiCC training webchat RERA training, SLE training Training in NH on security and risk in schools Webcast of SRGBV tool kit 83 Please list the events ECCN launch Jan. 2015; Conflict Sensitive Education Training co-sponsored with FHI; webcast re the DRC accelerated education program; RERA Training October 2018 Rera training, sel workshop, cies rera panels cies workshop Steering Committee Meetings; Urban Refugee Conference rera training Webcast on September 25th: "Perspectives on Education and Resilience". Webcast on September 12: "Perspectives on Humanitarian Development Coherence in Education". SEL Policy Round Table, Urban Refugee Round Table, SEL related webinars, RERA webinar, Steering Group Meetings Steering Group meetings, CIES events SLE Training, SLE CIES Workshops, multiple webcasts AEWG training in Uganda RERA training Washington DC October 2019 Support weak poor children education. RERA Q 26_9: In what areas have these activities been useful in your work? (Please select all that apply.) Other (please specify) - Text I used learning in the reports I was working on as a consultant. these both areas are for the AEWG training Q 27_4: For training, what modality did you find most valuable? Other (please specify) - Text Live trainings might have been nice but I don't recall there being any - the webinars were more around sharing information, not training folks. And they were called webcasts. Q 29a_7: Have you participated actively in any of the following Community of Practice activities? (Please select all that apply.) Other (please specify): - Text Not yet. I joined the community of Practice this month In the future I will share resources and knowledge with the community. Q 33a: Have you experienced or noticed any positive result of ECCN that you were not expecting? Please describe any unexpected positive result(s): Linkage between qualitative and quantitative indicators to measure education in crisis programs increased credibility when I was able to cite having researched my questions through the ECCN and or INEE sites and literature A wider dissemination of issues regarding alternative education New information so far and potentially new contacts improve standards of our practice; we also were able to spot new ideas on SEL that were never tackled before in our organization Overall understanding of considerations and evidence related to EiCC Opportunity to connect and engage with experts around the world the openness with which colleagues from different organizations are about sharing and collaborating on similar priorities information sharing, updates and other progress reports Guidelines and strategies with documented evidence about practicalities of providing education interventions in conflict context. 122 Strengthening and building networks with EiE community Keeping abreast of education theory and thinking. I think we expected ECCN to be challenged by the competitiveness of USAID contractors. It was much more collaborative than expected and I think we got closer as a community as a result. All the AEWG members were very available their share their accelerated education policies and strategies New relevant and applicable toolkits and more expertise information and research in the sector The need for Conflict and Crisis Zone are met more connected with colleagues in other organizations I learned about risk assessment and resilience indicators. Better awareness of the scope and diversity of the EiE community Organise variety of tools and resources available for wide range usage changemaker engaging youth during the RERA process helped in fostering community buying and created opportunities for community engagement Lots of knowledge I was able to pilot the SRGBV tool kit in uganda context Greater understanding across the community of practice - shared vocabulary, expectations, etc. Change in USAID policy and design One of my protection staff learned to integrate CSE with TPD to do multi sector programming One of the most significant aspects of being a member of the ECCN platform is having the feeling of being part of a huge community of experts around the world and being able to communicate with this community is very important for Venezuelan researchers at this time (November from 2019). The ability of steering group members to put their 'organizational hats' at the door when working together networking/working together with competitors A lot more collaboration and sharing across organizations and individuals than I expected - people giving their time without any clear and immediate benefit to themselves or orgs increased knowledge, culture shift in our organization in how we think about crisis and conflict I did not expect that AEWG training would allow so interesting discussions. It did. Q 34a: Have you experienced or noticed any negative results of ECCN that you were not expecting? Please describe any unexpected negative result(s): some measurements and indicators were difficult to apply for some reasons ECCN is not linked to other related global networks that could greatly enhance one another's work - esp. eg. Global Alliance for Disaster Risk Reduction and Resilience in the Education Sector, Global Coalition to Prevent Attacks on Education, Education Cannot Wait and so forth. At the moment it seems burdensome because it's another network going it on it's own. Instead of networks in relation to a single donor, it would be more impactful to have networks with multi-donor sponsorship I wouldn't call it negative, but a missed opportunity - less engagement with the 'field' where the rubber hits the road. Understand why this was the case, but a follow-on that meaningfully engaged with these folks - directed things to them or for them (and then directed to them) would be great to see We felt we were "obliged" to do a RERA when we wanted to do an education needs and conflict analysis. We did not use the RERA steps and conduct a wider study. The tools that helped us more was the JENA. However, we presented our work like a RERA in all ECCN events. Q 35_6: Which of the following describe why you wanted to be a member of ECCN? (Please select all that apply.) Other (please specify) - Text As part of AEWG Direct access to expert technical support Funding opportunities Application of research to practice We would like to have the platform to present our action and attention projects on the issue of education in crisis in the search for financing. 83 Other (please specify) - Text Valuable to my employer Q 38: Do you have any recommendations on how the Network might be improved? Do you have any recommendations on how the Network might be improved? Diversify the entities that are able to participate in working groups. Currently restricted - I believe - to contractors. Academics and researchers may be able to contribute as well. If possible, twice yearly networking events would be helpful. They are low key, seemingly low cost, and pay large dividends for networking and knowledge sharing in an informal environment. This effort should continue in some fashion. As said, I just joined recently and do not have a good view of the network opportunities yet to be able to make recommendations. But I appreciate the opportunity. Education in Emergency is a new area for me to work. The networks gives me a platform to learn on it. I’m highly grateful to the network. it would be great that ECCN offer MOOC platform for more interactivity with broad spectrum of members Open up such a network to other NGOs that are non-implementing USAID partners, both international and local. There is need for more voices to be included in the network. It appeared to be a bit of an 'insider' group. To engage more local organizations. Very USA focused- majority of members are USAID related or USAID grant holders and materials are specific to USAID, would encourage a more broader approach to get more broader/ global appeal and resources to be more generic and less USAID focused Continue to support the EiCC COP as possible. I hope there is a way for the network to continue. try to keep it going without a break. maintain the Steering Group somehow. Not at this time i propose to further strengthen this program, which is very useful in different sections of the community, and i hope you will implement like these program in Somalia. Have a good day. Thank you The network needs to improve on the delivery of capacity building sessions. Similarly, frequency of sharing reports/research findings especially what is working in the varied conflict contexts. I really hope that a network like ECCN will continue in the future as it helps provide those implementing USAID supported education programs feel like they have a voice with the donor. More succinct posts. Write for busy people and not educators. The challenge is always how to get more local involvement from our projects on the ground, so i think that's the next step to improving the network. Keep on with the information sharing through the Newsletter and organizing AEWG workshop to share best practices no Yes. Accessing should be more easier friendly. I had problems accessing because I forgot my password and it was a nightmare to reconnect. Do more webinar. Keep up the site and resources. Create local/country group Create awareness of availability of materials None In person capacity to building should be fixed ahead of time and Participants should be assisted to get visa to attend. Last time I was supposed to attend RERA training I missed the opportunity , because of Visa even when I had paid $320 for expedited visa as requeste the timing was too short so I got a date for the interview after the training which was not useful any more. keep it going It would be great if USAID found a way to keep the ECCN around in its current form or build something better without a gap/loss of the events, services, resources, website. ECCN is a wonderful platform. It would be great if all the resources and learning including recordings of the webinars are available even if ECCN is closed down. If possible, this platform should be run by USAID partners. Tbank you for the opportunity you are given to us. 124 Do you have any recommendations on how the Network might be improved? Linkages to other networks. Esp. Global Alliance for Disaster Risk Reduction & Resilience in the Education Sector. Linkages to donors with similar objectives. Explicit "collective impact" approach. More inclusive process so that we aren't torn between the tools of many different donors (even if you think USAID's are the best - which they often are...). We are tearing education authorities and INGO stakeholders in too many different directions, so that all their time is spent jumping through hoops. Let's work on putting collective shoulder to the wheel and achieving some consensus, so that we put forward our best evidence-based approaches, with one voice. No The should be regular opportunities for members More webinars for remotely located people to improve their implementation Make more network-oriented. Facilitate exchanges among partners. Much of the interactions are through the organizers or the working groups that the organizers set up. I'd be interested in hearing more about what others are doing in their work, what they're learning, what worked and why, what worked and why not, etc. There also needs to be a more grassroots approach to developing an advocacy agenda. I know that's hard since USAID is both an advocate and an entity advocated to, but the benefit of a network like this is the convening of ideas. If USAID wants to use this as a platform to push partners closer to their own approaches, the network won't be successful because partners will just use it to take direction on how to get USAID funds. But if they use it as an opportunity and platform for partners to learn from each other and collaborate, people will be interested in that because it'll help them in their work. I also think ECCN needs more voices from country and local levels. It's all very global-level heavy. None. USAID wastes too much time working with wasteful INGOs that have mixed added value in this space. Perhaps it could promote very specific events framed in the contexts of crisis education in Latin America and the Caribbean. Continue as is, and grow it from there. ECCN was more than just a resource for EiE professionals, it was collaborative problem solving between multiple actors in the field, especially between actors who compete with each other for USAID contracts. The collective skill and wisdom was unprecedented in my professional career and benefited us all. I give a very large amount of credit to the ECCN staff at EDC for running an excellent ship, and adeptly changing course as the needs of USAID emerged through new policies and initiatives. I hope the network will continue as it has multifaceted benefits for USAID and its partners and directly informs current USAID policy directives. Keep the momentum it was gaining, continue the model, reach out to the 'field' more often (perhaps have support team members based abroad to represent ECCN at the regional level. No I would suggest to : - Organize more and more activities intended to beneficiary countries people (NGO and Ministry) instead of "experts' events". - Make resources development more collaborative. Collaboration is seek only for piloting tools, after experts designed them. Local teams should be involved earlier. - be careful to the fact that "support" provided by the funder can be - or can be perceived as - an obligation. 1. Ensure even more explicitly that ECCN is not exclusively for USAID partners 2. More outreach to the field, for instance with more regional/national events focused on building capacity of field practitioners 3. Explore additional channels to build capacity, beyond the either face to face training or webinar (e.g. bite-sized learning opportunities) 4. Checklists, guidelines, etc: field practitioners do not have the time to figure out how to use them even when they are designed as user-friendly self-explanatory material. So explore all possible options to introduce the material, deliver training and follow ups. Really the network is very active and I say good job. Still new in the group and by the time learn more about the work of ECCN, I will be in a position to give suggestions on how the network can be improved. Would be great if the evidence gap maps also had an alternate offline version (e.g. excel) and also a way to view the full bibliography of the studies featured in the maps. While the interactive map is likely the common way people access the studies, it would be good if all the studies could alternatively be viewed all at one time. This appears to be more standard now on newer EGMs (e.g. 3ie) 83 GRN Survey Data Report Produced by NORC at the University of Chicago – 2/5/2020 Source: Evaluation Survey - GRN 1 – Which of the following best describes your institutional affiliation? # Answer % Count 1 USAID 10.4% 39 2 USAID Implementer 25.3% 95 3 Other international donor 10.9% 41 4 Government 5.6% 21 5 University/Academia 12.3% 46 6 Independent (consultant/researcher) 16.0% 60 7 NGO/International NGO 11.7% 29 8 Other (specify) 7.7% 44 Total 100% 375 1_8_TEXT - Other (specify)39 – Refer to Annex 2 – In which country are you currently based? 3 - Please indicate the regions in which you work most frequently on education projects. (Please select all that apply.) # Answer % Count 1 Afghanistan and Pakistan 4.8% 18 2 Asia 28.3% 105 3 Europe and Eurasia 3.5% 13 4 Latin America and the Caribbean 17.3% 64 5 Middle East 11.3% 42 6 Not Applicable 5.1% 19 7 Africa 61.0% 226 Total 100% 370 4 - For how long have you been a member of GRN? 39 All textual responses are provided in the Annex. # Answer % Count 1 Home-based 30.4% 114 2 Field 69.6% 261 Total 100% 375 # Answer % Count 1 Less than one year 11.9% 43 2 1-2 years 24.7% 89 3 2-3 years 28.1% 101 4 More than 3 years 35.1% 126 Total 100% 359 126 5 - How often would you say you looked at information from GRN in the last year? # Answer % Count 1 More than 10 times in the year 29.0% 142 2 5-10 times in the year 23.2% 106 3 1-4 times in the year 3.5% 85 4 Never 38.9% 13 5 Don’t know 5.2% 19 Total 100% 365 6 - Has your organization’s professional practice in reading programming changed as a result of GRN's work in a way that is clearly documented? (e.g. In reports, MEL plans, proposals, work plans, special studies, etc.) 7 - Have you ever accessed resources or knowledge products from GRN? # Answer % Count 1 Yes 81.1% 291 2 No 18.9% 68 Total 100% 359 8 - Have you accessed any of the following GRN products? (Please select all that apply.) # Answer % Count 1 Yes 44.4% 160 2 No 26.2% 94 3 Don't know 29.4% 106 Total 100% 360 83 # Answer % Count 1 Landscape Report on Early Grade Literacy 51.5% 147 2 Early Grade Reading Program Design and Implementation: Best Practices and Resources for Success (Comprehensive EGR Training Package) 50.8% 145 3 EGRA 2.0 Toolkit 49.8% 142 4 Enabling Writers Workshop Program Guides and Toolkits 37.8% 108 5 Towards the Design and Implementation of Comprehensive Primary Grade Literacy and Numeracy Programs 35.4% 101 6 Getting Early Grade Reading Right: A case for investing in quality early childhood education programs 32.2% 92 7 Coaching in EGR Programs: Evidence, Experiences and Recommendations 30.5% 87 8 Assessment to Inform Instruction: Formative Assessment 28.7% 82 9 UDL: Universal Design for Learning (UDL) to Help All Children Read: Promoting Literacy for Learners with Disabilities Toolkit 27.7% 79 10 Open Licensing of Early Grade Reading Materials: Information and Guidance 25.9% 74 11 Promoting Successful Literacy Acquisition through Structured Pedagogy 22.8% 65 12 Global Book Fund Study 18.6% 53 13 Latin America/Caribbean Reading Network Policy Papers 7.7% 22 14 None of these 2.4% 7 Total 100% 285 11 - Have you applied any of the GRN products or resources that you accessed? “Applied” could include instances where concepts, guidance, or research findings have informed programming, actions, or decisions. (Please select all that apply.) # Answer % Count 1 EGRA 2.0 Toolkit 36.8% 102 2 Landscape Report on Early Grade Literacy 24.5% 68 3 Early Grade Reading Program Design and Implementation: Best Practices and Resources for Success (Comprehensive EGR Training Package) 23.4% 65 4 I have not applied any of these resources 19.8% 55 5 Enabling Writers Workshop Program Guides and Toolkits 19.1% 53 6 UDL: Universal Design for Learning (UDL) to Help All Children Read: Promoting Literacy for Learners with Disabilities Toolkit 18.0% 50 7 Coaching in EGR Programs: Evidence, Experiences and Recommendations 17.6% 49 8 Getting Early Grade Reading Right: A case for investing in quality early childhood education programs 14.8% 41 9 Towards the Design and Implementation of Comprehensive Primary Grade Literacy and Numeracy Programs 14.4% 40 10 Assessment to Inform Instruction: Formative Assessment 13.0% 36 11 Open Licensing of Early Grade Reading Materials: Information and Guidance 11.5% 32 12 Promoting Successful Literacy Acquisition through Structured Pedagogy 10.4% 29 13 Global Book Fund Study 6.1% 17 14 Latin America/Caribbean Reading Network Policy Papers 3.9% 11 Total 100% 277 12 - In what ways have you applied resources or knowledge from GRN in your work? (Please select all that apply.) 128 # Answer % Count 1 Integrated into future program/activity design 43.8% 97 2 Theory of change development 23.9% 53 3 Improved implementation of an existing program 58.3% 129 4 Integrated into metrics/monitoring and evaluation plans 23.9% 53 5 Developed publications 13.5% 30 6 Developed training material 42% 93 7 Improved my personal and/or my team's understanding for future efforts 65.6% 145 8 Informed policies and/or strategies for promoting the new concepts/ideas/approaches in our organization, program, or community. 37.5% 83 9 Other (please specify) 6.3% 14 10 None of these 0.4% 1 Total 100% 221 12_9_TEXT - Other (please specify) – Refer to Annex 13 - For GRN resources you have accessed but never applied in your work, what are the main reasons why you did not apply the resource or resources? (Please select all that apply.) # Answer % Count 1 Some resources do not lend themselves to practical application 12.1% 33 83 # Answer % Count 2 Resources did not offer clear, succinct presentation of key actionable findings and lessons 3.6% 10 3 Resources were not relevant for my purposes 23.9% 65 4 Resources were duplicative of what I already do 21.0% 57 5 May use in the future, but have not yet had an opportunity to use 59.0% 160 6 Resource is in draft form and we prefer to wait for the final 1.0% 3 7 Other (please specify) 6.6% 18 8 None of these 4.8% 13 9 Not applicable - I applied all GRN resources I accessed 8.4% 23 Total 100% 271 13_7_TEXT - Other (please specify) – Refer to Annex 14 - Goal 1 of the 2011-2018 USAID Education Strategy is: “Improved reading skills for children in primary grades." To what extent do you think GRN contributed to the effective implementation of this goal? 15 - Have you ever looked at the GRN website? # Answer % Count 1 Contributed a lot 52.6% 20 2 Contributed some 21% 8 3 Contributed a little 5.2% 2 4 Did not contribute 0% 0 5 Don’t know 21.0% 8 Total 100% 38 # Answer % Count 1 Often 22.9% 80 2 Several times a year 37.0% 129 3 Once or twice 32.4% 113 4 Never 7.4% 26 Total 100% 348 130 16 - What do you look at most on the GRN website? (Please select all that apply.) # Answer % Count 1 Research findings/reports 58.2% 187 2 Webinars and webinar resources 52.0% 167 3 Reading program resources 49.8% 160 4 EdData datasets/Assessment data 13.7% 44 5 Event information 21.1% 68 6 News and/or newsletter 19.3% 62 7 Blog posts 5.6% 18 8 Other (please specify) 0.3% 1 9 I do not consult the website 2.8% 9 Total 100% 321 16_8_TEXT - Other (please specify) – Refer to Annex 83 17 - How do you most frequently obtain information from GRN? (Please select up to three.) # Answer % Count 1 I go directly to the website 31.7% 110 2 Emails from GRN 71.7% 249 3 Monthly newsletter 35.4% 123 4 Social media posts 3.4% 12 5 Blog posts 2.8% 10 6 Attend in-person events and training 6.6% 23 7 Attend online events/webinars/webcasts 23.6% 82 8 Other (please specify) 2.3% 8 9 I do not obtain information from GRN 3.1% 11 Total 100% 347 17_8_TEXT - Other (please specify) 18 - Do you receive emails from GRN? # Answer % Count 1 Yes 93.9% 326 2 No 6.1% 21 Total 100% 347 19 - Which of the following do you recall seeing in a GRN email? (Please select all that apply). # Answer % Count 1 Feature articles 50.3% 164 2 Spotlight 18.7% 61 3 Resource Repository Updates 40.4% 132 4 Upcoming Events 80.6% 263 5 Past Events 19.6% 64 6 Other (please specify) 1.5% 5 7 No specific items 4.2% 14 Total 100% 326 19_6_TEXT - Other (please specify) – Refer to Annex 20 - For each item you recall seeing in a GRN email, how useful was it? # Question Very useful Somewhat useful Not useful Don't know Total 1 Feature articles 52.7% 86 45.4% 74 1.8% 3 0.0% 0 163 2 Spotlight 40.9% 25 59% 36 0.0% 0 0.0% 0 61 3 Resource Repository updates 69.2% 90 28.4% 37 0.0% 0 2.3% 3 130 4 Upcoming events 54.4% 141 40.5% 105 5.0% 13 0.0% 0 259 5 Past events 43.7% 28 51.5% 33 3.1% 2 1.5% 1 64 6 Text entry 80.0% 4 0.0% 0 0.00% 0 20.0% 1 5 21 - How would you describe the number of emails you get from GRN? # Answer % Count 1 Too many 9.5% 31 2 The right amount 83.2% 270 132 # Answer % Count 3 Too few 7.3% 24 Total 100% 325 22 - Have you participated in any of the following GRN activities? (Please select all that apply.) # Answer % Count 1 In-Person knowledge-sharing events 20.9% 72 2 In-person training 16.3% 56 3 Collaborative research 4.6% 16 4 Webinars 58.6% 201 5 Interest groups on specific topics 15.4% 53 6 Joint development of open-source toolkits or guides 9.9% 34 7 On-line training courses 9.3% 32 8 Regional/sub-regional professional networks managed virtually 1.7% 6 9 Large yearly face-to-face network meetings 8.1% 28 10 Other (please specify) 1.4% 5 11 None of the above 24.2% 83 Total 100% 343 22_9_TEXT - Other (please specify) 23 - Are there any specific GRN events you recall participating in that were useful in your work? # Answer % Count 1 Yes 42.7% 145 2 No 57.2% 194 Total 100% 339 24 - Please list the events – Refer to Annex 25 - How useful were these activities? # Question Very useful Somewhat useful A little useful Not useful Total 1 In-Person knowledge-sharing events 68% 49 23.6% 17 6.9% 5 1.3% 1 72 2 In-person training 75% 42 21.4% 12 1.7% 1 1.7% 1 56 3 Collaborative research 68.7% 11 31.2% 5 0% 0 0% 0 16 4 Webinars 63.1% 127 31.3% 63 3.9% 8 1.4% 3 201 5 Interest groups on specific topics 56.6% 30 39.6% 21 3.7% 2 0% 0 53 6 Joint development of open￾source toolkits or guides 55.8% 19 41.1% 14 2.9% 1 0% 0 34 7 Online training courses 59.3% 19 34.3% 11 3.1% 1 3.1% 1 32 8 Regional/sub-regional professional networks managed virtually 66.6% 4 33.3% 2 0% 0 0% 0 6 9 Large yearly face-to-face network meetings 57.1% 16 39.2% 11 3.5% 1 0% 0 28 10 Text entry 66.6% 2 33.3% 1 0% 0 0% 0 3 26 - In what areas have these activities been useful in your work? (Please select all that apply.) 83 # Answer % Count 1 Future program/activity design 51.7% 132 2 Theory of change development 22.3% 57 3 Improved implementation of an existing program 48.2% 123 4 Metrics/monitoring and evaluation plans 22.7% 58 5 Developing publications 16.4% 42 6 Developing training material 32.5% 83 7 Improved my personal and/or my team's understanding for future efforts 57.2% 146 8 Informed policies and/or strategies for promoting the new concepts/ideas/approaches in our organization, program, or community 27.0% 69 9 Other (please specify) 1.9% 5 10 None 3.1% 8 Total 100% 255 26_9_TEXT - Other (please specify) – Refer to Annex # Answer % Count 1 Yes 42.7% 145 2 No 57.2% 194 Total 100% 339 27 - For training, what modality did you find most valuable? 27_5_TEXT - Other (please specify) – Refer to Annex 29a - Have you participated actively in any of the following Community of Practice activities? (Please select all that apply.) # Answer % Count 1 Learned new approaches & knowledge 41.1% 138 2 None of these 36.4% 122 3 Shared resources and/or knowledge with the community 27.7% 93 4 Used networking opportunities 21.7% 73 # Answer % Count 1 In-person training 25.7% 57 2 Dual in-person and online training 14.9% 33 3 Live webinar 39.8% 88 4 Recorded webcast (YouTube) 14.9% 33 5 Other (please specify) 4.5% 10 Total 100% 221 134 # Answer % Count 5 Formed relationships 20.3% 68 6 Participated in a working group 17.3% 58 7 Have shared my challenges with the community for help 13.1% 44 8 Other (please specify): 1.4% 5 Total 100% 335 29a_7_TEXT - Other (please specify) – Refer to Annex 29b - For the activities in which you participated actively, to what extent was the activity valuable? # Answer % Count 1 Very valuable 68.6% 46 2 Somewhat valuable 31.4% 21 3 Not valuable 0.0% 0 4 Don't know 0.0% 0 Total 100% 67 30 - How frequently do you engage with GRN in each of the following ways? # Question Often Sometimes Rarely Never Total 1 In person events (including trainings) 5.4% 16 20.6% 61 24.0% 71 49.8% 147 295 2 Webcasts/webinars 14.8% 45 38.6% 117 27.7% 84 18.8% 57 303 3 Newsletter 38.4% 117 39.4% 120 12.8% 39 9.2% 28 304 4 Website 22.0% 65 44.0% 130 23.0% 68 10.8% 32 295 31 - In your view, are there sufficient opportunities available to engage meaningfully with GRN? 32 - How frequently have you yourself actively shared knowledge through GRN in the following ways? # Question Often Sometimes Rarely Never Total 1 Contributed to development of knowledge products 11.7% 36 19.8% 61 17.5% 54 50.8% 156 307 # Answer % Count 1 More than enough 14.8% 50 2 Enough 52.0% 175 3 Not enough 21.7% 73 4 Don’t know 11.3% 38 Total 100% 336 83 # Question Often Sometimes Rarely Never Total (including review, piloting) 2 Participated in working groups or task teams 6.6% 20 22.9% 69 14.6% 44 55.8% 168 301 3 Contributed to a webcast 4.3% 13 14.2% 43 14.9% 45 66.5% 201 302 4 Provided training or other capacity building 5.6% 17 15.3% 46 13.3% 40 65.6% 197 300 5 Participated in consultations 4.5% 14 20.2% 62 17.2% 53 57.9% 178 307 6 Other (please specify) 4.5% 9 9.6% 19 7.6% 15 78.1% 154 197 33 - Have you experienced or noticed any positive result of GRN that you were not expecting? # Answer % Count 1 Yes 41.0% 133 2 No 58.9% 191 Total 100% 324 33a - Please describe any unexpected positive result(s): - Refer to Annex 34 - Have you experienced or noticed any negative results of GRN that you were not expecting? # Answer % Count 1 Yes 4.6% 15 2 No 95.3% 310 Total 100% 325 34a - Please describe any unexpected negative result(s): - Refer to Annex 35 - Which of the following describe why you wanted to be a member of GRN? (Please select all that apply.) # Answer % Count 1 Opportunity to interact with USAID and other partners around education 64% 210 2 Partner advocacy with USAID 26.5% 87 3 Knowledge sharing 71.6% 235 4 Access to tools and other resources 75.3% 247 5 Personal capacity building 70.7% 232 6 Other (please specify) 2.1% 7 7 None of the above 2.4% 8 Total 100% 328 35_6_TEXT - Other (please specify) – Refer to Annex 36 - To what extent were your expectations met in each of those areas? # Question Fully met Somewhat met Not met No answer Total 1 Opportunity to interact with USAID and other partners around education 19.8% 41 57.9% 120 18.3% 38 3.8% 8 207 2 Partner advocacy with USAID 22.0% 19 40.7% 35 23.2% 20 13.9% 12 86 3 Knowledge sharing 37.0% 86 50.4% 117 9.0% 21 3.4% 8 232 136 # Question Fully met Somewhat met Not met No answer Total 4 Access to tools and other resources 46.0% 112 46.5% 113 4.9% 12 2.4% 6 243 5 Personal capacity building 38.7% 89 51.3% 118 7.8% 18 2.1% 5 230 6 Text entry 33.3% 2 33.3% 2 33.3% 2 0% 0 6 37 - For the following aspects, to what extent would you say that the discontinuation of a network like this one would negatively affect your work? # Question Significantly Somewhat A little Not at all No answer Total 1 Opportunity to interact with USAID and other partners around education 45.6% 73 31.8% 51 14.3% 23 6.2% 10 1.8% 3 160 2 Partner advocacy with USAID 44.4% 24 35.1% 19 9.2% 5 9.2% 5 1.8% 1 54 3 Knowledge sharing 51.2% 104 29.5% 60 13.7% 28 4.4% 9 0.9% 2 203 4 Access to tools and other resources 53.1% 119 29% 65 13.3% 30 3.5% 8 0.8% 2 224 5 Personal capacity building 45.6% 94 34.9% 72 15% 31 3.4% 7 0.9% 2 206 6 Text entry 25% 1 50% 2 0% 0 25% 1 0% 0 4 38 - Do you have any recommendations on how the Network might be improved? – Refer to Annex 39 - Would you like to continue to be part of a network like GRN? # Answer % Count 1 Yes, very much 56.5% 186 2 Yes 35.2% 116 3 Maybe 6.0% 20 4 No 0.6% 2 5 Don't know 1.5% 5 Total 100% 329 83 Annex – Text Entry Options Q 1_7: Which of the following best describes your institutional affiliation? Other (specify) - Text International Organization Consultant Firm I coordinate a reading campaign from Civic Society that works independently from the government. We have been working since 2009, distributing books, and evaluating fluency and comprehension in the schools that are part of this effort. RTI & Data Coordinator for a private Mexican school district Intergovernmental agency Education and technology Company Publisher REACH staff Content distributor regional network Autonomous educational organisation I am independent social worker focused on the early childhood areas in dates Salaam. It’s been 10 yrs .this yr I started mentoring 30 local teachers on the need to nurture the child n how play n learn used at early stages of life Formerly contractor to USAID currently consultant Subscriber I worked with USAID implementer organization in projects but now I am independent/ consultant. social enterprise subcontracting to USAID programs A mother tongue language commission World Education Inc. Other international donor implementer Q 12_9: In what ways have you applied resources or knowledge from GRN in your work? (Please select all that apply.) Other (please specify) - Text Passed on these resources to international organizational partners we work with in education and development Would like to use it for all of the above. Since I an independent researcher, I have used these resources and the overall work of GRN to refine my own research conceptually as well as design-wise. Used them for my own continuous professional development. Addressed Open Licensing issues written criticism shared them with others a reference Programme Assessment Sharing with staff to enhance their understanding of education and early grade reading proposal development Used it as guidance for call for papers for the 11th Pan African Literacy Conference in Kampala in August, 2109 feedback on what we are doing correctly improve research questions/approaches Q 13_7: For GRN resources you have accessed but never applied in your work, what are the main reasons why you did not apply the resource or resources? (Please select all that apply.) 138 Other (please specify) - Text were for sharing purposes There was nothing useful in it. The research cited by GRN uses antiquated ideas. GRN has been firmly against the perceptual learning and neurocognitive reading research. The Early Grade Assessment tool Since my focus is M&E I directly apply the EGRA toolkit. My technical team colleagues may have applied other resources, Resources were used to inform our own guidelines and guidelines developed by the government. Resources provided by USAID to government and distributed to schools No time Need contextualization to the country I work in Instructional practices do not promote 21st century learning (e.g., higher-level thinking, problem solving, process writing, self-directed learning). In fact I liked your program and found it very beneficial to primary teachers and learners in Egyptian primary schools where teaching and learning Arabic (mother tongue & language of instruction) faces many problems & challenges which resulted badly in learning other subjects such as math, science, social studies, etc. As a freelance education consultant, I always conduct follow-up researches and new practices to be able to develop training material. I hope there could be an opportunity for me to apply some of your best practices and my knowledge and experience as a learner of linguistics to train Egyptian primary teachers. I don't know where to get the resources I applied the concept and matrices I am a consultant working to implement in organisations other than my own Didn't access the resources Buy-in from other organization members Q 16_8: What do you look at most on the GRN website? (Please select all that apply.) Other (please specify) - Text Jobs Q 17_8: How do you most frequently obtain information from GRN? (Please select up to three.) Other (please specify) - Text ATTENDED TRAINING my colleagues worked for GRN Google search Through my organisation and other partners office notifies Most GRN information is of very limited utility. The people it brings in simply do not know enough to bring about changes in poor countries. They talk English-based generalities about reading and math Q 19_6: Which of the following do you recall seeing in a GRN email? (Please select all that apply). Other (please specify) - Text Webinars I saw on the email but I didn't respond and didn't recognize that it is a reading program but reading is a very useful/important for the early reading in early years learning invitation to participate in webinar GRN webber work shops Webinars Q 22_9: Have you participated in any of the following GRN activities? (Please select all that apply.) 83 Other (please specify) - Text Advisory board, author of Global Reading Network Critical Topics Series An international planning event in DC online dissemination Registered in webinar a few times but never able to attend Q 24: Please list the events that were useful in your work Of 130 responses listed: - 52 mentioned webinars as one of the events attended - 13 mentioned workshops as being useful - 20 listed the EGRA toolkit and/or Program Design Please list the events: EGR Program Design and Implementation; Enabling Writers Projects - 2 sessions on Lessons Learned Universal Design Principles Policy Linking Workshops and Early Grade Reading Training Strategy, Policy and guidelines and manual Upper Grade reading; publishing Enabling Writers Training Workshops Primary education Benchmarks Reading Materials development I attended many of the seminars in the first year. Then, I attended the International Literacy Day event at the Peace Institute. I also attended a couple of webinars and networked with Allison and others from USAID that I met there, not to mention other implementers. I attended the Landscape report launch via webinar too. And I attended the closing event for GRN. Early Grade Reading Program evaluation in Kenya rural areas Printing specs and paper selection Tangerine training Education in conflict and crisis, EGR Courses Publishers Training by USAID in Nairobi interest group zoom meetings, reading comprehension training Inclusive education Open licensing of early grade learners materials in Nairobi, Kenya Publisher's training in Kenya (October) I do not recall that EW program Policy Linking (Social Moderation) workshop, GRN/BEC consultations on new USAID strategy; input via BEC to Literacy Landscape toolkit; input to Classroom observation toolkit; final event UDL Toolkit training Open Licensing, Disabilities Toolkit Open Licensing UDL literacy for all disabilities tool kit training g Investigations and experiences in other countries UDL Toolkit, EGRA 2.0, Literacy & Numeracy Training, Open License Training, etc CIES 2018 panel presentation on coaching/mentoring EGR Program Design and Implementation Non-Fiction Sector Interest Group 140 Please list the events: Simposio de investigadores Resultados de la investigación de RedLEI. II Convocatoria Becas RedLEI. 32 de 37 Panel Desafíos de la Lectura Inicial y la calidad educativa en Centroamérica Something to do with use of EGRA workshop EGR Tool kits Children's Literature There were few last year that I cannot remember now. Begin with Books Training held in Nairobi on Children's Books Publisher Training Publishers training workshop Universal Design for Learning to Help all Children Read, Using Bloom Books to add to a library for children Classroom Observation Toolkit webinar, Comparative & International Education Society (CIES) Annual Conference, Miami, Florida, World Literacy Summit, Oxford, United Kingdom, USAID 2020 Global Education Conference, Washington, D.C. Universal Design for Learning Literacy Toolkit How to improve EGR in larger scale Introduction of tool kits * Mentoring every school and classes whether they used properly or not and also encouraging them supporting where they feel confused UDL, EGRA, Benchmarking (with RTI), more than I can remember unfortunately UDL, Coaching, Open license, material development Disability tool kit Publishers Training Workshop in Nairobi, Kenya Early Grade Reading in-person training and webinar series; UDL in-person training Pretested a toolkit on parental engagement in literacy Sales seminars at CIES early Grade reading toolkit face to face training; UDL face to face training Procurement webinar A session on developing early grade reading materials Language in reading projects, informational texts, coaching Series of webinars on Early grade reading Publishers writing of learning materials GRN Training and Experience sharing event Policy linking Events related to book development and supporting learners with disabilities EGRA assessments and the results of these evaluations The EGR workshop, coaching workshop, close out event EGRA toolkit online dissemination and guide to early grade reading materials development Report on Mali's MLE history It was about membership and growth and it was meeting number; 799 995 295.The rest l have missed out due to the difference in time zone. Unpacking of the assessment tool pre workshop 2018 event, 2012 Asia regional GRN workshop Education assessments Universal design Early Grade Reading Program Design and Implementation: Best Practices and Resources for Success Conferences, seminars , in-service training for teachers, and sharing reports EGRA Reading Norms 83 Please list the events: UDL, training with Amy EGRA experience and way forward 2015 Events about assessment In person trainings for EGR; webinars; and comprehensive approach to reading and math Conferences and professional meetings EGRA training; workshop on parental engagement Comprehensive approach Continuous Professional Development in Early Grade Reading programs; Key EGR skills and strategies for effective instruction and assessment; Early Grade Reading Programming: From Conception to Scale; Coaching to improve early grade reading: Emerging evidence on effectiveness and sustainability; there were others but it takes too much time to list them all Q 26_9: In what areas have these activities been useful in your work? (Please select all that apply.) Other (please specify) - Text Passing on knowledge to international partners involved in education and development As I said earlier, in helping me refine my research conceptually as well as its design. a reference DISABILITY understanding what is considered relevant in the field Q 27_5: For training, what modality did you find most valuable? Other (please specify) - Text only did webinar don’t know about others I have not had training. Being in Senegal Webinars are the most accessible. Did not do a training I have only attended live webinar hence can’t provide feedback on other modalities It depends on the learning objectives. Introductions of resources could be done via recorded webcast. More depth, I prefer in-person. I participated in webinars only and cannot make that comparison only attended webinars, but did not find them particularly valuable All modalities would be good if they carried useful information Live webinar and recorded webcast. Live when I had time to attend live and recorded when I didn't have time to attend live. Q 29a_7: Have you participated actively in any of the following Community of Practice activities? (Please select all that apply.) Other (please specify): - Text signed up for one that I do not think got off the ground but was never notified discussion with others from same NGO who also attended the same events Teacher training Initiated COP on SCM under the GRN umbrella I was nominated to their advisory council, but I was omitted from the ballot Q 33a: Have you experienced or noticed any positive result of GRN that you were not expecting? Please describe any unexpected positive result(s): Creative ideas that have helped me link with others doing related kinds of work Making professional connections The connections with wonderful GRN Staff. 142 The upper grade reading materials development Webinar came at a time when we were doing the first of that activity in Hausa under DFID/UNICEF/FHI 360 RANA finding information for development of abstract for conferences It connected us to many different networks and we used the EWT Project for other projects with other funders and partners knowledge gained More engagement open sharing and acceptance Sustento teorico o evidencia actualizada Research reports are the good result to others for learning and planning for intervention Access to recorded webinars and materials GRN has been an indispensable source of expert advice and best practices for our EGR efforts in Guatemala Not remember well I got a greater understanding of the needs other practitioners experience in their work. My first contact with the GRN exposed me to some Resource on coaching and mentoring and since then I embarked on a research in that area. Also, I got insight as to how I’ll embark on my community outreach activities through the GRN. Improved skills for capacity development for teachers on early grade reading Network! Community building, comradery Mentioning the use of Bloom Software in producing books in a variety of languages CHILDREN ARE ABLE TO READ AFTER A SHORT TIME GRN became a continuous source of valuable and diversified information The richness of the content/resources. actually active and functioning community of practice The region embraces electronic data collection Improved learning There's seem to be more cohesion between partners and greater communication. Gained knowledge on Strategic management of operations Regular updates on every activity in my email that focus on workshop implementation capacity development The information shared is useful and I use it in many ways. The use of the reading toolkit to enhance reading training materials The knowledge sharing platform is very helpful for supporting work in the continent of Africa. Gained fresh insight on creating appropriate content for early learners Teaching skills improved It’s a resource platform which is referred by many professionals involved in Literacy Its provides detailed info about all reading activities and share good resources and materials to improve current programs. Learning better ways of doing decodable readers EW program results Access to more data and project research. Now feel more confident in assisting with designs for new Mission-based early grade reading activities. Forums such as Comprehension interest group and Non-fiction interest group Such a positive collaborative experience. Great to see overall sharing and evidence based resources to support learners to read globally Opportunity to provide meaningful input into coaching and mentoring strategy and tools (e.g., classroom observation) increased access to resources Global Networking collaboration Plenty of resources shared for free That in children's literature, you do not have to tell it all, let the young learners decipher some concepts. 83 The importance of reading proficiency for young children is increasingly acknowledged across the board. useful in implementation in schools nothing Opportunity for reading materials Gained knowledge about reading from different literature on children's book writing and publishing How to develop children levelled readers in a language that is simple. Secondly, we recently did a biography of Professor Francis Imbuga which eventually is the first one to be published by my company. What prompted the Biography to be done and published was as result of the skills I acquired during a seminar training held in Nairobi. Change in reading culture For the live training, Universal Design for Learning to Help All Children Read, the training was exceptional and the resources were very good. I did not expect the training to be so useful. The comprehensive information provided makes my own reading and research more guided and quick The enabling writers resources helped me to plan for a recent material development activity Low cost but higher impact Reading assessment tools In the context of Nepal, it is drastically changed but Vital posts people’s officer’s misused budget, and within two years they are not mentoring well. If this prog will be lunch in Private schools, it will be mile stones because they will apply soon and the outputs will be high. More students will be good than previous year. Thank you. IPs volunteering time and resources Resources and toolkits tend to be very practical and user friendly. Very helpful Organizations compete less; programming is more evidence based Feedback summary of Training interventions Support education process and strategies in poorest countries It was easy for me to use my knowledge to conduct researches and to also impact on my students and colleagues. greater influence within my organization for having attended and turn-key’d info on best practices learned via GRN Non for now The development and implementation of EGRA toolkits for Afghanistan. Better concentration Reading programs Increase knowledge in the area of early grade reading and adaptation, training and administration of the EGRA. Across development spectrum, the importance of train the trainer and multiple layers of accountability and M&E interest in further webinars and training keeping updated on new initiatives and research findings, updates on USAID policies and procedures (i.e. Theory of Change) Was able to revise reading program for the Ministry of Education in Haiti enhanced network From the knowledge received learners are improving in reading There is significant knowledge that is scattered and the network has brought these players at the table for sharing I have shared GRN materials with colleagues in Nepal. I didn't know if they would be interested, but they were VERY interested. And grateful. Because of my participation in GRN events, others have approached me for collaboration, expanding my network. I did not expect to be as invested in the GRN activities, particularly the webinars given the time difference but I found myself looking forward to listening and interacting in each session. It was great to be in the company of practitioners who can speak my language and bear the same challenges as I do in education work. forming new professional relationships knowledge of the international programas and tools developed and implemented by the GRN 144 interest groups were unexpected and helpful Reminder for further reading to develop professionally networking opportunities, direct learning opportunities Providing a community that support each other in learning more about the work we do. effective reading strategies for children The strength and momentum of the group. Networking is really exceptional Connections with other literacy practitioners professional networking with other link-minded professionals from various organizations Encouragement and support for making our products usable by others. Their involvement of people worldwide in what they are doing I received information on Newsletter which was very timely on assessment, which I needed in my work. When it began, I did not realize they would be supporting documents put together by working groups (as opposed to a hired consultant). While the process is cumbersome, I think it has led to some really good products/resources. training government officials to develop their in-depth understanding of reading programming Value of Reading CIES panels were very good and informative. It brought together experts to share best practices. Have used GRN materials in my own teaching and classroom visibility in Macedonia many children perform poorly in mathematics Information The development of policy-oriented documents and guides Network has provided a great resources that improved my knowledge skills in designing and implementing early grade related activities. This wasn't expected. Extended collaborations Gained knowledge from their research We have learned how to use software to develop relevant readers Is most usefully network which most of the people in education area are benefited and material provided is most useful when u compare to other network. I know more now Knowledge growth among stake holders E G R A More general awareness of educational issues. Neutral (non-competitive) sharer or knowledge It was good to understand what is going on in the world in regards to reading and literacy. our children being able to read some words by age 5 Q 34_a: Have you experienced or noticed any negative results of GRN that you were not expecting? Please describe any unexpected negative result(s): Still very often the same people within the beltway talking to each other, to the (unintentional) exclusion of others, particularly academics and government agents in the places where we work. Independent consultants doing small scale projects were not well represented in the focus and efforts of GRN. Implicit racism in describing "Africa" as a monolith Providing me the opportunity to participate/attend in conferences on specific programs when I don't have the means or opportunity to travel for participation weak network GRN resources highly relies on the context of a specific region, specifically researches, references of education philosophies. It should be more inclusive and have better understanding of the context and suitable pedagogy. What becomes of all the resources (website) and CoP that's been developed over the years now that GRN is closing. 83 Please describe any unexpected negative result(s): The unprecedented cancellation of the GRN Training Workshop for publishers in Tanzania. The guidance developed and sanctioned by USAID has been treated by some Mission as the bible, leading to requiring full compliance with all approaches, and suggestions. All activities held in USA and for use it difficult to enter this country Domination of EGR narrative and resources by organizations with deeper pockets and exclusion of alternate perspectives Siloing of technical areas led to some duplicative conversations and work across other networks, work streams, and groups. Supplementary reading material recommendation( Bloom) and to some extend Global Digital Library GRN essentially relies on antiquated, English-based, middle-class to transparent orthographies and convinces countries to take complicated actions. The methods don't work well, and USAID results have been negligible. But the network has mainstreamed these antiquated ideas and has trained thousands of teachers and officials in them. Work to speed up automaticity based on cognitive neuroscience has become harder because of GRN. GRN tries to control all EGR messages, activities and products. The leadership is toxic. Makes it difficult to collaborate. Q 35_6: Which of the following describe why you wanted to be a member of GRN? (Please select all that apply.) Other (please specify) - Text Jobs Keeping abreast of latest research Building a new COP on SCM Improve and revise in program designing and implementing knowledge to transmit support programs I worked on Q 38: Do you have any recommendations on how the Network might be improved? Do you have any recommendations on how the Network might be improved? I like what I've experienced and hope that such opportunities will continue now that the GRN is being dissolved. It would be good to consider time zones while considering the events and give opportunities for participants to share more. Connect with other donor agencies (DFID, GPE) and, especially, network organizations and associations in the places where we work - like research networks (ADEA in Africa, for instance) and ed ministry networks (like CONFEMEN in francophone Africa) to make it truly global. And more resources in French, please! No More opportunities for face to face interaction at country level. The recorded podcast/webinar on YouTube is preferable Not now The network, aside from the online resources and webinars must also have the opportunity to face to face interactions through Community of Practice. Further, the network can also build up other call for project partnerships, researches etc. where we can work together and learn at the same time. Time difference as per continent has to be clearly mentions to avoid any missed schedule. If you really want to be a regional network, it is important to expand the information to other languages such as Spanish. Expecting more face to face learning opportunity. Thank You Make it more regular Incluir artìculos en español, para latinoamerica Face to face training is the best tool to learning and sharing to each other. I am Alumni member of PAUN. So I have idea that I learning more thing during me USA visit. 1. Provide more training opportunities free of charge or at an affordable cost to individuals working in organizations within the community of practice. 2. Where possible, include direct recipients in the some 146 Do you have any recommendations on how the Network might be improved? webinars to share their own insights and stories on how they have directly benefited. 3. Create opportunities for mentor-ship within the network So far everything seems great to me More virtual events like webinars will be useful Sometimes on the network was very slow and I was feeling frustrated to wait and wait until the document is open but it was also couldn't open. I was really wanted to read and get some of the useful skills from GRN but sadly missed because of the internet problems. So please look for another way of sending of the documents so that I can easily receive them. I am also a member of our Early grade reading program in our country, and that is why I wanted to have other ideas or skills from your program and then I can share my ideas on how to do it for the betterment of younger children in the world. No, thank you Continue it, please! It's been excellent. Those of us working in the field need the expert knowledge and networking. We are seeing great results in implementing best-practices from USAID funded, GRN guides and research - especially those customized for Guatemala. You all do great work. Please keep it up! Summarize the main things, link the detailed references to next click. It saves my time reading the shared info. If you often did this kind of message, please continue doing so. Thanks! Continue the momentum that had been building, especially as new challenges become apparent. Look into the Early Childhood Development aspects of Reading. The network seemed deliverables/product driven, especially in its last year. It seemed like they were more interested in getting products done to meet their deliverable which seemed to affect the quality of the products. Not sure if the advocacy side worked well. It seemed that the network was eager to please USAID so when calls for advocacy came in (i.e. inputs on new Education Strategy, and the like), I'm not sure inputs were carried forward. There was no transparency with what was shared with USAID after the collective call for inputs. Other organizations/groups do share what was submitted. Based on my profile need exposure to assignments or jobs which gives us to grow in expertise in my education field. I think the Network was doing an amazing job! My own research interests are beyond literacy but I have learned a lot from their work which has helped me frame my own research, such as the centering of the mother tongue as well as the continued struggle with comprehension. I believe that giving independent global reading consultants a larger platform in GRN conferences, newsletter, blogs, etc. would add perspective and depth of knowledge to the mission we share in. Small scale, on the ground interactions between educators provide insight that massive projects miss. Consequently, the learning curve global education development travels is much longer than need be. Need for improvement in the strength of Networks during Webinars in some areas like in Nigeria. I sometimes find it very difficult to connect. My interest is in the area of coaching and mentoring, how do I join the community of practice? 1.More resources be made free to access 2.More opportunities for participating physically in EGR programmes Strengthening the Multilingual Education component, strengthening language policy advocacy efforts to influence educational policy. Connections with Boards of Education in the U.S. and around the world I have little benefit directly from the GRN so far - I think however the GRN is quite relevant in the field of EGR, especially for countries or regions that do not produce much research leading to knowledge sharing on good practices. It would be good to reinforce active participation from researchers and practitioners as much as possible. I would think I can seek to look at opportunities, resources and contacts to advice to other colleagues in the future. Keep up the great resources you are sharing. I would like more information on how I can participate more directly. Calls for presenters ahead of time. My consultants always lamented that they can do better than yours! EGRA has also been gravely criticized, please reach out to those who presented alternatives. None Shorten the timeline in making resources available after webinars. 83 Do you have any recommendations on how the Network might be improved? The network I think is very valuable and it offers a good range of interactions - professionally and with the USAID-related network. The only lacks are from my side - I have not engaged as much as I would have liked to do because reading has not been central to my work recently. I do think it would be very sad if the network were disbanded. Education is vital. Reading is fundamental to education. The investment of funds in maintaining the network is probably trivial in comparison with the professional enrichment and economy it represents in drawing together at no cost the best thinking on the topic, which then feeds into and improves the quality and effectiveness of every USD spent on reading programs around the developing world - and I am sure in USA schools as well. Please keep going: I hope to engage more in future than I have done of late. Not now. Creating network groups around thematic areas This is great! Keep it up and I hope there are ways this community of practice will continue. Organize more training sessions with Education and technology partners like Snapplify who are offering a free Basic digital library model to schools with a feed of over 46, 000 free eBooks to promote access of digital content Not really I feel it's adequate. - No To create a platform for attending training out of one's country. To continue sharing update on all programs To support by establishing focal persons or groups or organizations in my country (Sierra Leone) that support the development of education. frequent reminders to the members Stimulate participation from other agencies, such as UN. Regarding webinars we have had challenges with network connectivity, otherwise I appreciate the sharing. GRN should purchase readers for supply to schools in Kenya. More involvement of EGR professionals from different regions in presentations and product development. The main challenge for Education Officers at USAID is time. Most of us are stretched very thin and hardly ever have time to keep abreast of what's going on outside of missions. We have to make a concerted effort to keep abreast with any new development in the sector. Also, ensure that reports have an Executive Summary. I also find useful to share tidbits of information at a time. No one has the time to read a 75-page report, but we do have five minutes to read highlights or bullets on key important findings in the sector. Thanks for all you do to strengthen learning outcomes around the world. connect with INEE and help more with how to apply knowledge and best practices to emergencies TED talk like smaller events at different geographic locations especially in developing countries would be of substantial value to the local practitioners. And, that is where the acute need is. More webinars to enhance capacity and share resources I suggest that the network should continue and not be discontinued. I suggest that there should be opportunities for face to face either in-country or regionally to integrate some of the things that were learnt through webinars and online platforms. It would be helpful to also explore the possibility of using APP based learning or MOOCs for specific areas related to language, literacy and reading education. Create more linkages with editors and authors in Kenya who are the creators of content through the Kenya Publishers Association and launch more training and fora for knowledge sharing. NONE Address school readiness assessments. Emergent literacy, how to capture and to examine the predictability of assessments at 4 years of age to link to primary school achievement results on standardized assessments (e.g. PISA, SEA-PLM) Design the online events (webinar and others) in such a manner that people from different time zones can join in. I request if you develop and share a guide for reading project activities. This will be a good support for the reading project implementers. In addition to this, would like to receive more resources on multilingual education and literacy. Organize more in person trainings per year in Kenya More Innovative creation 148 Do you have any recommendations on how the Network might be improved? Not at this time. Thank you. Might be good to work or focus in individual country, rather than globally in general. Is there a way to Read articles free without data cost? Recommend adding webcasts pitched at a higher level of experience and expertise. --Offer more in-person workshops outside D.C./US --Provide funds to support bringing together working groups to contribute to resource development/PD --Engage more GRN members to deliver trainings and other forms of PD --Better package PD--not just long resources, but shorter, more interactive packages of materials on specific topics Spanish speaking activities increased. No. Just keep this group working! GRN should work with people from different cultural backgrounds who know the current challenges within their communities. Doing this will enable GRN to really meet up with the challenges of the people. No Consider a dues/fee based membership system for large US-based organizations as a way to cover costs and create some independence from USAID. None Not sure. I did not profit much from this network due to my limited availability. Responding to this survey I realize that I do not remember all the information GRN has been sharing. I think that the network should continue. It has been useful in various aspects. I have to make time to access and study the shared materials and be able to participate more in webinars. Improve translation to Spanish I am in early childhood development so GRN is not that closely connected to my work. But I appreciated the updates and sharing of information. Providing more outreach opportunities for USAID /university partners to collaborate and share ideas; disseminate reports; and build more opportunities for interaction. Fewer emails No Recommend to continue supporting the knowledge sharing and evidence around EGR through communities and platforms similar to GRN. More regular input solicited for the development of tools, materials, strategies, etc. To engage more with existing networks in education such as the Inclusive Education task group in IDDC ( International Disability Development Consortium) and GCE It is better to consider time zone differences while organizing live online events. I usually register to the events but the schedule doesn't match and couldn't attend. The network can also contribute towards organizing region wise webinars to provide opportunities to wide range of stakeholders. Further, GRN can also act as a repositories of early literacy and numeracy knowledge especially in regional languages for better cross learning and scaling up. GRN can further initiate cross-regional knowledge building. None Not for a moment. Materiales de aprendizaje, cursos e información en español Learning material, courses and information in Spanish, please I have often had issues access full PDFs of resources through the site. Sometimes when you click on a resource, there is no intuitive way to access the full resource. It would be nice to make this a little easier to navigate. Integrated into USAID's EduLinks website or other sustainable initiative, rather than a short-term activity. Continue with Webinars and shadings of research findings. Do a face to face meeting with members to share ideas and learn from each other Promote the formation of national networks and support them to grow. None at this time since I'm not an active participant. The website is a platform for sharing. However, finding documents is not easy and sometimes starting outside of the GRN - with Google as the search engine, is the better way to locate files. 83 Do you have any recommendations on how the Network might be improved? I have not been intentional about the use of the network. Thanks to the email, I occasionally read the articles. I would be more conscious in doing this but would also like it if there is a way the team could make it more engaging for we those who have subscribed. non No none Need for continued advocacy for the PLHIV AND CLHIV Keep it up good work Local chapters can be introduced. Face to face conferences to share can also be introduced at least once annually. Some podcasts with alerts on you tube helps to share and also access By creating a community of practice for people with similar expert-ism. It would be great to learn firsthand experiences where there has been clear turnaround literacy achievements from people on the ground. I am thinking of remarkable changes of 75% and above reading successes. I keep thinking is the teaching of reading so complex that in the developing world we can't seem to be turning around what in the past simple non-expert missionaries managed to do. I have come across non-reading professional parents score 100% success with their children what we professionals are struggling to replicate. In Zambia, we have had cases of community schools run by volunteer teacher, again non-trained achieve better results. In a number of cases, these schools have done better on average than schools run by our national government. I keep asking, are our researchers missing something? Are they asking the right questions? Even where there has been some moderate success, these have not been sustained beyond the tenure of literacy programs, at least in Zambia. If you asked me, I think, public teachers in Zambia have been over trained through many program initiatives to an extent that these same programs have now created a different expectation in the teachers. The programs are now viewed as sources for extra income through allowances. Not at all. Organizing several trainings and events especially related to publishing in Africa. Give more prior notice about live events. I travel to the US yearly, and would plan my time in the US around attending at least one live event, if it were applicable to my work in Indonesia. I would request the network to continue, allow for more opportunities for international participation (like that of ours) by sharing some event information early on. GRN tried to reach my inbox on any early grade reading relevant information; but it is more on my side who have not sufficient time to explore the useful resources available to me. Online training can be extended. Please share more update on EGR in the world. No Obviously, Through PABSON of Nepal or Bhaktapur Pabson Or Madhyapur Pabson. Thank you Develop other online techniques that allow implementers and participants to communicate with each other continuously to exchange their own experiences and lessons learned during the life of the project. - still having regional meetings for the GRN to be in touch with experts having the same agenda... - hosting a regional conferences tackling hot issues related to GRN work >>> - networking with regional networks having the same work and missions... - using regional platforms to publish and reach out GRN work None at this time More opportunities to create and participate in working groups and knowledge sharing activities Engage more smaller and medium size organizations working in education. Make your outreach more inclusive Continue it Address emerging trends and traditional ways of learning that can be incorporated into the material production and tools. None Maybe more opportunities to be provided for exposure of local practices by inviting local practitioners The search engine/finding resources I knew existed was difficult. I couldn't always find things and the way they were organized was not logically for me. 150 Do you have any recommendations on how the Network might be improved? I hope to participate in your network via live webinar and accessible websites none The Network might be improved if we at Africa and other countries outside USA can have the privilege of a face to face, interaction like in workshops, conferences and conducting researches with researchers from GRN... Do more regional live workshops especially around Africa. Nothing to say Although face to face approaches are expensive, I believe they are more meaningful than webinars. Increase opportunities for local partnerships I suggest there are more tailored material available for most of the countries. Network activities/ webinars etc. should be organized taking into account various time zones. For example, most webinars are held at a time that is very inconvenient to India. Also GRN can proactively reach out to members to share their stories, achievements and thoughts Show live example(s) that could convince government officials buy the idea easily More affordable programs Context-specific cluster of experience and lessons would benefit the members more to tap resources. Keep as many of the advantages going even if in different ways. Invite to join a similar/replacement network. Continue knowledge sharing, challenges, networking opps - all. No. No. Continue with the webinar More opportunities for discussion at a policy level on issues in early grade reading and learning both online and face to face Help link information to DEC and MAKE DEC Accessible. GRN is doing an awesome job of archiving information, webinars, etc. and making it accessible over time. THANK-YOU Continue and increase participation of people in the field no No. Thanks for you work! Maybe broader engagement with nonUSAID partners/contractors. And being careful to avoid very specific USAID language when engaging with non USAID contractors in training. The network provided logistics support for COPs to meet, and no plan in place. This is a waste of 4 years of work and effort. The network may be improved by considering that in some areas internet access is not easy and English is not used by everybody. Another way of giving access to members can be developed. For example having a local focal point who can make the network information available to local experts (in the DR Congo for example access to information in English is opened to few persons) and local information send to the Network. A library with produced documents is one of the solutions to that situation. Not at this point. The resources and collaboration that the GRN provides are hugely significant. But I will think about this!! The network is ok. Rather it's me who needs to interact with it meaningfully Keep up with the good work Connecting it to as many people at all levels (Government, CSR/AID, INGO, Universities etc.) I would like to be involved in online activities and open-source learning activities technical expert inclusion early on for publications/webinars as the managers of the network cannot have all tech knowledge The webinars are frustrating as staff want to engage but the quality is extremely poor and cannot usually be heard. They often hang and stop. In addition the timing is often at odd times that are difficult for staff to listen in at. We have more than 25 staff directly engaged in EGR work plus several hundred partner NGO staff. In addition we are working with thousands of teachers looking to develop skills. Short YouTube videos of the webinars or useful topics that can be shared through the local network would be a great help. The Global Reading Network is a terrific and promising initiative. It was great having that kind of technical support when we were implementing our early literacy project in the Philippines, though I wish I could have 83 Do you have any recommendations on how the Network might be improved? contributed more. Perhaps it would be good to create and/or strengthen regional networks/affiliates of the larger GRN so regional partners can have more opportunities to interact, share knowledge and resources and test/pilot initiatives. I wish there could have been a more deliberate effort or mechanism that can allow members to work together and produce a specific knowledge product, other than what the GRN consultants and key partners/CoP are doing. Translation of tools/resources into French would be an asset. Most francophone teams benefited very little from the resources and opportunities curated by GRN. Don't close it down! Facilitate cross-implementer research into promising practices for reading education in resource-poor contexts The network has to be local face to face experience sharing and an opportunity to participate in global experience sharing outside of the country especially out of Africa. And tools sharing how to improve reading at all level in softcopy, easily shared with others Not at this time No more Need to engage more in order to answer this question. More in person meetings. Webinars are not very useful. - More interaction and support - Two way communication - Support implementation of research findings by interested organizations provide more opportunities for USAID missions and implementing partners in other countries to participate - through targeted events, access to resources and discussion in other languages etc. It is wonderful! I would love for it to continue! It's been a great resource for our organization. It is pretty good as it is. Organization of local meetings or webinars for the rest of us not miss out. The network is doing well. It should continue providing information. I definitely would suggest to continue the network. Being based in Asia, the times of webinars and streamed live events almost always made it impossible for me to participate. In the future it would be good to rotate timing so that sometimes it is more feasible for people in other countries, rather than always revolving around convenient times for US-based participants. Some of the webinar resource persons need to be more resourceful. Rudimentary issues should not be allowed in the webinar as people across the world are joining with high expectations. Experiences of young professionals are important but there are others means of sharing those. One must feel that his/her time was well spent with enriching experiences when s/he joins a webinar. - Materials developed in various languages - Online courses launch The network is good but any improvement is welcome It should teach out to members to invite them to join working groups, and should email links to past webinars and tools developed. It would be great if EGRA instruments, evaluation reports and program reports were easier to locate on the website (have broken down categories for each of the above). The search options on the website are not easy to navigate to find such resources. All resources on the USAID Dec and other donor publications should be available on the GRN website. Would like for the network to develop a more robust communication approach in terms of getting information to the members of the network. No thanks Be even more internationally recognized and visible Contextualize teaching and learning materials It seems this network does not work on ground I suggest to more productive it Should be on ground like in Rwanda I don't think is known While all the sessions and knowledge products are so relevant and useful, the time of webinars etc. has been of issue all the time for non-USA residents. Also, if materials are archived somewhere, the information on how to access them would be so much valuable. At least, I never saw the information about YouTube materials. It is useful to organize online and face to face training to other organization in coming days. It can be organized in different regional level (Asia, Africa, and Europe) so that its importance can be more effective as the context 152 Do you have any recommendations on how the Network might be improved? may vary from place to place. I strongly recommend to have online and face to face orientation from GRN so that we can be benefited and individuals and organizations can bear the cost to participate in event. GNR should organize a face to face training and sharing program. Webinar or virtual sharing platform may not work well in developing countries like Nepal because of internet access. Furthermore, face to face training provides an opportunity for meaningful participation and opportunity for deeply engaging in the topic. It also ensures better learning. N/A Publications in Spanish wider group of presenters Need to use questionnaire to obtain information. There is a challenge participating in face to face discussion because of problems with internet connectivity. Regular network through correspondences (letters, emails etc.) Regular in-person and on-line training opportunities for GRN members Region-based capacity building opportunities None The webinars are very dry and not at all interactive. When there is virtual participation for in-person events, it has always felt like the people online are an afterthought. This has made the GRN feel like it is mainly for people in DC. Consider providing opportunities for professionals from Developing countries to attend in person workshops N/A NORC when selected people to participate in the survey please ensure that you have contacted the people who are in the database to be the user of the GRN More activities should be designed to help this network to actively work together. More innovative work. Most of what USAID does in relation to literacy is not making a sustainable difference. None Webinars at different times of day - usually in the middle of the night for Asia/Oceania Constant sharing No Fortunately the GRN is ending, so there is no need to improve it. In the 6 or so years of its existence, GRN has been run directly by USAID staff and by a few chosen contractors. They dictated who worked in it based on personal likes and dislikes. Directors who appeared to disobey were just removed. The staff also had very low salaries and easily left. So there was never intellectual leadership; just ability to respond to USAID requests. The directors also actively suppressed the research and views of people who were deemed unfriendly. Given the state of knowledge by the directors' friends, GRN disseminated the state of reading knowledge prevalent in the US and for English in the 1990s. So it relies on antiquated, English-based, middle-class to transparent orthographies and convinces countries to take complicated actions. The methods don't work well, and USAID results have been negligible. But the network activities have mainstreamed these antiquated ideas to hundreds of officials. Work to speed up automaticity based on cognitive neuroscience has become HARDER because of GRN. If a network were formed again, it should use the international cognitive neuroscience findings as a starting point. Many are published in the Scientific Society for the Studies of Reading and other psychology journals. no Including linguists in addition to specialists in education in reading projects. Including research component in the reading projects with the intention to implement it in reading projects (this apply to reading projects in the Arabic language.) No Organize face to face trainings to us. No It’s a great network but would like to have more in-country or regional meetings to build synergies. Keep up the great work! Get new leadership with better interpersonal skills. Collaboration is about listening, not trying to boss others around and claim all the credit for the contributions made by members of the Network. 83 Do you have any recommendations on how the Network might be improved? I was living overseas for a good portion of the time that the network was active. I used the print resources extensively and found these valuable. I wanted to use the webinars more but they seemed to stay at a more basic level throughout the 5 years while the field learned quite a bit about the subject and might have been at a higher level the final years. I wanted to be able to use the webinars (on YouTube) more but didn't. Perhaps the communities of practice served this purpose? no Have materials/resources available/ translated in other languages, provide further capacity development with more regularity. For improvements to take place the network should continue to be supported. My only complaint is that there could sometimes be hard to understand delays in the release of resources (for example, the classroom observation toolkit, which appeared to be finished at CIES, however isn't being released until November). I suppose my recommendation for improvement would be better communication about either way the product is delayed or when it will be released (when delayed). The network has a limited conception of who the experts are. Should cast the net more widely to include more than the "usual suspects." 154 Evaluation Addendum APRIL 24, 2020 USAID presents this addendum to represent the input of URC’s current leadership into this third-party, external evaluation of ECCN and GRN. During the preparation of this evaluation, the evaluators met with seven members of the URC team for the Reading within REACH/Global Reading Network contract. They also interviewed the REACH Project Director in a second, independent meeting. However, no members of URC’s current senior leadership team were present in those meetings. This addendum presents perspectives from URC senior executives and corrections of key informant statements that URC’s leadership brought to USAID’s attention after the Agency had the independent contractor finalize the evaluation. A. PERSPECTIVES USAID and URC agree that the REACH/GRN project was a very ambitious undertaking that underwent a tremendous transformation during its 6-year implementation period. In the last 18 months of the award, the highly qualified URC key personnel and the USAID team supervising the contract built productive and focused working relationships that resulted in the creation and dissemination to the field of a suite of valuable resources on improving reading instruction. Although the REACH project did experience, as do all programs, highs and lows during its period of performance, it finished with a crescendo of activity and impact that has resulted in the network “living on” after the contract and continuing to serve the education community. USAID and URC also wish to record that the time and materials contract for the REACH project created inherent and challenging constraints within which all parties had to work to achieve the contract’s positive end results. The contract had only two line items, one for labor and one for ODCs, and it was not possible to shift financial resources between them. Despite this restriction, the URC team produced knowledge products, training, research, events, and communications that, in the long run, allowed the Global Reading Network to prosper. Lastly, the URC leadership has brought to USAID’s attention that, because the report incorporates input from a wide range of key informants, some of the quotations from interview participants are inconsistent with one another. For example, the report contains both positive and negative reflections on the REACH project’s convening power. USAID agrees with URC that, by the end of the contract, the REACH team had the necessary convening power to grow the reading network and to deliver the required technical products. Likewise, comments in the report about staffing sometimes reference difficulties in filling select key personnel positions, but at other times praise the URC staff’s performance. USAID agrees that, particularly as the contract moved into its last years of performance, URC made sure to identify and employ staff and consultants capable of completing the required technical work. As a case in point, USAID considers URC’s successes in training (i.e. the delivery of an innovative “Reading for Transformation” course for USAID personnel, technical contributions to USAID staff training abroad, and presentations at high-level international meetings on technology and education), to be due to excellent staffing choices in the final years of the award. By the conclusion of the contract, USAID could verify that the REACH project had fulfilled its primary mission of creating a viable reading network. 83 B. CORRECTIONS OF STATEMENTS FROM KEY INFORMANTS The report quotes key informants who told the evaluators that URC is a small business. These key informants’ statements are in error. URC is not a small business. URC is included in USAID’s list of top forty vendors. Participants interviewed for the report described the policy linking toolkit as a GRN product. This is not correct. This toolkit is a joint donor product from an international working group whose first technical sessions were hosted at the GRN. The report references statements from key informants that “funding and staffing limitations limited GRN’s impact in metrics and protocols (page 31).” USAID recognizes that the REACH contract staff and consultants developed high-quality products under this domain of work. These included the revised EGRA toolkit, the classroom observation toolkit, and the prototypes for the literacy landscape analysis. On page 31, interviewees are quoted as saying that there were “no regional events.” This is not fully accurate. The REACH contract staff worked diligently, with USAID’s approval, to leverage conferences and meetings like the annual CIES meetings, International Literacy Day events, and convenings of the M￾ED Alliance to regroup representatives of ministries, NGOs research institutions, USAID missions, and others to discuss technical topics related to improving reading outcomes. USAID notes that no key informant described to the evaluation team the landmark accomplishment of The Global Book Fund Feasibility Study. This study, which was the result of a major subcontract to Results for Development, (R4D), was a seminal piece of work that stands as a principal deliverable of the REACH award and as a point of reference for the entire membership of the Global Book Alliance. USAID concurs that any record of URC’s work under REACH should highlight the completion of this groundbreaking work. 72 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523