USAID/West Africa Peace through Evaluation, Learning and Adapting (PELA) REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT Original Submission July 3, 2019 Resubmitted: October 28, 2019 Final Submission: April 17, 2020 This publication was produced for review by the United States Agency for International Development. It was prepared by International Business & Technical Consultants, Inc. (IBTCI) under AID-OAA-I-15-00022, TO No. 72062418F00001 DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. REACTING TO EARLY WARNING DATA IN WEST AFRICA: PERFORMANCE EVALUATION REPORT Contract No. AID-OAA-I-15-00022 TO No. 72062418F00001 Originally Submitted: July 3, 2019 Resubmitted: October 28, 2019 Final Submission: April 17, 2020 Prepared by: International Business & Technical Consultants, Inc. (IBTCI) One Airport Square, 6th floor Airport City, Accra, Ghana Office Tel: +233 (0)500079350 USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | III ACKNOWLEDGEMENTS This report was made possible by the efforts of a number of people. Jacques Bérard, Team Leader is the author of this report. He was aided by Kpedetin Oke, Ph.D., Subject Matter Specialist. They were supported by Kirsten Siebach, Qualitative Analyst, who participated in all interviews, wrote up interview notes, and entered all excerpts from documents and interviews into the qualitative data analysis software that the evaluation team used. Thanks to Carol Conragan and Miranda Weinstein, from International Business & Technical Consultants, Inc. (IBTCI), for their help in ensuring the quality control and formatting of the report. Thanks, also, to Erin Dowling for copy editing the document. Special thanks go to Kris Inman, Ph.D., Research Director for the Peace through Evaluation, Learning and Adapting (PELA) activity and to Robert Grossman-Vermaas, Vice President, Technical Services, Crisis, Conflict, and Governance, both with IBTCI, for their comments on successive drafts of the report. IV | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV TABLE OF CONTENTS Table of Contents .............................................................................................................................................. iv Acronyms............................................................................................................................................................. vi Executive Summary ............................................................................................................................................ 9 Project Background ................................................................................................................................................ 9 Evaluation Purposes ................................................................................................................................................ 9 Evaluation Questions ............................................................................................................................................ 10 Evaluation Method and Limits ............................................................................................................................ 11 Main Conclusions .................................................................................................................................................. 11 Conclusions for Additional Strategic Questions ........................................................................................... 16 Main Recommendations ...................................................................................................................................... 16 Introduction ....................................................................................................................................................... 19 Project Background .............................................................................................................................................. 19 ECOWAS Structure ............................................................................................................................................. 21 Evaluation Purpose and Objectives ................................................................................................................... 21 Evaluation Questions ............................................................................................................................................ 22 Evaluation Design, Methods and Limitations .............................................................................................. 24 Design....................................................................................................................................................................... 24 Methodology ........................................................................................................................................................... 25 Limitations ............................................................................................................................................................... 25 Conclusions, Findings and Recommendations ........................................................................................... 26 Evaluation Question 1 .......................................................................................................................................... 26 Evaluation Question 2 .......................................................................................................................................... 32 Evaluation Question 3 .......................................................................................................................................... 36 Evaluation Question 4 .......................................................................................................................................... 47 Evaluation Question 5 .......................................................................................................................................... 53 Evaluation Question 6 .......................................................................................................................................... 61 Evaluation Question 7 .......................................................................................................................................... 62 Evaluation Question 8 .......................................................................................................................................... 65 Further Evaluation Questions for ECOWAS ............................................................................................. 68 ANNEXES.......................................................................................................................................................70 Annex 1: Inception Report ............................................................................................................................ 70 Background ............................................................................................................................................................. 70 Purpose and Objectives ....................................................................................................................................... 70 Evaluation Questions ............................................................................................................................................ 71 USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | V Evaluation Design, Methods, and Data Collection Strategy ....................................................................... 74 Overall approach of the evaluation .................................................................................................................. 75 risks and limitations that may undermine the reliability and validity of the evaluation results .......... 76 Evaluation planning matrix .................................................................................................................................. 76 Further Evaluative Questions for ECOWAS.................................................................................................. 78 Timeline ................................................................................................................................................................... 80 Evaluation Team .................................................................................................................................................... 81 Logistics Considerations ...................................................................................................................................... 84 Annex 2: Bibliography..................................................................................................................................... 85 Annex 3: KIIs List ............................................................................................................................................ 88 Annex 4: KII Guides ........................................................................................................................................ 89 KII GUIDE 1 EWD ................................................................................................................................................ 89 KII Guide 2 DPAPS ............................................................................................................................................... 90 KII Guide 3 SA & Gender .................................................................................................................................... 92 KII Guide 4 REWARD ......................................................................................................................................... 94 KII Guide 5 USAID & DoS .................................................................................................................................. 95 KII Guide 6 ECOWAS Commissioners ........................................................................................................... 97 KII Guide 7 Development Partners .................................................................................................................. 99 Annex 5: Coding Matrix ............................................................................................................................... 101 Annex 6: Status of Task Order Activities ............................................................................................... 116 Annex 7: INDICATOR TABLE ................................................................................................................. 120 Annex 8: Evaluation statement of work compliance checklist ........................................................... 122 VI | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV ACRONYMS AAR After-action review COP Chief of Party COR Contracting Officer’s Representative CoW Council of the Wise CPAPS Commissioner for Political Affairs, Peace and Security CPX Crisis Response Planning Exercise (also known to ECOWAS staff as Crisis Response Planning or Crisis Scenario Planning) CRVA Country Risk and Vulnerability Assessment DANIDA Danish International Development Agency DFID Department for International Development (UK) DO Development Objective DOS Department of State DPA Directorate of Political Affairs (PAPS) DPKRS Directorate of Peace Keeping and Regional Security (PAPS) ECPF ECOWAS Conflict Prevention Framework ECPFS ECOWAS Conflict Prevention Framework Secretariat ECAF Early Warning Conflict Analysis Framework ECOWAS Economic Community of West African States ET Evaluation team EU European Union EWARP Early Warning and Response Partnership ECOWARN ECOWAS Warning and Response Network EWD Early Warning Directorate EWS Early warning system EWR Early warning response FBA Folke Bernadotte Academy (Sweden) FM Field monitor GSAD Gender and Social Affairs Department GIS Geographic Information System GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit (German Federal Enterprise for International Cooperation) HSAF Human Security Analysis Framework IDWG Inter-Directorate Working Group USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | VII IR Intermediate result KI Key informant KII Key informant interview ISC Internal Steering Committee (ECPF) IDWG Inter-Directorate Working Group (REWARD) JARP Joint Analysis and Response Planning MFD Mediation and Facilitation Division NCCRM National Centers for Coordination of the Response Mechanism PAPS Department of Political Affairs, Peace and Security (ECOWAS) PELA Peace through Evaluation, Learning and Adapting PITT Performance Indicator Tracking Table PSS Peace, Security and Stability (also known as EU-PSS) REWARD Reacting to Early Warning and Response Data in West Africa RPGO Regional Peace and Governance Office (USAID/WA) RRVA Regional Risk and Vulnerability Assessment SIDA Swedish International Development Cooperation Agency SMD Systems Management Division (EWD) SOP Standard operating procedures TA Technical assistance TO Task Order USAID United States Agency for International Development USAID/WA United States Agency for International Development Mission for West Africa USG United States Government V-P Vice-President WANEP West African Network for Peacebuilding EXECUTIVE SUMMARY PROJECT BACKGROUND On September 25, 2015, the United States Agency for International Development/West Africa (USAID/WA) awarded Creative Associates International (Creative) the Task Order (TO) AID-624-TO￾15-00004 (under Indefinite Quantity Contract AID-OAA-I-13-00005) to cover a four-and-a-half-year period (2015–2020) to implement its Reacting to Early Warning and Response Data (REWARD) program in West Africa. USAID anticipated an investment of $11.5 million for REWARD. REWARD addresses critical gaps in conflict early warning and response in the 15 member-countries of the Economic Community of West African States (ECOWAS). The program provides technical assistance to reinforce the capacity of the ECOWAS Commission’s Department of Political Affairs, Peace, and Security (PAPS) to collect and analyze conflict-risk data, develop strong recommendations, and improve the sharing of this information within ECOWAS, its member states, and other relevant actors. Specifically, REWARD strengthens the capacity of the PAPS Early Warning Directorate (EWD) to produce early warning recommendations, which appropriately reflect regional policies, best practices, and socio-cultural conflict dynamics. REWARD has two purposes: Purpose 1 is: “ECOWAS response to early warning recommendations improved” and Purpose 2 is: “National and local stakeholders’ response to electoral violence triggers improved.” This performance evaluation focuses on Purpose 1, which has four outputs: (1) EWD strengthened to collect early warning data; (2) EWD capacity to analyze, report, and generate recommendations improved; (3) Enhanced DPA response capacity; (4) Enhanced PAPS internal and external communication and collaboration on early warning and response. EVALUATION PURPOSES International Business & Technical Consultants, Inc. (IBTCI), under AID-OAA-I-15-00022, TO No. 72062418F00001, prepared this performance evaluation report for review by USAID. It was written on the basis of an Inception Report (March 11, 2019). This evaluation has two objectives: (1) to determine if REWARD is well positioned to achieve success in strengthening the ECOWAS Early Warning and Response Systems and (2) to determine if assistance to ECOWAS/PAPS/EWD is relevant to the sub-region in terms of advancing self-reliance from the regional to national level; what efforts were/are effective or not effective; and how assistance should be modified to better reach USAID strategic objectives for the region based on the most recent priorities of the U.S. Government (USG). 10 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV EVALUATION QUESTIONS The approved Inception Report poses the following questions: Evaluation Questions Question Sub-Question EQ 1: How has the regional conflict early warning system of ECOWAS changed or not since the inception of the USAID REWARD project? EQ 1.1: What were the best practices, lessons learned, successes and failures and why? EQ 2: How did gender integration efforts implemented by REWARD contribute, if at all, to a more gender-balanced Early Warning data collection and analysis? EQ 2.1: Do the Early Warning Directorate and the CPAPS demonstrate the institutional and professional capacity and will to sustain the approach of integrating gender in its peace and security efforts? EQ 3: How has the USAID REWARD project contributed to improving EWD’s ability to collect Early Warning data and strengthen EWD’s ability to analyze, report, and generate recommendations? EQ 3.1: What specific capabilities and skills were increased and how have they been applied by EWD? EQ 3.2: Does EWD demonstrate the institutional and professional capacity and the will to sustain early warning data collection, analysis, reporting, and recommendations without the assistance of REWARD? EQ 4: How has the USAID REWARD project contributed, if at all, to improving CPAPS ability to carry out its mandate of early warning, conflict prevention, and mediation, especially by enhancing CPAPS’ internal and external communication and collaboration on early warning and response? EQ 4.1: What are the specific instances of CPAPS’ better internal and external communication and collaboration on early warning response? EQ 4.2: Does CPAPS demonstrate the institutional and professional capacity and the will to sustain early warning, conflict prevention, and mediation without the assistance of REWARD? EQ 5: What internal and external factors prevented or support Creative from accomplishing the REWARD expected results? EQ 6: Which of the program design assumptions where no longer valid or changed during the activity implementation that might have affected the program implementation? EQ7: Which activities on the REWARD Task Oder should continue and why? Which should be modified and Why? EQ 8: To what extent was REWARD complementary to the State Department (or other) support to ECOWAS and EWD? Additional Questions: The Inception Report includes strategic questions which fall outside the scope of a performance evaluation of REWARD. The evaluation team (ET) was given the possibility of asking the following questions of USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 11 specific KIIs that USAID identified to help guide future strategic-level evaluations of ECOWAS. These questions are: Additional Questions 1. How many and what type of recommendations generated by the EWD early warning work have translated into action by the ECOWAS leadership or member states? 2. What role, if any, would the ECOWAS early warning system/efforts play in meeting USG’s current and evolving objectives, particularly, in assisting countries to achieve self-reliance? Should USAID continue to invest in these organizations and why? 3. Has USAID’s investment in ECOWAS led to national level impact, if any (though not part of REWARD’s objective)? 4. How has support to early warning and response improved Early Warning and Response? 5. Is ECOWAS recognized by regional governments as a force for regional integration across countries, and sectors? What evidence exists? 6. Is EWD conflict prevention work relevant to the region in terms of advancing countries to become self￾reliant in preventing conflict? If Yes, how does the work of the ECOWAS/EWD currently contributes to national development of member states? EVALUATION METHOD AND LIMITS To answer these questions, the ET applied a qualitative research approach using content analysis of documents and interviews with 20 of 35 key informants identified by the USAID Contracting Officer’s Representative (COR). These informants include REWARD staff, middle-level staff and managers from the ECOWAS Directorates with whom REWARD has been working, and donors. The ET held key informant interviews (KIIs) between April 19 and May 17, 2019. Excerpts from the interviews and documents were coded and analyzed using a qualitative data analysis software package. MAIN CONCLUSIONS Overall, findings indicate that REWARD has brought about significant changes in the ECOWAS early warning system. These are taking root informally and are beginning to bridge the gap between data production and response. ECOWAS’s cumbersome organizational culture is hindering REWARD’s progress, but the technical solutions being implemented are mitigating the situation, to some extent. Questions 1 and 1.1: How has the regional conflict early warning system of ECOWAS changed or not since the inception of the USAID REWARD project? What were the best practices, lessons learned, successes, and failures, and why? 1. REWARD has changed the ECOWAS early warning system (EWS) by improving the quality of the data that the EWD collects and by beginning to mainstream gender considerations into its data collection, analysis and, to a lesser extent, reporting functions. The REWARD program strengthened our capacity so that we can do all these things that we are able to do today. In 2014, we were not the EW that we are today. The reference from everywhere, up to the UN, is ECOWARN, and I think it is all this capacity-building which have been given to us by REWARD that put us there. —Interview with ECOWAS representative, May 17, 2019 12 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 2. REWARD has begun to have a positive effect on quality of the analysis by providing a new analytical framework and by holding joint analysis and crisis-scenario planning workshops. The above has also resulted in improved quality of reporting. 3. Although not within the REWARD scope of work, National Centers for the Coordination of the Response Mechanism (NCCRMs) have begun to be established but they are not functional and have yet to improve the flow of information between EWD and member states. It is too early for REWARD to answer calls to extend its capacity building efforts to these structures. 4. The bureaucratic, hierarchical, and politicized nature of ECOWAS is hindering REWARD’s progress toward its objectives, but REWARD’s strategy of working with directors on technical solutions is mitigating these obstacles. 5. ECOWAS has not adopted standard work processes whereby EWD is systematically called upon to work with other specialized ECOWAS Directorates to formulate recommendations grounded in others' response practices. The changes that REWARD has begun to implement are not yet sustainable for lack of formalization in business working processes. 6. The business process re-engineering project under way, with support from the European Union, provides an opportunity for REWARD to institutionalize the new methods and modus operandi it has introduced. 7. It is too early to tell what the placement of EWD under the Vice President’s (V-P’s) Office will have on the project but there may be early signs that this reorganization may negatively affect REWARD. The placement of EWD at the V-P's Office and the idea of placing the EWD at the center of a network of NCCRMs are not fully implemented with clear roles, responsibilities and procedures. 8. REWARD’s decision to start its activities by first improving the quality of the data being collected as a precondition to deploying efforts to improve its analysis (through joint processes and other tools in the Systems Management Division (SMD) of EWD), is a best practice; by doing so, REWARD has improved the quality of EWD reports. 9. Working within existing structures (namely, the Early Conflict Prevention Framework Secretariat or (ECPFS)) reduced resistance to change and accelerated the uptake of inter-directorate collaboration. REWARD’s decision to bring early warning (EW) data producers and responders closer together through the implementation of Country Risk and Vulnerability Assessments (CRVAs) and later, under the auspices of the ECPFS, the Joint Analysis and Response Planning (JARP), Crisis Response Planning Exercise (CPX), and After-Action Review (AAR), is also a best practice. This decision has helped circumvent the relative dysfunctionality of the bureaucratic structures within the inter-directorate committees. Questions 2 and 2.1: How did gender-integration efforts implemented by REWARD contribute, if at all, to more gender-balanced early warning data collection and analysis? Do the EWD and CPAPS demonstrate the institutional and professional capacity, and the will to sustain the approach of integrating gender in its peace and security efforts? 1. REWARD has succeeded in mainstreaming gender in the collection of data; this is less visible in their analyses, and even less so in the reports EWD generated. 2. PAPS and EWD have the professional capacity to mainstream gender in data collection. Primarily because ECOWAS’s senior management has not yet approved REWARD’s contributions in this USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 13 regard, EWD and PAPS do not have the institutional capacity to mainstream a gender-balanced approach to their work. The will to do so is apparent at the technical level and REWARD’s contributions are likely to be sustainable in the near future. 3. The Directorate of Political Affairs (DPA) and the Directorate of Peacekeeping and Regional Security (DPKRS) have no specific direction in how to mainstream gender in conflict mediation, facilitation, and military interventions. 4. The absence of templates for reports prevents the generalization of gender mainstreaming in EWD products. Questions 3, 3.1 and 3.2: How has the USAID REWARD project contributed to improving EWD’s ability to collect early warning data and strengthen EWD’s ability to analyze, report, and generate recommendations? What specific capabilities and skills were increased and how have they been applied by EWD? Does EWD demonstrate the institutional and professional capacity and the will to sustain early warning data collection, analysis, reporting, and recommendations without the assistance of REWARD? 1. REWARD’s work at introducing the EWD to the CRVA methodology significantly improved the quality of the data EWD collects. The recent Human Security Analysis Framework (HSAF) REWARD collaboration with the EWD provided a standardized structure under which analysts can review data by thematic area, country, gender, etc. EWD’s decision to specialize and organize its staff along the same thematic areas as those contained in the HSAF is a sign that EWD has internalized its use. 2. REWARD’s efforts at building the capacity of EWD analysts and that of EWD’s SMD have improved the quality of the unit’s analysis and reporting. However, EWD’s long-standing practice of isolating itself from other directorates to analyze data continues to have negative effects on the quality of analysis and formulation of actionable recommendations for response. 3. The formulation of actionable recommendations by the units responsible for responding to data is still weak, as EWD does not systematically work with other directorates in crafting them. The JARP workshop has begun to break down this isolation and, thus, to have a positive effect on improving the quality of analysis (and recommendation formulation). The workshop on CPX that the ECPFS held (at REWARD’s instigation) with its 15 focal directorates across the organization has also started to improve the quality of the recommendations flowing from data analysis. 4. By leveraging the platform that the ECPFS offers, and through events such as the JARP and the CPX workshops, REWARD has helped bring data producers and users together to improve the quality of analysis and to create more actionable recommendations for those responsible for responding to early conflict situations. ECOWAS is just starting to get Early Warning System (EWS) data to inform response. JARP and CPX are, to date, isolated events rather than standard work processes. Thus, their institutional sustainability as a work method for analysis is not ensured; they are not formally documented and operationalized in business processes. 5. Even though EWD conducted the Regional Risk and Vulnerability Assessment (RRVA) in January 2019, data suggest that EWD does not yet have the professional capacity to sustain the results of REWARD’s efforts. 6. The organizational culture of ECOWAS and of PAPS’ senior leadership, coupled with the lack of formalization of the methods that REWARD has introduced, makes it unlikely that REWARD’s 14 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV efforts will be sustained. ECOWAS did not allocate funds to support the results that REWARD has begun to achieve. Questions 4, 4.1 and 4.2: How has the USAID REWARD project contributed, if at all, to improving CPAPS’s ability to carry out its mandate of early warning, conflict prevention, and mediation, especially by enhancing CPAPS’s internal and external communication and collaboration on early warning and response? What are the specific instances of CPAPS’s better internal and external communication and collaboration on early warning and response? Does CPAPS demonstrate the institutional and professional capacity and the will to sustain early warning, conflict prevention, and mediation without the assistance of REWARD? 1. REWARD’s leveraging of the platform offered by the PAPS’ ECPFS (through the JARP and CPX workshops) has been successful in enhancing PAPS’s communications and collaborations with the ECPFS’s 15 focal point directorates across all ECOWAS departments that had not, until 2018, made full use of the EWD data. 2. The September 2018 reorganization of PAPS, whereby EWD was moved to the Vice-President’s Office, may negatively affect the already limited communications that CPAPS has with other ECOWAS directorates but, given the interest of technicians in JARP and CPX, it is too early to tell what repercussions this repositioning of EWD will have on inter-directorate collaboration. 3. Whereas the former leadership of the Commissioner for PAPS (CPAPS) helped to break down ECOWAS’s culture of internal competition—which, in turn, helped to enhance PAPS’s collaboration within the department and across ECOWAS—the current Commissioner’s leadership style has not resulted in the institutionalization of inter-directorate collaborations. This has prevented REWARD from fully reaching its objective of improving internal and external collaborations across ECOWAS’s departments and directorates. 4. The business process reengineering project that the European Union (EU)’s Peace, Security, and Stability (PSS) is about to finalize in 2020-21 offers REWARD with an opportunity to formalize the joint work processes (JARP and CPX) with which REWARD has experimented. This would enhance the quality of analysis of data and promote the formulation of more actionable recommendations, all of which may potentially improve ECOWAS’s ability to better respond to emerging crises. Question 5: What internal and external factors prevented or supported Creative from accomplishing REWARD’s expected results? 1. REWARD’s leadership and collaborative engagement styles with EWD and PAPS, its ability to bring high-quality technical support to bear on important REWARD initiatives when needed, and the fact that REWARD responded to the needs that EWD had identified prior to the REWARD project’s start, have all contributed to REWARD’s accomplishment of its objectives. 2. ECOWAS management and work cultures have prevented REWARD from accomplishing its objectives more quickly. However, data from the performance indicators tracking table (PITT) indicate that REWARD has surpassed its targets on most outputs. The only exceptions relate to the continued poor perception that ECOWAS staff have of the quality of the EWD reports and the lower-than-planned performance on after-action review (AAR) recommendations. 3. Because REWARD pursued a deliberate strategy of working at the level of directors and staff, and because it deployed technical solutions leveraging the ECPFS and the ECPF platform, REWARD USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 15 succeeded in bringing together directorates responsible for the production of data and those responding to those data. 4. Prior to REWARD’s launch, there was no legally binding general framework of agreement between the USG and ECOWAS and this—considering EWD was the only contributor to the crafting of the REWARD scope of work—resulted in slow uptake of REWARD project activities with directorates responsible for responding to early warning data. After communication efforts that the REWARD and the former CPAPS deployed, uptake improved in 2018. Question 6: Which of the program design assumptions were no longer valid or changed during the activity implementation that might have affected the program implementation? 1. All of the assumptions contained in the REWARD Performance Monitoring and Evaluation Plan were (and are still) valid and remained unchanged since the beginning of REWARD. 2. Because the removal of EWD from PAPS may either undermine or enhance inter-directorate collaboration, Assumption 4 (“the on-going organizational restructuring in ECOWAS supports inter￾directorate coordination and integration of early warning and early response processes”) may no longer be valid. Question 7: Which activities of the REWARD TO should continue and why? Which should be modified and why? 1. The project performed well in light of its output indicators. Besides the need to continue strengthening the practice of joint analysis and response planning, more efforts are required to improve the quality and relevance of reports, and to implement AARs. 2. Activities in the TO pertaining to the Council of the Wise (CoW) and the Mediation and Facilitation Division (MFD) were not started due to circumstances outside of REWARD’s control. 3. There is a need to continue enhancing links between data and response. The ECPFS has been useful in this regard, in spite of its financial and human resource constraints. ECOWAS interviewees asked REWARD to continue with its capacity-building efforts to enable ECOWAS to continuously move through the full cycle of joint data collection, analysis, scenario and response planning, and AAR. 4. Having REWARD push down analytical capacity-building to the level of the NCCRMs may seem a logical step to many, but it is premature as these national-level institutions do not currently have necessary functionality. It is also not within the current scope of the REWARD project. Question 8: To what extent was REWARD complementary to Department of State (DOS) (or other donor) support to ECOWAS and EWD? 1. USAID and DOS efforts are complementary within REWARD. 2. Weak operational and strategic coordination mechanisms between development partners are apparent. USAID (not REWARD) does not regularly attend donor meetings. In addition, ECOWAS senior leadership is not taking an active role in the coordination of donors in support of linking data to response. As a result, there is less-than-optimal information sharing and synergy between development partners and ECOWAS. This negatively affects their ability to efficiently and effectively leverage each other’s efforts to effect transformational change, improve links between data and response, or enhance organizational learning. 16 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV CONCLUSIONS FOR ADDITIONAL STRATEGIC QUESTIONS The documents that USAID/WA shared with the ET were of an operational nature. They did not contain information that provided answers to the additional questions, more strategic in nature. In addition, the ECOWAS staff identified by USAID/WA for KIIs were largely middle-level officers and technicians. The more senior-level directors interviewed during the evaluation did not have access to the types of authoritative information that is needed to answer the additional questions. MAIN RECOMMENDATIONS Based on the conclusions drawn from its findings, the ET recommends the following: 1. REWARD should consolidate the gains it made through ECPFS with the JARP and CPX processes and should continue leveraging the ECPFS platform as a means to convene hitherto siloed directorates. 2. REWARD should coordinate with the EU-PSS project more closely to ensure that the new joint data collection, analysis and response-planning methods and events it has introduced (CRVA/HSAF, JARP, CPX, and AAR) become part of the ECOWAS’s reengineered “business processes.” 3. Rather than train staff from NCCRMs, REWARD should ensure that events (such as JARP, CPX, and AAR) and methodologies like the CRVA/HSAF become standardized and regularly applied in the organization’s business processes through, among other tools, standard operating procedures (SOPs). Eventually, in a second phase of REWARD, capacity building of NCCRMs could be done by training EWD trainers in data analysis. 4. REWARD should not rely on seeing soon the formalization of SOPs for inter-directorate collaboration, but rather focus its attention on leveraging ECPFS through the JARP and CPX processes to bring directorates together. To arrive at SOPs that eventually institutionalize these work processes, REWARD should leverage the EU-PSS business process reengineering project. 5. REWARD needs to pursue and deepen its support to the ECPFS while recognizing its limitations. It could do so by using this mechanism more regularly to support inter-directorate collaboration in analysis and response planning, and by coordinating its financial and technical support to the ECPFS with that of the EU. 6. USAID should undertake efforts to have ECOWAS approve the gender manual so that REWARD can deepen the training of the SMD staff, the EWD analysts, and other directorates to better integrate gender considerations into their responses. REWARD should proceed as quickly as possible with the preparation of guidelines/templates for integrating gender into thematic reports. 7. REWARD should continue on its current course to build the capacity of ECOWAS headquarters staff to move through the full circuit of joint data collection, analysis, reporting, scenario response planning and after-action review. To enhance inter-directorate collaboration, REWARD needs to encourage the EWD to continue its efforts at joint analysis and recommendation formulation. 8. Because EWD is no longer a part of the PAPS structure, USAID should amend REWARD’s TO to reflect this change. 9. To appeal to ECOWAS departments and directorates, EWD should produce at least two types of reports: thematic and by country. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 17 10. If USAID decides to pursue a future REWARD-like project, it should ensure that it can frame ECOWAS’s commitments with a legal document that will spell out the mutual expectations of all parties involved. 11. REWARD should maintain the presence of an embedded advisor. 12. In light of EWD’s move to the V-P’s Office, USAID’s Regional Peace and Governance Office should track how Assumption 4 may or may not change and assess its effect on REWARD’s ability to meet its mandated objectives. Modify REWARD’s TO accordingly. 13. Modify or cancel the activities pertaining to the CoW and the MFD. 14. Modify outputs 1.3 and 1.4 to target the ECPFS more specifically, rather than DPA and PAPS only. 15. To bolster the relevance and uptake of the EWD reports, REWARD should support EWD in enhancing the quality of regional-thematic reports as well as producing country-level thematic reports. REWARD could do so by training EWD and other analysts (SMD and those of other directorates) on applying a HSAF lens to specific countries while insuring their conceptual linkages to, and consistency with, regional trends and thematic reports. 16. At the end of Year 4, hold a retreat with all stakeholders to conduct an AAR of REWARD before designing the next phase, should there be one. 17. Extend the project for another year to leverage the work that the EU-PSS is about to start in reengineering ECOWAS’s business processes in order to ensure that the JARP, CPX, and AAR work processes that REWARD has begun to establish in ECOWAS are institutionalized in the organization’s headquarters. 18. REWARD should not extend its capacity-building efforts to the NCCRMs during this phase of the project; rather, it should concentrate on building the EWD’s capacity to train their NCCRM counterparts. 19. While the aforementioned recommendation is being implemented, USAID should devise a future (post-2020) project of a REWARD-like nature (perhaps implemented by USAID’s country-level Missions and coordinated by USAID/WA) to extend capacity-building efforts to the NCCRMs if they are functional and supported by the member states. 20. USAID should work with development partners to establish the planned PAPS Management Cell as soon as possible. If need be, use part of the REWARD funding to pay part of that person’s emolument. 21. Change the agenda of the monthly donor thematic working group meetings to include workplan sharing as a standing item. 22. Distribute relevant reports to development partners to improve information flow and leverage the work of Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) in organizational development to improve organizational efficiency. This recommendation also refers to this evaluation report. 23. USAID/WA should attend donor meetings more frequently to coordinate high-level decisions and empower REWARD to adjust its work plans without jeopardizing its responsibilities under its TO. 18 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 24. USAID and REWARD should identify and work with a member of ECOWAS’s senior leadership, who will serve to champion the changes that REWARD is bringing forward. 25. Distribute relevant operational and strategic reports to development partners to improve information flow and leverage the work of other donors working in ECOWAS to improve organizational efficiency. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 19 INTRODUCTION PROJECT BACKGROUND On September 25, 2015, the United States Agency for International Development/West Africa (USAID/WA) awarded Creative Associates International (Creative) the Task Order (TO) AID-624-TO￾15-00004 (under Indefinite Quantity Contract AID-OAA-I-13-00005) to cover a four-and-a-half-year period (2015-2020) to implement its Reacting to Early Warning and Response Data (REWARD) program in West Africa. USAID anticipated an investment of $11.5 million in REWARD.1 REWARD is part of a broader U.S. Government (USG) initiative, the Early Warning and Response Partnership (EWARP), which was announced at the U.S.-Africa Leaders’ Summit in August 2014. The objective of EWARP is to bolster the capacity of the Economic Community of West African States (ECOWAS) Warning and Response Network (ECOWARN) to monitor, gather, analyze, and disseminate threat information to its 15 member states to support peace and security in the region. Within the EWARP framework, REWARD is one activity under a broader USAID/WA project entitled Peace through Elections (P/Elect). REWARD activities are to contribute primarily to Intermediate Result (IR) 1.3 and to IR 1.2 of the USAID/WA’s Regional Development Cooperation Strategy Development Objective (DO) 1.2 REWARD addresses critical gaps in conflict early warning and response in the 15 member-countries of ECOWAS. The program provides technical assistance to reinforce the capacity of the ECOWAS Commission’s Department for Political Affairs, Peace, and Security (PAPS)3 to collect and analyze conflict￾risk data, develop strong recommendations, and improve the sharing of this information within ECOWAS, its member states, and other relevant actors. Specifically, REWARD strengthens the capacity of the PAPS4 Early Warning Directorate (EWD) to produce early warning recommendations, which appropriately reflect regional policies, best practices, and socio-cultural conflict dynamics. REWARD also builds the capacity of the CPAPS5 Directorate of Political Affairs (DPA) to act on these recommendations in a timely, gender-sensitive, and politically appropriate manner to mitigate conflict through strengthening internal systems and tools. The program features a small grants mechanism to enable civil society groups to act on appropriate early warning recommendations regarding electoral security by engaging in national and community-level peace initiatives. This element requires close collaboration with the West Africa Network for Peacebuilding (WANEP) in four ECOWAS countries. Similarly, close coordination with the Department of State (DOS), its implementing partner(s), and other donors active in peace and security in West Africa are required for the successful implementation of this program.6 REWARD has two Purposes as illustrated in Figure 1 below. This evaluation focuses on Purpose 1. The outputs of Purpose 1 are:  Output 1: EWD strengthened to collect early warning data; 1 Of this amount, $9.0 million was to be dedicated to Purpose 1 and $2.5 million was destined to Purpose 2. 2 IR 1.2: Targeted government institutions strengthened to engage citizens on contentious issues; IR 1.3: Ability of institutions operating at the regional level strengthened to respond to governance challenges. 3 The Directorate of Political Affairs, Peace and Security is referred to only as PAPS, CPAPS refers only to the Commissioner of PAPS. 4 Ibid. 5 Ibid. 6 REWARD Performance Monitoring and Evaluation Plan, April 2019. 20 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV  Output 2: EWD capacity to analyze, report, and generate recommendations improved;  Output 3: Enhanced Department of Political Affairs (DPA) response capacity;  Output 4: Enhanced CPAPS internal and external communication and collaboration on early warning and response. Figure 1: REWARD Results Model USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 21 ECOWAS STRUCTURE REWARD has worked with PAPS, which is structured as depicted below and reports to a Commissioner (CPAPS). Figure 2: Structure of PAPS Department Prior to Reorganization of EWD to the Vice-President’s Office in September 2018 Figure 2 shows the PAPS structure before the EWD moved to the Vice-President’s Office in September of 2018. The dotted line at the bottom of the diagram shows the strategic placement of the Early Conflict Prevention Framework Secretariat (ECPFS) within PAPS; it is designed to be the mechanism through which the three PAPS directorates (and those of other Departments) coordinate their activities within the framework of the ECOWAS ECPF of 2008. In this organizational structure, the EWD is responsible for collecting data, analyzing it, and formulating analysis-based recommendations for the consumption of the DPA. This unit may decide to engage in preventive diplomacy, mediation, and facilitation and, if these efforts fail, recommend a stronger (military) intervention to be managed by the Directorate of Peacekeeping and Regional Security (DPKRS). EVALUATION PURPOSE AND OBJECTIVES On February 20, 2019, USAID/WA’s Regional Peace and Governance Office (RPGO) tasked the Peace through Evaluation, Learning, and Adapting (PELA)7 activity to carry out a performance evaluation of the REWARD activity. The evaluation began April 1, 2019, and the field work was concluded in May 2019. Per the Inception Report (Annex 1), the goals of this evaluation are to:  Determine if the project (REWARD) is well-positioned to achieve success in strengthening ECOWAS early warning and response systems that ultimately contribute to reducing the risk of violence in West Africa;  Determine if assistance to ECOWAS/PAPS/EWD is relevant to the sub-region in terms of advancing self-reliance from the regional to national level; what efforts were/are effective or not, 7 PELA is implemented by International Business & Technical Consultants, Inc. (IBTCI). 22 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV and how should assistance be modified to better reach USAID strategic objectives for the region based on the most recent strategic priorities of the USG. To meet these goals, this performance evaluation sought to achieve the following objectives:  Determine the effectiveness and relevance of the approaches used to meet activity objectives and document any lessons learned, best practices, and challenges to inform future programming.  Identify any internal and external factors that affected the implementation of the activity to date and the changes that occurred within ECOWAS that might have impacted implementation.  Based on the findings, propose any recommendations for maintaining or changing the course of action to help inform the remaining years of REWARD and future USAID programming around regional and national conflict prevention and early warning efforts.  Document any successes, best practices, lessons learned, and challenges encountered in the activity. EVALUATION QUESTIONS The REWARD performance evaluation addresses the following questions. Evaluation Questions Question Sub-Question EQ 1: How has the regional conflict early warning system of ECOWAS changed or not since the inception of the USAID REWARD project? EQ 1.1: What were the best practices, lessons learned, successes and failures and why? EQ 2: How did gender integration efforts implemented by REWARD contribute, if at all, to a more gender-balanced Early Warning data collection and analysis? EQ 2.1: Do the Early Warning Directorate and the CPAPS demonstrate the institutional and professional capacity and will to sustain the approach of integrating gender in its peace and security efforts? EQ 3: How has the USAID REWARD project contributed to improving EWD’s ability to collect Early Warning data and strengthen EWD’s ability to analyze, report, and generate recommendations? EQ 3.1: What specific capabilities and skills were increased and how have they been applied by EWD? EQ 3.2: Does EWD demonstrate the institutional and professional capacity and the will to sustain early warning data collection, analysis, reporting, and recommendations without the assistance of REWARD? EQ 4: How has the USAID REWARD project contributed, if at all, to improving CPAPS ability to carry out its mandate of early warning, conflict prevention, and mediation, especially by enhancing CPAPS’ internal and external communication and collaboration on early warning and response? EQ 4.1: What are the specific instances of CPAPS’ better internal and external communication and collaboration on early warning response? EQ 4.2: Does CPAPS demonstrate the institutional and professional capacity and the will to sustain early warning, conflict prevention, and mediation without the assistance of REWARD? EQ 5: What internal and external factors prevented or support Creative from accomplishing the REWARD expected results? USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 23 Evaluation Questions EQ 6: Which of the program design assumptions where no longer valid or changed during the activity implementation that might have affected the program implementation? EQ7: Which activities on the REWARD Task Oder should continue and why? Which should be modified and Why? EQ 8: To what extent was REWARD complementary to the State Department (or other) support to ECOWAS and EWD? The SOW included questions of a more strategic nature. As stated in the Inception Report, 8 these fall outside the scope of a performance evaluation of REWARD. These questions ask about ECOWAS (or components thereof) more generally, and about how the USG’s support (including, but not limited to REWARD) of ECOWAS has improved this organization’s work. The Inception Report pointed out that although the strategic evaluation questions (EQs 9-15) would not be part of the evaluation itself, the ET could ask these questions of specific KIIs that USAID identified9 to help guide future strategic-level evaluations of ECOWAS. The strategic questions posed in the SOW are as follows:  EQ 9: How many and what type of recommendations generated by the EWD early warning work have translated into action by the ECOWAS leadership or member states?  EQ 10: What role, if any, would the ECOWAS early warning system/efforts play in meeting USG’s current and evolving objectives, particularly, in assisting countries to achieve self-reliance? Should USAID continue to invest in these organizations and why?  EQ 11: Has USAID’s investment in ECOWAS led to national level impact, if any (though not part of REWARD’s objective)?  EQ 12: How has support to early warning and response improved Early Warning and Response?  EQ 13: Is ECOWAS recognized by regional governments as a force for regional integration across countries, and sectors? What evidence exists?  EQ 14: Is EWD conflict prevention work relevant to the region in terms of advancing countries to become self-reliant in preventing conflict? o If Yes, How does the work of the ECOWAS/EWD currently contributes to national development of member states? o If Yes, should USAID be investing in these organizations? If yes, why? o If No, why not?  EQ 15: How sustainable is ECOWAS capacity and level of resources? 8 See Annex 1. The Inception Report that USAID approved includes: a discussion of USAID’s purpose and objectives for the evaluation; a review and a revision of the evaluation questions posed by USAID in the SOW; an evaluation design that includes methods, data sources, data collection strategy, and notional analysis framework to analyze the evaluation questions; a recommendations framework for next steps USAID may consider for REWARD; a timeline for the evaluation; details of the evaluation team; and logistics and operations considerations. 9 USAID identified respondents that were not able to answer these questions because they were not in position to do so. Also, the ET was not mandated to interview senior officials of ECOWAS, its member states, USAID or DOS. 24 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV Evaluation Questions Question Sub-Question EQ 1: How has the regional conflict early warning system of ECOWAS changed or not since the inception of the USAID REWARD project? EQ 1.1: What were the best practices, lessons learned, successes and failures and why? EQ 2: How did gender integration efforts implemented by REWARD contribute, if at all, to a more gender-balanced Early Warning data collection and analysis? EQ 2.1: Do the Early Warning Directorate and the CPAPS demonstrate the institutional and professional capacity and will to sustain the approach of integrating gender in its peace and security efforts? EQ 3: How has the USAID REWARD project contributed to improving EWD’s ability to collect Early Warning data and strengthen EWD’s ability to analyze, report, and generate recommendations? EQ 3.1: What specific capabilities and skills were increased and how have they been applied by EWD? EQ 3.2: Does EWD demonstrate the institutional and professional capacity and the will to sustain early warning data collection, analysis, reporting, and recommendations without the assistance of REWARD? EQ 4: How has the USAID REWARD project contributed, if at all, to improving CPAPS ability to carry out its mandate of early warning, conflict prevention, and mediation, especially by enhancing CPAPS’ internal and external communication and collaboration on early warning and response? EQ 4.1: What are the specific instances of CPAPS’ better internal and external communication and collaboration on early warning response? EQ 4.2: Does CPAPS demonstrate the institutional and professional capacity and the will to sustain early warning, conflict prevention, and mediation without the assistance of REWARD? EQ 5: What internal and external factors prevented or support Creative from accomplishing the REWARD expected results? EQ 6: Which of the program design assumptions where no longer valid or changed during the activity implementation that might have affected the program implementation? EQ7: Which activities on the REWARD Task Oder should continue and why? Which should be modified and Why? EQ 8: To what extent was REWARD complementary to the State Department (or other) support to ECOWAS and EWD? Based on the evaluation’s Scope of Work (SOW) delivered to PELA by RPGO, PELA drafted an Inception Report which served as a basis for the crafting of the evaluation’s technical instruments (coding matrix and interview guides (Annex 5 and 4, respectively)). EVALUATION DESIGN, METHODS AND LIMITATIONS DESIGN This is a “before-and-after” type of evaluation design. It documents findings on the current situation in the early warning system (EWS) in light of REWARD’s TO and a baseline that was partially recreated by the ET through interviews (see description below). The approach for this performance evaluation flows from USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 25 the evaluation questions and the nature of the program itself. Since the “beneficiary” in this program is an intergovernmental body rather than a group of people, survey data are not appropriate. Instead, the REWARD outcomes and the evaluation questions are best answered through document review and key informant interviews (KIIs), to include group interviews. The majority of the data gathered are therefore qualitative and descriptive in nature. PELA worked with USAID and REWARD to detail the exact documents that needed to be reviewed and to determine the KII roster. The documents reviewed included REWARD program documents (i.e., quarterly and annual reports, monitoring and evaluation reports) and ECOWAS documents from EWD and PAPS. KIIs involved personnel that USAID identified from ECOWAS, USAID, REWARD implementing partners (Creative Associates and its subcontractors, Fund for Peace and the Karuna Center for Peacebuilding), and the DOS. 10 METHODOLOGY The evaluation uses a qualitative approach of content analysis of project-related documents11 and interviews with 20 of the 35 key informants whom USAID identified as sources of information. The ET12 collected the data from field interviews held between April 29 and May 17, 2019, in Abuja, Nigeria. The ET conducted KIIs with representatives of the ECOWAS DPA, the DPKRS, the EWD, the ECPFS, and the former Commissioner’s office, as well as with different development partners and stakeholders. 13 The ET Team Lead designed the KII guides14 and a preliminary deductive coding matrix15 with input from team members and PELA. Assisted by team members, the Team Leader conducted all interviews in English or in French, depending on the language chosen by the interviewee. The KIIs were recorded and notes were taken during the interviews. Referring to the recordings, all team members completed and double￾checked interview notes for completeness and integrity prior to coding them. The Team Leader provided quality assurance of the notes and their preliminary deductive coding using the NVivo 12 software. Once this was done, coded notes and excerpts were uploaded in a shared-drive folder with access restricted to the ET and PELA. The excerpts were compiled and analyzed applying an inductive coding approach. LIMITATIONS This evaluation has methodological limitations, but the ET applied mitigation measures to reduce their potential to bias the findings. Firstly, the ET did not independently pick the key informants to interview. This introduces a selection bias. To reduce the potential for this type of bias, the ET cross-checked the answers it got to its interview questions by (1) asking the same questions to different sources (interviewees and documents) and (2) phrasing the questions in different ways. Secondly, the ET did not interview all key informants USAID initially identified; some ECOWAS staff never responded to email and phone requests for interviews, others were on leave, and still others were excluded from interviews because of their own statement that they had no knowledge about the project. This situation introduces another selection bias (self-selection), whereby there could have been a misrepresentation of opinions— overly favorable or unfavorable—based on self-selection to participate in the evaluation data collection. Again, the ET applied information triangulation to reduce the potential for a self-selection bias by seeking 10 See Annex 3. 11 See Annex 2 for the bibliography of documents consulted. 12 The ET was composed of four members: a Team Leader and senior methodologist; a Subject Matter Specialist; an Analyst; and a Logistician. 13 The list of interviewees is shown in Annex 3. The representative from DOS did not meet with the ET as that person was unsure of what questions would be addressed with her. 14 See Annex 4. These were shared with USAID on May 2, 2019 and were approved. 15 See Annex 5 for the preliminary coding matrix. 26 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV a variety of informed interviewees and by relying on complementary document sources to verify data. Thirdly, the ET faced a potential for confirmation bias, whereby respondents may tell the interviewers “what they think they want to hear.” To mitigate this bias, the ET relied on triangulating sources of information and different lines of questioning and document-based fact-checking. A baseline for the REWARD organizational capacity building project of ECOWAS was not established using a standardized diagnostic tool.16 Thus, there were no verifiable pre-intervention measurements of EWD’s and other directorates’ behaviors in producing and leveraging early warning data to respond to emerging conflict situations. Therefore, although REWARD completed a diagnostic of the situation, there were no specific, verifiable, planned behavior targets with which to compare the current situation. To mitigate the impact of this limitation on the evaluation’s findings, the ET asked respondents to compare their recollection of the baseline situation with the current one and used the values written into REWARD’s performance indicator tracking table (PITT) as a baseline. Some findings are based on the insights of one or two persons only (this is due to an overall limited number of KIIs and reports). Therefore, they could not be properly triangulated. To mitigate this limit on the findings, the ET referenced the sources of all findings and reported upon them in a transparent fashion. Such findings were reported upon, however, because they came from influential and insightful staff (for instance, managers or more senior technical staff). Finally, the ET could not answer the evaluation’s strategic questions (EQ 9-15) because the documents provided did not contain relevant information (they were operational in nature) and the staff interviewed were not positioned at strategic levels within ECOWAS; also, the ET was not mandated or provided resources to interview key informants in the ECOWAS member states. CONCLUSIONS, FINDINGS AND RECOMMENDATIONS EVALUATION QUESTION 1 How has the regional conflict early warning system of ECOWAS changed, or not, since the inception of the USAID REWARD project? Evaluation Question 1.1: What were the best practices, lessons learned, success, and failures, and why? Evaluation Questions 1 and 1.1 are catch-all questions that encompass significant elements of all other evaluation questions. Therefore, many details of the changes or absence thereof, as well as the reasons for these, are discussed under other relevant sections of this report. In this section of the report, the focus is on major system-level changes in the EWS and the ECOWAS/USAID relationship. FINDINGS CHANGES IN EWS ATTRIBUTABLE TO REWARD REWARD contributed to changes in data collection, analysis, and response planning. Interviewees and project documents corroborate that the EWS has changed since 2014–15 due to REWARD’s methodological and work-process contributions. The project introduced the following methodologies to the EWD: the Country Risk and Vulnerability Assessments (CRVA), the Human Security Analysis Framework (HSAF), and gender mainstreaming. It has also introduced to the DPA and/or its ECPFS the after-action review (AAR), the Crisis Response Planning Exercise (CPX), and Joint Analytic and 16 USAID tools include: the Organizational Performance Index, the Organizational Capacity Assessment or the Human and Institutional Capacity Development Index. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 27 Response Planning (JARP). 17 According to one ECOWAS interviewee, the CRVA and its accompanying capacity-building are, to date, the most important lasting contributions that REWARD has made to change the EWS.18 Through the CRVAs and HSAF, the EWS has reported a significant improvement in the quality and comparability of the data across countries. This, alone, has improved the performance of the EWS. REWARD improved the quality of EWD reports. Through an enhanced data-collection framework (CRVA), the training of Field Monitors (FMs) and the training of EWD analysts in the HSAF, REWARD has changed the reports of the EWS. The quality of reporting has also improved because of the use of dynamic data-visualization techniques that REWARD introduced in the EWD’s Systems Management Division (SMD). 19 REWARD introduced new tools to the Systems Management Division (SMD). REWARD introduced new tools that enhanced the SMD’s ability to analyze, 20 visualize, and avoid duplication of data points.21 Data from interviews and documents confirm that REWARD has enabled a “rapprochement”22 between the SMD and the analysts of EWD. This is reflected in the improved use that EWD analysts make of data-visualization techniques to depict situations. Five of our respondents who addressed this issue commented that the quality of these reports has gradually improved and is “moving in crescendo.”23 REWARD improved linkages between data producers and users, but more needs to be done. REWARD co-financed, with the Danish International Development Agency (DANIDA), the ECPFS Retreat of Focal Points from January 20–31, 2019. It is an activity that saw a strong participation of the focal points and allowed us to link the [ECPF] 2018- 2020 Action Plans that we have just developed and also to see how we can turn this into projects with a focus on risk-taking, or how to deal with risks in the management of our projects, and how to prepare for risks. Emphasis was also placed on project management based on results. So, I think that we are achieving the objectives of creating synergy between the focal points, putting them together so that they work together because that is what was missing. How to coordinate our actions, how to work in synergy, how to have a platform where we share what each other do. So, in relation to these objectives, we have achieved them. In 2018, we had done the CRVA for all the member states and only Togo was missing. However, the link between EW and response should be created. This link is still missing.24 This last sentence is critical, because despite the CPX and JARP, it depicts a state of affairs that is still prevalent in ECOWAS in terms of REWARD’s objectives as shown in Figure 1. This is perhaps due to the fact that the key informants interviewed do not have a sense of the progress in relation to REWARD Purpose 2, which is more focused on the national response capacity, and is outside the scope of this evaluation. However, given the wording of Outputs 1.3 and 1.4, this perception may still be valid for those in the regional confines of the ECOWAS headquarters. 17 The AAR was introduced in Year 1, the CPX in Year 2 and the JARP, although planned for Year 4, was trialed in Year 3. 18 Interview, ECOWAS representative, May 3, 2019. 19 Interview, ECOWAS representative, May 3, 2019. 20 REWARD_AID-624-TO-15-00004_FY18-Annual report. 21 Interview, ECOWAS representative, May 8, 2019. 22 Phone Interview, REWARD representative, May 7, 2019. 23 Interview, ECOWAS representative, May 8, 2019; Interview, ECOWAS representative, May 2, 2019; Interview, ECOWAS representative, May 6, 2019; Interview, REWARD representative, May 9, 2019; Interview, ECOWAS representative, May 3, 2019. 24 Interview, ECOWAS representative, May 13, 2019. 28 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV REWARD helped close the gap between EWD and the 15 focal point directorates of ECPFS. REWARD has leveraged existing structures to bring about a “rapprochement” between all the directorates and to bring those units responsible for response together more closely with EWD.25 REWARD recognized the importance of working within existing structures and adjusted itself after a notable failure —it tried to establish a project-specific Inter-Directorate Working Group (IDWG), but this failed through a lack of interest. Invited parties cited that they did not understand how this could contribute to the work of their directorates. Rather, “working within the established mechanism of ECPF has brought success.”26 REWARD contributed to gender mainstreaming in the EWS. REWARD systematically incorporated a gender-sensitive lens into the data collection and, to a lesser extent, analysis sub-systems. The validation of the gender manual began in November 2018 and awaits formal approval from both ECOWAS and USAID. Notwithstanding this last step, ECOWAS has been applying it informally, which is discussed in detail in Evaluation Question 2. CHANGES IN EWS ATTRIBUTABLE TO ECOWAS OR OTHERS National Centers for the Coordination of the Response Mechanism (NCCRM) lack functionality.27 Under the DOS component (Purpose 2) of the Peace through Elections (P/Elect) project, national early warning centers have been established in five of countries: Mali, Burkina Faso, Côte d’Ivoire, Liberia, and Senegal. All interviewees questioned on this issue confirm that these NCCRMs are not yet operational.28 At the time of data collection for this evaluation, five others were being established with the support of German and European Union (EU) donors. Other ECOWAS partners are showing interest in having the last five be built.29 A more-effective ECPFS has an ECPF plan of action. With the support of EU findings, the ECPFS has published its ECPF Plan of Action 2018–2020. This plan lays out 15 main components, one of which is related to early warning . Since 2015, ECPFS has established focal points in the various ECOWAS directorates to better leverage EWD data into actionable responses.30 Institutionalizing inter-directorate work processes has begun. GIZ and the EU-PSS projects aim various aspects of organizational development of the units involved in the security portfolio (PAPS and those interacting with each other in the framework of the ECPF coordination mechanism). These donors’ projects have begun positively affecting ECOWAS’s business processes. For instance, these projects have spurred the (soon-to-be) establishment of a Management Coordination Cell reporting to CPAPS which may affect the relationship between PAPS directorates and EWD. Similarly, the EU-PSS business process reengineering efforts are just beginning to affect ECOWAS’s information management / information technology systems in the peace and security sector. Similarly, GIZ’s attempts at putting in place standard operating procedures (SOPs) to formalize the working (existing 25 Ibid. 26 Interview, REWARD representative, May 1, 2019. 27 Although this falls outside the scope of REWARD and of this evaluation, we report on this finding because, as will be seen later, it has a bearing on REWARD’s current and future orientation. 28 Interview, REWARD representative, May 9, 2019; Interview, REWARD representative, May 13, 2019; Interview, ECOWAS representative, May 17, 2019. 29 Interview, ECOWAS representative, May 17, 2019. 30 Interview, Development partner, May 14, 2019. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 29 or to-be-established) relationship between directorates are the beginning of the institutionalization of work processes. The move of EWD to the Vice-President’s Office may impact REWARD. The move of EWD from PAPS to the Vice-President’s Office is a major institutional change. It emanated from an argument which the Director of EWD formulated to elevate the data-collection function to a higher hierarchical level within ECOWAS, with the aim to render EWD more transversal and enable it to feed data across ECOWAS directorates (i.e., both inside and outside of PAPS). This may impact REWARD’s ability to effect change, because EWD attributions with ECOWAS are not formalized. It was easier when EWD was under PAPS because the institutional agreement was there. So, even now we are at the moment navigating how we are going to look at, how to operationalize the new structure. Because even though the move had already been approved by the Political Affairs, technically, we are still seeing that it has not been fully implemented. They know they are under the VP but in terms of relationship between the VP’s office and the PAPS, we have not seen any document and the move also was not informed by any empirical analysis that was conducted before.31 EWS CHANGES THAT DID NOT TAKE PLACE Institutionalization of organizational developments is unchanged. At the level of ECOWAS, organizational development and improvement have not been institutionalized. For instance, the EWD 2009 SOPs have not been renewed to take into consideration the new methods and work processes REWARD introduced, and to cover the existence of the NCCRMs. The same applies to the DPA SOPs ; despite REWARD’s assistance, the DPA has not been able to implement the SOPs developed under REWARD.32 ECOWAS member states’ exchange of information is unchanged. For reasons beyond the influence and scope of action of REWARD, the level of exchange between ECOWAS and member states around data collection and analysis is still unchanged. This may impact negatively on the quality of the EWD data and on the capacity of member states to craft and implement appropriate responses to lower-intensity national conflicts. Indeed, the DOS-supported NCCRMs are not yet fully functional33 because their analysts do not have the same capacity as the ECOWAS EWD—and because the staff are not getting paid, as there is no legal document which clearly identifies the responsibility of each party (DOS, other donors, national government) to pay for staff. Competing roles and responsibilities exist in analysis and recommendation formulation. ECOWAS is not taking the lead in clarifying the roles and responsibilities in the analysis of data and the formulation of recommendations. Changes in the work processes that the JARP and CPX imply are not being captured in systems and standard operating procedures (SOPs). 34 No update or institutionalization of SOPs for EWD, DPA, and PAPS has taken place. There has been no update to the EWD 2009 SOPs, and the unit’s move to the Vice-President’s Office is not formally reflected in a similar document (or under the EU-PSS’s business process reengineering 31 Interview, REWARD representative, May 13, 2019. 32 Interview, ECOWAS representative, May 17, 2019; Interview, REWARD representative, May 13, 2019. 33 Interview, ECOWAS representative, May 17, 2019. 34 Although there are specific quotes supporting this finding, it an observation that the ET noted and it is supported, inter alia, by our May 1, 2019 interview with a REWARD representative. 30 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV project, which has been slow to take off because of ECOWAS).35 Similarly, even though REWARD produced SOPs for the DPA, these have not been implemented nor do they reflect in detail how EWD and the DPA need to interact (especially in light of the unit’s move). Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) also prepared SOPs for PAPS, but ECOWAS has not yet implemented them.36 BEST PRACTICES Starting with high-quality data before working on analysis, reporting, and recommendation formulation. As an entry point to enhancing the SMD with analytical capacity, REWARD worked first on building up the quality of the data being collected. This was a first step, before analysis and reporting, to culminate in improving the ECOWAS capacity to respond to early warning data. As a result of data quality improvements, REWARD managed to bridge the gap between data producers (EWD) and users (other directorates within ECOWAS). For the first time, we were able to work together in close collaboration, so they could see what we do and how the info we provide to them is based on empirical data and this is starting to make sense to them.37 I am glad that REWARD has facilitated these conversations through the crisis scenario planning activity. There was a prior workshop that was an attempt to have these conversations that wasn’t very successful, but the practical simulation was very effective and very good.38 This strategy allowed EWD analysts to gradually make more extensive use of visual aids to display their analyses. REWARD is the only project working in this specific area of capacity-building. Changes brought to the system in this area are therefore attributable to REWARD. This strategy of improving the collection of data before attempting to improve the quality of analysis and later, reporting, was sound and constitutes a “best practice.” Working within existing ECOWAS structures. REWARD has learned from experience and has followed a deliberate strategy to ensure its institutional sustainability. After taking note of the lackluster success of various inter-directorate committees that PAPS (and REWARD) attempted to enhance collaboration between ECOWAS directorates inside and outside PAPS, REWARD then leveraged the ECPFS that ECOWAS established in 2010. This is the PAPS unit responsible for the coordinated preparation of the compendium of the 2018–2020 Plans of Action (for 15 directorates) for the implementation of the ECPF. The ECPFS is also a convening platform, even though it has staff, capacity, and resource limitations. Working with ECPFS can be seen as a best practice. One reason evoked for this being a best practice is it circumvented ECOWAS’s bureaucratic culture that does not always respond well to donor attempts to set up structures that are project-specific.39 In short, working through an existing structure made REWARD’s efforts more palatable for ECOWAS.40 The Joint Analysis and Response Planning has had the greatest obstacles. We were working closely with ECOWAS ECPF Secretariat and the EWD to try and make sure that REWARD is working within existing structure and not trying to overlay a new structure on to what exists.41 35 Interview, Donor partner, May 14, 2019. 36 Interview, Development partner, May 9, 2019. 37 Ibid. 38 Ibid. 39 Interview, Donor partner, May 8, 2019. 40 Interview, REWARD representative, May 1, 2019. 41 Interview, REWARD representative, May 1, 2019. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 31 Leveraging an approach that brings people together informally through workshops to change the system. When REWARD concluded that trying to improve inter-directorate collaboration through committees did not work, the project leadership decided to create demand for this by: (1) involving key stakeholders in CRVA data collection; (2) experimenting with JARP; (3) involving ECPFS focal point directorates in the CPX; and (4) workshopping AAR with the DPA. In short, they led staff through a series of experiential learning exercises that demonstrated the benefits of working across directorate lines. CONCLUSIONS CHANGES IN EWS ATTRIBUTABLE TO REWARD  At the level of the EWS, REWARD’s CRVA, the HSAF, AAR as well as the JARP workshop and the CPX are being informally42 used to promote change in inter-directorate collaboration, increased rigor and transparency in the analysis of data, and the crafting of responses.  As discussed in more detail in Evaluation Question 2, REWARD has contributed greatly to gender mainstreaming within the EWS. However, tardy approval of the gender-mainstreaming manual by the President of ECOWAS and USAID is slowing down its formal application. CHANGES IN EWS ATTRIBUTABLE TO ECOWAS OR OTHERS  ECOWAS, with support from DOS and other donors, has begun to establish NCCRMs and five remain to be implanted.  Donors (mainly the EU) have supported the ECPF, and its Secretariat, in becoming bridging mechanisms for inter-directorate collaboration (with REWARD co-financing) and have supported the development of a series of ECPF sectoral Action Plans, with the intent of increasing inter￾directorate connectedness of conflict prevention measures.  It is too early to tell what the placement of EWD under the Vice-President’s Office will have on the project but, as will be seen under Evaluation Question 4, there are early signs that this reorganization may negatively affect REWARD.  Institutionalizing inter-directorate work processes has begun with GIZ and EU support. This presents an opportunity for REWARD to institutionalize its results. CHANGES THAT DID NOT TAKE PLACE IN THE EWS BECAUSE OF ECOWAS  The changes that REWARD has begun to implement are not yet sustainable due to lack of formalization in working business processes.  The DOS-supported NCCRMs are not currently functioning well enough to have changed the way in which communication and coordination occurs in national- and regional-level responses. Similarly, NCCRMs have not improved the flow of early conflict communication between ECOWAS member states and EWD. 42 They have not been formally endorsed or approved by ECOWAS and hence, USAID. 32 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV  ECOWAS has not adopted SOPs whereby EWD is systematically calling upon other specialized ECOWAS directorates to formulate recommendations also grounded in their expertise and response practices.  The concepts of positioning the EWD at the Vice-President’s Office and at the center of a network of NCCRMs are not fully developed with clear roles, responsibilities, and procedures.  EWD’s recommendations remain less-applicable because they are not grounded in other directorates' response practices as a result of the absence of standard work processes ensuring that EWD systematically interfaces with them in formulating recommendations. BEST PRACTICES AND LESSONS LEARNED  REWARD’s decision to start its activities by first improving the quality of the data being collected as a precondition to deploying efforts to improve its analysis (through joint processes and other tools in the SMD of EWD), is a best practice; by doing so, REWARD has improved the quality of EWD reports.  Working within existing structures (ECPFS) reduced resistance to change and accelerated the uptake of inter-directorate collaboration.  Rather than relying on inter-directorate committees like the Internal Steering Committee (ISC) or the IDWG, REWARD’s decision to bring EW data producers and responders closer together through the implementation of CRVAs and later, under the auspices of the ECPFS, the JARP, CPX, and AAR, is also a best practice. This decision has helped circumvent the relative dysfunctionality of the bureaucratic structures within the inter-directorate committees. RECOMMENDATIONS 1. REWARD and USAID should support the Vice-President’s Office and the CPAPS in championing the formalization of its work processes reflecting the positioning of NCCRMs and EWD in the collection, analysis, reporting, and response recommendations to early warning signs. 2. REWARD should consolidate the gains it made at spurring inter-directorate collaboration by continuing its approach of bringing people together through events such as the JARP, CPX, and AAR (under the purview of the ECPFS). 3. More should be done to formalize the role of ECPFS focal points (and others in non-EWD directorates) in the joint analysis of data. 4. REWARD should direct its training in data analysis to the EWD staff, ECPFS focal points, and analysts from non-EWD directorates. 5. REWARD should coordinate with the EU-PSS project more closely to ensure the new analysis and response planning events it has introduced (JARP, CPX, and AAR) become part of the ECOWAS reengineered business processes. EVALUATION QUESTION 2 How did gender-integration efforts implemented by REWARD contribute, if at all, to a more gender-balanced early warning data collection and analysis? USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 33 Evaluation Question 2.1: Do the EWD and the CPAPS demonstrate the institutional and professional capacity and will to sustain the approach of integrating its peace and security efforts? FINDINGS Gender is being mainstreamed into the EWS. Following an assessment that PAPS published in early 2017, REWARD conducted a series of trainings to integrate gender into early warning and response mechanisms43. A post-training survey conducted July 2017 shows that, in comparison to pre-2017 measures, progress had been made to mainstream gender in all aspects of early warning and response – 43% of respondents stated they “always” report on gender. 44 Concerning Field Monitors more specifically: Sixty-seven Field Monitors participated in an online survey administered by USAID REWARD in late July and early August to assess the results of the May training in Dakar. The survey’s results confirm the training’s success on all counts. All 67 respondents reported that the FM training helped to improve the way they did their early warning incident reporting. Field monitors are more sensitive to gender in their reporting as a result of the training, with 87% of field monitors taking gender into account in at least half of their reports. 45 REWARD worked with nine EWD staff on October 13, 2017,46 to develop a framework for mainstreaming gender into data collection and analysis. REWARD also drafted the gender manual and accompanying training modules, which were submitted to EWD in August 2018.47 These are being applied only informally within ECOWAS, because the institution’s President and USAID have yet to approve it. (Note: ECOWAS’s website states the validation work began in November 2018.)48 The October 13, 2018, workshop on the gender framework (and prior trainings in 2017) were well received. Interviews corroborate that the training offered was a first and successful start for EWD in mainstreaming gender into their work.49 The REWARD Fiscal Year 2017 Annual Report rightfully points to the fact that REWARD has significantly changed the way in which EWD approaches gender issues in data collection and analysis.50 Progressively, gender is being disaggregated in the analysis and reports produced by EWD. The CRVAs use 55 indicators, all disaggregated by sex, and four of these are gender-specific. Through the data entry screens with drop￾down menus, field monitors are complying with gender-balanced data collection, but not all field monitors have systematically adopted this behavior. Nonetheless, interviews, documents, and reports provide evidence that gender mainstreaming is achieved through the CRVAs’ disaggregated indicators.51 These are 43 The following trainings occurred in 2017:  Gender Responsive Early Warning Training in Abuja, Nigeria (February 2017)  Workshop on Gender and Early Warning (May 2017)  Training on Gender Integration in Early Warning with EWD (July 2017)  Inter-Directorate Workshop on Gender and Conflict Sensitivity (July 2017) 44 REWARD_AID_624-TO-15-00004_FY17 Annual Report, p.11. 45 REWARD_ AID-624-TO-00004_FY17 Annual Report, p. 20 46 EWD Gender Integration Framework - October 2017.pdf. 47 REWARD_AID-624-TO-15-00004, p. 21. 48 https://www.ecowas.int/experts-begin-validation-of-the-ecowas-early-warning-gender-integration-manual/ 49 Interview, ECOWAS representative, May 2, 2019; Interview, ECOWAS representative, May 17, 2019; ECOWAS PAPS Gender Survey Report March/April 2017. 50 REWARD_ AID-624-TO-00004_FY17 Annual Report. 51 REWARD_ AID-624-TO-00004_FY17 Annual Report; Interview, ECOWAS representative, May 3, 2019. 34 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV now part of the data collection procedures and systems.52 There is now “an additional (6th) pillar in the ECOWAS EW system because of the achievement of the project [on gender mainstreaming].”53 The data indicate that the gender manual (not yet ECOWAS-approved and published) is being used informally as guidance to train field monitors for gender balance in data collection.54 REWARD is credited for the advances in mainstreaming gender. REWARD is not the only program working on gender mainstreaming within ECOWAS (other entities include GIZ and the Swedish Folke Bernadotte Academy (FBA)),55 but it is the only one doing so intimately with EWD. REWARD held multiple virtual and in-person conversations with FBA when it found out that the donor organization was providing gender-integration support to PAPS. When REWARD and FBA determined that FBA was largely supporting gender integration in response and mediation, it was determined that REWARD would continue its focus on gender integration in early warning, where it had already gained traction with the EWD. REWARD and FBA further shared programming materials to ensure coordination of efforts including workshop agendas and reports. The level of training REWARD has provided has never been offered before in ECOWAS. For instance, one of the interviewees pointed out that: “Nobody has done that with ECOWAS before and Creative brought a really good expert in to look at all the mandates and how it should apply.”56 MFD, DPA and DPKRS have not mainstreamed gender in their approach. The integration of gender balance in the formulation of political (under DPA), mediation, facilitation (under MFD), and military (under DPKRS) responses is not yet visible in the composition of the teams. Despite the usual training of military personnel on gender considerations, the DPKRS military interventions mostly use men, due to the fact that member states send few women. Therefore, although men are mostly fielded for military interventions, they are supposedly trained in identifying and responding to gender dimensions of conflicts. The DPA was not able to tell us how gender is being integrated in the directorate or in its operations. In short, DPA and DPKRS staff have attended the workshop on the gender-integration framework but still have no specific direction in how to mainstream gender in conflict mediation, facilitation, and military interventions. Interviewees referred the ET to the Gender and Social Affairs Department (GSAD) for further details on how gender is being integrated into response planning and other dimensions of PAPS functions.57 EWD has adopted a gender lens for data collection among field monitors (estimates show up to 40 percent of field monitors are women, from a total of 77) and there is gender balance of staff of EWD (up to 55 percent).58 Moreover, EWD has now a dedicated focal point for integrating gender in all of the HSAF analysis and for preparing reports on gender. There is no indication that there are any other gender focal points in other PAPS directorates. 59 52 Interview, USAID/REWARD representative, May 7, 2019. 53 Interview, REWARD representative, May 9, 2019. 54 Interview, ECOWAS representative, May 2, 2019. 55 REWARD AID-624-TO-15-00004_FY18 Annual Report. 56 Interview, USAID representative, May 3, 2019. 57 Interview, ECOWAS representative, May 2, 2019. The ET could not interview members of the GSAD because they were on leave. 58 Interview, ECOWAS representative, May 17, 2019. 59 Because the ET was not directed to interview members of other non-PAPS and EWD directorates, this evaluation cannot document the presence or absence of gender focal points elsewhere. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 35 Gender-balanced reporting is inconsistent. Yet the analysis reports on CRVA shared with the ET show some inconsistencies in addressing issues specific to women, girls, and elderly women. Gender mainstreaming is better in certain reports than in others. The female analysts are strong, the men are lagging. The [forthcoming] templates will help them be more systematic in how they mainstream. […] Up until now, they are very good at gender disaggregation in data collection, but they are less strong on the analysis/reporting side of things.60 Although the CRVAs consulted have a section dedicated to gender considerations, these are not systematically woven into the analysis of the 55 CRVA indicators. The new thematic reports do not necessarily focus on gender issues; only the one on the Nigeria elections was systematic in integrating gender considerations into the report. The ET was told there were no templates for other thematic reports that were issued, and REWARD is preparing a report template to ensure gender considerations are addressed in the forthcoming ones.61 This will further institutionalize gender mainstreaming in data analysis and reporting. EWD has been readied to mainstream gender. Regarding the sustainability of the gender-mainstreaming work that REWARD has done (Evaluation Question 2.1), the evidence at hand points to the fact that EWD now has the ability to train field monitors on gender integration (following the guidance of the gender manual) with gender-specific indicators. The EWD Director and his staff have become the flag-bearers within the PAPS for integrating gender into their technical work including requesting the development of a Framework for Gender in Early Warning, and integration of gender considerations into their new technical area indicator reviews and development.62 The EWD Gender Focal Point indicates that EWD is now capable of advocating for a more gender￾balanced collection and analysis within ECOWAS and beyond. REWARD has succeeded in this area, so much so that FBA has indicated its interest in disseminating the REWARD gender manual to other areas.63 CONCLUSIONS Based on the findings from documents and interviews, the ET draws the following five conclusions:  The REWARD-led training on the gender framework has ensured a more gender-balanced approach to data collection. The fact that ECOWAS has not yet approved the gender manual is delaying its implementation and institutionalization. Until the gender manual is formally approved, ECOWAS will continue to use it informally.  Although the indicators gathered during data collection are gender-disaggregated, there is still some work to be done to mainstream gender considerations in the analysis of data, reporting and especially in response planning.  The absence of report templates taking into account gender dimensions prevents the generalization of gender mainstreaming in EWD products. 60 Interview, implementing partner representative, May 7, 2019. 61 Interview, REWARD representative, May 7, 2019. 62 REWARD AID-624-TO-15-00004_FY17 Annual Report. 63 Interview, ECOWAS representative, May 2, 2019; Interview, REWARD representative, May 9, 2019. 36 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV  Even though the manual is not approved, the EWD Gender Focal Point and the field monitors are sustainably equipped to collect gender-balanced data. Gender-balanced analysis and reporting, however, remains a weakness.  Mainstreaming gender is not universal in ECOWAS. RECOMMENDATIONS In light of the above, the ET recommends that: 1. As soon as possible, USAID undertake efforts to have ECOWAS approve the gender manual so that it can do the same; 2. REWARD deepen the training of the SMD staff, the EWD analysts, and those of other directorates to better integrate gender considerations into their responses; 3. REWARD proceed as quickly as possible with the preparation of guidelines/templates for integrating gender into all EWD reports. 4. REWARD should work with the FBA, the GIZ and the ECOWAS/GSAD to ensure that gender considerations are mainstreamed into DPA, MFD and DPKRS, as well as across other non-PAPS directorates. EVALUATION QUESTION 3 How has the USAID REWARD project contributed to improving EWD’s ability to collect early warning data and strengthen EWD’s ability to analyze, report, and generate recommendations? Evaluation Question 3.1: What specific capabilities and skills were increased and how have they been applied by EWD? Evaluation Question 3.2: Does EWD demonstrate the institutional and professional capacity and the will to sustain early warning data collection, analysis, reporting and recommendations without the assistance of REWARD? The ways in which REWARD has impacted the EWS in its various function were discussed under Evaluation Question 1. This section provides further details on the specifics of REWARD’s impact on the EWD. FINDINGS DATA COLLECTION FUNCTION REWARD introduced better data collection and analysis methods and tools. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 37 REWARD introduced the CRVA/HSAF methodologies to EWD. They have been applied to 14 (of 15) CRVA reports and the Regional Risk and Vulnerability Assessment (RRVA) report for the Lake Chad Basin.64 The latter still required REWARD’s continued technical backstopping, albeit less intensely than what REWARD had to offer for the CRVAs. In one interview with a group of REWARD staff, the ET was told that on a scale of 1 to 10 for data collection capacity, EWD is currently rated at “8.”65 There is an increased level of interaction between EWD division staff. EWD’s SMD produces real-time data analysis as a result of REWARD’s training and the new software it installed.66 There are reports of increased levels of interaction between analysts and the SMD and a greater level of demand for visual representations of data. 67 In an effort to further improve the quality of analysis and the visual representation thereof, REWARD exposed the SMD to a tool named Beta-Test; this still ongoing. This new tool helps “to create the CRVA Index to measure vulnerabilities in each member state and for producing specialized thematic reports based on CRVA data.”68 REWARD notes in its review of this evaluation that “the Beta-Test Platform was an existing tool that was adapted for the project as a means of demonstrating some of the critical skill sets around best practices in data collection and online incident mapping that the project was/is imparting. It is an online incident-mapping and data-visualization platform; it does not create thematic reports, nor does it create indices. Moreover, it was introduced, in part, to help fill the gap in data visualization for the EWD as their regular mapping platform was being upgraded by Virtual Reality Associates (VRA), an independent firm contracted by ECOWAS which has responsibility for the EWD platform; therefore, it was not done by or through REWARD.” Collaboration improved through training in joint analysis and conflict-scenario planning. As REWARD unfolded, its focus gradually shifted from improving data collection to establishing a better linkage between the data EWD produced and the responses they should spur. This was done through the JARP, CPX, and AAR. As a result, the ET sees that a bridge has been created between the EWD and other directorates within ECOWAS: “For a first time, we were able to work together in close collaboration so they could see what we do and how the info we provide to them is based on empirical data and this is starting to make sense to them.”69 The Systems Management Division needs more tools and training. The SMD of EWD wants REWARD to provide further training on web-based applications for field monitors. Accordingly, a respondent (who did not know that this is outside of the REWARD scope of work) from this unit said: “We need to reinforce the capacity that has been built. Right now, there are online applications, on intelligent phones. It would be helpful to have an application that has localization function for the field monitors, for example during elections. We need training and tools for web mapping and web design to make things more attractive.”70 64 Interview, ECOWAS/EWD representative, May 3, 2019. 65 Interview REWARD representatives May 13, 2019. 66 The hardware, furniture, etc., come from DOS while the software is from Creative and its partners in implementation (Fund for Peace and Karuna Centre for Peacebuilding). 67 Interview, ECOWAS/GIS representative, May 8, 2019. 68 REWARD-AID-624-TO-15-00004-FY17 Annual Report. 69 Interview, ECOWAS representative, May 2, 2019. 70 Interview, ECOWAS representative, May 8, 2019. I am glad that REWARD has facilitated these conversations through the crisis scenario planning activity. There was a prior workshop that was an attempt to have these conversations that wasn’t very successful, but the practical simulation was very effective and very good. —Interview, ECOWAS representative, May 2, 2019 38 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV Updated technology is needed to quickly report on violent extremism. “For EWD, the technology was there for them to work with, but it needed some updating to track incidents of terrorism and violent extremism.”71 SMD interviewees report that REWARD needs to update the technology that has been put in place for them by equipping field monitors with portable devices preloaded with the data-entry screens in order to better keep on track issues related to terrorism and violent extremism. This is outside of REWARD’s scope of work. There is no collaboration mechanism between the field monitors and the NCCRMs. This may be explained by the lack of operational procedures between field monitors and the NCCRMs and by the fact that the staff of national centers have not received the same level of training as the field monitors. One more reason why the national centers are dysfunctional is reportedly: The staff at the national centers report directly on the ECOWARN system. They look at the issues and decide who should respond. As of now, I don’t think it’s operational and they are supposed to develop SOPs to guide their work. There’s new staff and we conducted orientation when I was part of WANEP. REWARD is planning to do a training at the national centers so they can use some of the tools. We should have started by now, but when we went to Dakar to look at the system and look at needs, it delayed the start. 72 REWARD and the EWD had settled on dates for REWARD to carry out training at the national centers earlier in FY 2019, but during the field monitor training in Dakar (March/April 2019), the EWD and national center directors decided that they wanted to conduct their own needs assessment at the national centers before REWARD’s training modules were finalized73. The time frame in which the EWD and national centers took to finalize their own needs assessment delayed REWARD’s ability to start the NCCRM trainings. ANALYSIS FUNCTION Thematic analysis is the new norm introduced by the EWD, but there are weaknesses. There is a perception (based in reality) that the EWD analysts are not expert enough in their assigned HSAF-thematic subject matters to provide a credible and nuanced analysis of issues that EWD data may reveal. There has been informal recourse to the thematic experts from other directorates or commission divisions to enrich EWD’s analysis ; this practice, however, is not systematic. Still, there exists the potential to broaden the analytical perspective that EWD brings to the table by involving others: I think USAID should bring capacity-building to all the analysts, in all the directorates. Each department brings their perspective. This is why the Crisis Scenario [sic] Planning Exercise (CPX) was successful because they were all working together. We all agree that the problem is not warning, but the response. This is why we believe REWARD has value because they are linking to response.74 Use of the CRVA/HSAF approach has improved analysis, but only to some extent. The CRVA methodology and the organization of indicators along thematic HSAF areas have improved the analysis work of field monitors and analysts. “The Human Security Analysis Framework brought them to improve their analysis by providing a basis for thematically organizing and analyzing data, thus improving the quality of the 71 Interview, ECOWAS representative, May 3, 2019. 72 Interview, REWARD representative, May 9, 2019. 73 It is questionable if this is, and should be, part of REWARD’s scope of work. 74 Interview ECOWAS representative, May 6, 2019. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 39 data reported upon.” 75 However, their level of mastering this tools for the development of analysis is still low and was scored by a group of implementing partner staff as a “6” on a scale 1 to 10. 76 An interviewee, reflecting what others have also said, states that “we can do better”77 and suggest that the DPA should be more involved in the realization of analysis: For me, the DPA should do more with the analysis. For me when you are analyzing data and finding trends, it’s different from when the crisis is there. When the crisis is there, it’s difficult to know which is EWD or DPA. To sit and look and provide analysis… we are doing security analysis. This is the difference between making something fast. For me, early warning does not mean competition to announce first. 78 EWD can now produce dashboards because of REWARD’s Beta-Test tool. The Beta-Test software platform that REWARD worked on with EWD is another tool that helps to transform data collected into charts for analysis. This seems to have greatly reinforced the capacity of EWD to produce rigorous analyses: One of the tools included in the proposal to EWD was the Beta-Test platform, which allows users to search through uploaded data to identify patterns, trends, and hotspots in ECOWAS member states. Users can also analyze data within the platform. Built-in tools are capable of transforming data into monthly, quarterly, and yearly charts for analysis. Users can create bar charts based on indicator, sub-indicator, country, or sub-national geographic levels. Users can narrow data by geography, date, or incident type. The tool therefore supports more streamlined and sophisticated analysis of early warning data.” 79 Using the HSAF improves the quality of analysis. REWARD hired an international consultant to assist the EWD team of analysis to develop the Early Warning Conflict Analysis Framework (ECAF). In November 2018, this framework became the Human and Security Analysis Framework (HSAF). It an instrument that allows EWD analysts a basis on which they can provide high-quality analyses. We focused on improving the quality and usability of conflict analysis in EWD through the development of the ECAF, an innovative tool that is specifically tailored to work in an integrated fashion with EWD’s systems data-collection enhancements and analysis that USAID REWARD has been supporting. The USAID REWARD team considers the ECAF to be a key instrument in strengthening the EWD’s capacity to produce analysis that is of high quality that can facilitate decision-making on response by different categories of end users.80 JARP opens a door to resolving unclear roles in analysis. The organization of the JARP was the first exercise that made all the directorates work together and enrich the EWD analysts’ examination of data. Compared to the situation in which EWD analysts informally called on colleagues to enrich their analysis, the JARP workshop was a more formal way to bring together expertise from across the organization. REWARD and its stakeholders within ECOWAS are attempting to clarify roles and responsibilities in the analysis of data and the formulation of recommendations functions. The JARP points to a possibly sustainable change in EW system behavior by 75 Interview USAID/REWARD, May 1, 2019. 76 Interview REWARD representatives May 13, 2019. 77 Interview ECOWAS representative, May 6, 2019. 78 Interview ECOWAS representative, May 6, 2019. 79 REWARD_AID_624_TO-15-00004-FY17 Annual Report. 80 REWARD_AID_624_TO-15-000I-04-FY18 Annual Report. 40 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV spreading the task of analysis among entities within ECOWAS that are as, or more, knowledgeable than EWD in the five areas of the HSAF. As one knowledgeable and senior ECOWAS member states: “the analysis would never be able to be complete at the EW level because those who are responsible for the action have access to data the EW doesn’t use to inform their response, whether it be DPKRS, agriculture, etc.” 81 The ET noted that REWARD slated further work on the JARP process for Year 4.82 REPORTING FUNCTION REWARD ensured that data have been cleaned of duplicates; this has improved their exploitation and representation. REWARD provided training to the EWD and the SMD on CRVAs, event geo-localization and time￾dynamic mapping, Beta-Test, and the Deduplication tool.83 Even though the Deduplication tool is very useful in eliminating duplicate records, SMD staff had to complete the cleaning process manually. Ultimately, staff from this unit have managed to clean up to ten years of quantitative and qualitative data sets. As a result, staff confirm that their capacity has greatly improved and they are confident about the data being used for reporting. Positive feedback was received on the quality of reports. By adopting the HSAF, EWD has improved the quality of the analysis and, therefore, the reports it produces. However, as a representative of the implementing partner stated, the CRVA and the HSAF are just ways of “cataloguing” data and the analysis needs to flow from them. REWARD has been working with Human Security Analysis framework this year with reports that include actionable items and digestible recommendations. The Human Security Analysis framework brought them to improve their analysis by providing a basis for thematically organizing and analyzing data, thus improving the quality of the data reported upon.84 Analysis of the interviews reveals that ten interviewees commented positively on the quality of reports that EWD produces. One respondent captures the general trend of the data: “EWD is becoming technical as far as its products are concerned. I think they improved on several domains with regard to the quality of their products. Their reports improved.”85 Otherwise, while interviewees voiced86 an improvement in report quality, one EWD representative thinks it is too early to assess their progress in reporting, due to the fact that the 14 CRVA reports, for lack of approval, have not yet been “launched” (i.e., released).87 More progress is needed to standardize reports. REWARD helped to develop the report template for the report on the Nigeria elections (in the HSAF domain of democracy and governance), where gender integration was included throughout the report. 81 Interview ECOWAS representative, May 7, 2019. 82 REWARD_Year4 Work Plan Narrative Final. 83 Interview ECOWAS representative May 8, 2019. 84 Interview, REWARD representative, May 1, 2019. 85Interview, ECOWAS representative, May 13, 2019. 86 Interview, ECOWAS representatives, May 3, 2019; Interview, Development partner, May 8, 2019; Interview, ECOWAS representative, May 13, 2019; Interview, ECOWAS representative, May 6, 2019; Interview, REWARD representative, May 9, 2019; Interview, REWARD representatives, May 13, 2019; Interview, ECOWAS representative, May 8, 2019; Interview, REWARD representative, May 7, 2019. 87Interview, ECOWAS representative, May 2, 2019. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 41 REWARD plans this year to continue developing such report templates.88 However, there remain issues with regard to standardizing the products, notably because of the absence of report templates to structure the writing.89 The CRVA was a mechanism to enhance capacities for data collection, methodology, integration, but then when it goes to the analysis [there is an issue]. The CRVA is just a catalogue – the analysis flows out of it. That should be informing their EW products. We are trying to work with them now to strengthen the products. Some products are better than others, and there is not a lot of standardization.90 EW reports are being produced elsewhere. There remains an issue with the fact that other (non-EWD) directorates can, and do, continue to produce and rely more upon their own reports than those EWD produces. This is essentially because of the state of inter-directorate competition, the historically poor quality of EWD data being reported upon, and the absence of inter-directorate collaboration in analysis. 91 Also, as is discussed below, due to the inadequate nature of the EWD analysis there is no senior management decision on which reports are authoritative (i.e.: EWD’s or those of other technical directorates not located within PAPS). RECOMMENDATION-FORMULATION FUNCTION EWD formulates recommendations with little input from others. EWD is performing the recommendation-formulation function, which it sees as being within its purview more than anyone else’s.92 Prior to (and in spite of) the REWARD facilitated JARP and CPX events, there is no formal and functional process for involving responders in formulating recommendations. When crafting recommendations, EWD analysts seldom consult with those who are responsible for responding to issues—and who ask to be part of the process to better own93 those recommendations. Recommendations are being formulated by the EWD analysts but, depending on the issue, [they] may call on staff from other units to co-develop recommendations. Because of REWARD, we have increased our collaboration with other directorates; before, it was based on who you know.94 The system is not designed to have EWD alone formulate recommendations. According to a senior ECOWAS officer, the system was not designed to have EWD formulate recommendations on its own. Commenting on the situation, he said that: Before the JARP workshop happened, EWD was the only one to do analysis and recommendations but in the design, it wasn’t supposed to be that way. There’s early warning data analysis and response data analysis. With EW data analysis, you have people who are experts in the thematic areas. But with response data analysis, you need different perspectives, but how often is that happening? Reacting to EW data is political at some point. States may deny they have issues. When is it the right time to intervene? When is it the right time to take the data and use it to act? This depends on different factors from leadership of 88 REWARD Year 4 Work Plan Narrative_Final 89 Interview, ECOWAS representative, May 3, 2019. 90 Interview, REWARD representative, May 7, 2019. 91 Interview, ECOWAS representative, May 2, 2019. 92 Interview, ECOWAS representative, May 2, 2019. 93 Ibid. 94 Interview, ECOWAS representative, May 3, 2019. 42 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV the Commission to the leadership of the director, the capacity of analysts, and leadership of our policy organization.95 Some find EWD recommendations vague and, thus, not actionable. Because EWD recommendations are very general in nature, they allow responders to adopt the specific stance they prefer to take on a given issue. Although the ET did not have the opportunity to interview staff from non-PAPS or EWD units about their appreciation of the EWD recommendations, those from DPA and DPKRS do not necessarily rely exclusively on EWD reports to decide how to respond to events.96 Therefore, even though REWARD’s aim is to have responders act on the basis of the recommendations that EWD produces (or ideally, co-produces with experts from other directorates), they are not doing so. This is in spite of the fact that EWD staff are generally happy with what they produce: The feedback we received on our monthly thematic reports has been quite phenomenal. Sometimes we have directorates writing to us, or even to senior authorities asking for the monthly reports, and sometimes writing to our director, or even beyond, saying that “this month’s thematic was exceptionally good.” One ambassador even commented that he reads highlights to know how he should start his day.97 Use of the HSAF is being discussed to improve the formulation of analysis-based recommendations. Even if the HSAF has brought structure to analysis and reporting, the institutionalization of this approach for the recommendation-formulation function remains outstanding: We were unsure how to proceed in the response component after recommendation. We had the framework talks about data collection and data analysis, scenario building… For the purpose of this framework [HSAF], there are discussions over where the recommendation function should go. The recommendation is part of the EWD mandate, but response is not part of the EWD mandate. The HSAF is still a work in progress – early warning has done its part. If USAID wants to expand it to include different directorates to contribute to the recommendations, that is acceptable.98 The JARP and CPX help to bridge the data producer-user gap. The JARP and CPX that REWARD piloted have been critical in bringing together, to formulate recommendations, EWD human security analysts and those from other directorates responsible for response planning and execution. There are calls to ensure that events like these do not merely live on as one-off affairs99 and that workshops like these become part of the organization’s SOP. 100 SUSTAINABILITY In the context of this evaluation, the notion of sustainability has three dimensions, spanning both institutional and professional capacity and the will to sustain the project’s thrust. For the ET, the notion of “will to sustain” has two dimensions: (1) the attitude of ECOWAS towards the future and (2) financial 95 Interview, ECOWAS representative, May 7, 2019. 96 The representative from the Mediation Division of PAPS did not show up to his appointment and attempts to reestablish one with him were to no avail. 97 Interview, ECOWAS representative, May 3, 2019. 98 Interview, ECOWAS representative, May 2, 2019. 99 USAID/REWARD see these as processes that are to be embedded, in the longer term, into ECOWAS. 100 Interview, ECOWAS representative, May 2, 2019 and May 13, 2019; REWARD_AID-624-TO-15-00004_FY18 Annual Report. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 43 commitments from ECOWAS to pay for the continuation of the activities, processes, and human resources associated with the EWS. No SOPs have been formalized. From an institutional perspective, the work methods and processes that REWARD implemented have not been captured formally into work processes and SOPs. This is not because REWARD did not try to accomplish this. One report states that the development of SOPs has been affected by: …(1) turf issues and fear of staff in other directorates taking away jobs was related to poor communication at multiple levels and lack of clarity about the respective roles and responsibilities; (2) an inordinate amount of staff time is consumed by inefficient and/or deficient administrative processes/infrastructure; and (3) time pressures, silent competition, negative dynamics, and habits in the institutional culture are preventing collaboration.101 One senior ECOWAS official confirms what is written in the REWARD Fiscal Year 2018 Annual Report102 and further explains why this could not be accomplished: We are working on [SOPs] with the EU, the business processes and the workflow. It was not completed because the staff are very overwhelmed. They are involved in a lot of things and it’s difficult to get them to create something by themselves. Because recruitment was frozen, we rely on [development] partners for positions. We would not be able to function very well without [development] partners. Because the member states are easing their restrictions on recruitment, we hope that, soon, we will have the right number and quality of staff.103 Although this quotation comes in the context of questions pertaining to SOPs and institutional sustainability, it also captures information about the financial constraints affecting ECOWAS’s ability to sustain the project; this aspect is discussed further below. REWARD works with directors to improve sustainability. The vertical move of EWD to the Vice-President’s Office may help to institutionalize the changes that REWARD has introduced ; it may present an opportunity to better connect the ECOWARN system with the rest of the organization. However, from a change-management perspective, the impetus that REWARD has given to the organization may be irreversible and hence, sustainable because: Most of what REWARD does … it is at the directorate level. The directors don’t change and have been there for a decade or two. The staff doesn’t really change, either. The capacity-building has been with the staff and the directors give approval. Everything comes with the correct political approval, but the directors are brought into the activities.104 As will be seen in further detail under Evaluation Question 8, the EU-PSS has met with the same constraints mentioned in the preceding quotation and they informed the ET that they have received clear indications that their business process reengineering activities will start soon.105 101 REWARD_AID-624-TO-00004_FY17 Annual Report. 102 REWARD_AID-624-TO-15-00004_FY18 Annual Report. 103 Interview, ECOWAS representative, May 7, 2019. 104 Interview, REWARD representative, May 1, 2019. 105 The EU-PSS has expressed interest in working closer with REWARD to ensure that its new data collection, analysis, and response planning processes (and their accompanying methods) are written into the SOPs of the organization. This is something that EU-PSS confirms it will not do, thus leaving it to REWARD to take advantage of the exercise. 44 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV The same tree cannot grow everywhere at the same place. One interviewee summarized what others who were questioned on the subject of professional capacity said: “anything that works well will be sustained because it will be in the interest of the organization. You cannot remove the new tools and methods, but you can’t be rushing things. The same tree cannot grow everywhere at the same place.”106 The professional capacity of the organization is likely to be sustainable because REWARD was timely in answering the needs that EWD had identified for itself. Thus, EWD staff bought into to the new methods that REWARD brought to the directorate. 107 “Capacity has been built; we want to take the lead on studies based on what we learned [from REWARD]. If REWARD were to leave, we are capable. It [REWARD] is sustainable and this is unlike other projects who come and go without leaving anything behind.”108 Although EWD respondents generally express confidence in their ability to continue doing their work following the methods and processes that REWARD introduced into the organization, others do not think the same way. Although they recognize that EWD’s data collection capacity has been reinforced through CRVAs (and that the cataloguing of data has been improved through the HSAF), they continue to doubt the expertise of EWD analysts to perform quality analysis of thematic areas. Indeed, two senior level officials (not with EWD) question analysts’ thematic expertise.109 One of them even goes so far as to state that the “competency of analysts is not a condition for recruitment.” 110 They see a continuing need to have EWD analysts mentored in data analysis. They also think that EWD needs to completely master their training on CRVA-based data collection—that the Lake Chad Basin RRVA attests to the EWD’s inability to go on its own in the area of research planning: “It is just like somebody who is a millionaire he has to make serious investment in order to remain a millionaire: if you don’t tomorrow, everything will be gone. So, this capacity [building] needs to continue and be strengthened more.” 111 For the reasons evoked above, EWD analysts are not yet the masters of HSAF-based analysis. Senior ECOWAS management shows no overt signs of “will to sustain.” The ET was informed that the former CPAPS was actively supportive of the project, but there are no indications that the same holds for the current Commissioner.112 Given the challenges to break down organizational silos within ECOWAS and to effectively link data producers and users through existing mechanisms, this poses a problem; senior management is perceived as having little interest in developing SOPs for inter-directorate collaboration (even for the existing ECPFS). More will be said on this in answering Evaluation Question 5. However, as discussed earlier, given that REWARD is working hard with directors and staff from the whole of ECOWAS to socialize the changes in organizational behaviors that would further bridge the data producer-user gap, the absence of clearly expressed senior management support may be a temporary setback to REWARD’s efforts. No ECOWAS financial commitments sustain REWARD’s results. Financial commitments can be interpreted as a sign of ECOWAS’ will to pursue REWARD’s thrust. Interviews reveal that ECOWAS receives 90 percent of its funding from the community tax levy (0.05 106 Interview, ECOWAS representative, May 7, 2019. 107 Interview, ECOWAS representative, May 3, 2017. 108 Interview, ECOWAS representative, May 3, 2017. 109 Interview, ECOWAS representatives, May 6, 2019 and May 7, 2019. 110 Interview, ECOWAS representative, May 6, 2019. 111 Interview, ECOWAS representative, May 17, 2019. 112 It is noteworthy that the CPAPS refused to meet the ET. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 45 percent) on imported goods and 95 percent of this amount is dedicated to funding staff salaries.113 ECOWAS relies on partners (donors) for any project that develops the organization (such as USAID is doing through REWARD). As addressed under Evaluation Question 1, the absence of a legal document that frames the partners’ mutual expectations may lead to situations where ECOWAS fails to pick up the cost of new work processes (such as JARP and CPX workshops or, even, REWARD induced staff) when REWARD ends. Although what follows is outside the scope of this evaluation of REWARD’s Purpose 1, it has a bearing on REWARD. EWD interviewees shared with the ET that part of the problem with the EWS is that NCCRMs are not linked to the EWD. These are national-level organizations that REWARD is under pressure to support in order to build the ECOWARN system that EWD would oversee and manage. The five NCCRMs that DOS has financed are not functional and even less sustainable at this point because financial expectations were not legally spelled out for ECOWAS (and national authorities). Therefore, the NCCRMs failed to pay their staff for three months; ECOWAS has recently, and reluctantly, picked up those costs while they await future financial commitments from national authorities. From the perspective of financial sustainability of REWARD’s efforts, the ET is told that there is no ECOWAS financial commitment to guarantee the future. However, given that REWARD has developed, tested, and begun to institutionalize its tools and methods, and that EWD has adopted them, there should be no need for added funding to sustain REWARD’s results. Staff will be paid out of ECOWAS’s general funds to continue using the REWARD tools as part of the standard work processes—if ECOWAS adopts these and their accompanying SOPs. CONCLUSIONS Data collection function  The REWARD-organized training on new methodologies and tools for EWD staff have improved the rigor, completeness, and accuracy of the data collected. Dynamic data visualization has improved. Data are better-organized and this facilitates analysis.  NCCRMs are not established well enough to receive technical assistance from REWARD.  EWD’s professional capacity has improved but EWD cannot yet collect data using the CRVA methodology without REWARD’s technical assistance—even though the latest experience of the RRVA is promising for EWD’s future mastery of this method.  SMD needs for additional GIS tools, although relevant, are outside of REWARD’s current scope. Data analysis function  EWD staff are not experts in the thematic areas of the HSAF analysis and do not systematically call on colleagues to enrich their analysis. This reduces the credibility and completeness of the analysis and as a result, the uptake of EWD products. Nonetheless, these products have been reported as “improved” since 2014.  Low levels of EWD analytical expertise are compounded by the lack of systematic coordination with other directorates. Additional barriers include the absence of a structured mechanism to 113 Interview, ECOWAS representatives, May 7, 2019 and May 14, 2019. 46 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV analyze data in a collaborative fashion and the continued questioning around the proper use of tools, such as the HSAF. JARP may offer a solution to this issue.  Data visualization has improved  Data are better organized through the use of the ECAF/HSAF, and this facilitates analysis  Although it is not within REWARD’s current scope to do so, there are well-founded calls for additional GIS tools to better geolocalize events of violent extremism Reporting function  Progress is noted in the quality of the reports and efforts at their needed standardization are under way, even though some directorates produce their own versions of early warning reports. Recommendation formulation function  EWD continues to view as its responsibility the formulation of recommendations; it (occasionally and informally prior to REWARD’s JARP) calls on directorates responsible for response planning to craft recommendations that are actionable.  A workshop such as the JARP and the CPX is seen as a unique, one-off event, even though USAID/REWARD see them as longer-term endeavors. It is not a SOP and is not embedded in the organization’s culture. There is unanimous consent that these promote inter-directorate collaboration and improve the appropriateness of the response by enhancing its evidence-based foundation.  EWD formulates recommendations that are sometimes too vague to act upon.  Attempts at applying the HSAF to guide the formulation of analysis-based recommendations are pending a decision on its suitably for the whole of ECOWAS  The processes and methods that REWARD has introduced are not formalized into the organizations' work processes at this stage. Sustainability  In spite of the absence of openly supportive leadership from the current CPAPS, the institutional sustainability of REWARD’s work is being reinforced, but not guaranteed, by REWARD’s decision to systematically work with directors.  There is no evidence that ECOWAS intends to rely on its own financial resources to sustain the implementation of REWARD methodology. The ET’s analysis concludes that if these “new ways” are integrated in the organization’s SOPs, there may not be a need to add financial resources to ensure their sustainability.  ECOWAS is just starting to get EWS data to inform response. JARP and CPX are, to date, isolated events rather than standard work processes. Thus, their institutional sustainability as a work method for analysis is not ensured; they are not formally documented, operationalized business processes. RECOMMENDATIONS USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 47 1. REWARD should continue on its current course to build capacity of enabling ECOWAS headquarters staff to move through the full circuit of joint data collection, HSAF-grounded analysis, scenario response planning, and after-action review. 2. REWARD should wait for NCCRMs to be made functional before extending its technical training to them. Rather than train staff from NCCRMs, REWARD should ensure that what have been, so far, one-off events (like the JARP, CPX, and AAR) and new methodologies like the CRVA/HSAF become standardized in the organization’s business processes through formally approved tools and SOPs. 3. REWARD should continue to support EWD as it did for the RRVA in the next round of CRVAs until specific and measured organizational capacities are met. 4. USAID should allow REWARD to provide geolocation tools to the SMD to better geolocate events of violent extremism. 5. REWARD needs to pursue and deepen its support to the ECPFS while recognizing its limitations. One way to do this would be to further leverage this mechanism to support inter-directorate collaboration in analysis and response planning. 6. EWD, now positioned in the Vice-President's Office and led by REWARD in its inter-directorate work, needs to adjust its products to answer the needs of all ECOWAS directorates by improving consistency of analysis and by producing two types of reports, at least: country-level and (regional) thematic ones. 7. REWARD should continue focusing its attention on behavior change in the organization through training and inter-directorate activities, and it should work with the EU-PSS to ensure new behaviors are integrated into its forthcoming business process reengineering project. 8. REWARD should continue to work with directors and USAID should further engage the senior ECOWAS leadership in championing for change within the organization. EVALUATION QUESTION 4 How has the USAID REWARD project contributed, if at all, to improving CPAPS ability to carry out its mandate of early warning, conflict prevention, and mediation, especially by enhancing CPAP’s internal and external communication and collaboration on early warning and response? Evaluation Question 4.1: What are the specific instances of CPAPS’s better internal and external communication and collaboration on early warning and response? Evaluation Question 4.2: Does CPAPS demonstrate the institutional and professional capacity and the will to sustain early warning, conflict prevention, and mediation without the assistance of REWARD? FINDINGS: During interviews, unless pressed to do so, respondents did not distinguish clearly between early warning, prevention, and mediation functions in their answers to the ET’s questions. Similarly, they did not 48 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV distinguish clearly between internal and external communications and collaborations.114 The ET did so in its analysis. INTERNAL COMMUNICATIONS AND COLLABORATIONS DPA and DPKRS are reluctant to act on EWD recommendations. As we have discussed, REWARD began to work with EWD by working on the duplications and inconsistencies in the data sets. In this context, work that REWARD is tasked to do on enabling PAPS to better perform its conflict prevention and mediation115 mandate is delayed. At its beginning, evidence indicates that REWARD struggled to bring DPA and DPKRS to work together with the EWD and make use of the data or act on recommendations EWD produced.116 Also, “it was difficult to work with DPKRS because many of their staff are military and there are USAID restrictions to support military-related activities.” 117 Interviews reveal that part of this problem rests in the fact that EWD has taken responsibility for crafting recommendations that are seen as too vague to act upon by DPA and DPKRS.118 DPA and DPKRS find that the country-based CRVAs are more useful to them than the thematic reports that EWD now produces, as their units’ interventions are country-based and not necessarily regional in nature. An added reason for this lack of cooperation is that representatives from DPA and DPKRS do not believe EWD analysts have the required knowledge to be thematic experts.119 SOPs for the DPA have not been implemented. For reasons that were not clearly explained to REWARD120 or the ET, the SOPs that REWARD has developed for the DPA have not been implemented. The EWD’s move to the Vice-President’s Office has further delayed this work as the DPA SOPs that REWARD developed refer to the EWD when it still dwelt within PAPS. Furthermore, there are GIZ121 and EU-PSS122 projects working in the same thematic area of peace of security and both, like REWARD, aim at the organizational development and performance improvement of the sector. Although the REWARD team claims123 to have daily contact with staff from these projects, representatives from both projects have negated this statement.124 In spite of REWARD’s efforts, little has been done with the Mediation and Facilitation Division. REWARD supported PAPS in conflict prevention through the CRX and AAR125 but limited attention was given to the MFD function.126 During its work planning sessions with DPA, REWARD offered to support the MFD under DPA, but they did not request this support. REWARD was informed by DPA that the directorate was receiving support from other donors for its mediation support needs. REWARD could 114 Prior to conducting interviews and coding documents, the ET defined “internal” communications and collaborations as those occurring within a directorate (for example, between EWD analysts and SMD staff) or a department (within PAPS), whereas “external” ones refer to those taking place with non-PAPS directorates, the national centers, or member states. 115 The ET did not interview anyone from the Mediation and Facilitation Division of PAPS because the interviewee did not show up for the interview meeting and subsequent attempts to reschedule the interview were unsuccessful. 116 Interview, ECOWAS representatives, May 6, 2019 and May 7, 2019. 117 Interview, REWARD representative, May 1, 2019. 118 Interview, ECOWAS representatives, May 6, 2019 and May 7, 2019. 119 Interview, ECOWAS representative, May 6, 2019 and interview, ECOWAS representative, May 7, 2019. 120 See AID-624-TO-15-00004_FY18 Annual Report. 121 Support to the ECOWAS Peace and Security Architecture and Operations (ESPAO), November 30, 2018. Interview, Development partner, May 9, 2019. 122 Interview, EU-PSS representative, May 14, 2019. 123 Interview, REWARD representative, May 13, 2019. 124 Interview, Development partner, May 9, 2019 and interview, Development partner, May 14, 2019. 125 REWARD AID-624-TO-15-00004 FY17 Annual Report. 126 Interview, ECOWAS representative, May 2, 2019. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 49 not provide significant support to the DPKRS analysis function due to the USG policy against assisting the military.127 EXTERNAL COMMUNICATIONS AND COLLABORATIONS Senior-level support for inter-directorate SOPs is absent. The 2018 EWD move to the Vice-President’s Office has reorganized PAPS. Even though the move has not changed REWARD’s day-to-day relations with EWD itself, it has stalled the development of SOPs for EWD. Inter-directorate SOPs “were a priority for previous Commissioner and they are not a priority for the new Commissioner.”128 EWD’s move may have hindered REWARD. EWD’s move to the Vice-President’s Office already has repercussions for the REWARD SOW and has already forwarded recommendations to USAID/WA’s RPGO to ensure that its TO is amended to reflect that reorganization—this may bring the project to interface with the Vice-President’s Office. Indeed, given the already limited working relationship between EWD and other PAPS directorates, the EWD’s move to the Vice-President’s Office raises fears that collaboration could suffer. There are already negative repercussions being felt: “the EWD being moved to Office of the VP has created a gap with how information is created and shared. Currently the method in which the EWD and this department communicate has not been spelled out. The synergy has suffered. Our interventions will lack substance…”129 But “it is too early to see the repercussions of the [EWD] move out of PAPS.”130 REWARD is challenged by ECOWAS’s culture. The working environment in ECOWAS is highly politicized, dominated by bureaucratic processes and a hierarchical, command-and-control organizational culture. This is a major challenge for REWARD in bridging the data-response gap. It is a situation where other directorates asked to establish their own early warning systems but were denied this by the former ECOWAS President.131 However, one interviewee said the Agricultural Directorate continues to have its own early warning system. 132 This is a sign that EWD may not be producing the sort of data or analysis that this particular unit needs. Although there are no data to indicate that the same finding may apply to other directorates, the issue is raised as a possibility. REWARD is using the ECPFS platform to increase external collaborations. 127 Interview, REWARD representative, May 1, 2019. 128 Interview, REWARD representative, May 1, 2019. 129 Interview, ECOWAS representative, May 2, 2019. 130 In this vein, it should be noted that EWD and PAPS are physically located in different buildings, a situation that aggravates the pre-existing miscommunication between PAPS and EWD. The Government of the People’s Republic of China is building new headquarters for ECOWAS, one in which PAPS would be brought physically closer to EWD. Also see: Interview with USAID representative, May 16, 2019. 131 Interview, USAID representative, May 9, 2019. 132 Ibid. 50 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV PAPS’s collaboration with others in ECOWAS is supposed to be ensured by the ECPFS. To facilitate this collaboration, there was an attempt (in 2010) to establish an Internal Steering Committee (ISC) for the ECPFS. 133 The scheme did not work because of lack of added-value information and relevance to the invited ECOWAS units.134 Later, REWARD attempted to establish a similar platform (the Inter-Directorate Working Group or IDWG), but it also failed to sustain interest in ECOWAS. This attempt failed for the reasons described above, but also because it was never institutionalized through a set of SOPs and clear business processes between units. REWARD then decided to work within the existing structure which the ECPFS offers.135 It was only when REWARD, through the ECPFS platform, facilitated a 2018 retreat that inter-directorate collaboration and communication were invigorated.136 Now, with the improvements in the quality of EWD data, and with REWARD’s funding, the ECPFS has found a second support for its business—the first being the just-published137 ECPF 2018–2020 Action Plans. The ET finds that by involving staff from directorates other than EWD in the completion of CRVAs (and of the JARP and CPX), REWARD has generated two benefits: (1) non-EWD staff have come to appreciate the quality and rigor of the EWD data and (2) non-EWD staff are brought together with EWD to better understand how they work and the nature of the information they collect. About the Conflict Scenario training of March 2019, I can testify that it was an excellent workshop. It generated a culture of collaboration and pulling in expertise from different departments when developing reports and identifying which person would require action in response to conflict. ECPFS believes this is just a start for other things we can do to strengthen ECOWAS’s capacity to recommend to the President and management on how to intervene.138 Interviews are rich with testimonies of staff from DPA, EWD, and DPKRS seeing the value of working together. As one interviewee put it, involvement in the CRVA, specifically, was a “light bulb moment”139 that has “opened the eyes” 140 of the non-EWD staff to the value of that directorate’s work. Interviews suggest that, as a result, the resistance that the DPA and DPKRS exhibited at the beginning of the project has decreased progressively. 133 Interview, ECOWAS representative, May 13, 2019. 134 It also suffered from a leadership challenge (when the former CPAPS and the Director of DPA who were invited to launch the meeting did not attend at the last minute) in a context where some perceived the ECPFS as encroaching on the prerogatives of some directorates. 135 REWARD funded the ECPFS Directorate Retreats for ECPFS Focal Point directorates (2018 and 2019). REWARD held a training of ECPFS Focal Points in Conflict Prevention Project design (June 2018), and a Crisis Scenario Planning Exercise (March 2019). Interviews reveal that all meetings before 2018 were characterized by a lower-than-desired attendance because staff from other directorates could not see what they could gain for their work by attending them. Thus, rather than attending ECPFS￾sponsored meetings, directors from invited units sent lower-level representatives who did not have the authority to commit their units to “decisions” pertaining to analysis findings or response planning. All interviewees responding to our questions on the matter admitted that the meetings were not well-attended. 136 Ibid. 137 With EU funding. 138 Interview, ECOWAS representative, May 2, 2019. 139 Interview, ECOWAS representative, May 2, 2019. 140 Interview, ECOWAS representative, May 7, 2019. REWARD’s technical contributions are seeds that help us. So, today, when we invite people to our meetings, they come. —Interview, ECOAS/ECPFS representative, May13, 2019 USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 51 REWARD ensured that other directorates have a reason to collaborate. Our sources added that REWARD, offering something useful at ECPFS retreats and events, has helped to increase participation in such meetings. While these are positive developments attributed to REWARD, the ECOWAS decision to position ECPFS within PAPS limits its ability to bring about inter-directorate collaborations. Right now, it’s under PAPS, and under DPA. It’s not a Secretariat, it’s a desk. There are few staff. Strategically I don’t think it was smart to place it under a division (DPA). They needed to put it under the Office of the Commissioner. It would be easier to put it under other department heads in order to engage other departments. All the staff in ECPFS are just contract staff, none are permanent. The more you elevate the more value you give it. We have been engaging with them on this matter.141 Collaboration between EWD and the national centers is nascent. Given that the NCCRMs are relatively new and are not fully functional, EWD has not developed a clear working relationship with analysts in those structures. Although EWD spoke of this relationship,142 it could not share with the ET any document that models the triangular relationship between EWD HQ, the field monitors, and the NCCRM analysts. REWARD has not supported building the capacity of NCCRMs under Purpose 1, but it is under pressure to do so by training NCCRM analysts. There are plans to do this training in FY 2018–19.143 ECOWAS does not have the money to sustain REWARD’s gains, but it may not need it. The ECOWAS community levy (created in 2003) requires the member states to transfer to the ECOWAS Community Fund 0.5 percent of the taxes they collect on the value of imported goods. These amounts cover about 90 percent of the ECOWAS budget.144 ECOWAS is largely self-financed and 95 percent of this transferred amount covers recurrent costs (e.g., salaries, election monitoring). However, the organization relies on donors for its continued institutional development, capacity-building, and to support the cost of other functions such as those that the ECPFS fulfills. ECPFS receives funding for its operations from DANIDA (at 25 percent of budget), the EU (10 percent), GIZ (3 percent) and REWARD (5 percent); the ET was told that the ECPFS is totally dependent on donors, but there are no other numbers available to confirm or negate this.145 CONCLUSIONS  Improvements in inter-directorate collaboration through work processes that REWARD is introducing are not supported as much now as they were under the leadership of the previous CPAPS.  REWARD’s TO does not consider the fact the EWD has now moved to the V-P’s Office.  It is too early to foresee what affect the EWD move to the Vice-President’s Office will have on PAPS’s internal and external communications and collaborations. The EWD move has opened channels of communication between it and other non-PAPS directorates. This reorganization provides REWARD with an opportunity to contribute to the creation of SOPs that would flesh 141 Interview, REWARD representative, May 9, 2019. 142 Interview, ECOWAS representative, May 2, 2019. 143 REWARD_AID-624-TO-15-00004_FY18 Annual Report. 144 Interview, ECOWAS representative, May 7, 2019. 145 Interview, ECOWAS representative, May 2, 2019. 52 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV out the relationship between EWD and all other departments and their directorates within the ECPF mechanism, which brings together 15 directorates (not only PAPS).  There is not enough evidence to state categorically that the JARP and CPX have improved relations between EWD, DPKRS, and MFD. However, REWARD can be credited with contributing to improved communication and collaboration between PAPS (when EWD was within it) and other departments.  Senior management approaches to force inter-directorate collaboration in leveraging EW data have not worked well in the past. Despite its strategic position (and mandate) within ECOWAS, ECPFS is still unable to provide a viable framework to facilitate inter-directorate collaboration to implement programs and activities. In addition to having few staff and limited financial resources, ECPFS suffers from the absence of clear terms of reference and SOPs to formalize the function of this platform for inter-directorate collaboration.  PAPS does not have the institutional and professional capacity to sustain the project’s contribution without stronger management support, funding for ECPFS, and a clarification of the unit’s role and positioning with regard to all ECOWAS directorates.  The thematic nature of EWD’s reports makes it hard for DPA and DPKRS to carry out their mandates, but through ECPFS retreats such as the JARP and CPX events, REWARD was able to contribute to improving internal communications between EWD and DPA.  Work that has begun with ECPFS has shown the unit’s potential as a useful, but weak, mechanism to support inter-directorate collaboration in support of PAPS’s mandate.  Knowledge gains from the project ensure partial sustainability of the project's results, but sustaining lasting changes will rely on continued financial support from REWARD. RECOMMENDATIONS 1. To enhance inter-directorate collaboration, REWARD needs to encourage the EWD to continue efforts at joint analysis and recommendation formulation currently under the aegis of the ECPFS. 2. Because EWD is not part of the PAPS structure any longer, USAID should amend REWARD’s TO to reflect this change. 3. REWARD should not work on SOPs for inter-directorate collaboration, but rather focus its attention on leveraging the ECPFS and the ECPF 2018-2020 Plans of Actions, through the JARP and CPX, to bring together directorates. 4. As is discussed under Evaluation Question 8, to the extent that these types of workshops are seen as important work processes, REWARD should leverage the EU-PSS business process reengineering project and have ECOWAS itself draw up the SOPs that will institutionalize these work processes. 5. EWD should produce two kinds of reports that will appeal more to ECOWAS departments and directorates: thematic and by country (still organized following HSAF thematic areas). 6. REWARD should build the EWD data collection, analysis, recommendation-formulation, and reporting capacity functions to meet specific targets of organizational performance in delivering its mandate and work with other donors to ensure that funding streams are mutually reinforcing. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 53 EVALUATION QUESTION 5 What internal and external factors prevented or supported Creative from accomplishing the REWARD expected results? Many of the conclusions, findings, and recommendations to this question have been provided in the preceding sections of this report. This section attempts to regroup them and highlights additional elements that USAID could consider. FINDINGS INTERNAL ENABLING FACTORS Leadership was critical to success. The leadership of Creative, Fund for Peace, and the REWARD Chief of Party (COP) are mentioned as success factors for the project. The fact that REWARD hired a COP who was, formerly, the liaison of WANEP in ECOWAS facilitated the success of all the activities at this stage of REWARD program. It took a lot of communication and feedback and a lot of collaboration in order to work through each and every activity […] Trust and collaboration was established due to the fact that Mr. Diallo has worked with EWD before helped a lot.146 [Success factors include the] the leadership role of the Creative team — they have a very good track record of managing high-value projects and they have the tools for that… The Creative team at HQ makes sure that any high-value activity — they come down… The whole project was designed in such a way that ECOWAS can take ownership. They tell us every year what they want — it is demand-driven.147 For major events, Creative has travelled to the site to help facilitate and guide key activities, a reflection of its leadership style. Collaborative approach enhanced respect. Respondents inside and outside ECOWAS cited a collaborative, respectful approach on capacity-building, consistent involvement from Creative’s leadership, responsiveness to needs, and flexibility from implementing partners as factors of success. “It wasn’t about telling them what they were doing wrong but working shoulder-to-shoulder to add value to what they were doing… .” 148 Embedding a technical staff enhanced collaboration. “Embedding technical staff has also been critical in success— they are seen as ECOWAS staff.”149 For those who spoke about this topic, this type of posting has ensured close collaboration between REWARD and EWD, as well as the establishment of a mechanism that facilitates making REWARD a demand-driven project. Just-in-time technical expertise was provided. The just-in-time provision of expert technical staff to ECOWAS from Creative, the Karuna Center for Peacebuilding, and Fund for Peace is mentioned as an enabling factor, specifically in regard to training, the 146 Interview, REWARD representative, May 9, 2019. 147 Ibid. 148 Ibid. 149 Interview, REWARD representative, May 9, 2019. 54 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV gender manual, capacity-building on the GIS and data collection system, the CPX, the implementation of the CRVA methodology, and the revision of the indicators. Well targeted, high quality, training [One success factor is the] reinforcement of capacity. Our domain is the prevention of conflict and the analysis of data. We have had a lot of modules of very high quality, and even the training personnel are highly qualified, and now we have a framework of what we need to follow. The pedagogical method is well-targeted.150 Working within existing structures assisted coordination. According to interviewees both inside151 and outside ECOWAS,152 Creative prioritized sustainability in their implantation approach. This approach includes utilizing existing ECOWAS structures to pursue project outputs. Multiple interviewees specifically mentioned the leveraging of the ECPFS as an inter￾directorate coordinating mechanism, especially after the limited success of establishing the IDWG.153 Working at the director level helped progress. REWARD points out that despite the need being felt for more openly supportive leadership from the senior level of ECOWAS, the project has ensured that directors and staff (who do not change as frequently as senior leaders) are targeted for interventions and that directors approve all REWARD initiatives with their staff before they are implemented.154 This has helped REWARD circumvent the relative absence of senior ECOWAS leadership and the lack of formalization of inter-directorate business processes. Open communication was a best practice. Another best practice that characterized REWARD’s approach to the project was its openness to communication. REWARD’s ability to listen155 was reported as an important factor that has made REWARD a success. As ECOWAS is a highly politicized environment, it takes a lot of trust building for donors to be able to implement a successful program. Relationship and trust building…. There have been a lot of donors coming in and out of ECOWAS and a previous USAID project was unsuccessful. It is a very politicized institution and people have to be very careful of not stepping on toes of superiors.156 REWARD responded to EWD needs. In conjunction with the responsiveness of the Creative team to EWD’s needs, interviewees within the EWD noted that the internal assessment completed prior to the beginning of the project enabled the EWD to have a clear grasp on its needs. This needs assessment framed the contribution and scope of the REWARD project.157 150 Interview, ECOWAS representative, May 8, 2019. 151 Interview, ECOWAS representative, May 2, 2019. 152 Interview, REWARD representative, May 9, 2019. 153 Interview, ECOWAS representative, May 7, 2019; May 13, 2019; Interview, REWARD representative, May 9, 2019. 154 Interview, REWARD representative, May 2, 2019. 155 Interview, ECOWAS representative, May 2, 2019. 156 Ibid. 157 Interview, ECOWAS representative, May 3, 2019. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 55 REWARD leveraged pre-existing coordination mechanisms. The ECPFS provided an existing structure through which REWARD can contribute to improved communication and collaboration, such as the coordination of the CPX. The ECPFS also established focal points at the level of ECOWAS directorates, a network that REWARD also leveraged to coordinate between directorates.158 EXTERNAL ENABLING FACTORS Senior-level leadership helped REWARD. An interviewee from REWARD and one from ECOWAS mentioned the commitment from high-level individuals within ECOWAS to REWARD activities as an enabling factor, especially at the outset of the project. These individuals include the former CPAPS and the former Executive Assistant to the Commissioner of PAPS.159 INTERNAL INHIBITING FACTORS There is an inappropriate legal framing of the project. This factor has an internal facet as well as an external one. Internally, USAID/REWARD decided to not craft a legal document framing the mutual expectations of USAID/REWARD and ECOWAS (unlike what exists for EU projects) upon initiation of the project. As will be seen below, two interviewees mentioned that ECOWAS and directorates other than EWD did not understand the mutual obligations of partners or that REWARD would entail financial and other commitments from ECOWAS. 160 Changes in COP affected the project. There was an initial in-country COP whose style negatively affected the relationship between REWARD and EWD.161 REWARD replaced this person with the current COP, who was formerly known to EWD, and this has helped redress the relationship. EXTERNAL INHIBITING FACTORS ECOWAS did not require a legal framework for REWARD and this slowed down the project’s acceptance by units other than EWD. The ET finds that the absence of legal documentation to circumscribe the ECOWAS/USAID/ DOS spheres of action led to confusion and territorial conflicts between the EWD and other PAPS directorates. 162 This situation has been beneficial to EWD because it could ask USAID for what it needed. It also allowed EWD and USAID to be more flexible and created the conditions for REWARD to be a demand-driven project and to be adaptive in its work planning.163 However, the absence of this legal framework has also slowed down implementation of the project by other, non-EWD, directorates. REWARD was seen, at first, as only an EWD project—not one aimed at building the capacity of all units responsible for leveraging EWD data to craft responses to emerging issues. It was only after repeated contacts and attempts with DPA and DPKRS that these directorates accepted to join the program: 158 Interview, REWARD representative, May 1, 2019. 159 Interview, ECOWAS representative, May 7, 2019; Interview, USAID representative, May 3, 2019. 160 Interview, ECOWAS representative, May 6, 2019; Interview, ECOWAS representative, May 7, 2019. 161 Interview, USAID representative, May 3, 2019. 162 Interview, Development partner, May 9, 2019. 163 Interview, ECOWAS representative, May 6, 2019. 56 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV When it comes to the design of this project, I think we need to be very clear that the project enters ECOWAS by the window, not the main gate. Why? I recall attending the opening meeting at the Main Hall where a memo was being signed with USAID and, at that signing ceremony, later on I thought that this was the REWARD project that we were signing. We were asking what the legal basis of the partnership was? Because for me, for all the projects I’ve managed, it starts by a project document and intention to cooperate. Then you budget it for it, and then we agree and sign. But there was never a REWARD document. This has some consequences, meaning that if I am the direct counterpart, I can contribute to what I want and what I don’t want. I am working with all the EU projects in peace and security and the action document is my Bible, I refer to it, and I can tell which activity can be done and which cannot. There is no agreement between the US government and us. When it is like that, you have to compensate and agree piecemeal. For me, it was an early warning project until the end of 2017 when the former Commissioner requested that we have a retreat on REWARD. What our EW colleagues were presenting to us was different than what USAID was saying. The then PAPS Commissioner Halima Ahmed (now the Commissioner of Finance) had to force for a retreat with USAID saying that “I want to understand.” Personally, I refused twice to receive them because I thought that I had nothing to do with the project. It was at that retreat my colleague from the Political Affairs, and myself that we learned, along with PAPS, that we have something to do with this project. So, it’s only from then that we got involved in early warning.164 ECOWAS interviewees report that ECOWAS recruited several staff at REWARD’s behest (and with partial funding from DOS). Although USAID did not expect that REWARD would pay their salaries, ECOWAS officials think that that these staff were left without pay for several months because there was no clear legal instrument that supported how the charges are to be shared between ECOWAS and REWARD. As a result, DPA and DPKRS were left with a financial burden which was not previously budgeted for by ECOWAS.165 REWARD was seen as an EWD project. The initial design and approach of the project brought about the perception from other directorates that the REWARD project was only for EWD, specifically because there was also no legal document signed to guide the project and because they knew that the EWD director was “holding the pen.” The support by the US government to ECOWAS was very much tied to support to early warning…When the USAID contribution was announced it was going to EW, and the rest of the PAPS wondered if there was support for others. When the attempt was made to extend it to other parts of the department, there was some pushback.166 ECOWAS’s organizational culture is an impediment to REWARD. 164 Interview, ECOWAS representative, May 6, 2019. 165 Interview, ECOWAS representative, May 2, 2019 and May 6, 2019. 166 Interview, Development partner representative, May 9, 2019. There is a tendency for us to sit down in our own office and think on how things affect us. But when I attended that scenario [CPX] workshop, I saw that there is a need for us to coordinate together, and the exercise brought together different technical directorates of the commission. And you can where the other Directorates are coming from, where USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 57 In the field of security where REWARD operates, ECOWAS receives support from GIZ and the EU-PSS to build up its organizational capacity. Even so, ECOWAS’s organizational culture and hierarchical systems continue to slow down the pace at which the impact of these projects can be felt. Various attempts, including ECPFS retreats prior to 2018 and the establishment of its ISC as well as REWARD’s IDWG, have not been very successful in bridging the “data-to-response” gap.167 To circumvent this problem, REWARD has successfully implemented an approach of engaging staff from other directorates in CRVAs and events such as the JARP and CPX. This approach seems to have produced better results compared to previous ECOWAS attempts to reform its use of data to respond to emerging risks and vulnerabilities through the establishment of formal Committees. ECOWAS’s organizational dynamics and culture hinder the project’s success. Multiple factors within ECOWAS and thus outside the Creative/REWARD sphere of control pose challenges to achieving expected results. In addition to the complexity of regional peace and security issues themselves, interviewees highlighted the hierarchical, siloed, and slow-moving nature of ECOWAS as an inhibiting factor, as illustrated in the following interview excerpts: Working with ECOWAS is challenging — scheduling and changing. ECOWAS is not used to the system we have in place. The requirement for USAID approval — in the context when you have to move fast, it can be difficult.168 One of the challenges in ECOWAS is duplication of work and working in silos. Joint planning has evolved from first just getting focal points to work together. Many of the focal point directorates weren’t familiar with early warning reports and tools, but through retreats they have become familiar.169 An information-sharing gap is possible with EWD’s move out of PAPS. The ECOWAS decision to move EWD outside of PAPS to the Vice-President’s Office affects REWARD because this change created a gap in how information is now produced and shared. There are no SOPs for how EWD should communicate with other departments.170 Furthermore, the same interviewee stated that there is difficulty with maintaining an institutional memory within ECOWAS. The institutional memory [is absent]. Individuals work and gather experience and they understand the justification for certain mechanisms. But once these people leave, those who are left behind or are new may not adequately carry out these mechanisms. There are some mechanisms that have been painstakingly created that are changed with new management.171 For REWARD, this move has the potential to create problems if there is no change in its TO. Though the move did not necessarily affect day-to-day activities, the original TO focuses on PAPS, including EWD. 167 Interview, ECOWAS representative, May 2, 2019. 168 Interview, REWARD representative, May 9, 2019. 169 Interview, ECOWAS representative, May 2, 2019. 170 Ibid. 171 Ibid. they can feed an input. And with the fictional country they used, you now understand that conflict analysis has multiple dimensions, you might be thinking that political, environment issues might be involved, legal issues might be involved, good governance issues might be involved. So, it’s kind of broadened our understanding of conflict. And then the other thing is the network, it brought colleagues together, they are not communicating effectively among themselves so that particular experience was really helpful to me. It began to streamline our thoughts. —Interview, ECOWAS Officer, May 14, 2019 58 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV Yet, now that EWD is outside of PAPS, REWARD must navigate additional systems in order to achieve the project’s intended results. Changes in ECOWAS management priorities negatively affect REWARD. Interviewees note that, with change in leadership positions in ECOWAS, there comes change in priorities between commissioners. For example, one REWARD respondent (echoed by USAID and a donor) highlights the fact that the “previous Commissioner of PAPS prioritized inter-directorate SOPs but is not a priority for the current Commissioner.” This leads to disjointed progress.172 Interviews and documentary evidence point to the fact that there has been great difficulty from ECOWAS in following through with agreed-upon commitments and there is constant change, delay, or cancelling of actions, activities, and travel. For example, the beginning of the CRVA was postponed due to internal politics; the Council of the Wise has yet to be established, and: the Youth Assessment has yet to begin. The organizational structure of ECOWAS is not built to move quickly, which can be required with a project such as REWARD. 173 There is no apparent ECOWAS senior management support to sustain REWARD’s gains. There is a desire on the part of interviewees to continue with the project (and even to have it extend to NCCRMs), 174 but the ET confirmed with those it questioned on the topic that there is no documentary or verbal evidence of any ECOWAS plans, commitments, messages, or budgets to make this “wish” come true. There are also no work plans to further consolidate efforts at the level of the ECOWAS headquarters. USAID decision-making processes are slow. Some interviewees lamented about the amount of time it takes USAID to make decisions. Examples provided by interviewees included the absence of a decision on the CRVAs, as well as the time that it has taken for USAID to approve the gender manual once ECOWAS technicians approved it.175 CONCLUSIONS  REWARD laid the foundation for a sustainable project through consistent communication and a collaborative and responsive approach to an already completed EWD needs assessment, as well as by working within existing structures at the levels of directors and staff.  The REWARD style of leadership, openness to communication, flexibility, and the demand-driven approach of REWARD implementing partners contribute to REWARD’s success with EWD and PAPS.  The just-un-time provision of high-level technical expertise and training from Creative, the Fund for Peace, and the Karuna Center for Peacebuilding are essential for the success of the project. 172 Interview, REWARD representative, May1, 2019; Interview, USAID representative, May 3, 2019; Interview, Development partner, May 9, 2019. 173 REWARD_AID-624-TO-15-00004_FY18 Annual Report’. 174 Interview, ECOWAS representative, May 7, 2019; May 8, 2019; and May 17, 2019. 175 Interview, ECOWAS representative, May 2, 2019. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 59  Creative/REWARD diligently mitigated the difficulties from the beginning of the program. These have been successfully corrected, but the absence of an internal and external mechanism of coordination within ECOWAS may impede the impact of REWARD on the EWS.  Senior-level support within ECOWAS has helped REWARD progress. The absence of higher-level management support at ECOWAS is dangerous for REWARD’s ability to reach its goals of bridging the gap between data and response, notably through the under-resourced and technically limited ECPFS.  Leadership changes within Creative/REWARD and ECOWAS inhibited the progress of the project, but the leadership qualities and prior experience of the current COP with ECOWAS helped achieve the project’s outputs.  The absence of a legal framework for the project is important in a rule-bound organization steeped in formal agreements and accords, such as ECOWAS. While respondents from non-EWD directorates have come to understand and appreciate REWARD, the mode and point of entry of REWARD into ECOWAS generated a perception, lasting until 2018, that the project’s focus was only on EWD instead of on creating a link between EWD data and other directorates responsible for leveraging it for response purposes.  The organizational structures within ECOWAS and USAID, specifically their hierarchical, slow￾moving nature, have inhibited progress.  The ET concludes that the institutional and administrative environment of ECOWAS is characterized by a legalistic mindset, bureaucracy, politicization, and a culture of doing work in non-communicating silos. ECOWAS’s organizational culture and processes undermine the establishment of data-to-response bridge between EWD, other PAPS, and non-PAPS directorates that the project seeks to achieve. ECOWAS’s bureaucratic and silo dynamics have prevented REWARD from bridging the data-to-response gap, even though the JARP, CPX, and AAR events have laid the foundation of change in ECOWAS’s behavior.  The SOP for the EWD and DPA are not updated nor implemented to reflect the change in organizational structure following the EWD’s reorganization from PAPS to the Vice-President’s Office. The EU-PSS project provides an opportunity to do so in a way that will bring coherence to the whole of the ECOWAS business processes. RECOMMENDATIONS 1. USAID and REWARD need to maintain their demand-driven approach to ECOWAS while working with other donors to steer the course of linking response to data. 2. REWARD should maintain the presence of an embedded advisor. 3. USAID may wish to review the TO for REWARD in order to reflect the current and foreseeable situation in ECOWAS. 4. REWARD and USAID should seek to enroll a member of ECOWAS's senior management to champion REWARD and other projects aiming to improve inter-directorate collaboration. 5. The REWARD COP should continue his approach and remain in place until the end of the project. 60 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 6. If USAID decides to pursue its support to ECOWAS in a future REWARD-like project, it should ensure that it can frame ECOWAS’s commitments with a legal document that will spell out the mutual expectations of all parties involved. This should apply throughout the ECOWAS constellation of stakeholders, including a member of ECOWAS management who can serve as a champion for change. 7. REWARD should work with the EU-PSS project to ensure that the SOPs for the EWD and DPA are amended to reflect organizational practices and needs. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 61 EVALUATION QUESTION 6 Which of the program design assumptions are no longer valid or changed during the activity implementation that might have affected the program implementation? FINDINGS Since this evaluation question is relevant to only a select category of interviewees, specifically those involved in the initial design of the project, there are a limited number of responses. The REWARD Performance Monitoring and Evaluation Plan lists the following assumptions:176 1. Willingness of ECOWAS and its directorates and other partners to cooperate with REWARD in the project implementation; 2. Political and security stability in the region; 3. Response partners will have the requisite capacity to effectively implement activities and/or recommendations; 4. The on-going organizational re-structuring (sic) in ECOWAS supports inter-directorate coordination and integration of early warning and early response processes; 5. ECOWAS and potential partners will fund measures to sustain the commitment and motivation of community monitors and analysts; 6. National stakeholder’s interest is maintained; 7. Political atmosphere remains conducive for advocacy on early response. All assumptions remain valid, except the fourth one. The only two respondents to this question stated that all assumptions remain valid, though Assumption 4 may need special attention because of the move of the EWD to the Vice-President’s Office.177 This move may lead to some repercussions on REWARD’s ability to bring data and response closer, though it is too early to predict any impact. CONCLUSIONS  Particular attention needs to be given to the impact that the EWD move to the Vice-President’s Office will have on REWARD’s ability to fulfill its mandate. RECOMMENDATIONS 1. Based on the above conclusions, the ET recommends that USAID/Regional Peace and Governance Office management track how Assumption 4 may or may not change and impact REWARD’s ability to meet its mandated objectives in light of EWD’s move to the Vice-President’s Office; modify REWARD’s TO accordingly. 176 REWARD Performance Monitoring and Evaluation Plan, April 2019 version, p. 17. 177 Interview, REWARD representative, May1, 2019; Interview, USAID representative, May 3, 2019. 62 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV EVALUATION QUESTION 7 Which activities of the REWARD Task Order should continue and why? Which should be modified and why? The findings and conclusions reported upon here reflect those found in the documents the ET analyzed and the interviews it conducted. The ET formulates its own recommendations in light of this question’s findings and conclusions. FINDINGS All activities in the TO are ongoing and making progress except for those regarding the Council of the Wise and the Mediation and Facilitation Division.178 The CoW does not exist except in name, due to lack of movement on the part of ECOWAS. The MFD was absorbed into the DPA during organizational restructuring. Given the demand-driven and consultative nature of the REWARD project, REWARD initiated specific activities where EWD expressed need. Overall, the project is exceeding its planned outputs except for AARs and 60 percent of EWD report readers do not see reports as being of “high quality.” An analysis of the REWARD PITT that the Mission shared shows that the project is reaching a performance level of 310 percent with regard to its EOP projected outputs. Analysis of the latest PITT indicators to which the ET has access reveals that 40 percent of respondents179 see the reports as being of high quality. The implementation of AAR recommendations is not occurring (0 percent) and only 18 percent of AARs are jointly held with ECOWAS early warning and early response units.180 This provides an indication of REWARD’s challenges moving forward and a direction in which to concentrate the activities for the remainder of the project’s life. SHORT-TERM ACTIVITIES TO CONTINUE Continue capacity-building in analysis. Interviewees unanimously recognized181 the importance of the capacity-building that REWARD brought to ECOWAS and hope that the program continues to better support the work of EWD analysts in producing analysis and reports. I hope it [the project] won’t stop now because I think this project has created structure and it goes beyond this for me. Beyond these structures, you have the other aspect which is to equip people with skills to be able to read data. I’m not yet convinced that we have succeeded in linking the response to the analysis.182 Continue REWARD’s accompaniment. For others, capacity is essential in order to be prepared to face new challenges and also to establish mechanisms to improve collaboration between directorates. “Capacity-building does not end, there are new 178 See Annex 7 for a table showing the progress to date against current indicators. 179 Performance Monitoring & Evaluation Plan 2015–2020. 180 Ibid. 181 Interview, ECOWAS representatives, May 2, 2019; May 3, 2019; May 6, 2019; May 8, 2019; and May 17, 2019. 182 Interview, ECOWAS representative, May 6, 2019. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 63 challenges, new tools, it is still good that REWARD accompanies us as we are developing a culture of working together.”183 Consider answering calls for more GIS tools Interviewees from the technical level noted that further training needs to be done with the GIS staff on web-based applications for field monitors. Although this is outside REWARD’S SOW, USAID should consider the following data point to further build the EWD analytical capacity. We need to reinforce the capacity that has been built. Right now, there are online applications, on smartphones. It would be helpful to have an application that has localization function for the field monitors, for example during elections. Web mapping, web design to make things more attractive.184 Consider calls for another regional risk and vulnerability assessment to consolidate skills Interviewees strongly recommended that REWARD continue on its current course of building ECOWAS’s capacity to move through the cycle of joint data collection, analysis, scenario planning, response, and after-action review.185 One interviewee within EWD suggested replicating the Lake Chad assessment exercise with the Sahel region, with support from REWARD, to practice skills gained throughout the project.186 Prepare country-level reports to facilitate the responses The recent shift from country-level analytical reports to thematic ones makes it more difficult for the DPA/MFP and DPKRS as well as other response mechanisms in ECOWAS to respond, at their responses are country-specific. 187 There are calls to add country-level reports to the thematic ones that EWD already prepares. Approach the work with ECPFS with caution. Regarding the use of the ECPFS as a convening platform for inter-directorate collaboration, there were mixed responses among those interviewed. One interviewee from REWARD stated that it is important to work within existing structures, such as the ECPFS, in order to improve the sustainability of the project.188 The ECPFS demonstrated weak but real potential in its role during the JARP and CPX. Due to its placement within the organizational hierarchy, it could be used to play the role that REWARD intended for its IDWG. However, not only is it under-resourced and not a technical body, it does not have broad power to coordinate Commissioner-level buy-in.189 This raises questions as to how this unit could be approached. REWARD has an opportunity to tighten its working relationship with the EU-PSS project to ensure that ECPFS’s role as a coordinating body is fully exploited.190 LONG-TERM ACTIVITIES TO INITIATE 183 Interview, ECOWAS representative, May 17, 2019. 184 Interview, ECOWAS representative, May 8, 2019. 185 Interview, ECOWAS representatives, May 2, 2019; May 3, 2019; May 6, 2019; May 8, 2019; May 17, 2019. 186Interview, ECOWAS representative, May 8, 2019. 187 Interview, ECOWAS representative, May 2, 2019. 188 Interview, REWARD representative, May 9, 2019. 189 REWARD_AID-624-TO-15-00004_FY18 Annual Report; Interview REWARD representative May 9, 2019. 190 This role could be further heightened if the ECPFS was moved to the Vice-President’s Office alongside the EWD, something that REWARD could promote with that Office and the EU-PSS. 64 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV Push capacity-building down to the national level. Developing the capacity of staff in NCCRMs is seen by many as the next logical step in REWARD’s Phase 2, should there be one. Although outside the scope of the current project, those who hold position within ECOWAS stated that it would be beneficial for capacity building to push down its capacity efforts in data collection and analysis to the national and community level, specifically with regards to the NCCRMs. Ultimately, the response in early warning is done at the national level, and interviewees from EWD state it is critical that these centers have the skills to perform their function well. One interviewee recommended that REWARD develop SOPs for the national centers and train staff in data collection and response methodology using the HSAF. I would explore how we can support ECOWAS in engaging with the Member States. If you want to promote peace and security and the community engagement approach is best. We need to capacitate the Member States to take ownership of the conflicts.191 Even though member states respond to conflicts within their own borders, the NCCRMs are not yet functional enough (competency-wise and financially) to be formally linked to the EWD. A functional process of seamless interaction between the NCCRMs and the ECOWAS’s EWD is not defined, let alone, in place. CONCLUSIONS  The project performed well with regard to its output indicators. Besides the need to consolidate the practice of joint analysis and planning, more efforts are required to implement AARs and to further improve the analytical quality of the reports.  Activities regarding the Council of the Wise and the MFD were not started due to circumstances outside of REWARD’s control. Therefore, TO activities regarding them are outside of REWARD’s control.  Enhancing the GIS with additional tools could enhance the EWD’s ability to more quickly geolocate events.  There is a need to strengthen the links between response and data. The ECPFS has been useful in this regard, in spite of its financial and human resource constraints.  ECOWAS interviewees ask REWARD to continue with its capacity-building efforts to enable ECOWAS to move through the full cycle of joint data collection, analysis, scenario planning, response, and after-action review.  Pushing down analytical capacity-building to the level of the NCCRMs may seem a logical step to many, but it is premature given the dysfunctional status of these national-level institutions. REWARD needs to wait for the establishment of functional NCCRMs, with the support of other donors, before extending its training activities to that level.  The production of country-specific reports, in addition to thematically organized regional ones, will facilitate ECOWAS’ capacity to respond to emerging conflict situations. RECOMMENDATIONS 191 Interview Evaluation Team 9 May 2019. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 65 Based on the conclusions, the ET recommends the following: 1. Modify or cancel the activities pertaining to the Council of the Wise. 2. Modify or cancel the activities pertaining to the Mediation and Facilitation Division. 3. Modify outputs 1.3 and 1.4 to target more specifically the ECPFS rather than DPA and PAPS only. The ECPFS, although weak, could be supported on a case-by-case basis and in coordination with other donors. In this way, it could implement joint analyses and planning exercises to better link the EWD data with ECOWAS’s response and support the integration of this work process into the SOPs of ECOWAS. 4. Consider responding favorably to SMD’s request for additional GIS tools to more quickly geolocate events 5. Provide support to EWD to produce country-level analysis by theme to improve the uptake of its products by PAPS and other directorates; do not relinquish the production of thematic reports for other ECOWAS directorates. 6. At the end of Year 4, hold a retreat with all stakeholders to conduct an AAR of REWARD before designing the next phase, should there be one. 7. Extend the project for another year to leverage the work that the EU-PSS is about to start in reengineering ECOWAS’s business processes. This would ensure that the work processes (JARP, CPX, AAR) that REWARD has brought to bear on ECOWAS are institutionalized at the level of the organization’s headquarters. 8. REWARD should not extend its capacity-building efforts to the NCCRMs during this phase of the project. Rather, it should concentrate on building the EWD’s capacity to train their NCCRM counterparts (through training-of-trainers). 9. While the aforementioned recommendation is being implemented, USAID should devise a future (post-2020) phase of a REWARD-like activity (perhaps implemented by USAID’s country-level Missions and coordinated by USAID/WA) to extend capacity-building efforts to the NCCRMs— if they are deemed functional and supported by the member states. EVALUATION QUESTION 8 To what extent was REWARD complementary to the State Department (or other donors) support to ECOWAS and EWD? Evaluation Question 8.1: How can USAID REWARD further complement the State Department’s or other donors’ assistance to ECOWAS going forward? FINDINGS DOS 66 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV USAID and DOS efforts are complementary within REWARD. Interviewees state that there is complementarity between the mandate of the DOS and REWARD.192 This is said in the context where the infrastructure, equipment, and software is provided by the DOS, especially at the national centers, while REWARD provides capacity-building, as well as technical expertise and assistance. One respondent at USAID noted that, at the inception of REWARD, there was some duplication of activities; Creative has done well to liaise with DOS to understand its plans and adjust REWARD’s accordingly. Another respondent within EWD confirmed that the strong collaboration between Creative and DOS was facilitated in part by the EW Advisor embedded within the EWD.193 The development of SOPs to build onto DOS work constitutes an opportunity. One area of potential complementarity that has yet to be fully implemented is the SOPs for the EWD. DOS hired a consultant designated to revise the SOPs to improve the communication and collaboration within the EWD. An interviewee stated that the consultant completed the development and proposed the SOPs; they have yet to be reviewed and are therefore not operational. The consultant has since left.194 OTHER DONORS REWARD presented its Year 4 work plan to other donors active in the peace and security portfolio. Given that there are many development partners working with ECOWAS, it is critical that REWARD’s activities complement those of other donors in order to avoid duplication, inefficiency, and waste. Donor interviewees 195 stated that there was initial duplication of activities between REWARD and donors, but since then the COP has undertaken concerted efforts to present REWARD’s annual work plan to complement, rather than duplicate, activities.196 REWARD does not leverage the work of other donors also working with EWD/PAPS. REWARD and USAID were criticized for their regular absence in the donor-coordination meetings.197 Furthermore, donors find that there was not enough information shared between REWARD, GIZ, and EU/PSS, who are the leaders of the Peace & Security thematic group of donors.198 The few times that the REWARD team was present were for strategic meetings, most of which were limited exchanges of information on the future, or events to clarify the financial responsibilities of each. REWARD could do better coordinate its activities with those of donors. Throughout the course of the interviews, it became clear that GIZ and the EU work in close proximity to REWARD, though interviewees also mentioned DANIDA, DFID, and the Swedish International Development Cooperation Agency (SIDA). GIZ complemented U.S. efforts at the national level by funding five of the national centers and mobilizing the civil society organizations. There is hope that other partners will fund the remaining five. GIZ also contributes in field-monitor training, in technical capacity for PAPS and DPKRS, and in organizational development for PAPS. SIDA works in gender, as does REWARD, yet 192 Interview, REWARD representative, May 13, 2019. 193 Interview, USAID representative, May 3, 2019. 194 Interview, REWARD team, May 13, 2019. 195 Interview, Development partner May 9, 2019 196 Interview, REWARD representative, May 1, 2019. 197 Interview, Development partner, May 9, 2019. 198 Interview, ECOWAS representative, May 14, 2019. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 67 based on the interviews it is unclear if there is complementarity in activities between the two donors. DANIDA complements REWARD’s work in its funding of the ECPFS.199 REWARD cannot adequately speak for USAID at donor meetings. While REWARD has begun to be more present at donor meetings , it serves as an implementing partner with no decision-making power on whether, when, and how to implement the activities that USAID has approved for its work plans. It is bound by its contract with USAID and cannot change its workplan activities without risking an effect on its contractual obligations. Therefore, REWARD concludes that it cannot adequately represent USAID’s interests, but only those of the project for which it is responsible. ECOWAS is not taking the lead in transparently coordinating donor support. Despite areas of complementarity, six respondents highlighted significant challenges in the coordination of efforts between donors and ECOWAS.200 Sparse attendance, weak commitment, lack of capacity, and lack of leadership from ECOWAS hampered coordination mechanisms, such as the monthly peace and security thematic meeting for donors and ECOWAS. One respondent explained that even though the leadership of the donors’ thematic working group is supposed to rotate every two years, the current leader of the group has been in place for seven years and is not from ECOWAS, but from a donor organization. Furthermore, scheduled annual meetings occur rarely ; there were three meetings in the last seven years and the purpose of the meetings was to share information rather than share work plans and coordinate activities. Furthermore, a different respondent explained that ECOWAS uses the lack of coordination among development partners to leverage its own immediate priorities. He added that this results in duplication of efforts and resource waste. Contrary to other regional organizations, they have financial limitations, but they can run themselves. Where they need donor money is capital expenditures and projects. They are not very dependent on donors, contrary to other organizations, and they don’t really feel a need to pamper donors. Engaging with donors can be a bit of a burden… if we don’t take initiative on the donor side, ECOWAS is not going to move.201 Senior ECOWAS leadership in donor coordination is absent. In addition to organizational characteristics that bring challenges in coordination within ECOWAS and with donors, individual leaders affect synergy between donors and ECOWAS. One donor interviewee202 notes that the previous CPAPS was much more “amenable and accessible” when it came to development partner relations, but the current Commissioner is not. An interviewee from a partner development organization noted that they had little information from ECOWAS about REWARD activities, despite overlapping areas of work and REWARD attendance at the monthly thematic work group meetings. The same interviewee recognized the need for more organic exchange of information, especially among those with core involvement in ECOWAS (USAID, GIZ, DANIDA, SIDA, and EU). 199 Interview, Development partner, May 9, 2019; Interview, REWARD representative, May 13, 2019; Interview, ECOWAS representative, May 14, 2019. 200 Interview, Development partner, May 8, 2019; Interview, Development partner, May 9, 2019; Interview, REWARD representative, May 9, 2019; Interview, REWARD representative, May 13, 2019; Interview, ECOWAS representative, May 7, 2019; Interview, ECOWAS representative, May 14, 2019. 201 Interview, Development partners, May 8, 2019. 202 Interview, Donor representative, May 8, 2019. 68 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV To rectify the challenges with coordination within and between donors and ECOWAS, interviewees also noted that discussions have begun among development partners to finance a coordinator position that can serve as a liaison between ECOWAS and donors. CONCLUSIONS  USAID and DOS efforts are complementary within REWARD.  Weak coordination mechanisms between development partners are apparent at the project operational level and at the more strategic level of the donors’ meetings. In addition, ECOWAS senior leadership is not taking an active role in the coordination of donors in support of linking data to response. As a result, there is less-than-optimal sharing of information or synergy between development partners and ECOWAS to efficiently and effectively leverage each other’s efforts to effect transformative change—for ECOWAS to better link data to response and enhance organizational learning. RECOMMENDATIONS In light of the above conclusions, the ET recommends the following: 1. Collaborate with development partners to ensure that the senior-level Management Cell that is planned for the office of CPAPS is established as soon as possible. If need be, use part of the REWARD funding to pay part of that person’s emolument. 2. Distribute relevant reports to development partners to improve information flow and leverage the work of other donors working in ECOWAS’s security sector to improve organizational efficiency. 3. USAID/WA should attend donor meetings more frequently to coordinate high-level decisions and empower REWARD to adjust its work plans accordingly, without jeopardizing its contractual obligations under its TO. FURTHER EVALUATION QUESTIONS FOR ECOWAS The Inception Report203 lists a series of additional questions that the ET attempted to answer, based on the following. The SOW included a number of strategic questions that fall far outside the scope of a performance evaluation of REWARD. These questions ask about ECOWAS (or components thereof) more generally, and about how USG support (including but not limited to REWARD) has improved ECOWAS’s work. Though these questions are not part of the evaluation itself, the Inception Report stated that the ET could ask the following questions to specific key informants (KIs) that USAID identified to help guide future strategic-level evaluations of ECOWAS. The strategic questions posed in the SOW are as follows: 1. How many and what type of recommendations generated by the EWD early warning work have translated into action by the ECOWAS leadership or member states? 203 See page 19 of the Inception Report. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 69 2. What role, if any, would the ECOWAS early warning system/efforts play in meeting USG’s current and evolving objectives, particularly, in assisting countries to achieve self-reliance? Should USAID continue to invest in these organizations and why? 3. Has USAID’s investment in ECOWAS led to national-level impact, if any (though not part of REWARD’s objective)? 4. How has support to early warning and response improved Early Warning and Response? 5. Is ECOWAS recognized by regional governments as a force for regional integration across countries, and sectors? What evidence exists? 6. Is EWD conflict-prevention work relevant to the region in terms of advancing countries to become self￾reliant in preventing conflict? 7. If Yes, how does the work of the ECOWAS/EWD currently contribute to the national development of member states? While this REWARD performance evaluation partially responds to Question 4 above (see discussion in Evaluation Questions 1, 3, and 4 above), the ET would need to interview personnel in member states, leadership positions within ECOWAS who consume early warning data, and other USAID and USG personnel who make policy to be able to properly assess the other questions. The list of ECOWAS KIs that USAID provided to the ET identified people mainly located within PAPS/EWD. The KIs, however, were mostly middle-level staff and first-level executives with limited, if any, insight into these strategic-level questions. Crucially, the ET did not interview staff in member states or senior leadership in ECOWAS who consume the early warning data; such interviews are necessary to answer these questions. In addition, the documents the ET reviewed are of an operational nature and focus on the REWARD project only; the documents do not address issues at the level that the questions were formulated. Therefore, the ET could not ascertain the extent to which early warning has impacted other departments within ECOWAS or its member states. The ET recommends that to answer these questions, USAID design an ECOWAS assessment that provides resources including travel to member states and a longer period of data collection in Abuja for an assessment team to adequately address these gaps. 70 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 1: INCEPTION REPORT BACKGROUND On February 20, 2019, USAID West Africa’s Regional Peace and Governance Office (RPGO) tasked the Peace through Evaluation, Learning, and Adapting (PELA)204 activity to carry out a midterm performance evaluation of the Reacting to Early Warning and Response Data in West Africa (REWARD) program. Based on the evaluation’s Scope of Work (SOW) delivered to PELA by RPGO, the first task PELA is to undertake is drafting an inception report in response to the SOW. This document is the inception report and includes: a discussion of USAID’s purpose and objectives for the evaluation; a review and revisions of the evaluations questions posed by USAID in the SOW; an evaluation design that includes methods, data sources, data collection strategy, and notional analysis framework to analyze the evaluation questions; a recommendations framework for next steps USAID may consider for REWARD; a timeline for the evaluation; details of the evaluation team; and logistics and operations considerations. The REWARD activity involves two purposes: 1. Work with the ECOWAS Commission for Political Affairs, Peace, and Security (CPAPS) and the Early Warning Directorate (EWD) to enhance conflict early warning and response systems to reduce the risk of violence in the 15 ECOWAS member states; 2. Support national and local stakeholders in Niger, Ghana, Sierra Leone, Mali, and Togo to mitigate electoral violence triggers before they escalate into violence. According to the SOW, this midterm performance evaluation is intended to evaluate Purpose 1 only. The outcomes of Purpose 1 are: - Outcomes 1: EWD strengthened to collect early warning data - Outcomes 2: EWD capacity to analyze, report, and generate recommendations improved - Outcomes 3: Enhanced Department of Political Affairs (DPA) response capacity - Outcomes 4: Enhanced CPAPS internal and external communication and collaboration on early warning and response PURPOSE AND OBJECTIVES The USAID-stated purpose of this evaluation is to “determine if the project (REWARD) is well positioned to achieve success in ‘strengthening ECOWAS Early Warning and Response Systems that ultimately contribute to reducing the risk of violence in West Africa’” and what efforts were/are effective or not, and how should assistance be modified to better reach USAID strategic objectives for the region based on the most recent strategic priorities of the U.S. Government, specifically the USAID West Africa Development Objective 1 (DO1) “Systems of Non-violent Conflict Management Strengthened in West Africa.” Although not a priority when the program was designed, this evaluation will also serve the purpose of informing USAID on how conflict prevention efforts under REWARD can or could better advance self-reliance at both the regional and national levels.” In support of the purpose, the midterm performance evaluation will address the following objectives: 204PELA is implemented by International Business & Technical Consultants, Inc. (IBTCI). PELA and IBTCI are used interchangeably in this inception report. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 71 1. Document any successes, best practices, lessons learned, and challenges encountered in the activity; 2. Determine the effectiveness and relevancy of the approaches used to meet activity objectives and document any lessons learned, best practices, and challenges to inform future programming; 3. Identify any internal and external factors that affected the implementation of the activity to date as well as identify the changes that occurred within ECOWAS that might have impacted implementation; 4. Propose any recommendations for maintaining or changing the course of action, based on the findings, to help inform the remaining years of REWARD and future USAID programming around regional and national conflict prevention and early warning efforts. EVALUATION QUESTIONS The questions originally posed by USAID in the SOW, are as follow: 1. Has the institutional capacity of ECOWAS/EWD/CPAPS and other directorates increased as a result of USAID/WA REWARD assistance, and do they have sustainable capacity to effectively implement their respective mandates? a. What are the biggest gains to ECOWAS and the U.S. government as a result of assistance to ECOWAS? What are the examples of key successes/failures? b. In what manner or how have USAID Investments paid off and created value for ECOWAS? c. How has the USAID REWARD project contributed to improving the regional conflict early warning system of ECOWAS? Which capabilities and skills where increased and how was knowledge applied? 2. How did USAID REWARD assistance enable ECOWAS and the Commission for Political Affairs, Peace and Security (PAPS), to better carry out its mandate in terms of early warning, conflict prevention and mediation? How did USAID funding contribute to improving the effective operationalization of the ECOWAS Conflict Prevention Framework (ECPF)?” 3. How many and what type of recommendations generated by the EWD early warning work have translated into action by the ECOWAS leadership or member states? 4. What contribution did the USAID REWARD project make in integrating gender in early warning and response? Will ECOWAS sustain the approach of integrating gender in its peace and security efforts, in line with UNSCR 1325? If, yes, how, and if no, why not?” 5. Going forward, how will ECOWAS conflict prevention work support the implementation of USAID/WA’s strategic objectives, particularly with respect to current USG priorities for national security, and moving countries along the path to self-reliance? Which of the conflict prevention efforts would be most important in this regard? a. Which tasks of the REWARD Task Order are no longer relevant and should be modified? Why? b. Which type of assistance to ECOWAS should USAID provide going forward? c. To what extent was REWARD complimentary to the Department of State support to ECOWAS and EWD as well as of the assistance provided by other donors present in Abuja? How could USAID further complement Department of State efforts in its assistance to ECOWAS? IBTCI expressed initial concern that there are too many evaluation questions in the SOW that can be adequately and appropriately addressed in the expected timeframe with the requested evaluation team. 72 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV In a meeting with the PELA Contracting Officer Representative (COR), PELA Research Director (RD), and REWARD COR on February 21, 2019, all parties agreed that IBTCI would submit revised evaluation questions in the inception report. Accordingly, IBTCI proposes the following revised evaluation questions to guide this midterm performance evaluation. Some of the questions remain the same. For each evaluation question, IBTCI includes a narrative explaining our understanding of the question, how it supports the evaluation objectives, and when relevant, explains our understanding of ambiguous and/or undefined terms such as “effectiveness,” “success/failures,” “relevancy,” “institutional capacity,” and the like. Questions to support Objective 1: Document any successes, best practices, lessons learned, and challenges encountered in the activity The SOW proposes two questions that support Objective 1: 1. How has the regional conflict early warning system of ECOWAS changed, or not, since the inception of the USAID REWARD project? What were the best practices, lessons learned, successes and failures and why? (Proposed Evaluation Question 1) This question revises the original question posed in the SOW to focus on the specific performance of REWARD in improving the ECOWAS regional conflict early warning system. PELA understands this question to be aimed at evaluating a specific “success” (or possibly failure) of the REWARD program: the transformation of EWD early warning into actions in the region. This question also serves to evaluate the performance of REWARD in achieving Outcome 1. 2. How did gender integration efforts implemented by REWARD contribute, if at all, to more gender-balanced early warning data collection and analysis? (Proposed Evaluation Question 2) a. Do the EWD and the CPAPS demonstrate the institutional and professional capacity and will to sustain the approach of integrating gender in its peace and security efforts? PELA understands the essence of this question to elicit an evaluation of the extent to which ECOWAS has integrated gender in early warning, as well as an assessment of how it may continue to do so in the future. By “integrate gender into early warning and response,” PELA understands this to mean that there is evidence of a more gender balanced approach – women and gender dynamics – being incorporated in the early warning system data collection and analysis. By “institutional and professional capacity,” PELA understands this to mean that ECOWAS staff have the expertise required to: understand gender roles and contributions in data collection and analysis; and to incorporate this understanding into the early warning conflict system, and that ECOWAS’ bureaucratic systems and processes support inclusion of gender balanced approaches in early warning data collection and analysis. By “will to sustain the approach,” PELA understands this to mean that even if ECOWAS has the expertise and institutional capacity to mainstream gender in its early warning conflict system, it also has staff and leadership who buy-in to a gender-conscious approach and have the will to ensure that gender integration continues. This question does not appear to evaluate any of the REWARD outcomes (as provided in the SOW); however, it does evaluate an overarching best practice and policy guidance of USAID programming as set forth in ADS 205 “Integrating Gender Equality and Female Empowerment in USAID’s Program Cycle.” As a question aimed at evaluating a USAID best practice, it supports Objective 1. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 73 Questions to Support Objective 2: Determine the effectiveness and relevancy of the approaches used to meet activity objectives and document any lessons learned, best practices, and challenges to inform future programming The SOW proposes several questions that support Objective 2. “Effectiveness” and “relevancy” are vague terms; accordingly, IBTCI has revised the questions into the following two proposed evaluation questions (with subsidiary questions) that focus more tightly and specifically around the REWARD’s intended outcomes: 3. How has the USAID REWARD program contributed, if at all, to improving EWD’s ability to collect early warning data and strengthen EWD’s ability to analyze, report, and generate recommendations? (Proposed Evaluation Question 3) a. What specific capabilities and skills were increased, if at all, and how have they been applied by EWD and other PAPS directorates (DPA, DPKRS, Mediation Division, etc.)? b. Do EWD and ECOWAS demonstrate the institutional and professional capacity and the will to sustain early warning data collection, analysis, reporting, and recommendations without the assistance of REWARD? c. How did the ECOWAS consumers of EWD products report on the quality of the early warning information they received? This question is a revision of Question 1 and 1.C in the SOW to focus specifically on the gains made in EWD and ability to carry these gains on in future work. The question as revised will help the evaluation team empirically assess Outcomes 1 and 2 of the REWARD program. The “specific capabilities and skills” should directly align to REWARD training and capacity building activities and the evaluation team will investigate instances where these skills and capabilities were applied in the early warning system after the REWARD activity. The last part of this question will help USAID understand the extent to which EWD can carry on its work without the support of REWARD. To the extent that it still needs support, answering this question can help USAID determine ways to change future programming. The first two subsidiary questions under Question 1 in the SOW have been omitted here because they are too broad for a midterm performance evaluation of REWARD. Answering the questions as written in the SOW would require a “return on investment” analysis of all USAID funding toward ECOWAS, which is beyond the scope of what PELA can accomplish with the USAID-requested Evaluation Team, timeframe, and budget for this evaluation. 4. How has the USAID REWARD project contributed, if at all, to improving CPAPS ability carry out its mandate of early warning, conflict prevention, and mediation, especially by enhancing CPAPS’ internal and external communication and collaboration on early warning and response? (Proposed Evaluation Question 4) a. What are the specific instances of CPAPS’ better internal and external communication and collaboration on early warning and response? b. Does Commission for PAPS demonstrate access to money for adaptation, sustainability, and development capacity and the will to sustain early warning, conflict prevention, and mediation without the financial and technical assistance of REWARD? This question is a revision of Question 2 in the SOW to focus more specifically on the gains made by PAPS Commission. The revised question will help the evaluation team empirically assess Outcome 4 of the REWARD program. The last part of this question will help USAID understand the extent to which 74 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV CPAPS can carry on its work without the financial and technical support of REWARD. To the extent that it still needs support, answering this question can help USAID determine ways to change future programming. Questions to Support Objective 3: Identify any internal and external factors that affected the implementation of the activity to date as well as identify the changes that occurred within ECOWAS that might have impacted implementation Suggested questions to address Objective 3: 5. What internal and external factors prevented or supported Creative from accomplishing the REWARD expected results: Result/Output 1.1: EWD strengthened to collect early warning data Result/Output 1.2: EWD capacity to analyze, report, and generate recommendations improved Result/Output 1.3: Enhanced DPA response capacity Result/Output 1.4: Enhanced CPAPS internal and external communication and collaboration on early warning and response. (Proposed Evaluation Question 5) 6. Which of the program design assumptions where no longer valid or changed during the activity implementation that might have affected the program implementation? How should these be modified/addressed by USAID going forward? (Proposed Evaluation Question 6) Questions to Support Objective 4: Propose any recommendations for maintaining or changing the course of action, based on the findings, to help inform the remaining implementation of REWARD and future USAID programming around regional and national conflict prevention and early warning efforts The SOW proposes several questions in support of Objective 4, some of which go beyond the scope of a midterm performance evaluation of REWARD. PELA has revised the original SOW questions into the following two recommendation questions: 7. Which activities of the REWARD Task Order should continue and why? Which should be modified and why? (Proposed Evaluation Question 7) This is a slight revision from SOW Question 5.A. It leaves open the opportunity to provide recommendations to continue what is working or change/eliminate what is not working. It removes the “no longer relevant” wording in the original question to allow more room for determining what may not be working beyond relevancy. 8. To what extent has REWARD work been complementary to the State Department’s support to ECOWAS and EWD? How can USAID REWARD further complement the State Department’s or other donors assistance to ECOWAS in going forward? (Proposed Evaluation Question 8) This is a very slight revision from SOW Question 5.C. PELA understands this question to provide evidence about how REWARD has and may work together in an interagency capacity in the future in support of ECOWAS. EVALUATION DESIGN, METHODS, AND DATA COLLECTION STRATEGY This section provides a detailed discussion of the evaluation approach, as requested in the SOW. We take each point in turn. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 75 OVERALL APPROACH OF THE EVALUATION The approach for this midterm performance evaluation flows from the evaluation questions and the nature of the program itself. Since the “beneficiary” in this program is an intergovernmental body (IGO) rather than a group of people (such as individuals who participate in, for example, a vaccination program), survey data are not appropriate here. Instead, the REWARD outcomes and the evaluation questions are best answered through document review and key informant interviews (KIIs), to include group interviews. The vast majority of the data gathered will therefore be qualitative and descriptive in nature (with the exception of a few counts of events – see Table 1 below). We will work with USAID to detail the exact documents that need to be reviewed, as well as to determine the KII roster. Notionally, the documents to be reviewed will be REWARD program documents (such as quarterly and annual reports, monitoring and evaluation reports, etc.) and ECOWAS documents from EWD and CPAPS. KIIs will entail relevant personnel from ECOWAS, USAID, REWARD implementing partners (Creative Associates and its sub-contractors, Fund for Peace and the Karuna Center for Peacebuilding), State Department, other US Government (USG) entities, ECOWAS member states, and other donors. Data collection methods As discussed above, the primary data collection sources and methods for this evaluation are documents and KIIs. The documents will be any relevant reporting, notes, participant lists, the REWARD M&E Plan, and/or other documents related to REWARD and the ECOWAS divisions that REWARD supports. The main KIIs will be comprised of USAID staff, REWARD staff, ECOWAS staff, US State Department implementing partners, and may also be comprised of people from recipient countries, other donors, and other governmental officials as time permits (note: there is not time or budget to support the Evaluation Team to travel throughout the region to visit ECOWAS member states). The team will work with USAID to develop appropriate interview tools for each main type of KII (e.g., the tool that is used for ECOWAS staff will be different than the tool used for USAID). The interview protocols will be semi-structured: while the main evaluation questions will be asked of the interviewees, the semi￾structured nature of the collection tool will allow the Evaluation Team to probe further with KIIs as needed. The interview protocols will be conducted in French and English. The data will be collected via note taking. If the interviewee grants permission, interviews may be recorded. If recorded, the team will store the recordings on a shared Google drive with USAID and take notes on key information from the interview. Data analysis Document sources will be analyzed for content that is relevant to answering the research questions. As shown in the last column of Table 1, the analysis will mostly be descriptive in nature. In response to Evaluation Question 1, the team may also analyze for counts of events. All documents will be uploaded and stored into Dedoose, where the Qualitative Analyst on the team will create a structure to analyze both the documents and the KIIs. The first part of analyzing the KIIs will be to translate the interviews into English if they are conducted in French. Once the interview notes (either from the interview or from the recording) are translated, 76 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV they will be uploaded into Dedoose and analyzed according to the structure set up by the Evaluation Team Leader and Qualitative Analyst. The analysis structure will start with the themes and indicators in the Evaluation Questions and will evolve as the evaluation continues and the Team learns from the KIIs and documents. The Evaluation Team Leader and Qualitative Analyst, in close consultation with the Subject Matter Specialist, will also analyze the documents and interviews for any emerging themes that come out of the data sources that are important or relevant for the evaluation, but not necessarily captured in the main Evaluation Questions. This will also facilitate future lines of inquiry for research and evaluations (e.g., a USAID￾wide strategic evaluation of ECOWAS programming). Sampling methods and details Surveys are not applicable to this midterm performance evaluation, given the nature of the evaluation questions. Field design testing (testing on survey tools especially if translated) Surveys are not applicable to this midterm performance evaluation. The Evaluation Team will draft KII questions in close consultation with the REWARD COR in English and French as appropriate. IBTCI recommends that the Evaluation Team vet the KII questions with a small group of stakeholders to ensure they will elicit the desired data, if time and budget permit. RISKS AND LIMITATIONS THAT MAY UNDERMINE THE RELIABILITY AND VALIDITY OF THE EVALUATION RESULTS The thoroughness and completion of this evaluation rests on the assumption that documents and interviewees will be accessible to the Evaluation Team in a timely manner. Delays in the evaluation team acquiring documents that are required to answer evaluation questions or inaccessible interviewees could result in the inability of the Evaluation Team to answer all evaluation questions, either in part or in whole. To the extent that the Evaluation Team has sufficient access to data, the reliability of the evaluation results will be ensured by carefully documenting all reports and KIIs, as well as KII transcripts and notes. We will also carefully document the analysis of the qualitative data using Dedoose. However, we cannot ensure that interviewees would answer the same way if interviewed again in the future, and therefore, the data may not be reliable or valid. This is a common limitation to reliability and validity when using qualitative data. EVALUATION PLANNING MATRIX Table 1 details the indicators that map to each evaluation question, as well as the data collection and analysis strategies to assess each indicator. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 77 TABLE 1: EVAULATION PLANNING MATRIX EVALUATION QUESTIONS INDICATORS SUGGESTED DATA COLLECTION METHODS/SOURCES205 DATA ANALYSIS METHODS Eval Question 1: How has the regional conflict early warning system of ECOWAS changed or not since the inception of the USAID REWARD project? What were the best practices, lessons learned, successes and failures and why? 1. Early warning recommendations generated by EWD (number and type) 2. Recommendations that were translated into action 1. Document review 2. KIIs 1. Overall count of early warning recommendations made by EWD 2. Count by type of early warning recommendations 3. Count of recommendations that were translated into action 4. Description of the types of recommendations made 5. Description of the recommendations that were translated into action, including who took action Eval Question 2: How did gender integration efforts implemented by REWARD contribute, if at all, to more gender-balanced early warning data collection and analysis? 1. Gender integration in early warning 2. Gender integration in early warning response 3. EWD institutional capacity 4. EWD professional capacity 5. EWD will 6. CPAPS institutional capacity 7. CPAPS professional capacity 8. CPAPS will 1. Document review 2. KIIs 1. Description of instances when gender was integrated in early warning and response 2. Description of EWD institutional and professional capacity and will to integrate gender in future work 3. Description of CPAPS institutional and professional capacity and will to integrate gender in future work Eval Question 3: How has the USAID REWARD program contributed, if at all, to improving EWD’s ability to collect early warning data and strengthen EWD’s ability to analyze, report, and generate recommendations? 1. Early warning data collection. 2. Early warning data analysis 3. Early warning reporting 4. Early warning recommendations made by EWD 5. EWD institutional capacity, professional capacity, and will 1. Document review 2. KIIs 1. Description of EWD’s early warning data collection, plus evaluation of quality 2. Description of EWD’s early warning data analysis, plus evaluation of quality 3. Description of EWD’s early warning reporting, plus evaluation of timeliness and quality 4. Description of EWD’s recommendations (this will also be part of answering Eval Question 1) 5. Description of EWD’s institutional and 78 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV FURTHER EVALUATIVE QUESTIONS FOR ECOWAS 205 The specific documents and KII sources will be provided by USAID prior to onboarding of the Evaluation Team. professional capacity and will to continue this work without the support of REWARD Eval Question 4: How has the USAID REWARD project contributed, if at all, to improving CPAPS ability carry out its mandate of early warning, conflict prevention, and mediation, especially by enhancing CPAPS’ internal and external communication and collaboration on early warning and response? 1. CPAPS internal communications on early warning response 2. CPAPS external communications on early warning response 3. CPAPS institutional capacity, professional capacity, and will 1. KIIs 2. Possibly document review, if the evaluation team can acquire email or other communications in support of evaluating this question 1. Description of CPAPS internal communications about early warning conflict response 2. Description of CPAPS external communications about early warning 3. Description of CPAPS’ institutional and professional capacity and will to continue this work without the support of REWARD Eval Question 5: What internal and external factors prevented or supported Creative from accomplishing the REWARD expected results 1. Internal factors that impacted Creative’s execution of REWARD 2. External factors that impacted Creative’s execution of REWARD 1. Document review 2. KIIs 1. Identify and describe the internal and external factors that contributed to Creative’s execution of REWARD Eval Question 6: Which of the program design assumptions where no longer valid or changed during the activity implementation that might have affected the program implementation? 1. Program assumptions 1. Document review 2. KIIs 1. Description of initial program assumptions and how and why they changed over time. 2. Description of how these changes affected program implementation, if at all. Eval Question 7: Which activities of the REWARD Task Order should continue and why? Which should be modified and why? 1. REWARD activities 1. Document review 2. KIIs 1. Description and analysis of what is working and why 2. Description and analysis of what is not working and why Eval Question 8: To what extent was REWARD work has been complementary to the State Department’s support to ECOWAS and EWD? How can USAID REWARD further complement the State Department’s or other donors assistance to ECOWAS in going forward? 1. State Department support to ECOWAS 1. KIIs 1. Description of State Department support to ECOWAS 2. Evaluation of the extent to which REWARD complements it USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 79 The SOW included a number of strategic questions that fall far outside the scope of a performance evaluation of REWARD. These questions ask about ECOWAS (or components of ECOWAS) more generally, and about how the USG’s support (including, but not limited to REWARD) of ECOWAS has improved ECOWAS’ work. Though these questions will not be part of the evaluation itself, the Evaluation Team can ask the following questions of specific KIIs that USAID identifies to help guide future strategic-level evaluations of ECOWAS. The strategic questions posed in the SOW are as follows: ● What role, if any, would the ECOWAS early warning system/efforts play in meeting USG’s current and evolving objectives, particularly, in assisting countries to achieve self-reliance? Should USAID continue to invest in these organizations and why? ● Has USAID’s investment in ECOWAS led to national level impact, if any (though not part of REWARD’s objective)? ● How has support to early warning and response improved Early Warning and Response? ● Is ECOWAS recognized by regional governments as a force for regional integration across countries, and sectors? What evidence exists? ● Is EWD conflict prevention work relevant to the region in terms of advancing countries to become self-reliant in preventing conflict? a. If Yes, How does the work of the ECOWAS/EWD currently contributes to national development of member states? b. If Yes, should USAID be investing in these organizations? If yes, why? c. If No, why not? ● How sustainable is ECOWAS capacity and level of resources? Responses to these questions will be incorporated into a “Further Evaluation Questions for ECOWAS” section of the evaluation. The Evaluation Team will report the questions as answered and will not make these questions part of the evaluation analysis. Instead, USAID can expect a narrative of different responses to these questions in the “Further Evaluation Questions for ECOWAS” section. As part of this section, the Evaluation Team will also seek to develop further lines of inquiry that may help guide future strategic ECOWAS evaluations. 80 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV TIMELINE USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 81 EVALUATION TEAM PELA proposes the following 4-member Evaluation Team: A senior evaluation expert/specialist to lead the evaluation, a subject matter specialist with expertise in ECOWAS, early warning conflict systems, conflict resolution, and/or governance, a qualitative analyst, and a logistics coordinator. The evaluation expert and SMS positions will be filled by consultants with French language skills. The ideal qualitative analyst will also have French language skills. Details of each team member’s respective roles and qualifications as follow: TEAM LEADER – LEAD EVALUATION SPECIALIST (KEY POSITION): The Team Leader will have a minimum of a Master’s Degree in a relevant discipline (Conflict Prevention and Mitigation, Peacebuilding, Democracy and Governance, etc.) and at least seven years’ of experience in donor-funded performance evaluations in Africa. The Team Leader will have excellent analytical and report writing skills as well as demonstrated skills in applied political analysis, project management, and survey administration. ● Demonstrates the expertise and communications skills necessary to guide, coordinate, and successfully execute a performance evaluation. The evaluation team includes the Evaluation Specialist/Expert, a subject matter specialist (SMS), a logistics coordinator, and a qualitative analyst ● Provides technical guidance for all aspects of the performance evaluation ● Leads and conducts a document review of pertinent reports and other documents relevant to the performance evaluation ● Finalizes the evaluation tools (interview and focus group protocols) and work plan ● Performs interviews and focus groups with senior officials from ECOWAS and other stakeholders in Accra, Ghana and Abuja, Nigeria. Performs remote or in-person interviews with stakeholders in Washington DC ● Ensures that the data collected in the field are of the highest possible quality and are recorded and stored in a manner that meets human subjects ethics and USAID regulations ● Ensures that transcripts and/or notes of each key informant interview (KII) and focus group discussion (FGD) are typed and made available to the PELA team within 48 hours of the KIIs or FGDs ● Reviews the evaluation data analysis to ensure the data are of the highest-quality, responds to the evaluation questions, and follows the methodology outlined in the statement of work (SOW) and inception report ● Coordinates the process of assembling the evaluation findings and recommendations ● Drafts a high-quality 40-50 page midline evaluation report with clearly stated, well-grounded findings, corresponding conclusions and actionable recommendations ● Briefs PELA and USAID on the evaluation progress on at least a weekly basis, either in person (when in Accra) or by electronic communication (when in Abuja or remote) ● Communicates any delays or complications to PELA to allow quick resolution and to minimize any disruptions to the evaluation ● Identifies and discusses emerging opportunities to strengthen the evaluation as they arise ● Ensures that all deliverables outlined in the SOW are completed and submitted within the given timeframe ● Addresses PELA comments in draft deliverables prior to USAID submission ● Reviews USAID comments in the draft and revises the draft accordingly 82 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV ● Implements the performance evaluation as stipulated in the SOW Preferred Knowledge, Skills and Abilities: ● Familiarity in one or more of the following subjects is strongly preferred: ECOWAS, early warning conflict systems, conflict resolution, peacebuilding, governance, and/or democracy ● Past experience working in West Africa ● Proficiency in Microsoft Office Suite ● Ability to communicate well through written work with diverse stakeholders, including donors, beneficiaries, and local officials ● Ability to translate qualitative findings into clear and actionable recommendations for the client ● Past experience and demonstrated success in leading a cross-cultural evaluation team Minimum Requirements: ● 7+ years of conducting evaluations for USAID or other international development agencies ● 5+ years of experience conducting Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs) with senior officials, preferably in West Africa ● Ability to meet tight deadlines ● Strong communication and inter-personal skills required for working with multi-cultural, multi￾lingual and multi-ethnic teams ● Oral and written fluency in French is required. Professional-level oral and written communication in English SUBJECT MATTER SPECIALIST (CONFLICT/EARLY WARNING) Tasks include, but are not be limited to: ● Contributes to a document review of pertinent reports and other documents relevant to the midline evaluation ● Contributes to the development of the midline tools (interview and focus group protocols) and work plan ● Researches and identifies relevant key informants and conducting key informant interviews in Accra, Ghana and Abuja, Nigeria, as assigned by the Lead Evaluator ● Transcribes Focus Group Discussions (FGD) and Key Informant Interviews (KII) and notes; translates as needed ● Ensures that transcripts and/or notes of each KII and FGD are typed and made available to the PELA team within 48 hours of the KII or FGD ● Analyzes the evaluation data in response to the substantive evaluation questions, following the methodology outlined in the statement of work and inception report ● Assists the qualitative analyst in analyzing the data, as assigned by the Lead Evaluator ● Assists the Lead Evaluator in drafting a 40 to 50-page report of the evaluation and findings ● Assists with the development and presentation of findings to stakeholders Preferred Knowledge, Skills and Abilities: ● Demonstrated expertise in one or more of the following areas: ECOWAS, early warning conflict systems, conflict resolution, peacebuilding, governance, and/or democracy ● Proficiency in Microsoft Office Suite USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 83 ● Ability to communicate well through written work with diverse stakeholders, including donors, beneficiaries, and local officials is required ● Previous work experience in qualitative data collection and data analysis methods Minimum Requirements: ● At least 5 years of experience in contributing subject matter expertise – in one or more of the following areas: ECOWAS, early warning conflict systems, conflict resolution, peacebuilding, governance, and/or democracy – to evaluations or research in support of international development agencies ● At least 2 years of experience conducting interviews and focus group discussions with senior officials, preferably in West Africa ● Ability to meet tight deadlines ● Strong communication and inter-personal skills required for working with multi-cultural, multi￾lingual and multi-ethnic teams ● Oral and written fluency in French is required. Professional-level oral and written communication in English required QUALITATIVE ANALYST Tasks include, but are not be limited to: ● Contributes to the development of the midline qualitative tools (interview and focus group protocols) and work plan ● Develops a coding system to sort, filter or group different themes from the interviews and focus groups ● Instructs and guides the evaluation team in applying the proposed categorization and coding system ● Researches and identifies relevant key informants and conducts key informant interviews as assigned by the Evaluation Team Leader ● Provides quality assurance for the all qualitative data collected during the evaluation ● Analyzes qualitative data, prepares findings, and drafts and reviews sections of the performance evaluation report and presentation as assigned by the Evaluation Expert/Specialist Preferred Knowledge, Skills and Abilities: ● Previous work experience in qualitative data collection and data analysis methods ● Ability to translate qualitative findings into actionable recommendations for clients ● Experience with interviewing and engaging with senior level stakeholders to collect qualitative data ● Substantive expertise in early warning conflict systems, conflict resolution, peacebuilding, governance, democracy, and/or ECOWAS is strongly preferred Minimum Requirements: ● At least 5 years of experience with qualitative data collection and analysis in support of midline evaluations ● At least two years of experience using Dedoose ● Fluent in French and English ● Excellent oral and written communication skills in French and English ● Ability to communicate well through written work with diverse stakeholders, including donors, beneficiaries, and local officials 84 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV ● Excellent organizational skills and attention to detail ● Ability to meet tight deadlines ● Strong communication and inter-personal skills required for working with multi-cultural, multi￾lingual and multi-ethnic teams LOGISTICS COORDINATOR Tasks include, but are not be limited to: ● Organizes and schedules interviews for the evaluation, including refreshments when needed ● Organizes and books accommodations and travel for the evaluation team ● Helps the evaluation team with translations and transcription support as needed ● Helps the evaluation team develop and stick to a work plan and agenda ● Serves as the main point of contact for coordinating day-to-day implementation of the evaluation activities and ensures logistical requirements for the evaluation team are met ● Coordinates with the PELA office administrator to ensure compliance with USAID and IBTCI regulations ● Maintains filing system for all hard and soft copy evaluation documents, as directed by the Lead Evaluator and the PELA Research Director ● Maintains supplies and equipment needed for the evaluation, ensures that supplies are available and equipment is in proper working order at all times ● Perform general office clerk duties and errands, such as printing and photocopying, as needed by the evaluation team ● Ensures the evaluation team has adequate communications tools, such as SIM cards and internet, in each evaluation site (Ghana and Nigeria) LOGISTICS CONSIDERATIONS The Evaluation Team will include a logistics coordinator who will help ensure smooth evaluation operations, travel, administration, record keeping, and clerical work in support of the evaluation. An important logistics consideration that may impede the timeline of the evaluation is obtaining Nigerian visas for the Evaluation Team Lead, SMS, and Qualitative Analyst (if any of the above are not Nigerian nationals; most likely, none of them will be Nigerian nationals). Recent rules changes in the Nigerian Foreign Ministry have made it much more cumbersome and time consuming to obtain a visa to travel. Increasingly, the Nigerian Embassies in Western countries require applicants to appear in person in the capital city for an interview. This will add costs and time to the project. We will rely on USAID to secure visa invitation letters in a timely manner. We should factor in two to four weeks for the team to secure their visas. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 85 ANNEX 2: BIBLIOGRAPHY “Achievements and Challenges of ECPF Implementation.” ECPF Secretariat, January 30, 2018. “Annexes to DPA SOPs,” n.d. “ANNUAL WORK PLAN – YEAR 4.” Reacting to Early Warning and Response Data in West Africa (USAID REWARD). USAID & CREATIVE, n.d. “Country Risk and Vulnerability Assessment (CRVAs).” Key Findings 2016 - 2018. USAID & ECOWAS, n.d. “CPAPS Structure at Launch of REWARD,” n.d. “CRVA Capstone Training.” Reacting to Early Warning and Response Data, Activity Report. Abuja, Nigeria: USAID & ECOWAS, September 18 – 21, 2018. “CRVA Data Analysis Methods Trainings with Early Warning Directorate (EWD).” Activity Report. USAID & ECOWAS, July 2017. Daniel K.B. Inkoom. “Evaluation of the Operationalisation of the ECOWAS Conflict Prevention Framework (ECPF) at the Level of ECOWAS Directorates from 2013- June 2016.” Final Report. GIZ/ECOWAS, July 2017. “ECOWARN DATA CLEANING EXERCISE.” Status Report. Early Warning Directorate, ECOWAS, June 2018. “ECOWAS Conflict Analysis Framework Consultation Workshop.” Activity Report. Abuja, Nigeria: USAID & ECOWAS, November 2018. “ECOWAS Conflict Prevention Focal Points Retreat – Resourcing and Implementing Conflict Programs.” Activity Report. Goshen City, Nigeria: USAID & ECOWAS, January 2019. “ECOWAS CONFLICT PREVENTION FRAMEWORK.” ECPF Secretariat - ECOWAS, n.d. “ECOWAS Crisis Analysis Framework Consultation Workshop.” Reacting to Early Warning and Response Data, Activity Report. Abuja, Nigeria: USAID & ECOWAS, November 2018. “ECOWAS Crisis Scenario Planning Exercise.” Reacting to Early Warning and Response Data, Activity Report. Abuja, Nigeria: USAID & ECOWAS, March 2019. “ECOWAS Field Monitors Training Workshop.” Reacting to Early Warning and Response Data, Activity Report. Accra, Ghana: USAID/West Africa, November 1 - 4, 2016. “ECOWAS Field Monitors Training Workshop.” Reacting to Early Warning and Response Data, Activity Report. Dakar, Senegal: USAID/West Africa, May 2 – 6, 2017. “ECOWAS Field Monitors Training.” Reacting to Early Warning and Response Data, Activity Report. Saly, Senegal: USAID/West Africa, April 1 – 4, 2019. “ECOWAS Field Monitors Training.” Reacting to Early Warning and Response Data, Activity Report. Abidjan, Cote d’Ivoire: USAID/West Africa, October 17 – 19, 2018. “ECOWAS Gender Survey Response” Activity Report. USAID & ECOWAS, May 2017. 86 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV “ECOWAS Gender Survey.” Activity Report. USAID & ECOWAS, May 2017. “ECPF SECRETARIAT COORDINATING FUNCTIONS AND RESPONSIBILITIES.” ECPF Secretariat - ECOWAS, n.d. “Findings Report: Follow-Up Survey of ECOWARN Field Monitors.” Activity Report. USAID & ECOWAS, August 2017. “Gender Framework for Early Warning.” Reacting to Early Warning and Response Data, Activity Report. USAID & ECOWAS, October, 2017. “GENDER FRAMEWORK FOR EARLY WARNING.” USAID & ECOWAS, October 2017. “Gender Matters.” Reacting to Early Warning and Response Data, Activity Report. Abuja, Nigeria: USAID & ECOWAS, February 20, 2017. “Gender Matters.” Reacting to Early Warning and Response Data, Activity Report. Abuja, Nigeria: USAID & ECOWAS, July 2017. “Indicator Refinement Workshop.” Reacting to Early Warning and Response Data, Activity Report. Saly, Senegal: USAID, April 6 – 13, 2018. “Leveraging Tracks I, II, and III Actors for Early Response Workshop Report.” Reacting to Early Warning and Response Data, Activity Report. Abuja, Nigeria: USAID/West Africa, October 18, 2016. “Minutes of the CPAPS-REWARD Inter Directorate Working Group Launch Meeting.” CPAPS-REWARD Inter-directorate Working Group (IWG), n.d. “PERFORMANCE MONITORING AND EVALUATION PLAN (Updated Version).” Reacting to Early Warning and Response Data in West Africa (USAID REWARD) USAID & CREATIVE, April 2019. “Rapport de La Table Ronde Sur Les Enseignements Tirés de La Prévention de La Violence Electorale Au Mali.” Activity Report. Bamako, Mali: USAID, December 18, 2018. “Reacting to Early Warning and Response Data in West Africa (REWARD).” Annual Report – Fiscal Year 2018. USAID/West Africa, October 30, 2018. “Reacting to Early Warning and Response Data in West Africa (REWARD).” Annual Report – Fiscal Year 2016. USAID/West Africa, October 31, 2016. “Reacting to Early Warning and Response Data in West Africa (REWARD).” Annual Report – Fiscal Year 2017. USAID/West Africa, September 30, 2017. “REACTING TO EARLY WARNING AND RESPONSE DATA IN WEST AFRICA (REWARD) - Conflict Analysis Workshop Report.” Activity Report. Abuja, Nigeria: USAID/West Africa, July 20 - 21, 2016. “Special Workshop Retreat for ECPF Focal Points.” Reacting to Early Warning and Response Data, Activity Report. Goshen City, Nigeria: USAID & ECOWAS, January 30 – 31, 2018. “Standard Operating Procedures (SOPs) for Directorate of Political Affairs (DPA), ECOWAS Commission Political Affairs, Peace and Security (PAPS).” USAID & ECOWAS, February 2018. USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 87 “Standard Operating Procedures (SOPs) for Directorate of Political Affairs (DPA), ECOWAS Commission Political Affairs, Peace and Security (PAPS).” USAID & ECOWAS, February 2018. “Summary Results of ECPF M&E Questionnaire.” USAID & ECOWAS, January 2018. “SUMMARY RESULTS OF M&E QUESTIONNAIRE.” USAID & ECOWAS, February 1 - March 31, 2018 “Survey for EWD on Application of Learning.” Activity Report. USAID & ECOWAS, n.d. ”Survey for PAPS Directorates on Application of Learning” Activity Report. USAID & ECOWAS, n.d. “TERMS OF REFERENCE – ECOWAS CONFLICT PREVENTION FRAMEWORK (ECPF) INTERNAL STEERING COMMITTEE.” ECOWAS, n.d. “Training Workshop for ECPF Focal Points.” Reacting to Early Warning and Response Data, Activity Report. Abuja, Nigeria: USAID & ECOWAS, January 30 – 31, 2018. “USAID REWARD ECPF Application of Learning,” October 25, 2018. “USAID REWARD Quarterly M&E Questionnaire.” USAID & ECOWAS, Quarter 1, 2019 “Workshop on Foundations for an Integrated Early Warning/Early Response.” Activity Report. Abuja, Nigeria: USAID/West Africa, June 8 - 9, 2016. “WORKSHOP ON GENDER-MAINSTREAMING IN ECOWAS EARLY WARNING AND RESPONSE SYSTEM.” Activity Report. Dakar, Senegal: ECOWAS, May 8 – 10, 2017. 88 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 3: KIIS LIST Akpasom Yvonne, Head of Unit, Peace and Security, GIZ, 5-9-2019 Alimou Diallo, Chief of Party, REWARD-Creative Associates International, 5-9-2019 Babatunde Idowu, Project Officer – External Relation, ECOWAS, 5-14-2019 Constant Gnancaja, ECPF, ECOWAS, 5-13-2019 Dr. Abdou Lat Gueye, Director, EWD, ECOWAS, 5-17-2019 Dr. Cyriaque Agnekethom, Director – DPKRS, PAPS, ECOWAS, 5-6-2019 Dr. Habibu Yaya Bappah, EA to Commissioner of Finance, ECOWAS, 5-7-2019 Dr. Remi Ajibewa, Director, PAPS, ECOWAS, 5-2-2019 Esther Daramola, PO Analyst, EWD, ECOWAS, 5-2-2019 Ina Pislaru, Senior Regional Governance Advisor, USAID-West Africa, 5-3-2019 Ingnatius Onyekwere, Chijioke Ogbogu, and Alimou Diallo, REWARD, 5-13-2019 Ismaila Gueye, GIS Program Officer, EWD, ECOWAS, 5-8-2019 Kebba Touray, PPO, EWD, ECOWAS, 5-3-2019 Leora Addison, Project Director, REWARD, Creative Associates International, 5-1-2019 Nate Haken, Programs Director, Fund for Peace, 5-7-2019 Pascal Hollinger, Senior Political Advisor – ECOWAS Peace and Security, Embassy of Switzerland, 5-8-2019 Sa’adatu Shuaibu and Abigail Ogah, Program Officers, ECPF – DPA, ECOWAS, 5-2-2019 USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 89 ANNEX 4: KII GUIDES KII GUIDE 1 EWD KII-1-EWD Date: Start Time: Interviewer: End time: Note taker: Others present? Y / N Positions/Titles Name Name Name Interviewee Name: Consenting to Record? Organization: Position/Title: Unit: Since? Sex: Email: Phone: This interview is being conducted to inform an evaluation of the REWARD project under USAID funding. We are ____ and ____ from IBTCI, a firm external to the project that USAID has tasked to conduct this evaluation. The responses you will provide to our questions will remain anonymous and confidential. Do you give us the permission to record the interview to help us with our notetaking? 1. What history/role in unit and involvement with REWARD project activities 90 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 2. What changed in the EWD unit since 2015? Which projects brought about changes and how done? (EQ 1 /EQ3) - Probe for relative contribution of REWARD vs. Others - Probe for reasons of move of EWD from DPAPS and repercussions on DPAPS 3. How and to what extent REWARD met what capacity needs in collection, analysis, reporting and formulating actionable recommendations (EQ 3) 3a. Probe for specific skills/examples applied in each of Refer to training received/systems & procedures developed and : - data collection - analysis - reporting - recommending actions (EQ 3.1) 3b. Probe for presence and adherence to SOPs/other signs (forms, software programs, required training, etc.) of integration of new skills into each of data collection, analysis, reporting and recommendation formulation processes and practices (EQ 3.2 - institutional capacity) 3c. Probe for applications of new skills in each of: data collection; analysis; reporting; recommending (EQ 3.2 - professional capacity) 3d. What feedback gotten from consumers re.: quality of reports (EQ 3.3) 3e. Probe for gender integration (EQ2) and institutional/professional capacity (EQ 2.1) 4. Identify what worked well and not so well, and why (EQ 1.1) 5. What factors positively/negatively affect achieving the REWARD objectives? (EQ 5/EQ 6) 6. What REWARD activities should continue, stop or change (how) and why? (EQ 7) 7. How do the REWARD project activities complement efforts of other projects in EWD? (EQ 8) 8. What recommendations would you formulate for REWARD? KII GUIDE 2 DPAPS KII-2-DPAPS Applicable to Early Warning, Prevention and Mediation units Date: Start Time: Interviewer: End time: USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 91 Note taker: Others present? Y / N Positions/Titles Name Name Name Interviewee Name: Consenting to Record? Organization: Position/Title: Unit: Since? Sex: Email: Phone: This interview is being conducted to inform an evaluation of the REWARD project under USAID funding. We are ____ and ____ from IBTCI, a firm external to the project that USAID has tasked to conduct this evaluation. The responses you will provide to our questions will remain anonymous and confidential. Do you give us the permission to record the interview to help us with our notetaking? 1. What history/role in unit and what involvement with REWARD project activities 2. What changed in the DPAPS since 2015? What brought about these changes? (EQ 1 /EQ3) - Probe for relative contribution of REWARD vs. other projects - Probe for reasons of move of EWD from DPAPS and repercussions on DPAPS 3. How and to what extent REWARD contributed, if at all, to improve PAPS ability to fulfill its mandate in each of: early warning, prevention and mediation (EQ 4)? 3a. How, if at all, have EWD products/reports changed/improved over time since 2016? (EQ 3.3) - Probe for what change in their use by DPKRS, DPA, Mediation? 92 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 3b. How and to what extent REWARD met PAPS needs in communicating internally and externally in each of: conflict early warning, prevention, mediation (EQ 4.1) 3c. Probe for specific examples of changes/improvements in internal and external communications in each of: early warning, prevention and mediation (EQ 4.1) 3d. Probe for gender integration (EQ2) and institutional/professional capacity (EQ 2.1) 3e (i). What plans, if any, does the unit have to ensure its ongoing development? LEADER: what messages/communications did you deliver to your staff on the plans for the future unit; STAFF: what messages have you heard from your management concerning the future of the unit? (EQ 4.2 - will) 3e (ii). What budgets/sources of funding does the unit have to ensure its ongoing development? (EQ 4.2) 3e (iii). Probe for other signs showing "will" to sustain/adapt/develop (EQ 4.2) 4. Identify what worked well and not so well, and why (EQ 1.1) 5. What factors positively/negatively affect achieving the REWARD objectives? (EQ 5/EQ 6) 6. What REWARD activities should continue, stop or change (how) and why? (EQ 7) 7. How do the REWARD project activities complement efforts of other projects in EWD? (EQ 8) 8. What recommendations would you formulate for REWARD? KII GUIDE 3 SA & GENDER KII-3-SA&Gender Date: Start Time: Interviewer: End time: Note taker: Others present? Y / N Positions/Titles Name USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 93 Name Name Interviewee Name: Consenting to Record? Organization: Position/Title: Unit: Since? Sex: Email: Phone: This interview is being conducted to inform an evaluation of the REWARD project under USAID funding. We are ____ and ____ from IBTCI, a firm external to the project that USAID has tasked to conduct this evaluation. The responses you will provide to our questions will remain anonymous and confidential. Do you give us the permission to record the interview to help us with our notetaking? 1. What history/role in unit and involvement with REWARD project activities 2. What changed in the unit since 2015? What brought about these changes? (EQ 1 /EQ3) 3. How and to what extent REWARD met need to integrate gender balance in each of collection, analysis, reporting and formulating actionable recommendations in EWD (EQ 2) 3a. Probe for specific skills/examples applied in each of: data collection, analysis, reporting and recommending (EQ 3.1) 3b. Probe for SOPs/other signs (forms, programs, etc.) of integration of gender balance into data collection, analysis, reporting and recommendation formulation processes and practices (EQ 2.1 - institutional capacity) 3b. Probe for applications of new skills to gender balance in each of: data collection; analysis; reporting; recommending (EQ 2.1 - professional capacity) 4. Identify what worked well and not so well with REWARD activities, and why (EQ 1.1) 5. How and to what extent REWARD integrate gender balance in DPAPS (EQ 2.1) 94 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 5a. Probe for examples applied in each of: early warning, conflict prevention and mediation (EQ 2.1) 5b. Probe for SOPs/other signs (forms, programs, etc.) of integration of gender balance into DPAPS processes and practices (EQ 2.1 - institutional capacity) 5c. Probe for applications of new skills related to gender balance in each of conflict early warning, prevention and mediation (EQ 2.1 - professional capacity) 6. What factors positively/negatively affect achieving the REWARD objectives? (EQ 5/EQ 6) 7. What REWARD activities should continue, stop or change (how) and why? (EQ 7) 8. How do the REWARD project activities complement efforts of other projects in EWD? (EQ 8) 9. What recommendations would you formulate for REWARD? KII GUIDE 4 REWARD KII-4- REWARD Date: Start Time: Interviewer: End time: Note taker: Others present? Y / N Positions/Titles Name Name Name Interviewee Name: Consenting to Record? Organization: Position/Title: Unit: Since? Sex: USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 95 Email: Phone: This interview is being conducted to inform an evaluation of the REWARD project under USAID funding. We are ____ and ____ from IBTCI, a firm external to the project that USAID has tasked to conduct this evaluation. The responses you will provide to our questions will remain anonymous and confidential. Do you give us the permission to record the interview to help us with our notetaking? 1. What history/role in unit and involvement with REWARD project activities 2. What changed in the EWS since 2015? Who and how changes brought about? (EQ 1 /EQ3) - Probe for relative contribution of REWARD vs. others 3. REWARD was based on assumptions (read them) and the management of risks. What has been the impact of each on the project? (EQ 6) 3a. How has each changed/evolved? (EQ 6) 3b. How will these evolve in the future? (EQ 6) 3c. How were they managed? What else needs to be done to mitigate/manage the negative impact it may have on the future of REWARD? (EQ 6.1) 3d. What new assumptions should the project take into account? (EQ 6.1) 5. What are REWARD's five most important achievements in the EWS/EWD/DPAPS? Why are these significant? (EQ 3-4) - Probe for institutionalization and professional capacity building. 5a. How have other projects/donors contributed to these changes? How significant is their contribution compared to REWARD's?(EQ 8) 5b. How does REWARD complement the projects of the USG and of other partners? (EQ 8.1) - Probe for examples of coordinated programming activities or other forms of coordination 6. What worked well/not so well with REWARD activities, and why (EQ 1.1) 7. What factors positively/negatively affect achieving the REWARD objectives? (EQ 5/EQ 6) 8. What REWARD activities should continue, stop or change (how) and why? (EQ 7) KII GUIDE 5 USAID & DOS 96 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV KII-5- USAID/DOS Date: Start Time: Interviewer: End time: Note taker: Others present? Y / N Positions/Titles Name Name Name Interviewee Name: Consenting to Record? Organization: Position/Title: Unit: Since? Sex: Email: Phone: This interview is being conducted to inform an evaluation of the REWARD project under USAID funding. We are ____ and ____ from IBTCI, a firm external to the project that USAID has tasked to conduct this evaluation. The responses you will provide to our questions will remain anonymous and confidential. Do you give us the permission to record the interview to help us with our notetaking? 1. What history/involvement with REWARD project activities 2. What changed in the EWS since 2015? How brought about? (EQ 1 /EQ3) - Probe for AID contribution relative to others' USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 97 3. REWARD was based on assumptions (read them) and the management of risks. What has been the impact of each on the project? (EQ 6) 3a. How has each changed/evolved? (EQ 6) 3b.How will these evolve in the future? (EQ 6) 3c. How were they managed? What else needs to be done to mitigate/manage the negative impact it may have on the future of REWARD? (EQ 6.1) 3d. What new assumptions should the project take into account? (EQ 6.1) 4a. How have other projects/donors contributed to these changes and how significant is their contribution compared to REWARD's? (EQ 8) 4b. How does REWARD complement/not complement its activities with those of other USG or donors' projects? (EQ 8) 5. What are REWARD's three biggest achievements in the EWS? Why are these significant? (EQ 3- 4) - Probe for institutionalization and professional capacity building. 6. What worked well and not so well with changing the ECOWAS EWS, and why (EQ 1.1) 7. What factors positively/negatively affect achieving the REWARD objectives? (EQ 5/EQ 6) 8. What REWARD activities should continue, stop or change (how) and why? (EQ 7) 9. How can the REWARD activities better complement efforts of other projects in EWS (EWD + DPAPS)? (EQ 8) KII GUIDE 6 ECOWAS COMMISSIONERS KII-6-ECOWAS Commissioners Date: Start Time: Interviewer: End time: Note taker: Others present? Y / N Positions/Titles Name 98 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV Name Name Interviewee Name: Consenting to Record? Organization: Position/Title: Unit: Since? Sex: Email: Phone: This interview is being conducted to inform an evaluation of the REWARD project under USAID funding. We are ____ and ____ from IBTCI, a firm external to the project that USAID has tasked to conduct this evaluation. The responses you will provide to our questions will remain anonymous and confidential. Do you give us the permission to record the interview to help us with our notetaking? 1. What history/involvement with REWARD project activities 2. What changed in the EWS since 2015? How brought about? (EQ 1 /EQ3) - Probe for AID contribution relative to others' 3. What are REWARD's three biggest achievements in the EWS? Why are these significant? (EQ 3-4) - Probe for institutionalization and professional capacity building. 4a. How have other projects/donors contributed to these changes and how significant is their contribution compared to REWARD's? (EQ 8) 4b. How does REWARD complement/not complement its activities with those of other USG or donors' projects? (EQ 8) 5. What worked well and not so well with changing the ECOWAS EWS, and why (EQ 1.1) 6. What factors positively/negatively affect achieving the REWARD objectives? (EQ 5/EQ 6) 7. Which REWARD activities should continue, stop or change (how) and why? (EQ 7) 8. How can the REWARD activities better complement efforts of other projects in EWS (EWD + DPAPS)? (EQ 8) USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 99 9. What recommendations would you formulate for REWARD? KII GUIDE 7 DEVELOPMENT PARTNERS KII-7-Development Partners Date: Start Time: Interviewer: End time: Note taker: Others present? Y / N Positions/Titles Name Name Name Interviewee Name: Consenting to Record? Organization: Position/Title: Unit: Since? Sex: Email: Phone: This interview is being conducted to inform an evaluation of the REWARD project under USAID funding. We are ____ and ____ from IBTCI, a firm external to the project that USAID has tasked to conduct this evaluation. The responses you will provide to our questions will remain anonymous and confidential. Do you give us the permission to record the interview to help us with our notetaking? 1. What history/involvement with/knowledge of REWARD project activities 100 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 2. What changed in the EWS since 2015? How brought about? (EQ 1 /EQ3) - Probe for AID contribution relative to own 3a. How, if at all, have your projects contributed to these changes? (EQ 8) 3b. How does REWARD complement/not complement its activities with those of your projects? (EQ 8) 4. What are REWARD's three biggest achievements in the EWS? Why are these significant? (EQ 3-4) - Probe for institutionalization and professional capacity building. 5. What worked well and not so well with REWARD in changing the ECOWAS EWS, and why (EQ 1.1) 6. What factors positively/negatively affect achieving the REWARD objectives? (EQ 5/EQ 6) 7. Which REWARD activities should continue, stop or change (how) and why? (EQ 7) 8. How can the REWARD activities better complement efforts of your projects in EWS (EWD + DPAPS)? (EQ 8) 9. What recommendations would you formulate for REWARD? USAID.GOV REACTING TO EARLYWARNING DATA: PERFORMANCE EVALUATION REPORT | 101 ANNEX 5: CODING MATRIX EQ1: How has the regional conflict early warning system of ECOWAS changed or not since the inception of the USAID REWARD project? 11 What positively changed (general) 111 Institutional capacity 1111 Lessons Learned 11111 Examples of Successes 1112 Contributing projects 11121 Examples of Contributions 1113 Contributing factors 11131 Internal 11132 External 112 Professional capacity 1121 Lessons Learned 11211 Examples of Successes 1122 Contributing projects 11221 Examples of Contributions 1123 Contributing factors 11231 Internal 11232 External 12 What did NOT change (general) 121 Institutional capacity unchanged 1211 Lessons Learned 12111 Examples of Failures 102 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 1212 Contributing projects 12121 Examples of Failures 1213 Contributing factors 12131 Internal 12132 External 122 Professional capacity unchanged 1221 Lessons Learned 12211 Examples of Failures 1222 Contributing projects 12221 Examples of Failures 1223 Contributing factors 12231 Internal 12232 External factors 123 Baseline Information EQ3: 206 How has the USAID REWARD project contributed to improving EWD’s ability to collect early warning data and strengthen EWD’s ability to analyze, report, and generate recommendations? EQ 3.1 What specific capabilities and skills were increased and how have they been applied by EWD and other PAPS directorates (DPA, DPKRS, Mediation Division, etc.)? EQ 3.2 Does EWD demonstrate the institutional and professional capacity and the will to sustain early warning data collection, analysis, reporting, and recommendations without the assistance of REWARD? EQ 3.3 How did the ECOWAS consumers of EWD products/reports on the quality of the early warning information they received 206 Evaluation Question 2 has been treated as a cross-cutting one for both EWD and CPAPS. For instance see, codes 3114, 3124, etc. The same coding pattern is repeated in EQ 4. USAID.GOV REACTING TO EARLYWARNING DATA: PERFORMANCE EVALUATION REPORT | 103 311 General contribution to EWD 3111 EWD Institutional ability 31111 Enhanced internal communication 31112 Enhanced external communication 3112 EWD Professional ability 3113 EWD Will to sustain w/out REWARD 31131 Access to money 31132 Capacity for sustainability 31133 Capacity for development 3114 EWD Gender balance (EQ2) 31141 EWD Institutional capacity (EQ2) 31142 EWD Professional capacity (EQ2) 31143 EWD Will to sustain w/out REWARD (EQ2) 312 Contributions to EWD Collection Function 3121 Specific skills EWD acquired (EQ 3.1) 31211 Specific EWD skills applied 312111 By EWD 312112 By DPA 312113 By DPKRS 312114 By Mediation Division 312115 By others 3122 EWD Institutional capacity w/out REWARD 3123 EWD Professional capacity w/out REWARD 3124 EWD Will to sustain w/out REWARD 104 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 31241 Access to money 31242 Capacity for sustainability 31243 Capacity for development 3125 EWD Gender balance (EQ2) 31251 EWD Institutional capacity w/out REWARD (EQ2) 31252 EWD Professional capacity w/out REWARD (EQ2) 31253 EWD Will to sustain w/out REWARD (EQ2) 313 Contributions to EWD Analysis Function 3131 Specific EWD skills acquired 31311 Specific EWD skills applied 313111 By EWD 313112 By DPA 313113 By DPKRS 313114 By Mediation Division 313115 By others 3132 EWD Institutional capacity 3133 EWD Professional capacity w/out REWARD 3134 EWD Will to sustain w/out REWARD 31341 Access to money 31342 Capacity for sustainability 31343 Capacity for development 3135 EWD Gender balance (EQ2) 31351 EWD Institutional capacity (w/out REWARD EQ2) 31352 EWD Professional capacity w/out REWARD (EQ2) 31353 EWD Will to sustain w/out REWARD (EQ2) USAID.GOV REACTING TO EARLYWARNING DATA: PERFORMANCE EVALUATION REPORT | 105 314 Contributions to EWD Reporting Function 3141 Specific EWD skills acquired 31411 Specific EWD skills applied 314111 By EWD 314112 By DPA 314113 By DPKRS 314114 By Mediation Division 314115 By others 31412 Others’ appreciation of quality of EWD products (EQ 3.3) 314121 Positive 314122 Negative 3142 EWD Institutional capacity w/out REWARD (EQ 3.2) 3143 EWD Professional capacity w/out REWARD (EQ 3.2) 3144 EWD Will to sustain w/out REWARD (EQ 3.2) 31441 Access to money 31442 Capacity for sustainability 31443 Capacity for development 3145 EWD Gender balance (EQ2) 31451 EWD Institutional capacity w/out REWARD (EQ2) 31452 EWD Professional capacity (w/out REWARD EQ2) 31453 EWD Will to sustain w/out REWARD (EQ2) 314431 Access to money 314432 Capacity for sustainability 314433 Capacity for development 315 Contributions to EWD Recommendation Function 106 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 3151 Specific EWD skills acquired 31511 Specific EWD skills applied 315111 By EWD 315112 By DPA 315113 By DPKRS 315114 By Mediation Division 315115 By others 3152 EWD Institutional capacity w/out REWARD 3153 EWD Professional capacity w/out REWARD 3154 EWD Will to sustain w/out REWARD 31541 Access to money 31542 Capacity for sustainability 31543 Capacity for development 3155 EWD Gender balance (EQ2) 31551 EWD Institutional capacity w/out REWARD (EQ2) 31552 EWD Professional capacity (w/out REWARD EQ2) 31553 EWD Will to sustain w/out REWARD (EQ2) 315531 Access to money 315532 Capacity for sustainability 315533 Capacity for development 321 Issues with EWD (general) 3211 EWD Institutional ability 3212 EWD Professional ability 3213 EWD Will to sustain 32131 Access to money USAID.GOV REACTING TO EARLYWARNING DATA: PERFORMANCE EVALUATION REPORT | 107 32132 Capacity for sustainability 32133 Capacity for development 3214 EWD Gender balance (EQ2) 32141 EWD Institutional capacity (EQ2) 32142 EWD Professional capacity (EQ2) 32143 EWD Will to sustain (EQ2) 322 Issues with EWD Collection Function 3221 Issues with specific skills EWD acquired (EQ 3.1) 32211 Issues with skills EWD applied 322111 By EWD 322112 By DPA 322113 By DPKRS 322114 By Mediation Division 322115 By others 3222 EWD Institutional capacity 3223 EWD Professional capacity 3224 EWD Will to sustain 3225 EWD Gender balance (EQ2) 32251 EWD Institutional capacity (EQ2) 32252 EWD Professional capacity (EQ2) 32253 EWD Will to sustain (EQ2) 323 Issues with EWD Analysis Function 3231 Issues with specific EWD skills acquired 32311 Issues with specific EWD skills applied 323111 By EWD 108 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 323112 By DPA 323113 By DPKRS 323114 By Mediation Division 323115 By others 3232 EWD Institutional capacity 3233 EWD Professional capacity 3234 EWD Will to sustain 3235 EWD Gender balance (EQ2) 32351 EWD Institutional capacity (EQ2) 32352 EWD Professional capacity (EQ2) 32353 EWD Will to sustain (EQ2) 324 Issues with EWD Reporting Function 3241 Issues with specific EWD skills acquired 32411 Issues with specific EWD skills applied 324111 By EWD 324112 By DPA 324113 By DPKRS 324114 By Mediation Division 324115 By others 3242 EWD Institutional capacity 3243 EWD Professional capacity 3244 EWD Will to sustain 3245 EWD Gender balance (EQ2) 32451 EWD Institutional capacity (EQ2) 32452 EWD Professional capacity (EQ2) USAID.GOV REACTING TO EARLYWARNING DATA: PERFORMANCE EVALUATION REPORT | 109 32453 EWD Will to sustain (EQ2) 325 Issues with EWD Recommendation Function 3251 Issues with Specific EWD skills acquired 32511 Issues with Specific EWD skills applied 325111 By EWD 325112 By DPA 325113 By DPKRS 325114 By Mediation Division 325115 By others 3252 EWD Institutional capacity 3253 EWD Professional capacity 3254 EWD Will to sustain 3255 EWD Gender balance (EQ2) 32551 EWD Institutional capacity (EQ2) 32552 EWD Professional capacity (EQ2) 32553 EWD Will to sustain (EQ2) 326 Baseline Information (general) 3261 Collecting 3262 Analyzing 3263 Reporting 3264 Recommending EQ 4 How has the USAID REWARD project contributed, if at all, to improving CPAPS ability carry out its mandate of early warning, conflict prevention, and mediation, especially by enhancing CPAPS’ internal and external communication and collaboration on early warning and response? 110 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV EQ 4.1 What are the specific instances of CPAPS’ better internal and external communication and collaboration on early warning and response? EQ 4.2 Does Commission for PAPS demonstrate access to money for adaptation, sustainability, and development capacity and the will to sustain early warning, conflict prevention, and mediation without the financial and technical assistance of REWARD 411 Contribution to Commission for PAPS (general) (EQ 4) 4111 Institutional capacity w/out REWARD (EQ 4.2) 41111 Access to money 41112 Capacity for sustainability 41113 Capacity for development 412 Capacity contributions to CPAPS in early warning mandate (EQ4) 4121 Contributions to CPAPS’ enhanced communications (general) (EQ 4) 41211 Internal communications on EW warning and response (EQ 4.1) 41212 External communications on EW warning and response (EQ 41) 4122 Contributions to CPAPS’ enhanced collaborations (EQ 4.1) 41221 Enhanced internal collaborations (EQ 4.1) 41222 Enhanced external collaborations (EQ 4.1) 4123 Professional capacity w/out REWARD (EQ 4.2) 4124 Gender balance in implementing early warning mandate (EQ 2) 413 Capacity contributions to CPAPS in conflict prevention mandate 4131 Contributions to CPAPS’ enhanced communications (general) (EQ 4) 41311 Internal communications on EW warning and response (EQ4.1) 41312 External communications on EW warning and response (EQ41) 4132 Contributions to CPAPS’ enhanced collaborations (EQ4.1) 41321 Enhanced internal collaborations (EQ4.1) USAID.GOV REACTING TO EARLYWARNING DATA: PERFORMANCE EVALUATION REPORT | 111 41322 Enhanced external collaborations (EQ4.1) 4133 Institutional capacity w/out REWARD (EQ 4.2) 41331 Access to money 41332 Capacity for sustainability 41333 Capacity for development 4134 Professional capacity w/out REWARD (EQ 4.2) 4135 Gender balance in implementing early conflict prevention mandate (EQ 2) 414 Capacity contributions to CPAPS in mediation mandate 4141 Contributions to CPAPS’ enhanced communications (general) (EQ 4) 41411 Internal communications on EW warning and response (EQ4.1) 41412 External communications on EW warning and response (EQ41) 4142 Contributions to CPAPS’ enhanced collaborations (EQ4.1) 41421 Enhanced internal collaborations (EQ4.1) 41422 Enhanced external collaborations (EQ4.1) 4143 Institutional capacity w/out REWARD (EQ 4.2) 41431 Access to money 41432 Capacity for sustainability 41433 Capacity for development 4144 Gender balance in implementing conflict mediation mandate (EQ 2) 415 Contributions to CPAPS’ enhanced communications (general) (EQ 4) 4151 Internal communications on EW warning and response (EQ4.1) 4152 External communications on EW warning and response (EQ41) 4153 Gender balance in implementing mandate in communications (EQ 2) 416 Contributions to CPAPS’ enhanced collaborations (general) (EQ 4) 4161 Enhanced internal collaborations 112 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 4162 Enhanced external collaborations 4163 Institutional capacity w/out REWARD (EQ 4.2) 41631 Access to money 41632 Capacity for sustainability 41633 Capacity for development 4164 Gender balance in implementing mandate in collaborations (EQ 2) 421 Issues with PAPS 4211 Early Warning 4212 Conflict Prevention 4213 Conflict mediation 4214 Communications 42141 Internal 42142 External 4215 Collaborations 42151 Internal 42152 External 4216 Institutional capacity (EQ 4.2) 42161 Access to money 42162 Capacity for sustainability 42163 Capacity for development 4217 Gender balance in implementing mandate (EQ 2) 431 Baseline Information 4311 Early Warning 43111 Communications 43112 Collaborations USAID.GOV REACTING TO EARLYWARNING DATA: PERFORMANCE EVALUATION REPORT | 113 4312 Conflict Prevention 43121 Communications 43122 Collaborations 4313 Conflict Mediation 43131 Communications 43132 Collaborations 4314 Gender balance in implementing mandate (EQ 2) EQ 5: What internal and external factors prevented or supported Creative from accomplishing the REWARD expected results? 51 Enabling factors 511 Internal factors (Creative) 512 External Factors 52 Inhibiting factors 521 Internal factors (Creative) 522 External factors EQ 6: Which of the program design assumptions where no longer valid or changed during the activity implementation that might have affected the program implementation? EQ 6.1 How should these be modified/addressed by USAID going forward? 61 Changed assumptions 611 Managed how 114 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 612 Not managed 613 Impact on project 62 Assumptions no longer valid 621 Managed how 622 Not managed 623 Impact on project 63 Assumptions that still hold EQ 7: Which activities of the REWARD Task Order should continue and why? Which should be modified and why? 71 Activities to continue 711 Reasons to continue 72 Activities to be modified 712 Reasons to modify 713 How to modify 73 Activities to be initiated EQ 8: To what extent was REWARD complementary to the State Department (or other donors) support to ECOWAS and EWD? EQ 8.1 How can USAID REWARD further complement the State Department’s or other donors’ assistance to ECOWAS in going forward? 81 Complements DOS support USAID.GOV REACTING TO EARLYWARNING DATA: PERFORMANCE EVALUATION REPORT | 115 811 How 812 Why 813 How to further complement 82 Complements other donors’ support 821 Who/how 822 Why 823 How to further complement 83 No complement to DOS support 831 Who/how 832 Why not 833 How to further complement 84 Not complement to other donors’ support 841 Who/how 842 Why not 843 How to further complement 116 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 6: STATUS OF TASK ORDER ACTIVITIES The table below details the Task Order activities, the year REWARD planned to begin the activity, the actual year the activity began, the completion status of the activity, and additional comments, if pertinent. STATUS OF TASK ORDER ACTIVITIES TASK ORDER ACTIVITIES AND OUTPUTS PLANNED STARTED COMPLETED COMMENTS 1.1 EWD strengthened to collected early warning data 1.1.1 Embed an early warning adviser in the EWD to advise on technical issues and assist in early warning methodology. Year 1 Year 1 Year 1 This activity is completed. 1.1.2 Assist the EWD to evaluate the 66 indicators in the early warning system (to include a gender audit of these indicators) and suggest ways in which they can be further refined and targeted. Ongoing Year 1 This activity is ongoing. 1.1.3 Conduct a needs assessment of EWD staff and develop a training and plan modules in coordination with other donors. Year 1 Year 1 Year 1 This activity is completed. 1.1.4 Hold a series of staff trainings on topics such as data collection techniques. Ongoing Year 1 This activity is ongoing. 1.1.5 Refine standard operating procedures, reporting templates, and guidance tools. Ongoing Year 1 This activity is ongoing. 1.1.6 Periodic review and updating of ECOWARN Reporter, the database into which early warning data is imputed. Ongoing Year 1 This activity is ongoing. 1.1.7 Assist the EWD to consider how ICT can be further incorporated into early warning data collection. Ongoing Year 1 This activity is ongoing. 1.1.8 Conduct quality control on field data submitted by monitors and on data entry to gauge the accuracy and integrity of the information. Year 2 Year 2 This activity is ongoing. USAID.GOV REACTING TO EARLYWARNING DATA: PERFORMANCE EVALUATION REPORT | 117 1.1.9 Mentor EWD staff and community monitors over the life of the project, for example, through periodic monitoring visits to member states. Year 2 Year 2 This activity is ongoing. 1.2 EWD capacity to analyze, report, and generate recommendations improved 1.2.1 Review the accuracy, timeliness, and relevancy of previous EW report and recommendations. Ongoing Year 1 This activity is ongoing. 1.2.2 Conduct a survey of ECOWAS end users of conflict early response reports and recommendations to pinpoint strengths and weaknesses. Ongoing Year 1 This activity is ongoing. 1.2.3 Review current system for achieving early warning reports and recommendations and suggestions for upgrades to make the system more user-friendly and searchable. Year 3 Year 3 This activity is ongoing. 1.2.4 Assess existing data synthesis methods and current analysis practices, capabilities, and systems. Ongoing Year 1 This activity is ongoing. 1.2.5 Provide guidance to the EWD to improve its data analysis and reporting by taking full advantage of improved IT hardware and software and GIS tools and equipment, should they be provided by another donor. Ongoing Year 1 This activity is ongoing. 1.2.6 Refine data analysis tools that examine contextual factors, long-term structural risk issues, sequential and historical events, ethnic, socio￾cultural, religious, and internal conflict dynamics. Ongoing Year 1 This activity is ongoing. 1.2.7 Conduct a series of staff trainings on how to integrate gender into early warning analysis and recommendations by considering how men and women are affected differently and their respective roles in exacerbating and/or mitigating a given conflict. Year 1 Year 2 This activity was delayed due to a focus on launching the CRVA process. This activity is ongoing. 1.2.8 Provide technical assistance in the drafting and socialization of handbooks and/or internal policies. Year 2 Year 3 This activity was delayed for reasons not outlined in the work plan or annual report. The activity has since started and is ongoing. 1.2.9 Assist the EWD to create response development tools and templates. Year 2 Year 2 This activity is ongoing. 118 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 1.2.10 Provide technical assistance to operationalize a Communications Division within the EWD should another donor provide funding for additional personnel to staff it. Year 1 (Q4) Year 2 This activity was postponed due to a focus with the EWD on the launch of the first CRVA. The activity is ongoing. 1.3 Enhanced DPA response capacity 1.3.1 Organize a limited number of after-action reviews to determine how early warning recommendations were used and to gauge the effectiveness of the response. Ongoing Year 1 This activity is ongoing. 1.3.2 Assess DPA staff capacity in the area of early response and develop a training plan and modules, in coordination with other donors. Year 1 Year 1 Year 1 Completed. 1.3.3 Train staff in tools such as advanced mediation and facilitation techniques. Year 2 Not started This activity has not started because the DPA absorbed the MFD. 1.3.4 Provide limited technical assistance to the soon￾to-be established Mediation and Facilitation Division (MFD) to enhance the effectiveness and coordination of ECOWAS mediation efforts in the region. Year 2 Canceled This activity was canceled due to lack of response from DPA and the MFD absorption into the DPA. 1.3.5 Organize training sessions for officials from ECOWAS mediation organs (Council of the Wise, Special Representatives Offices, etc.) on negotiation and arbitration and new conflict prevention, management, and resolution strategies, methodologies, and techniques. Year 2 Not started This activity has not started because the COW has not been established. 1.3.6 Organize an orientation for the Council of the Wise members to deepen their understanding of ECOWAS policies, principles, and roles and finalize the draft statute for the Council. Year 2 Not started This activity has not been started because the COW has not been established. 1.3.7 Develop linkages between Council of the Wise members and relevant ECOWAS offices such as the MFD, the AU Panel of the Wise, national structures such as the NERMS, and civil society actors to promote conflict prevention and peacebuilding. Year 2 Not started This activity has not been started because the COW has not been established. 1.3.8 Conduct annual regional meetings of the Council of the Wise members to share lessons learned; discuss the security and political situation in the region and develop recommendations for ECOWAS leadership; develop annual work plans; and receive training in areas such as advanced mediation and facilitation techniques and gender integration. Year 3 Not started This activity has not been started because the COW has not been established. USAID.GOV REACTING TO EARLYWARNING DATA: PERFORMANCE EVALUATION REPORT | 119 1.4 Enhanced CPAPS internal and external communication and collaboration on early warning and response 1.4.1 Support the establishment of an internal and external communication strategy for CPAPS around early warning and response. Year 2 Year 2 This activity is ongoing. 1.4.2 Provide technical assistance to the Commission in implementing discrete activities and/or developing tools identified in the strategy such as the refinement of websites and training for staff in areas such as public speaking and media relations. Year 2 Year 2 This activity is ongoing. 1.4.3 Conduct a series of workshops between ECOWAS/EWD and Abuja-based client offices (other CPAPS directorates, other ECOWAS commissions and Ambassadors of ECOWAS countries in Abuja) to socialize ECOWARN products and identify communication gaps and client needs. Year 1 Year 1 This activity is ongoing. 1.4.4 Assist in establishing a distribution list and mechanisms for communicating early warning reports to key client offices. Year 2 Year 2 This activity is ongoing. 1.4.5 Provide technical assistance to establish regular feedback loops and coordination mechanisms between the three CPAPs directorates and other client offices to include the eventual establishment of a joint situation room, should the directorates eventually be co-located. Ongoing Year 1 This activity is ongoing. 1.4.6 Conduct as series of seminars for CPAPS leadership on evidenced based and client-oriented decision-making, including review of historical responses to various types of conflict. Year 2 Year 2 This activity is ongoing. 1.4.7 Support the development and adoption of decision-making tools that integrate high-level review and synthesis of early warning recommendations by all relevant actors within CPAPs. Ongoing Year 1 This activity is ongoing. 1.4.8 Advise on methods to strengthen information￾sharing and collaboration between ECOWAS, the NERMs, WANEP, and ECOWAS Permanent Representative's offices. Year 2 Year 2 This activity is ongoing. 1.4.9 Promote enhanced communication and collaboration with the AU and other regional economic communities (RECs) to share best practices related to early warning and response through semi-annual study tours and workshops. Year 2 Year 2 This activity is ongoing. 120 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 7: INDICATOR TABLE INDICATORS NO. PERFORMANCE INDICATOR UNIT OF MEASUREMENT BASELINE EOP TARGET TOTAL TO DATE RATIO Goal : Enhance early warning and response systems to reduce the risk of violence in West Africa 3: Annual Level of ECOWAS responsiveness to identified vulnerabilities and risk factors Likert scale with 1 being low responsiveness and 5 being high responsiveness 0 Targets 4 Actuals 6 150% 4: Quarterly Number of USG-sponsored forums, learning summits, or conferences to improve learning, and promote regional communication and networking (USAID WA PMP Indicator) # of events 0 Targets 94 Actuals 102 109% 5: Quarterly # of personnel working at regional institutions that apply knowledge and skills acquired through USG supported training in organizational capacity building including internal communications (USAID WA PMP Indicator) Number of staff 0 Targets 55 Actuals 179 325% Cross-cutting indicators CC2: Quarterly # of local women participating in a substantive role or position in peacebuilding processes supported with USG assistance (F 1.6-6). (USAID WA PMP IR1.1 Indicator) Number of women participating in a substantive role or position 0 Targets 197 Actuals 4063 2062% CC3: Quarterly # of training and capacity building activities conducted with USG assistance that are designed to promote the participation or the integration of gender perspectives in security sector institutions or activities (F 1.3-9)* Number of training activities 0 Targets 53 Actuals 177 334% USAID.GOV REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT | 121 Output 1.1: EWD strengthened to collect early warning data 1.1.1 Annual Number of early warning data reports from local, national and regional sources to EWD ECOWAS that meet the criteria of high quality (ECOWAS EWRS Action Plans (2015-2020) Number of reports 2.64 Targets 4 Actuals 3.22 81% 1.1.2 Quarterly # of ECOWAS staff and Field Monitors trained to improve data collection (ECOWAS EWRS Action Plans (2015- 2020) # of ECOWAS staff and Field Monitors 0 Targets 258 Actuals 531 206% Output 1.2: EWD capacity to analyze, report and generate recommendations improved 1.2.1: Bi-annual # of early warning reports generated by EWD that are of high quality Number of reports 0 Targets 52 Actuals 21 40% 1.2.2: Annual # of conflict, early warning end users who report increased quality of EWD products (USAID WA PMP Indicator) Number of end users 0 Targets 20 Actuals 74 370% Output 1.3: Enhanced DPA response capacity 1.3.1: Bi-annual # of recommendations from after action reviews (AARs) implemented Number of recommendations 0 Targets 55 Actuals 0 0% 1.3.2: Bi-annual # of After-Action Reviews (AARs) jointly held with ECOWAS early warning and early response units # of After-Action Reviews 0 Targets 11 Actuals 2 18% 1.3.3: Quarterly Percent of staff who report an improved level of knowledge and skills in conflict analysis, prevention and response tools. Percentage of ECOWAS staff trained 0 Targets 21.18% Actuals 87.76% 414% Output 1.4: Enhanced CPAPS internal and external communication and collaboration on early warning and response 1.4.2: Annual % of CPAPS Staff reporting of an increased collaboration on early warning and response Percentage and/or # of CPAPS staff 0/0 Targets 80% Actuals 100% 125% 122 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 8: EVALUATION STATEMENT OF WORK COMPLIANCE CHECKLIST 1. INFORMATION ABOUT THE STRATEGY, PROJECT, OR ACTIVITY EVALUATED  COMMENTS 1.1. Does the SOW identify the evaluation as either an impact or performance evaluation, per the definitions in Automated Directives System (ADS) 201? 1.2. Does the SOW identify the specific strategy, project, activity, or intervention to be evaluated? 1.2.1. Award number(s) listed? 1.2.2. Award dates listed (start and end dates)? 1.2.3. Funding level listed? 1.2.4. Implementing partner(s) listed? 2. BACKGROUND INFORMATION 2.1. Does the SOW provide country and/or sector context?  2.2. Does the SOW describe the specific problem or opportunity the intervention was designed to address?  2.3. Does the SOW describe how the intervention addresses the problem?  2.4. Does the SOW specify what existing and relevant strategy, project, or activity documents or performance information sources will be available to the evaluation team?  3. PURPOSE 3.1. Does the SOW state why the evaluation is being conducted (purpose)?  3.2. Does the SOW state who will use the results of the evaluation (audience)?  3.3. Does the SOW state the anticipated use(s) of the evaluation?  4. EVALUATION QUESTIONS 4.1 Does the SOW include a list of 1-5 questions that are answerable with empirical evidence and relevant to future programmatic decisions or learning?  4.2. Does the SOW identify all questions requiring sex-disaggregated data, the use of gender-sensitive data collection methods, and analysis of differential impacts on males and females?  If Impact Evaluation: 4.3. Are the questions about measuring the change in specific outcome(s) attributable to a specific USAID intervention? 5. DATA COLLECTION AND ANALYSIS METHODS USAID.GOV REACTING TO EARLYWARNING DATA: PERFORMANCE EVALUATION REPORT | 123 5.1. Does the SOW specify data collection and analysis methods or request that prospective evaluators propose qualitative and/or quantitative methods?  5.2. Does the SOW communicate methodological strengths and limitations or request that the prospective evaluators do so?  If Impact Evaluation: 5.3. Does the SOW require specific experimental or quasi-experimental methods or request 6. EVALUATION DELIVERABLES 6.1. Does the SOW request a written design that includes key questions, methods, main features of data collection instruments, and a data analysis plan?  6.2. Does the SOW require a draft report? 6.3. Does the SOW require a final report with (at minimum) the following? 6.3.1. An executive summary 2-5 pages in length that summarizes key points (purpose and background, evaluation questions, methods, findings, conclusions) 6.3.2. The Evaluation SOW in an annex  6.3.3. Any “statements of differences” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team in an annex 6.3.4. All data collection and analysis tools used—such as questionnaires, checklists, survey instruments, and discussion guides—in an annex  6.3.5. All sources of information properly identified and listed in an annex  6.4. Are dates or timeframes specified for deliverables? 6.5. Are quantitative data collected by the evaluation requested to be provided in an electronic file in easily readable format and organized and fully documented for use by 7. EVALUATION TEAM INDEPENDENCE AND QUALIFICATIONS 7.1. Does the SOW identify expectations about the methodological and subject matter expertise and composition of the evaluation team, including expectations concerning the involvement of local evaluation team members and evaluation specialists? 7.2. Does the SOW require team members provide a written disclosure of conflicts of interest (COI) and require key personnel to submit their COI disclosure with the proposal? 7.3. Does the SOW describe intended participation of USAID staff, implementing partners, national counterparts, or beneficiaries in the design or conduct of the evaluation? 8. SCHEDULE AND LOGISTICS 124 | REACTING TO EARLY WARNING DATA: PERFORMANCE EVALUATION REPORT USAID.GOV 8.1. Does the SOW state the expected period of performance? 8.2. Does the SOW specify any scheduling, logistics, security requirements, or other support that USAID will provide? 9. LEVEL OF EFFORT (LOE) AND BUDGET 9.1. Does the SOW include illustrative information about the LOE expected? 9.2. Is the SOW accompanied by an independent government cost estimate (if applicable)?