DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION OF THE CONFLICT ASSESSMENT FRAMEWORK 2.0 JULY 2019 Contract No. AID-OAA-I-13-00044, Order No. AID-OAA-TO-16-00036 ACKNOWLEDGEMENTS This evaluation would not have been possible without the contribution of many individuals who offered their knowledge, time, and support. We are indebted to all stakeholders who participated in the survey and remote interviews for the evaluation of USAID’s Conflict Assessment Framework 2.0. Specifically, the evaluation team would like to express its appreciation to the Conflict Assessment Team Members and USAID Mission representatives of Cambodia, Georgia, Guatemala, Iraq, Kenya, Kosovo, Liberia, Morocco, Niger, Nigeria, Senegal, South Sudan, Southern Thailand, Sri Lanka, Tanzania, Uganda, Washington DC, Zimbabwe, and elsewhere for contributing their time and sharing their experiences and knowledge. Special thanks to the leadership and staff of the USAID’s Office of Conflict Management and Mitigation for their valuable comments throughout various stages of the evaluation and logistical support. Stakeholders’ thoughtful contributions, reflections, and recommendations provided the foundation of this report, and the authors can expect to find their views and opinions accurately reflected herein. CONTENTS EXECUTIVE SUMMARY i INTRODUCTION 1 BACKGROUND ON CONFLICT ASSESSMENT FRAMEWORK 2.0 1 EVALUATION PURPOSE, AUDIENCE AND INTENDED USE 1 EVALUATION METHODOLOGY AND APPROACH 1 DESIGN AND ANALYTICAL FRAMEWORK 1 OVERARCHING EVALUATION QUESTIONS 2 METHODOLOGY 2 DATA ANALYSIS 3 DATA LIMITATIONS 3 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS 3 EQ1: WHAT ASSESSMENT MODELS HAVE BEEN USED IN DIFFERENT CONFLICT ASSESSMENTS SINCE 2012? 3 CONCLUSIONS ON EVALUATION QUESTION 1 7 RECOMMENDATIONS FOR EVALUATION QUESTION 1 7 EQ2: TO WHAT EXTENT DO ASSESSMENT TEAMS FOLLOW THE CAF APPLICATION GUIDE RECOMMENDATIONS ACROSS ALL PHASES OF THE CAF? 7 CONCLUSIONS ON EVALUATION QUESTION 2 11 RECOMMENDATIONS FOR EVALUATION QUESTION 2 12 EQ3: WHAT SYNTHESIS PROCESS ARE TEAMS MOST COMMONLY USING? 13 CONCLUSIONS ON EVALUATION QUESTION 3 15 RECOMMENDATIONS FOR EVALUATION QUESTION 3 15 EQ4: TO WHAT EXTENT HAVE MISSIONS UTILIZED CONFLICT ASSESSMENT REPORTS IN PROGRAMS AND OPERATIONS? 16 CONCLUSIONS ON EVALUATION QUESTION 4 20 RECOMMENDATIONS FOR EVALUATION QUESTION 4 20 ANNEXES 23 ANNEX A: TECHNICAL DIRECTION FOR TASK TWO: EVALUATION OF CAF A-1 ANNEX B: EVALUATION DESIGN MATRIX B-1 ANNEX C: LIST OF DOCUMENTS REVIEWED C-1 ANNEX D: DATA COLLECTION TOOLS D-1 ANNEX E: LIST OF STAKEHOLDERS CONSULTED E-1 ANNEX F: CONFLICT OF INTEREST FORMS F-1 ACRONYMS APS Annual Program Statement CAF Conflict Assessment Framework CDCS Country Development Cooperation Strategy CMM Conflict Management and Mitigation CSAID Conflict Sensitive Aid CS Conflict Sensitivity CVE Combating Violent Extremism DC District of Columbia DI Democracy International DNH Do No Harm DRG Democracy, Human Rights and Governance EQ Evaluation Question ET Evaluation Team GI Group Interview ICAF Interagency Conflict Assessment Framework ISIS Islamic State of Iraq and Syria KII Key Informant Interview MOU Memorandum of Understanding PD Program Description POC Point of Contact RFA Request for Application RFP Request for Proposal SMS Short Message Service SOW Statement of Work UNMIL United Nations Mission in Liberia USAID United States Agency for International Development MAP OF EVALUATED CAF COUNTRIES i | CAF 2.0 EVALUATION EXECUTIVE SUMMARY In 2012, the USAID Office of Conflict Management and Mitigation (CMM) revised and updated their Conflict Assessment Framework (CAF 2.0, in its update form) designed to improve USAID’s capacity to address the causes and consequences of violent conflict. In 2018, CMM commissioned Democracy International (DI) to conduct an evaluation of the CAF 2.0 application during the course of field-based conflict assessments since 2012 in 17 countries.1 The evaluated CAF countries are located in Africa (10), Asia (3), Europe (2), Latin America (1), and the Middle East (1). The evaluation utilized a mixed method design including quantitative and qualitative data collection tools to assess the use of the CAF 2.0 methodology and Application Guide and the extent to which Missions are implementing assessment findings and recommendations. The evaluation team (ET) also identified good practices and lessons learned relating to the CAF 2.0 methodology. Evaluation results will inform future revisions and adaptations of the CAF methodology, Application Guide, and training materials. Specifically, the evaluation focused on answering four overarching questions: 1. WHAT ASSESSMENT MODELS HAVE BEEN USED IN DIFFERENT CONFLICT ASSESSMENTS SINCE 2012? The ET identified three overarching CAF models of implementation: • The Standard CAF is procured through a contractor, led/co-led by a contractor Team Lead and CMM representative, and generally adheres to the CAF Application Guide. This is the majority of conflict assessments, representing 77% of the assessments reviewed. • The CAF Lite is commissioned directly by the Mission and has a more limited scope than a Standard CAF. It is primarily led and staffed by USAID. • The CAF Combo/Hybrid model uses parts of the CAF methodology combined with other tasks or tools. For example, USAID merged the Niger CAF with a transnational organized crime assessment, and Southern Thailand combined the CAF with an evaluation of a CMM-funded grant. Selection of the most appropriate model is based on a combination of assessment objectives and Mission resources and constraints, for example, the cost, scope and timing of the assessment. All models draw upon the CAF 2.0 methodology. Respondents agree that the diagnostic component of the CAF offers a strong analytical framework but would benefit from additional elaboration on concepts of resilience, bright spots and key mobilizers. However, they indicated that they perceive the response component of the CAF 2.0, which includes the synthesis process and developing recommendations, as weaker. Recommendations: 1 The 17 CAF evaluated countries include: Cambodia, Georgia, Guatemala, Iraq, Kenya, Kosovo, Liberia, Morocco, Niger, Nigeria, Senegal, South Sudan, Southern Thailand, Sri Lanka, Tanzania, Uganda and Zimbabwe. CAF 2.0 EVALUATION | ii • As USAID continues its reorganization, consider ways of applying different CAF models in the context of the new bureau. Explore the potential of offering a menu of models that may be used in different situations and contexts. • Elaborate and refine the resiliencies, bright spots and key mobilizer concepts in the CAF methodology. 2. TO WHAT EXTENT DO ASSESSMENT TEAMS FOLLOW THE CAF APPLICATION GUIDE RECOMMENDATIONS ACROSS ALL PHASES OF THE CAF? Conflict assessment reports are more likely to reflect the diagnostic components of the methodological framework than the response component. The most challenging parts of the CAF Application Guide to implement are preparation (pre-departure), identifying and securing interviews, conducting synthesis, and developing recommendations. Constrained timeframes, team skills, role confusion, and limited understanding of the Mission portfolio contributes to these challenges. Teams benefit when there is adequate time to prepare, roles are clear, and they have the resources to identify potential interviewees aside from existing USAID partners. Teams need more time and the right skill sets to conduct synthesis. Given the option, adding time to synthesis and generating recommendations is more important than adding time for data collection. CMM follow-up plays an important role in supporting Mission utilization of findings and recommendations. Routinely incorporating follow-on activities may require a shift from thinking about a conflict assessment as a discrete event to thinking about it as one element of a longer-term relationship between CMM and the Mission that doesn’t end with delivery of the report. Recommendations: • Revisit and update the CAF Application Guide to incorporate changing realities of conducting conflict assessments, for example, updating timeframes and expectations of different phases of the assessment, as well as good practice and lessons learned from seven years of assessments. • Allow more time for synthesis and generating recommendations. • Consider incorporating CMM follow-up activities into the Application Guide. 3. WHAT SYNTHESIS PROCESSES ARE TEAMS MOST COMMONLY USING? A combination of factors determine the timing and process used for synthesis, including the number and location of sub-teams and the skills and preferences of the Team Lead and CMM representative. Most teams use a headlines approach. Several USAID respondents indicated an interest in using systems mapping but did not feel confident leading synthesis using this approach. Recommendations: • Continue to further refine and expand systems mapping beyond the way teams currently use it for synthesis. Consider developing a guide of synthesis methodologies. • Continue piloting innovative mid-point and rolling synthesis approaches as opportunities arise. iii | CAF 2.0 EVALUATION 4. TO WHAT EXTENT HAVE MISSIONS UTILIZED CONFLICT ASSESSMENTS REPORTS IN PROGRAMS AND OPERATIONS? Missions apply conflict assessment findings and recommendations to enhance programming and institutionalize conflict sensitivity. Many continue to use the assessment as a source of information to orient new staff and to inform project/activity design. Two scenarios appear to improve the likelihood that Missions will implement recommendations. The first is when CMM maintains an ongoing relationship with the Mission. In these cases, the CAF is one part of a longer-term partnership that begins before the assessment and continues afterwards. The second is when a Mission considers the conflict assessment as one part of a broader self-driven conflict sensitivity agenda. Recommendations: • Make the CAF one part of an ongoing relationship between CMM and the Missions. • To maximize the likelihood that Missions will utilize recommendations, undertake the following before the assessment: o Work with Missions early in the process to clarify expectations around the level of recommendations that the team will provide. o Consider developing a checklist or tool describing steps for cultivating broad-based support, understanding the Mission context, portfolio, and programmatic parameters. o Understand what is driving the assessment. o Encourage teams to understand the Mission context and portfolio as it will help to cultivate support, buy-in and facilitate developing relevant, actionable and meaningful recommendations. o Strongly encourage CSAID training at the Mission before the assessment to begin socializing the CAF methodology and conflict sensitive programming across teams. o Advocate for including sector staff, in addition to DRG representatives, on the assessment team. • Request a Mission point of contact (POC) for a year or more following a conflict assessment to help CMM to identify and provide appropriate follow-on support. CROSS-CUTTING RECOMMENDATIONS The ET offers the following cross-cutting recommendations. • Keep learning. Continue using feedback reports to document team learning on process and methodology. Review feedback reports annually and improve efficiency and effectiveness by applying lessons learned. • Build CMM staff Conflict Assessment capacity. CMM staff play a critical role in guiding assessment teams throughout the process and supporting Missions to apply conflict assessment recommendations. CMM should continue the good practice of sending less experienced CMM staff to participate, but not lead, a conflict assessment prior to assuming a leadership role. 1 | CAF 2.0 EVALUATION INTRODUCTION The USAID Office of Conflict Management and Mitigation (CMM) commissioned Democracy International (DI) to conduct an evaluation of the Conflict Assessment Framework (CAF) 2.0 application during the course of field-based conflict assessments since 2012 in 17 countries.2 The evaluated CAF countries are located in Africa (10), Asia (3), Europe (2), Latin America (1) and the Middle East (1). BACKGROUND ON CONFLICT ASSESSMENT FRAMEWORK 2.0 In 2012, CMM revised and updated the CAF, which is designed to improve USAID’s capacity to address the causes and consequences of violent conflict. The CAF 2.0 incorporates lessons learned from more than 60 conflict assessments used to inform agency policy, strategic planning, programming, and implementation.3 The CAF 2.0 offers updated guidance on assessing the drivers of conflict and generating practical recommendations for USAID Missions. EVALUATION PURPOSE, AUDIENCE AND INTENDED USE The evaluation had three objectives: 1. Assess how the CAF 2.0 methodology and Application Guide have been used; 2. Determine how and to what extent Missions utilize assessment findings and recommendations; 3. Help to identify good practices and lessons learned concerning the CAF 2.0 methodology. The primary audience for the evaluation is USAID/CMM. The evaluation results will inform future revisions and adaptations of the CAF methodology, Application Guide, and training materials. CMM will disseminate the evaluation findings to Mission- and Washington-based staff. EVALUATION METHODOLOGY AND APPROACH DESIGN AND ANALYTICAL FRAMEWORK The evaluation utilized a mixed method design including quantitative and qualitative data collection tools. An “Evaluation Design Matrix”4 guided the design and the ET implemented it in four stages: Stage 1: Evaluation work plan, a desk review, and online survey Stage 2: Conduct remote key informant and group interviews Stage 3: Draft evaluation report, incorporate CMM feedback, and finalize report Stage 4: Dissemination of the evaluation findings and recommendations 2 The 17 CAF evaluated countries include: Cambodia, Georgia, Guatemala, Iraq, Kenya, Kosovo, Liberia, Morocco, Niger, Nigeria, Senegal, South Sudan, Southern Thailand, Sri Lanka, Tanzania, Uganda and Zimbabwe. 3 USAID/CMM, Conflict Assessment Framework, version 2.0: Letter from the Director 4 See Annex B for the Evaluation Design Matrix, which describes in detail the analytical approach, data collection methodology, and data sources that the evaluation team used. CAF 2.0 EVALUATION | 2 OVERARCHING EVALUATION QUESTIONS The evaluation focused on answering the following four overarching questions: 1. What assessment models have been used in different conflict assessments since 2012? 2. To what extent do assessment teams follow the CAF Application Guide recommendations across all phases of the CAF? 3. What synthesis processes are teams most commonly using? 4. To what extent have Missions utilized Conflict Assessments reports in programs and operations? METHODOLOGY The evaluation used qualitative and quantitative methods, aligned to the Evaluation Design Matrix, to deliver evidence-based answers to the four questions above. Mixed methods included a desk review, online survey, and key informant interviews (individual and group). Desk Review. The evaluation team (ET) reviewed documentation from 17 conflict assessments conducted from 2012-2017. In addition to the assessment reports, the ET also reviewed statements of work (SOWs), background reports (desk studies, literature reviews, and issues papers), field workplans, feedback papers, briefing presentations, and other relevant documentation. The ET categorized desk review findings according to: a) model; b) adherence to the CAF Application Guide; c) the application of the CAF methodology; and d) synthesis process. Survey. The online survey collected perception data to further the ET’s understanding of desk review findings and fill gaps in understanding of the evaluation matrix questions. The ET distributed the survey to 147 representatives from CMM, USAID staff in both Washington and the field, and conflict assessment team members.5 There were 69 of those surveyed (47%) who completed the survey (36 female and 33 male). Respondents self-identified as “CAF team member” (43), “Mission point of contact” (14), “report reviewer” (3), “other” (8), or “unidentified” (1). However, a number of those surveyed may in fact have played more than one of these roles, so the survey may have undercounted some categories.6 Key Informant Interviews (KIIs) and Group Interviews (GIs). The ET conducted individual and group interviews with 1) team members, including Mission￾based, CMM, other USAID, international and local consultants, and 2) end-users, including Mission staff who may or may not have also participated in field work but participated in the assessment as reviewers, points of contact for the team, or were otherwise in a position to use the conflict assessment report.7 Overall, the ET interviewed 53 people (35 female and 18 male) in 23 individual and 14 group interviews (see Figure 1 for a breakdown of respondents by type). 5 See Annex D for Survey Questionnaire. Of the 147 people invited to take the survey, 129 had unique email addresses. In other words, some people participated in multiple assessments. 6 The “team member” role was varied and included the following: local conflict specialist (19), CMM staff (6), Team Lead (5), technical specialist (5), Mission staff (4), other USAID staff (2) and other (2). 7 See Annex D for Interview Protocols. Some people participated in more than one interview if they were engaged in multiple conflict assessments. 3 | CAF 2.0 EVALUATION Combining surveys and interviews, the ET consulted 122 respondents (71 female and 51 male). This number represents data points because some individuals participated in both the survey and interviews. DATA ANALYSIS The ET used an Excel-based data management matrix to capture desk review findings, statistical analysis, and trends and themes emerging from each round of data collection. Using Excel, the ET generated cross￾tabulations of survey results by country, model, and role of respondents, allowing a deeper look into the survey findings. The ET then compared the desk review and survey findings against the Evaluation Design Matrix to identify information gaps and draft interview protocols. DATA LIMITATIONS To understand the evaluation findings, conclusions, and recommendations, it is important to acknowledge some of the data limitations. The survey response rate was only 47%,8 most likely because the survey launch coincided with a month-long government shutdown. In the survey, participants were only able to select one option to identify their role in relation to the conflict assessment. This contributed to an incomplete understanding of respondent profiles because, given the option, some may have selected multiple roles for example, a Mission-based team member who was also an end user. The ET corrected for this in the round of interviews that followed. The ET conducted interviews across all 17 conflict assessments. This resulted in only 2-3 interviews per conflict assessment, limiting the team’s ability to triangulate or validate information from respondents. Group interviews were particularly challenging, because of the need to coordinate with people in multiple time zones. In many cases, confirmed interviewees did not call in at the scheduled time, leaving the team to conduct the interview with only those who did, some of whom had limited knowledge of the assessment. Group interviews often combined individuals in a way that may have inhibited candid discussion, for example, CMM with Mission staff, local USAID staff with Foreign Service Officers, and USAID staff with consultants. The size and dynamics of some groups made it difficult to complete the interview in the time allocated. Other interview protocol limitations include the limited recall of many informants, and the difficulty of engaging knowledgeable end-users with visibility into utilization of the assessment beyond their team. The team did not have the opportunity to interview implementing partners to understand and triangulate information on process successes and challenges, for example, those relating to preparation, timelines, hiring team members, logistics, etc. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS EQ1: WHAT ASSESSMENT MODELS HAVE BEEN USED IN DIFFERENT CONFLICT ASSESSMENTS SINCE 2012? 1.1 Types of models used 8 Of the total number that did respond, the lowest response rate was from Mission staff (35%), followed by USAID/Washington (52%), with the highest number of responses coming from consultants (62%). CAF 2.0 EVALUATION | 4 CMM defines a model as a mix of parameters relating to geographic scope (covering the entire country or regions within country), thematic focus (related to a specific issue, for example electoral violence, natural resource management, or crime), rapid assessment (conducted in two weeks) or desk-only assessment (excludes field data collection).9 The ET initially thought that these parameters constitute the type of model used in conflict assessment. However, one of the first findings of this evaluation was that the assessment objectives define the model; that is, they determine the assessment’s geographic or thematic parameters. In other words, the model is a functional description for achieving the desired objectives using available resources (money, staff, time, etc.). Following this functional categorization, the ET identified three overarching CAF models of implementation: 1) Standard CAF, 2) CAF Lite, and 3) CAF Combo/Hybrid. However, not every CAF fits neatly into one model. For example, see South Sudan, Kosovo and Cambodia (below under Standard CAF). The Standard CAF is procured through a contractor, led/co-led by a contractor Team Lead and CMM representative. It generally adheres to the CAF Application Guide. Of the 17 assessments reviewed, 1310 (77%) used the Standard CAF model. The outliers were South Sudan, Kosovo, and Cambodia. Despite being full assessments, Missions contracted these CAFs directly. Although CMM was involved, the teams did not fully follow the Application Guide, often lacking desk studies and memorandums of understanding (MOU). USAID used the CAF Lite model in Georgia and Uganda, and it completed both of these assessments in two weeks. In both cases the Mission commissioned the assessments directly, they had a more limited scope than a standard conflict assessment, and USAID primarily led and staffed the team. Georgia did not include external consultants and the Uganda team included two local consultants. CAF Lite uses the CAF methodology but does not necessarily follow the Application Guide due to the need for rapid turnaround. Synthesis occurs quickly given the smaller team. In the case of Uganda, the team never finalized the report. The CAF Combo/Hybrid model uses parts of the CAF methodology combined with other tasks or tools. For example, USAID/Niger merged the CAF methodology with a transnational organized crime assessment, and Southern Thailand combined the CAF methodology with an evaluation of a CMM-funded grant. Similar to Uganda, the Southern Thailand assessment did not complete a desk study or issues paper, and there were no external consultants on the team. Data collection in the Deep South of Thailand only lasted for a week due to security concerns. 1.2 What factors determine selection of a specific model? As discussed above, conflict assessment objectives and Mission resources define the selection of the model. The ET found that conflict assessment objectives can generally be clustered into three broad categories that apply across different models and often overlap in the same assessment. These categories include: 9 Technical Direction for Task Two: Evaluation of the Conflict Assessment Framework, see Annex A. 10 These 13 Standard CAF countries include Cambodia, Guatemala, Iraq, Kenya, Kosovo, Liberia, Morocco, Nigeria, Senegal, South Sudan, Sri Lanka, Tanzania and Zimbabwe. CAF is the best conflict assessment tool on the market. It allows people to see all different aspects of society and identify places you can potentially influence dynamics. T Mb 5 | CAF 2.0 EVALUATION To inform Mission strategies and programs. Based on the desk review, 71% of CAFs included an objective related to informing Mission strategy (CDCS or Transition Strategy) and programming (new and existing). This includes efforts to integrate conflict sensitivity at both the strategy and programming levels. These assessments may go beyond areas of current and anticipated programming if the Mission wants a broader understanding of the conflict context. For example, in Tanzania, the Mission wanted to increase USAID’s understanding of existing or potential sources of conflict and how they manifest in formal and informal settings. To understand a change in context and/or potential trigger. When a Mission is looking for increased visibility on anything from an uptick in protests (Morocco), to risk of violence around upcoming elections (Nigeria and Kenya) to withdrawal of the United Nations Mission in Liberia (UNMIL) forces. Objectives in this category seek to understand changes in conflict dynamics in order to inform conflict-sensitive programming. To better understanding an issue or set of issues. This includes focusing on thematic issues, updates of previous assessments or reviews of program implementation. For example, the Niger assessment focused on gaining better understanding of transnational organized crime and youth gang violence risk factors in Northern and Central/Southern regions of the country. In Senegal, the Mission took advantage of the team presence to better understand a recent change of government in neighboring The Gambia, and the potential implications for a population living along the border. In addition to objectives, Mission resources and constraints also inform the selection of a specific model. Data indicates that cost implications of contracting directly versus through an implementing partner was a factor in the Niger, Georgia, South Sudan, and Morocco assessments. In the end, only Morocco decided to work through an implementing partner. The anticipated scope of the assessment was another factor that informs selection of a model. Will the assessment feed into a long-term planning process or is there a targeted need, such as updating a prior assessment as in the case of Uganda? Timing, or how quickly the information is required, also plays a role. The Mission in Georgia needed to move quickly to design a new project in response to a window of opportunity for engagement in the occupied territories. 1.3 What challenges or opportunities did teams face in implementing and adapting the designated model across all phases of CAF implementation? All of the models discussed above come with inherent challenges and opportunities. The Standard CAF is generally more structured and standardized, adhering to the CAF methodology and Application Guide. The team tends to be larger and more diverse, benefiting from government and implementer experiences, local expertise, and international perspectives to provide a more holistic look at conflict. The team may also include representatives from other USG agencies. However, a Standard CAF generally takes longer to plan and produce a report. It generally requires significant investment of time, funds, and people. The CAF Lite can be conducted very rapidly within two weeks with a small team of mostly USAID staff. It uses the CAF methodology, but does not necessarily follow the Application Guide. The CAF Lite requires fewer resources, but due to the limited size and scope of the team, offers less geographic coverage and depth, and therefore less data. Assessment of conflict dynamics rests with a more homogeneous team with less diversity of perspectives, especially if the team consists of only USAID staff. Preparation and field work are time constrained, and a smaller team can do the synthesis rapidly. Without the presence of an international consultant, however, it may be harder for the team to complete the report. Timing of CAF was right as everybody’s eyes were on the elections. ~ USAID Mission CAF 2.0 EVALUATION | 6 The CAF Combo/Hybrid is flexible and follows either the Standard CAF or CAF Lite approach, combined with another tool or task. It may not adhere as closely to the CAF methodology or Application Guide in order to accommodate a dual purpose. 1.4 Did the scoping of the CAF build off of prior USAID assessments or was it planned to feed into findings from another type of assessment? The desk review findings show that 13 assessments were informed by previous conflict assessments.11 Scoping of the Sri Lanka CAF was informed by a of Rule of Law assessment. Based on desk review findings, only two assessments were planned to complement findings from other assessments, for example, the integration of a focus on transnational organized crime to create a modified CAF framework for Niger. The other 15 assessments (88%) did not report on whether they planned to feed into other types of assessments. 1.5 Reported strengths and weaknesses of CAF methodology The feedback papers and interviews provided evidence on the strengths and weaknesses of the CAF methodology. Both sources recognize the CAF as a rigorous and useful analytical framework. According to the 20 interviewees who commented on the methodology, the CAF offers a systematic look at the underlying issues and provides shared vocabulary and concepts to analyze conflict dynamics. They agree that the diagnostic component of the methodology is strong and provides an excellent framework for conceptualizing and categorizing conflict dynamics. The survey data support these perceptions. However, the evaluation findings suggest that resiliencies and bright spots require additional elaboration and emphasis in the methodology. Similarly, survey respondents (53%) reported challenges with identifying key mobilizers. The ET found three key challenges based on responses to open-ended survey questions, feedback papers, and interviews: 1) Challenges of identifying key mobilizers for violence and peace in fragile contexts (Sri Lanka, Morocco, Senegal, Guatemala and Liberia) versus contexts experiencing active conflict. In fragile settings, identification of key actors is not always obvious when conflict is not prevalent. 2) Accounting for mobilization when there is no clear key actor. For example, in situations where people are mobilized using technology, such as SMS, or through decentralized networks, such as ISIS. 11 List of CAFs that were informed by previous conflict assessments: Georgia, Zimbabwe, Liberia, Southern Thailand, Cambodia, Nigeria, Guatemala, Tanzania, South Sudan, Kosovo, Kenya, Senegal and Uganda. CAF is useful tool for diagnosis. It helps to understand dynamics and conflict potential. ~ Team Member CAF’s key actor analysis does not go far enough. Understanding interests and incentives is a gap. ~ Team Member 7 | CAF 2.0 EVALUATION 3) Understanding interests and incentives of key actors is a gap, particularly in how these link to other aspects of a conflict system. (Southern Thailand, Guatemala). Interviewees indicated that the response section of the CAF 2.0 is weaker than the diagnostic component. This validated findings from the desk review. For further discussion of the CAF 2.0 response component, see 2.2 below. CONCLUSIONS ON EVALUATION QUESTION 1 The combination of assessment objectives and available resources determines the selection of the CAF model. The majority of assessments reviewed by the ET used the Standard CAF model, indicating that this continues to be the preferred option in most circumstances. Adherence to the Application Guide by teams implementing Standard CAFs appears to have increased over time. The CAF 2.0 methodology is strong but would benefit from further elaboration on resiliencies, bright spots, and identification of key mobilizers to reflect evolving trends in conflict contexts. Respondents to the survey and interviews agree that the response component of the CAF 2.0, which includes synthesis process and developing recommendations, is weaker. RECOMMENDATIONS FOR EVALUATION QUESTION 1 As USAID continues its reorganization, and in an effort to continue refining the CAF models and methodology, CMM should: • Consider how to apply different CAF models in the context of the new bureau. For the last four years, CMM mainly used the Standard CAF model. However, the evaluation shows that CMM has employed other models such as the CAF Lite or CAF Combo/Hybrid. As the bureau and CMM itself evolve, consider revisiting different CAF models and reflect on the pros/cons of moving forward with models other than the Standard CAF. Explore the potential of offering a menu of models that allow for use in different situations and contexts. • Elaborate and refine the concepts of resiliencies, bright spots and key mobilizers for violence and peace in the CAF methodology, including the Key Mobilizer Analysis Tool (Annex B of the CAF), in order to provide guidance on how to identify actors in fragile contexts and in cases when multiple actors mobilize groups using technology and/or decentralized networks. EQ2: TO WHAT EXTENT DO ASSESSMENT TEAMS FOLLOW THE CAF APPLICATION GUIDE RECOMMENDATIONS ACROSS ALL PHASES OF THE CAF? 2.1 To what extent do teams adhere to the CAF Application Guide? Findings from all three rounds of data collection indicate that teams working on Standard CAFs generally follow the Application Guide process. In the preparation phase, 88% of all survey respondents indicated that the team conducted some form of literature review before beginning field work. Interviews confirmed that most teams had some orientation and training before beginning field work, and many identified these as very helpful. It is hard to transition from diagnosis to response. ~ Team Member CAF 2.0 EVALUATION | 8 Survey responses indicate that teams spent an average of 10 days collecting data in the field. All teams conducted some form of synthesis to get to recommendations (see EQ3 below for additional information on synthesis). Given time constraints, some teams such as those in Liberia, Guatemala,12 and South Sudan developed or fleshed out more detailed recommendations after returning to the US. In the case of Uganda, the team never finalized the report. Interviewees suggested that this was because there was no international consultant to write the report. It is worth noting that in other cases, like Kosovo, the CMM Team Lead produced the report rather than an international consultant. 2.2 To what extent do Conflict Assessment reports reflect the methodological framework? Findings from the literature review demonstrate that conflict assessment reports are more likely to reflect the diagnostic components of the methodological framework than the response components. All CAF reports reviewed by the team included analysis of the context and drivers of conflict. The majority (88%), included discussion of resiliencies. The concept of key mobilizers appeared in most of the reports, although they did not always label them as such and the extent of key mobilizer analysis varied (see section 1.5 above for a discussion of challenges identifying key mobilizers). Most of the reports (88%) included some analysis of trajectories, identifying trends, triggers, or both. The response component (pictured) of the CAF reports did not follow the methodology as closely as the diagnosis. All reports included recommendations although less than half of the reports (47%) included theories of change. Of 17 CAF reports, eight included theories of change (TOC). The logic of the TOC and their linkages between conflict dynamics and recommendations varied. Only two of the 17 reports included an explicit operational filter, although several others offered operational considerations for implementing programming such as comparative advantage (Sri Lanka) and strategic program sequencing (South Sudan). 2.3 Which parts of the CAF Application Guide are more challenging to follow or implement? Several challenges to implementing the Application Guide emerged from the document review. The most frequently mentioned challenge in the feedback papers is a lack of clarity about the desired skills and expectations of a Team Lead. Three feedback papers pointed out that the Team Lead, and most team members, lack detailed understanding of USAID programming in the country where the CAF took place. The Team Lead may have been hired for their country expertise, but this still leaves a gap when trying to craft recommendations that are meaningful in the context of broader US Government assistance parameters, CDCS planning, or an existing Mission portfolio. 12 The Guatemala team had to leave the country one day early due to violent protests. 9 | CAF 2.0 EVALUATION Other challenges emerging from the desk review included expectations that teams conduct field work in multiple locations in a limited timeframe in difficult security and infrastructure environments, leaving very little time to collect data in each area. These findings were reinforced by interviews (Guatemala, Iraq, Kenya, Georgia, Morocco, Liberia, Zimbabwe, Cambodia and Southern Thailand). For example, in Zimbabwe, the team had just one part-time local consultant, which constrained their ability to cover significant geographic areas. The desk review and interviews also revealed that some teams used innovative approaches to expand geographic coverage in light of team size and security constraints. The Kenya and Zimbabwe teams subcontracted with local organizations to collect data in highly insecure areas. The Iraq team used telephone interviews to contact informants who were physically inaccessible due to insecurity. Survey results echoed these challenges with 43% of surveyed team members selecting “field work” as the biggest challenge in the assessment process, followed by “synthesis process” (35%), “preparation” (21%), and “report writing” (14%). However, review of the data revealed differences in perceptions. Local conflict specialists identified field work as the most challenging part of the Application Guide, whereas CMM, international consultants, and Mission staff indicated that the most challenging part was synthesis and generating recommendations. Interviews reinforced findings from the survey. Some local conflict specialists indicated that there should have been more time for field work. With one exception, USAID staff and international consultants did not echo this. Challenges that emerged in the interviews largely aligned with the survey data with a few exceptions. Most interview participants felt that the time allocated for field work was reasonable, although some local conflict specialists indicated the need for more time to collect data. Several interviewees raised the challenges of short lead times, which had implications for logistics, security, and the extent to which teams were able to prepare for the assessment before arriving in country. Many of the local conflict specialists felt that the time for synthesis and developing recommendations was adequate, while international consultants and CMM team members indicated that there was not enough time for synthesis and developing recommendations. Open-ended survey responses elaborated further on field work challenges including: access to field sites or to interview government officials (Kenya, Sri Lanka), security issues (Southern Thailand, Kenya, Iraq, Zimbabwe), harsh weather and infrastructure conditions such as poor road conditions, vehicle related challenges and poor internet connections (Iraq, Liberia, Kenya), some level of sensitivity around interview questions (Kenya, Zimbabwe, Iraq, Senegal), and language barriers (Nigeria, Kenya, Guatemala). Seventy percent (70%) of survey respondents reported that having an in-country orientation was an operational success. Open-ended responses suggest that the days spend on orientation and training provided an opportunity for the team to work together, offered a shared conceptual framework and helped to ensure consistency. Interviewees also highlighted challenges identifying and interviewing sources of information aside from high profile actors and USAID partners. These included lengthy or complicated protocol requirements to meet with government, police or military personnel; US Government restrictions on speaking to different parties Field data collection was the most difficult part: time for data collection was tight to get adequate information. Always a rush. ~ Team Member Respondents did not show up for an interview or the location of interview was wrong. We could use more time for the preparation. ~ Team Member There is not much time for the preparation and desk study. ~ Team Member CAF 2.0 EVALUATION | 10 to the conflict; host country surveillance and fear of endangering interviewees; and the difficulty of identifying non-USAID partners for interviews (as in Kosovo, Morocco, Kenya, and Senegal). Similar challenges related to interviewing informants emerged in the document review in the Georgia, Zimbabwe, Kenya, Iraq, Guatemala, and South Sudan assessments. Respondents elaborated on the difficulty of synthesizing vast quantities of data collected in very complex environments in a limited timeframe. Some Team Leads and CMM staff also indicated that they would like to try different synthesis approaches, for example systems mapping, but feel that they lack the skills. International consultants and CMM staff also raised the challenges of transitioning from diagnosis to response, citing lack of familiarity with the Mission portfolio and not enough time to complete synthesis and recommendations. Even with a DRG or conflict specialist on the team, they generally do not have the deep enough knowledge of other parts of the Mission portfolio to formulate targeted programmatic recommendations. A number of Mission interviewees emphasized the importance of including representatives from other sectors on the team. The role of CMM support in keeping the assessment findings and recommendations on the front burner for the Mission was raised in multiple interviews by CMM and Mission representatives. They emphasized that post-assessment engagement by CMM was central for making sure that Missions utilized recommendations and integrated them into conflict-sensitive programming. However, there is no discussion of CMM follow-up activities in the CAF Application Guide.13 The evaluation findings suggest that follow-up activities are important to help Missions integrate assessment findings and strengthen utility of the assessment. 2.4 What are the most important operational issues when conducting conflict assessments? Survey respondents indicated that the two greatest operational challenges facing teams are timeline (43%) and logistics (38%). Open-ended responses indicate that the timeline challenges refer primarily to a compressed timeframe for processing data, conducting synthesis and developing recommendations. Logistical challenges included access to field sites or to interview government officials (Kenya, Sri Lanka), security issues (Southern Thailand, Kenya, Iraq, Zimbabwe), harsh weather and infrastructure conditions such as poor road conditions, vehicle related challenges and poor internet connections (Iraq, Liberia, Kenya), some level of sensitivity around interview questions (Kenya, Zimbabwe, Iraq, Senegal), and language barriers (Nigeria, Kenya, Guatemala). The interviews echoed these findings, revealing that nine teams experienced significant logistical constraints (South Sudan, Nigeria, Morocco, Senegal, Uganda, Southern Thailand, Sri Lanka, Liberia and Georgia). Significant constraints included poor roads and limited flights hampering timely access to more remote areas, hiring qualified translators, setting up meetings, and communications. In some cases, communications were unreliable. In others, teams were concerned about using communication channels fearing that they were being monitored. Some team members were not able to travel to locations that 13 The ET defines “follow-up activity” as the post-report submission process to gather the Mission’s feedback on the usefulness of the CAF findings, conclusions and recommendations, and to find out whether any follow-on services are needed in order to integrate CAF recommendations into the Mission’s interventions. Logistics and preparation. Team was requested to visit particular sites; however, security approval process was challenging. ~ Team Member 11 | CAF 2.0 EVALUATION were off-limits to US citizens. In other cases, incidents such as protests left teams unable to access parts of the country perceived as being volatile at the time. Despite these challenges, teams made adjustments to mitigate security and logistical hurdles in the field. For example, the Iraq team conducted phone interviews with key informants in areas occupied by Al￾Qaeda in Iraq. Similarly, upon learning of Al Shabab activity in designated interview areas, the Kenya team rerouted, and key informants were able to travel out of these areas to meet them. Both the Kenya and Zimbabwe teams contracted with local organization to collect data in areas that were inaccessible to team members. In Kosovo, the team worked through local NGOs and used interns to conduct the interviews, since the USG staff were unable to travel to speak with them directly. For eight of the 17 conflict assessments, respondents mentioned the size, composition, qualifications, and dynamics of conflict assessment teams. The primary concern in three of these assessments was the size of the team vis-a-vis the number of field work locations requested. In other cases, interviewees raised issues with team qualifications or dynamics with the potential to affect the quality of the analysis or the ability to produce a report. 2.5 How well do timelines for each phase of the Application Guide support optimal utilization of data captured? According to survey respondents, time constraints were most challenging during synthesis (42%), in￾country data collection (40%), generating recommendations (35%), and sharing information across teams (35%). This aligns with the findings is section 2.3 above. CONCLUSIONS ON EVALUATION QUESTION 2 The conflict assessments reviewed by the team largely reflect the diagnostic components of the CAF methodology with slightly less consistency in identifying resiliencies and key mobilizers. The reports demonstrated less adherence to the response component of the CAF methodology. While all reports offered recommendations, more than half did not explicitly state the theory of change logic linking these recommendations to the conflict dynamics, or the logic linking them was not clear. Likewise, very few reports included an overtly stated operational filter, an important piece of the CAF response that ensures recommendations align with the Mission’s strategic, programming, and resource considerations. Teams need more support to transition from diagnostic to response components of the CAF. The most challenging parts of the CAF Application Guide to implement are preparation (pre-departure), identifying and securing interviews, synthesis, and developing recommendations. Constrained timeframes, team skills, role confusion, and limited understanding of the Mission portfolio contributed to these challenges. Teams benefit when there is adequate time to prepare, roles are clear, and they have the resources to identify potential interviewees aside from existing USAID partners. Teams need more time and the right skill sets to conduct synthesis. When teams are expected to generate recommendations that plug in to planned or existing Mission programs, they need to be able to access programmatic information and parameters. There is a need to provide more guidance on synthesis methodologies and response part of the framework. Given the option, adding time to the synthesis and recommendations process is more important than adding time to data collection in the field. This will provide teams adequate time to process data and generate recommendations that are relevant and meaningful for the Mission. CMM follow-up plays an important role in supporting Mission utilization of findings and recommendations. CAF 2.0 EVALUATION | 12 There are diverse operational constraints on conflict assessment teams including compressed timeframes, staffing, security, logistics, and Mission constraints. At times, these constraints have adversely affected conflict assessment teams’ efficiency and effectiveness. While it is impossible to eliminate these operational constraints, there are opportunities to mitigate and manage issues when they arise, such as approaches taken in Kenya, Zimbabwe, and Iraq to expand the reach of the assessment into insecure areas (specific examples provided in findings above). RECOMMENDATIONS FOR EVALUATION QUESTION 2 CMM developed the CAF Application Guide in 2012 to coincide with the release of CAF 2.0. The guide contains a wealth of information that is largely still relevant and useful today. Since 2012, however, CMM has continued to learn about conducting conflict assessments. In order to ensure that the Application Guide remains a useful resource for teams, CMM should: • Revisit and update the CAF Application Guide to incorporate changing realities of conducting conflict assessments as well as good practice and lessons learned from seven years of assessments. In particular, CMM should review the sections that address preparation, team composition, conducting interviews, implementing synthesis and developing targeted recommendations. Currently, the Application Guide describes a typical assessment as requiring, “...a minimum of eight weeks from start to finish, including one week of preparation, one week to prepare a desk study, three weeks of field work, and, finally, three weeks for final analysis and writing.” (p. 7) CMM should extend the window for preparation to reflect the realities of planning a complex assessment. These include logistics and identifying local support such as translators and drivers, conducting a literature review, reviewing program information, developing a field implementation plan, confirming areas of inquiry based on Mission priorities and literature review findings, drafting interview protocols and beginning to identify candidate interviewees in different locations. CMM could also add preparation good practices such as having the consultant Team Lead and logistics support arrive one week in advance. Encourage Team Leads to familiarize themselves with the various Mission programs. In addition to document reviews, this may include pre￾departure interviews with the different teams to better understand their objectives and projects. Team composition, including roles and responsibilities, would also benefit from review. Experience indicates that country expertise may not be the most important qualification for a consultant Team Lead. Depending on the circumstances, experience with the CAF 2.0, team facilitation skills, demonstrated leadership abilities, and strong writing skills may be more important qualifications for a Team Lead. While all of these appear in the Application Guide as important skills on the team, teams are often structured so that these Team Lead skills are required. For example, there are very few conflict assessments where the Team Lead is not responsible for writing the report. There are also areas of team member roles and responsibilities that would benefit from additional clarification in the Application Guide. For example, the relationship between the CMM staff and the consultant Team Lead. Guidance does not need to be overly prescriptive, but a simple checklist that guides the two through a discussion and agreed upon allocation of important roles and responsibilities might help to avoid confusion later on. Likewise, there is an opportunity to clarify expectations of the local conflict specialists, Mission team members, and logisticians regarding roles and responsibilities for identifying and scheduling interviews. Clear expectations can then convey to consultant scopes of work and harmonize roles for consultants and USAID. 13 | CAF 2.0 EVALUATION CMM should also include ideas and techniques for identifying non-USAID partner interviewees and accessing populations in difficult to access areas in the Application Guide. Guidance should take into consideration different approaches that are appropriate for secure versus highly insecure contexts. Include a more user-friendly discussion of how to use systems mapping approaches for synthesis. The Application Guide focuses on conflict archetypes and the benefits of systems thinking but does not necessarily guide readers through an illustrative process, such as the one describing the headlines approach. In addition to updating the Application Guide, CMM will need to provide training and hands on experience using systems mapping for synthesis (see capacity building recommendation below). Include concrete steps to support transitioning from conflict diagnosis to response. Expand the guidance for moving conflict assessment teams through the response elements of formulating theories of change, applying programmatic filters, and applying an operational filter to craft recommendations that are relevant and usable for Missions could be included. Good practice, such as vetting conflict dynamics and candidate recommendations with a carefully selected red team14 prior to a Mission-wide read out should also be included. Encourage teams to work with Missions to develop and apply an operational filter to ensure that recommendations are feasible within the parameters of Mission resources (programs, staff, budget, and time). For example, the Kenya Mission wanted recommendations that it could implement in the year leading up to elections and within the context of existing programs assuming no new funds. Allow more time for synthesis and generating recommendations. When feasible, consider adding additional time for synthesis and response. An additional day for synthesis would allow deeper refinement and give teams opportunities to advance application of systems mapping by testing whether linking a given set of feedback loops does, or does not, contribute to a different understanding of a conflict system. Alternatively, teams could use an additional day to move from synthesis to a well thought out, agreed upon set of recommendations that a red team can vet before presentation to the Mission. The loss of this time is a missed opportunity for teams to finalize recommendations once they leave the country. In this case, the task falls largely to the Team Lead and CMM counterpart, missing the vibrant discussion of a consensus-based process with the entire team contributing in real time. When CMM and Team Leaders develop recommendations after leaving the country, the team also misses the opportunity to elicit feedback from the broader Mission or Embassy community during the debrief. Consider incorporating follow-up activity in the Application Guide. As discussed in the Findings section, follow-up activities increase Mission utilization of assessment findings in programs and support integration of conflict-sensitive programming. This may require a shift from thinking about a conflict assessment as a discrete event to thinking about conflict assessment as a process that does not necessarily stop once the team has delivered the report. EQ3: WHAT SYNTHESIS PROCESS ARE TEAMS MOST COMMONLY USING? 3.1 What synthesis process is more useful or appropriate in specific circumstances and why? 14 The use of a red team is a process whereby the assessment team presents their findings and recommendations to a small cross-sectoral group of Mission staff in order to test validity and linkages to Mission priorities and programs, before the Mission-wide debrief. For example, in Senegal, the team presented their findings to senior local staff representing different sectors including economic growth, health, and education. In Kenya, the red team included a member from the political section as well because the findings were a priority for the Ambassador. CAF 2.0 EVALUATION | 14 The desk review revealed that almost half (47%) of conflict assessments did not document a synthesis approach. Of those that did, 12% reported using some form of headlines, 18% reported using systems, 6% reported using a combination of headlines and systems mapping, and 17% reported using other methods. Feedback reports, which begin in 2014 and are available for every CAF except Uganda (2015), were helpful sources of information about the process of conducting a CAF versus the actual report, which reflects the results of applying the CAF methodology. The ET asked survey respondents to select which approach their assessment used, but the data was not consistent with the desk review or interviews. Respondents appeared to be unclear about the different methodologies listed. By triangulating the desk review and interview data, the team found that 47% of the teams (Cambodia, Iraq, Kosovo, Liberia, Nigeria, South Sudan, Uganda, and Sri Lanka) reported using a headlines approach; 24% of the teams (Georgia, Guatemala, Morocco, and Southern Thailand) reported using mostly headlines with elements of systems mapping; 6% of the teams (Tanzania) reported doing a combination of systems and headlines; 18% of the teams (Kenya, Senegal, and Zimbabwe) reported doing systems mapping. The remaining assessment (Niger) reported conducting a network mapping of sources of information that may be more aligned with the Interagency Conflict Assessment Framework (ICAF) approach used by the team than those typically used in CAF synthesis. The timing of synthesis varied depending on circumstances and strategy. Most teams conducted synthesis at the end of field work. Only one team (Morocco) held a formal mid-point synthesis bringing all sub￾teams back to the capital and then redeploying to gather more information and fill gaps. In Kosovo, the team did not split into geographic sub-teams and met twice weekly to conduct rolling synthesis. Although the South Sudan team did split into two sub-teams, both teams returned to Juba every weekend and participated in informal sharing rather than a structured synthesis which was done at the end. Three teams (Zimbabwe, Kenya, and Southern Thailand) reported constraints that limited their ability to conduct synthesis within and across sub-teams. These included security concerns, fear of communications monitoring and having to travel with armed guards. Two teams (South Sudan and Morocco) exchanged sub-team members mid-way through the assessment to support cross-fertilization. 3.2 What is the perception of CAF teams regarding the synthesis process? Thirty five percent (35%) of survey respondents identified synthesis as among the most challenging aspects of conducting a conflict assessment due to the volume of data, complex nature of the conflict, and a limited understanding of the Mission portfolio for transitioning to recommendations. An even higher number, 43% flagged insufficient time for data processing, synthesis, and developing recommendations. For additional discussion of time constraints relating to the synthesis process and generating recommendations, refer to sections 2.3 and 2.5 above. We did headlines and then systems mapping. Systems mapping with novice do not work well; however, it works well for smaller teams who are aware of the systems. ~ Team Member The weakest part of CAF is the synthesis. There is little guidance how to do it. Section on synthesis should be improved. ~ Team Member 15 | CAF 2.0 EVALUATION Despite these challenges, more than half of survey respondents (51%) rated synthesis as “very effective.” This was followed by 32% of respondents who considered synthesis “effective.” Open-ended responses indicated that other success factors for effective synthesis included strong Team Lead guidance to facilitate the process and inclusion of local consultants. Perceptions of synthesis shortcomings included not enough time to effectively conduct systems mapping, difficulty transitioning to solid recommendations when using headlines, and losing “richness” of the data collected when working in such a compressed timeframe. Interviews validated the finding that most respondents felt the synthesis process, whether using headlines or systems mapping, produced good results. The majority of non-CMM respondents reported that the headlines approach was a very satisfactory process. Team members who have worked with both systems mapping and headlines express a preference for the systems mapping approach. Perceived benefits of using systems mapping include more rigorous analysis, a focus on relationships versus lists, and ease of transitioning from diagnosis to response, particularly identifying theories of change. Several CMM affiliated interviewees expressed an interest in using systems mapping for synthesis but did not feel confident that they had the skills to lead the exercise or questioned whether systems mapping would render synthesis “too academic.” Some indicated that they tried incorporating elements of systems mapping into the synthesis process but did not feel that they had the skills to rely on systems mapping alone. 3.3 What other synthesis process may be included / how do other assessment frameworks conduct synthesis? Respondents offered few concrete alternatives to current synthesis processes. One interviewee suggested hiring someone to research alternative synthesis approaches. Another reflected on conducting stakeholder analysis but concluded that it is not feasible to incorporate into field data collection given time constraints. The team also reviewed other USAID assessment frameworks, including the DRG Strategic Assessment Framework and Political Economy Analysis, and consulted with practitioners. The documents did not offer synthesis approaches and practitioners suggested that the Team Lead decides the approach used. There does not appear to be guidance on synthesis methodologies or consistent approaches utilized across assessment teams. CONCLUSIONS ON EVALUATION QUESTION 3 Synthesis timing and process are determined by a combination of enabling conditions (for example, number and location of sub-teams) and the preference and skills of the Team Lead and their CMM counterpart. During the timeframe of the evaluation, headlines is the most commonly used approach and the most familiar for CMM staff. Team members are interested, but do not feel confident leading a synthesis using systems mapping. Regardless of which process is used, team members and Mission end users are generally satisfied with both the process and the resulting recommendations. A level of uncertainty remains among CMM staff about using systems mapping for synthesis that inhibits broader application of this approach. RECOMMENDATIONS FOR EVALUATION QUESTION 3 In order to improve synthesis processes, CMM should: CAF 2.0 EVALUATION | 16 • Continue to further refine and expand systems mapping beyond its current use for synthesis. Explore opportunities to bring practitioners together with systems experts to pilot processes that move beyond developing feedback loops to connecting loops into a larger system and identifying strategic points of leverage. When possible, pilot these approaches, adding an additional day to ensure teams can complete the process. Assess and document if systems approaches provide a clearer pathway to theories of change and eventually to recommendations linked to drivers of conflict (the conflict system). • Consider developing a guide to synthesis methodologies, including elaboration on the headlines, systems mapping, and other synthesis options focusing on what does and does not work in different situations. External expertise may be needed to expand on synthesis methodologies and present them in an accessible manner. • Continue piloting innovative mid-point and rolling synthesis approaches as opportunities arise. Continue to explore the utility of bringing sub-teams together mid-CAF for synthesis. This may also include exchanging team members if it is not deemed too disruptive. Assess and document the costs and benefits of these approaches. Also look for opportunities to build in more structured or semi-structured rolling synthesis opportunities tailored to team circumstances. In some cases, this might include intentionally setting a few minutes aside at the end of each day for a sub-team debrief or, when possible, conducting a cross team readout. EQ4: TO WHAT EXTENT HAVE MISSIONS UTILIZED CONFLICT ASSESSMENT REPORTS IN PROGRAMS AND OPERATIONS? The majority of surveyed Mission POCs, report readers, and team members agree that the assessment report was “very useful” (65%) or “useful” (22%) for informing the understanding of the conflict dynamics in the targeted area. Similarly, the respondents believe that the reports were “very useful” (41%) and “useful” (35%) for identifying relevant programmatic recommendations in response to conflict dynamics. Interviews reinforce survey findings with all key informants recognizing the importance, relevance, and usefulness of the CAF. Of the 19 interviewees who were knowledgeable about report utilization, 17 confirmed that Missions use the results actively, although the degree of this varied. One assessment had a utilization challenge as the results came out late, mainly due to the delayed Mission comments (South Sudan). In Iraq, despite strong endorsement from Mission staff, the Mission was unable to use the report because it dramatically reduced staff as security deteriorated. The interviews also revealed that Missions continue to use the assessment reports as a source of information long after their completion. Kenya respondents said that the conflict assessment report is still very alive and relevant, and they are using it in the design of the new CDCS. The Liberia Mission still refers to the report when they program new funds. The ET found that CAF reports and recommendations supported programming by: CAF exercise is a learning forum and a good way to bring people on board. ~ USAID Mission The conflict assessment was received well as the recommendations were relevant and usable. ~ USAID Mission 17 | CAF 2.0 EVALUATION Informing Mission strategies and projects/activities Cambodia recommendations informed a program addressing transitional or restorative justice. Southern Thailand recommendations were used in 2017 for a new Deep South conflict mitigation project design. Niger recommendations informed the design of project working with youth. Georgia recommendations supported design of a $7.5 million project in the occupied territories with large health and education components. Liberia used the assessment to reintroduce a conflict work stream. Kenya used the assessment results to draft parts of a Conflict and Electoral Systems assistance project and the Embassy implemented a peacebuilding project informed by CAF recommendations. Morocco switched the geographic focus of their Combating Violent Extremism (CVE) and youth inclusion activities to the middle of the country. Providing a basis for Reconciliation Fund applications Liberia’s DRG team designed and implemented the “Community Dialogue for Conflict Mitigation and Reconciliation Program,” and the “Connect for Peace” activities immediately following the assessment. Kosovo received funds to implement activities aimed at the north of the country. Southern Thailand hopes to update the 2013 CAF to inform future Reconciliation Fund applications. Senegal received funds to engage communities in peacebuilding activities and the assessment is reflected in the Annual Program Statement (APS) Senegal Annex describing the desired focus of peacebuilding programs. Kenya used funds to respond rapidly to potential election violence. Guatemala used funds to implement the Communities Building Peace Together project in Western Highlands. Nigeria received funding for the “Community Initiative to Promote Peace” program. Validating what Missions think they know and facilitating consensus around conflict dynamics within the Mission and the broader country team The Morocco report helped to address discussions and disagreements about the cause of protests. After the briefing, people on the country team started using common terminology and wanted copies of the report. The Cambodia process helped to build consensus around conflict dynamics within the broader country team. The Iraq findings helped to facilitate consensus among the office directors who were initially skeptical about the assessment. The Guatemala assessment team became a conduit between technical offices and Mission leadership to convey the importance of conflict sensitivity and peacebuilding. The Senegal report contributed to a broader Embassy dialogue about the peace process. Liberia initiated an International Narcotics and Law Enforcement Affairs (INL) project connecting civil society and citizens with security councils. Advocacy Senegal recommendations led to a nationwide advocacy campaign for integrating women into the peace process. Georgia felt that they had an advocate in Washington once CMM understood the context. In Kenya the Ambassador used the report for his engagements and briefings informing policy. The report also provided a foundation for CMM advocacy to USAID/Washington on behalf of decisions made by the Mission. Morocco assessment findings helped maintain budget levels to support stability as administration priorities changed. For operationalizing or institutionalizing conflict sensitivity, the assessment reports informed: Incorporating conflict sensitivity across the Mission portfolio and new activities Liberia and CMM conducted a follow-on portfolio review looking at each program with a conflict sensitivity lens. They have also incorporated conflict sensitivity into Mission-wide reviews of new projects. Tanzania integrated Do No Harm (DNH) principles as a result of the CAF report. Kosovo created a list of assessment recommendations to use in portfolio reviews. Guatemala worked with CMM to review RFAs/RFPs and integrate conflict sensitivity language into Program Descriptions (PD), SOWs and evaluation documents. Nigeria Mission leadership requested conflict sensitive implementation of all activities. The DRG team collaborated with technical teams to conduct conflict sensitivity audits on all new activity designs. Senegal Mission leadership requested that all technical staff consult the assessment when designing activities. Drafting Mission Orders incorporating conflict sensitivity CAF 2.0 EVALUATION | 18 Guatemala drafted a Mission Order on conflict sensitivity Incorporating conflict sensitive indicators into M&E Nigeria includes a logframe indicator requiring implementing partners to collect data and conduct activities in a conflict sensitive manner, effectively mandating application of a conflict-sensitive approach. 4.1 What made these outputs achievable in some instances and not others (timelines, funding, staffing level, Mission capacity to support follow-on activities, follow-on support from CMM)? Buy-in from leadership and other teams was found to be a driving force for implementing recommendations. Fifty three percent (53%) of survey respondents indicated that “commitment/buy-in of Mission leadership” contributes to the Mission’s utilization of CAF recommendations. This is supported by interviewees who pinpointed prioritization of the CAF by Mission leadership, engagement of the Program Office, and outreach across teams as critical for broad-based buy-in to the process and recommendations. CMM and USAID/Kenya’s DRG team spent significant time engaging other sector teams, Mission leadership and other parts of the Embassy before the assessment team ever arrived in country. In Iraq, early in the process, a CAF champion at the Mission recognized that support from the Program Office and Front Office were needed to find funds for the assessment. In cases such as Tanzania, where these factors are absent, the assessment may gain little traction beyond the DRG or conflict team. Other factors identified as limiting utilization included competing priorities and understaffed Mission. Despite CMM’s efforts to provide support, there was only limited integration of the assessment into the Uganda CDCS because the assessment was not a top priority. The ET found that consideration of feasibility within Mission resources and programmatic parameters is another force driving implementation. At the request of the Mission, the Senegal team focused on recommendations anchored in existing programs. In Niger, although the Mission valued the diagnostic part of the assessment, it viewed the recommendations as too high level to be useful. Perceived relevance of the conflict assessment process and recommendations. According to 63% of survey respondents, “timing of the assessment” contributes to utilization. Interviewees confirmed that perceived and immediate need, whether informing a strategy or experiencing activity delays, played an important role in utilization. Examples include contributing to CDCS or project design (Senegal, Kosovo, Liberia, Georgia), or preparing for elections or understanding social protests (Kenya, Nigeria, Morocco). CMM follow-on played a critical role in increasing utilization, according to interviews with end users. In Guatemala and Kenya, CMM provided sustained support, continuing to engage with the Missions to help with implementation. In Liberia, CMM returned to conduct a portfolio review for conflict vulnerability. Interviews with end users revealed that several Missions did not request CMM follow-on support (Georgia, Nigeria, Southern Thailand, Niger), although some reflected that it would have been helpful. CAF helped tremendously for us as we continue to grapple with issues. It gives us credibility because we won’t be seeing as trying to separate development from conflict. ~ USAID Mission Recommendations need to be tailored to Mission resources in order to get higher chance of utilization. ~ Team Member CAF came up with a lot of practical recommendations but DRG doesn’t have the money to implement. ~ USAID Mission 19 | CAF 2.0 EVALUATION Interview subjects consider availability of reconciliation funds to be an important avenue for implementing assessment recommendations in many countries (see matrix above for Missions that used Reconciliation Funds). Availability of the Reconciliation Fund helps Missions finance peacebuilding activities. 4.2 What challenges prevented or limited the use of CAF findings to inform follow-on activities or outputs used by the Mission? Some Missions limit distribution of the assessment report beyond the USG due to concerns about political dynamics and sensitivities, preventing wider utilization (Kenya, Senegal). Respondents also identified changes in Mission leadership as another challenge to implementing recommendations (Kenya, Iraq). In both Tanzania and Cambodia, respondents cited limited inclusion of technical teams in the assessment process, or perceptions that conflict is a DRG issue, as limiting utilization. Conversely, respondents identified the presence of cross-sectoral representation on the team as improving conflict sensitivity in education programming. Delayed submission of the assessment reports reduces the likelihood of utilization. In South Sudan and Niger, the final reports were delayed waiting for Mission comments. In Morocco, the consultant Team Lead struggled to produce work that met USAID’s standards of quality and the contractor eventually replaced him. The serious illness of the Team Lead of the Cambodia CAF delayed submission for several months. 4.3 What could improve the extent to which Missions adopt recommendations? The ET found cultivating broad-based support up front contributes to improved utilization. It is important to determine the extent to which the conflict assessment is a priority for Mission and Embassy leadership. One approach to this is to encourage the Mission POC to actively engage the Program Office, sector/team leads, Embassy leadership, and key members of the country team. This good practice was evident in the Kenya conflict assessment. Understanding what is driving the assessment is another important means of gaining broad-based support. Assessment teams need to clarify with Missions exactly how they will use the recommendations and what is motivating the conflict assessment. As the Georgia and Kenya experiences demonstrate, the more immediate and pressing the need for a conflict assessment, the greater the likelihood of utilization. Develop recommendations that are relevant, actionable, and meaningful vs. “what we already know” or just plain “wrong” in the eyes of the Mission. Understanding concerns of the Mission about where they see the possibility of conflict impeding their ability to reach their objectives, as well as how a conflict assessment can add value/advance the success of their activities as it relates to conflict, can help to develop practical recommendations as well as to increase a sense of ownership. The ET found having non-DRG staff on the assessment team to be an effective approach to designing recommendations linked to different parts of the Mission portfolio. The evaluation found that when Mission staff participate in all aspects of a conflict assessment, it builds their capacity to assess conflict dynamics and formulate responses that remain with their respective teams long after the assessment is finished. The Senegal conflict assessment is evidence of this. A number of Mission representatives and Fundamentally conflict assessment should be driven and owned by Mission to ensure the utilization. ~ USAID Mission CAF 2.0 EVALUATION | 20 CMM affiliates interviewed, in retrospect, reflected on the importance of inclusion of sector staff on the assessment team. Existing relationships and established credibility with the Mission matter. Interviews revealed that building relationships with the Mission, and in some cases the country team, helps with acceptance of the report and the ability to implement recommendations. In Guatemala, CMM capitalized on relationships with staff and Mission leadership to effectively convey the importance of a dedicated peacebuilding program. Similarly, the Kosovo Team Lead’s relationships and understanding of Mission context and portfolio allowed for delivery of difficult messages that contributed to meaningful recommendations. Respondents mentioned leveraging or cultivating relationships with the Mission in Kosovo, Guatemala, Kenya and Iraq as furthering the credibility of assessment recommendations. CONCLUSIONS ON EVALUATION QUESTION 4 The evaluation concludes that Missions have utilized conflict assessments results largely to enhance programming and institutionalize conflict sensitivity. They continue to use CAF reports as a source of information for new staff, to inform new project / activity design, and to support conflict sensitivity across Mission portfolios. Two scenarios appear to improve the likelihood that Missions will implement recommendations. The first is when CMM maintains an ongoing relationship with the Mission. In these cases, the CAF is one part of a longer-term partnership that begins before the assessment and continues afterwards, as was the case in Guatemala and Kenya. CMM has a deep understanding of the Mission context and portfolio. They are able to work with the POC to cultivate broad-based support at all levels upfront. The second is when a Mission considers the conflict assessment as one part of a broader self-driven conflict sensitivity agenda. This was the case in Liberia and Nigeria, where the conflict assessment was one of several activities initiated by the Mission to foster a robust conflict workstream or institutionalize conflict sensitivity. There are factors that may detract from Mission utilization that are largely out of CMM’s control, for example, changes in leadership or decrease in security. RECOMMENDATIONS FOR EVALUATION QUESTION 4 In order to increase Missions utilization of CAF results, CMM should: Make the CAF one part of an ongoing relationship between CMM and the Mission. Understanding the Mission context, portfolio, and level of support from leadership, technical teams, and, in some cases the country team help to set the Mission up for success. After completing the assessment, follow-on support plays an important role to help Missions integrate assessment findings and strengthen utility of the assessment. In the cases of Guatemala and Kenya, the relationship with CMM continued for several years. • To maximize the likelihood that Missions will utilize recommendations, undertake the following before the assessment: o Work with Missions early in the process to clarify expectations around the level of recommendations that the team will provide. Section four of the Application Guide states that “the assessment will provide high-level recommendations to inform this planning process, but detailed implementation of those findings may require additional technical 21 | CAF 2.0 EVALUATION assistance, which CMM can provide.” 15 However, as this evaluation’s findings show, Missions might need detailed recommendations instead of high-level. Therefore, it is important to clarify whether the Mission is expecting high-level recommendations that they can then tailor and apply to projects and activities, or recommendations grounded in existing programming that require knowledge of the portfolio. CMM should clearly communicate these expectations to the Team Lead and be clear about the Team Lead skills required and resources necessary to meet these expectations. o Consider developing a checklist or tool describing steps for cultivating broad-based support, understanding the Mission context, portfolio, and programmatic parameters. This will help teams to effectively plan, implement and strengthen utilization of the report. o Understand what is driving the assessment. Is there an upcoming opportunity to apply the findings and recommendations? Clarify with Missions exactly how they will use recommendations and what is motivating the conflict assessment at this time. o Encourage teams to understand the Mission context and portfolio, as it will help to cultivate support, buy-in and facilitate developing relevant, actionable and meaningful recommendations. When preparing for the assessment, seek to understand concerns of sector/Development Objective teams and the ways that the conflict assessment can add value or advance the success of their activities. Is the Mission expecting high-level cross￾cutting recommendations that they can apply to CDCS development, or more detailed recommendations linked to specifics of an existing portfolio? o Strongly encourage CSAID training at the Mission before the assessment to begin socializing the CAF methodology and conflict sensitive programming across teams. This proved to be an effective approach for cultivating broad Mission support, increasing the sector teams’ understanding on conflict sensitivity, and strengthen utilization of assessment results. As practice shows, it will encourage technical staff to think in a conflict-sensitive way and draw linkages between their activities and conflict dynamics. o Advocate for including sector staff, in addition to DRG representatives, on the assessment team. This will help to better understand the Mission’s context and portfolio, focus conflict assessment recommendations, cultivate CAF champions, and build on the capacity of staff across sector teams for conflict sensitive implementation. Explore opportunities that may make diverse Mission participation easier or more attractive. • Request a Mission point of contact (POC) for a year or more following a conflict assessment. This will help CMM to identify and provide appropriate follow-on support. It will also encourage CMM learning about Mission successes and challenges with implementing assessment findings and recommendations. Interviews clearly indicate that even a few years after the assessment, Missions would like continued support to update or conduct conflict analysis, and continue integrating conflict sensitivity in their program designs and implementation. CROSS-CUTTING RECOMMENDATIONS The following cross-cutting recommendations arise directly from the CAF evaluation findings: 15 Conflict Assessment Framework: Application Guide, June 2012, USAID, p. 27 CAF 2.0 EVALUATION | 22 • Keep learning. Continue using feedback reports to document team learning on process and methodological issues that will not likely appear in the report or other conflict assessment documentation. Set aside time to review feedback reports annually and look for opportunities to improve efficiency and effectiveness by applying lessons learned. • Build CMM staff Conflict Assessment capacity. CMM staff play a critical role in guiding and supporting assessment teams throughout the process. CMM staff support has also been instrumental for helping Missions apply conflict assessment recommendations, for example in the cases of Kenya, Guatemala, and Liberia. Recently, CMM has experienced significant staff turnover, and USAID is undergoing substantial structural changes. In the midst of this evolution, considerable institutional expertise has been lost, detailed, or unable to actively participate in technical work due to other obligations. CMM should continue the good practice of sending less experienced CMM staff to participate, but not lead, a conflict assessment prior to assuming a leadership role. A similar principle applies with training more CMM staff to use systems mapping in the field. Training in systems mapping is an indispensable first step but leading this process in the field is a big leap. Having a CMM staff member “shadow” others who are familiar with the methodology and application is an important way of ensuring that CMM staff have the right mix of tools and approaches to draw upon when conducting conflict assessments. 23 | CAF 2.0 EVALUATION ANNEXES ANNEX A: TECHNICAL DIRECTION FOR TASK TWO: EVALUATION OF CAF ANNEX B: EVALUATION DESIGN MATRIX ANNEX C: LIST OF DOCUMENTS REVIEWED ANNEX D: DATA COLLECTION TOOLS • Survey Questionnaire • Conflict Assessment Team Member Protocol • Conflict Assessment Mission Engagement (Non-Team Member) Protocol ANNEX E: LIST OF STAKEHOLDERS CONSULTED ANNEX F: CONFLICT OF INTEREST FORMS A-1 | CAF 2.0 EVALUATION ANNEX A: TECHNICAL DIRECTION FOR TASK TWO: EVALUATION OF CAF Technical Direction for Task Order AID-OAA-TO-16-00036 Task Two: Evaluation of the Conflict Assessment Framework (CAF) A. Evaluation Purpose This technical direction provides guidelines for a Conflict Assessment Framework (CAF) 2.0 Evaluation. The main purpose of the evaluation is to: evaluate the different methodologies utilized by DCHA/CMM since July of 2012 to analyze qualitative data collected from key informant interviews during the course of field-based conflict assessments. The evaluation will: (a) define under what circumstances and for what reasons particular methodologies were utilized by CAF teams (b) identify the strengths and weaknesses of the different methodologies based on feedback from assessment teams (c ) evaluate the utility and/or shortcomings of the CAF findings and recommendations from the point of view of the client Mission (d) delineate if/when a specific methodology (including alternate methodologies not currently being utilized) would be more suited to producing the necessary outputs based on the stated purpose, objectives and key issues/questions in the CAF scope of work. The evaluation results will be used to: (a) inform revisions of the CAF, [including CAF guidance documents, synthesis processes and CMM and Mission planning processes]; and (b) update training materials related to conducting conflict assessments. CMM will also disseminate findings through technical briefs, a dissemination workshop and other roundtables and meetings as discussed in the deliverables section of the TO. These dissemination activities will help inform Mission and Washington-based staff about the suite of CAF and related technical output services available, and best practices and timelines for achieving these. B. Background Objectives and Intended Use of the CAF 2.0 Evaluation Part of what makes any tool effective is not just the design of the tool but how it is used in practice. USAID’s Conflict Assessment Framework was updated to version 2.0 in 2012. While there is general consensus within USAID/DCHA/CMM that the CAF 2.0 is a strong analytic tool for understanding the dynamics of conflict in the field, CMM is interested in better understanding and capturing how assessment teams are applying the framework and incorporating existing guidance for conducting a CAF in practice. For example, due to the variety of different needs and resource challenges faced by USAID Missions, it is not uncommon to tailor a CAF to specific regional and or thematic foci. CMM is interested in understanding what the optimum CAF models are, why and how they have been employed, in which scenarios they have been most successful and what methodologies are most effective when considering the various CAF models used to inform a variety of technical outputs. CAF APPLICATION MODELS In practice, Conflict Assessments may be tailored to meet particular Mission demands. Therefore, this evaluation will seek to understand the various ways or models of CAFs that have been implemented as CAF 2.0 EVALUATION | A-2 these were tailored to meet USAID Mission needs. Model types typically include a mix of parameters relating to: (a) Intensity/regional scope of the CAF - including: ● Macro Assessments - (whole of country) ● Sub-Regional Assessments ● Rapid Assessments ● Desk-Only Assessments ● Issue-Area Assessments (related to a specific issue, such as natural resource management and conflict, etc.) (b) Outputs and/or follow-on activities. For example, specific CAF objectives, end goals or outputs such as: ● Informing CDCS processes ● Informing new programing (Mission or CMM supported) ● Tracking changes in conflict dynamics to inform existing programming -- either for peacebuilding purposes or conflict sensitivity ● Providing additional understanding of cross-sectoral linkages that may have implications for the conflict dynamics or conflict sensitivity ● Creating general literacy within the Mission to inform thinking about conflict including heightening awareness of Mission staff to conflict dynamics and sensitivity ● Sensitizing Mission to relationships between stakeholders, helping decide who to engage with (or not), how to partner with government, civil society (CS) and others, and how to manage these relationships given conflict dynamics ● Informing contingency scenario planning and adaptive management processes ● Generating conversation and advocacy in Washington in support of Mission budget requests around priority conflict-related needs. This evaluation will recognize that model variations exist, and will disaggregate its analysis across these various CAF models. This will advance understanding of best practices for conducting each phase of a CAF, according to different CAF models (including both those currently being used and others that may be identified by the contractor). Evaluation Findings will serve a number of purposes. ● They will help inform decisions about when different models of CAFs might be most useful. ● They will provide technical guidance for assessment teams tailored to different CAF models and the outputs those models are intended to inform. ● They will help inform decisions at post and in CMM around ‘best practices’ regarding the timelines and staffing needed to conduct the various CAF models and the kinds of outputs and activities CAF recommendations are used to inform. ● They will help inform CMM’s understanding of how well data gathered from the collection phase of the CAFs is being fully mined and used to inform operations at the Mission level. C. The Evaluation Part I. Analysis 1. Lessons and Best Practices for various models of CAFs: A-3 | CAF 2.0 EVALUATION The CAF tool is flexible and able to respond to a variety of Mission needs. The first set of evaluation questions therefore will examine how the tool has been applied to date. What are the variations in the scope of inquiry that have been conducted -- or could be conducted -- and how have those variations supported specific types of outputs, such as those identified in the CAF APPLICATION MODELS (above). How can flexible application guidance be maintained to allow for additional unforeseen applications? Additional guiding questions: ● Scope and framing: ○ How have CAF teams adapted the framework to suit various contexts s (i.e. sub-regional, rapid, thematic focus)? ○ What were the successes and challenges of these adaptations and in which scenarios did these models work best? ● Programmatic/Operational: ○ As the CAF has been adapted to specific contexts and scopes, how have these models been used to inform and support follow-on services? ○ What are the range of outputs that have been successfully completed following a CAF? ○ What made these outputs achievable in some instances and not others (timelines, funding, staffing level and Mission capability to support follow-on activities)? ○ What current practices have not been successful in leading to follow-on services? ○ What were the challenges to some of these models? ○ What are additional models and outputs that could be considered? 2. Application Guidance vs. Practice: The Conflict Assessment Framework Application Guide provides practical guidance to assessment teams as they organize their activities across all phases of the CAF (1. Preparation 2. Orientation 3. Data Collection 4. Synthesis /Response 5. Reporting). CMM wants to better understand how closely this guidance is followed in practice (and why or why not), and whether variations in methodology -- including how resources (financial and human) are being applied and allocated across all CAF phases and models -- could be best fit to meet variations in Mission needs. Guiding questions: ● How do CAF teams follow the Conflict Assessment Framework Application Guide recommendations across all phases of the CAF: 1. Preparation 2. Orientation 3. Data Collection 4. Synthesis /Response 5 Reporting? How do CAF teams follow the Guide for all models of CAFs? What is working? What is not working and why not? ● Which practices best ensure that the upfront investments made in data collection result in the strongest analysis across the different CAF models? ● Which practices best ensure that the CAF process results in optimum follow-on activities? What are the enabling / inhibiting factors? ● How well do the timelines actually given to assessment teams to meet each of the five CAF phases seem to be serving the needs to make optimum utilization of the data that is captured given the different CAF models? ● How can CMM and Missions best balance timelines, staffing, and resource concerns while still achieving the best CAF outcome to meet their needs? ● Are there other considerations that should be incorporated into the application guide? CAF 2.0 EVALUATION | A-4 ● What data collection and data distribution sharing processes have been used and are most effective, realizing these may differ according to the scope of assessment being conducted? ● How can the findings from this evaluation better inform planning processes when thinking of assessments not as the end goal, but as the first step to informing a broader array of possible technical outputs? ● Are there any concepts central to the CAF that are causing challenges during the synthesis or recommendations process? ● How can CMM better advise USAID Missions and planners about expectations, timelines, and sequencing appropriate to each model? 3. Synthesis Processes / Synthesis to Response: Synthesis approaches help assessment teams bring together and cull meaning from the extensive set of data collected via field work. Synthesis approaches help CAF researchers shape data into meaningful information that USAID Missions can easily apply in their work (tailored to the kinds of outputs a given CAF is intended to inform). Current guidance provided in the Conflict Assessment Framework Application Guide includes discussion of both Headlines and Systems Mapping approaches to synthesis. Yet the way assessment teams actually perform synthesis varies in practice. In the past, CAF teams have used headlines, systems, and in some cases may have applied a blended approach or used other synthesis approaches. CMM is interested in understanding how past synthesis approaches have varied, how their use has affected analysis quality and types of recommendations generated, and how these methods contribute to utilization or uptake of recommendations. This should look at both the challenges and strengths of the approaches. Does one type of synthesis approach lend itself better to specific models of CAFs or particular CAF characteristics (i.e. size of team, complexity of data collection, etc.)? This information will help assessment teams determine which type of synthesis approach may be most appropriately suited to particular assessment goals. Guiding questions: ● What types of synthesis processes are teams most commonly using? What is working and what are the major challenges with the process and the resulting recommendations? ● Does one process provide richer synthesis? ○ In what cases and in what particular circumstances? ○ Do different synthesis processes yield different kinds of (or different qualities of) recommendations and levels of utilization of those recommendations by Missions? ○ When is it most appropriate to utilize a particular process based on team composition, complexity of CAF, size of CAF, flavor of CAF (i.e. intensity/regional scope, intended outputs and/or suites of follow-on activities) or other factors. ● What is missing from the approaches to synthesis that are being used? ● How do CAF teams feel the processes they followed (either systems mapping, headlines or a hybrid) worked? ○ Did the process allow for a ‘best case’ or even ‘good enough’ data capture, synthesis, and response outputs? ● How are other assessment frameworks doing synthesis? ○ What does the assessment literature have to offer about best practice for synthesis? ○ Are there alternative approaches CMM should be exploring? Part II: Dissemination A-5 | CAF 2.0 EVALUATION CMM has identified outreach -- both to Missions and in Washington -- as a priority action. Outreach (dissemination and discussion) is critical for ensuring CMM is better connected to meet the needs of those we are here to serve. Following CMM’s review of the completed final evaluation report, CMM will work with DI to fine tune an appropriate dissemination plan which will consist of the deliverables listed on p.26 of the TO. This includes providing technical briefers and a dissemination workshop. The workshop may include a mix of field and DC based audiences for the purpose of highlighting the models of ‘CAF to output’ packages Missions can choose, using information gleaned from part I. of this evaluation. Outputs may also include discussion of timelines and LOE requirements that Missions can expect from each undertaking. D. Methodology The recommended methodology may involve: 1. CMM will work with DI to identify 20 or fewer CAFs for selection into the evaluation study from the mid-2012 to mid-2018 time period. 2. Document review: Prior to arranging interviews, DI will conduct a desk study of a sample of identified CAFs together with corresponding Country Development Cooperation Strategies (CDCSs) and project appraisal documents (PADs, assuming our STTAs can sign NDAs), post￾CAF assessment reports, and post CAF Tech Team PowerPoint presentations from CMM staff where those assessments were conducted. CMM will provide all documentation under consideration for the review sample with any previously collected metadata pertaining to the documents prior to the start date of the study team. 3. Interviews: As appropriate, DI shall convene and interview a subset of informants to inform an interview protocol/s (to be approved by the TOCOR or Activity Manager). 4. Email Survey: DI may create an email survey(s) and conduct a global electronic survey of all CMM identified users involved in CAF assessments at Missions (CMM staff, local staff, and Foreign Service Officers who may now be in other posts or bureaus). Key informant interviews: In concert with associated USAID/ Washington offices, associated USAID Missions and key partners, DI shall identify key informants to interview for the evaluation corresponding to the completed assessment set. These interviews may include at minimum, a CMM POC, Mission POC, contracted assessment lead or co-lead, and several Mission staff and local staff who took part in the assessment. The contractor is responsible for safeguarding the rights and welfare of human subjects involved in research supported by USAID, and must comply with the Common Federal Policy for the Protection of Human Subjects as found in Part 225 of Title 22 of the Code of Federal Regulations (22 CFR 225). Additional guidance is available in the ADS 200 Mandatory Reference, “Protection of Human Subjects in Research Supported by USAID” (http://www.usaid.gov/sites/default/files/documents/1864/200mbe.pdf). The contractor will proposed a detailed methodology as part of the work plan. Work Plan and methodologies will address any potential conflicts of interests faced by the contracting company -- given their role in prior assessments -- and a plan for how these will be addressed. Reference AID-OAA-TO-16-00036 p 26 - 42. Period of performance: CAF 2.0 EVALUATION | A-6 Work will start in 2018. The final report and briefing presentations will end prior to the conclusion of the task order. Recommended timelines Technical Direction and Work Plan Timeline ● Technical Direction to DI (June 22) ● DI delivers evaluation plan (includes work plan, staffing plan, evaluation design, and literature review) to CMM in 6 weeks (August 3). Evaluation report due date is provided in the work plan. ● TOCOR reviews (gets stakeholder input and disseminates comments back to DI) ● DI - revised evaluation plan back to CMM in 10 days. Evaluation Report Timeline ● First draft of evaluation report to CMM (DI to propose) ● USAID Stakeholders review and to comment ● TOCOR sends comments to DI ● DI provides final report to CMM in ten days ● TOCOR must review and accept ● Evaluation data is submitted to the development library (see info). ● Periodic meetings and conference calls: As needed. Dissemination Timelines ● PEACE IDIQ Knowledge Management Meeting: - within 1 month of completion of final report. The purpose is to present the results/findings. TOCOR and DI collaborate in setting agenda. ● Roundtable Report - (max 10 pages) within 3 weeks of event. ● Technical Briefs: up to 2 may be required. (1-6 pages). ● Dissemination Workshop: for up to 100 people in DC area B-1 | CAF 2.0 EVALUATION ANNEX B: EVALUATION DESIGN MATRIX CAF Application Models Evaluation Question Sub-questions Type of question Methodological Approach to analyze data Data collection methodology and tools Sources 1. What models using the CAF methodology have been used in different conflict assessments? The question will look at both scope and objective of the assessments Descriptive question (describing the types of models that have been used in different Conflict Assessments) • Desk Review • Identification and classification of conflict assessment according to the model used • Desk assessments • Work Plans • MOUs with Missions • Conflict assessment reports • Feedback reports 1. What factors determine selection of a specific model? Descriptive/Explanatory question (requires explaining why CMM/USAID Mission selects one particular model and not others; whether they are flexible to changing the model on the ground as context changes) • Personal interviews • with USAID and CMM officials • USAID • CMM 2. What challenges or opportunities did teams face in implementing and adapting the designated model across all phases of CAF implementation? Was the implementation methodology adapted appropriately/sufficiently given the way the CAF was being tailored? Descriptive question Describing the challenges or opportunities encountered during field work; identification of strengths and weakness of the model selected; comparing across cases and looking for emerging patterns across cases; descriptive statistics. • Desk Review • Surveys to USAID • and CMM officials • Personal Interviews • with Team Leads, members of team (if available) and CMM officials participating in the team • Desk assessments • Conflict assessment reports • Work Plans • MOUs with Missions • Feedback reports • CMM • Team leaders CAF 2.0 EVALUATION | B-2 3. What are the consequences of the type of model selected for the quality and/or utility of the report? 4. Did the scoping of the CAF build off of prior USAID assessments or was it planned to feed into findings from another type of assessment? Descriptive/normative question Requires an assessment of whether a particular model was better utilized by the Mission; is there a relationship between the model and the utilization? Is there a relationship between the model, how it was implemented, and the quality of the report (as judged by the evaluators and CMM—to respond to the question of what works best) ● Desk Review ● Personal interviews with CMM and USAID officials participating in assessment teams ● Broad search on the DEC • Conflict assessment reports • Feedback reports • CMM • Work Plans • MOUs with Missions 5. What other models could be used? This is a recommendation that will be formulated as a result of the analysis and analysis of other models used in different types of assessments (if relevant) ● Desk Review ● Personal interviews with CMM and USAID officials participating in assessment teams ● Assessment reports ● Work Plans ● MOUs with Missions ● Methodologies for conducting assessments of youth, DRG, etc. ● Feedback reports ● USAID ● CMM Adherence to the Conflict Assessment Framework Methodology and Application Guide Evaluation Question Sub-questions Type of question Methodological Approach to analyze data Data collection methodology and tools Sources 2. To what extent do assessment teams follow the Conflict ** Why or why not? Normative question ● Desk Review ● Survey to CMM and USAID officials ● Conflict Assessment reports ● Feedback Reports ● Work Plans B-3 | CAF 2.0 EVALUATION Assessment Framework Application Guide recommendations across all phases of the CAF? (preparation, orientation, data collection, synthesis/ response; reporting?) Assessing to what extent the assessment teams adhered to the Framework Application Guide with a view to understanding what challenges there were to following the framework and/or why (in some cases) the implementation guidelines were not followed comparative analysis; content analysis and descriptive statistics. ● Personal Interviews with Team Leads, CMM officials who participated in assessment teams ● MOUs with Missions ● CMM ● Team Leaders 1. To what extent did Conflict Assessment Reports reflect the Methodological Framework? Normative Question Assessing to what extent assessment teams adhere to a norm or standard; evaluating the extent to which reports adhere to the framework—whether they identify drivers of conflict; key mobilizers; triggers and trends; and whether the recommendations include theories of change—as perceived by the team and reflected in the reports; descriptive statistics; comparative analysis and identification of trends/patterns ● Desk Review ● Survey to CMM and USAID officials ● Personal Interviews with Team Leads, CMM officials who participated in assessment teams ● Conflict Assessment reports ● Feedback Reports ● CMM ● Work Plans ● MOUs with Missions ● Team Leaders 2. What parts of the Framework are more challenging or difficult to follow or implement? Descriptive question Requires finding from teams what they found useful and what they found challenging; what concepts central to the CAF are more difficult to implement; descriptive statistics ● Personal interviews with CMM and team members ● Survey to USAID and CMM officials ● USAID ● CMM ● Team members CAF 2.0 EVALUATION | B-4 3. What are the most important operational issues (staffing, timeline, budget, training, etc.) CMM faces in conducting conflict assessments? Descriptive question Describes what different stakeholders involved in CAF assessments identify as the most important challenges ● Desk Review ● Personal Interviews with CMM and USAID officials ● Survey to USAID and CMM officials ● USAID ● CMM ● Feedback reports ● Work Plans ● MOUs with Missions 4. How well do the timelines given to assessment teams for each of the five CAF phases support optimum utilization of the data that is captured? Normative question Define the standard in terms of the quality of the data or quality of the assessment or utilization by the Mission. Explain whether timeline affects the quality of the assessment, either by curtailing lead/prep time ,data collection, synthesis or recommendation generating process Disaggregate by type of model used if possible ● Personal Interviews with CMM and USAID officials ● USAID ● CMM 5. What other considerations should be incorporated into the guide? 6. How can CMM do more to set assessment teams up for success? This is a Recommendation ● Survey to USAID and CMM officials participating in teams ● Personal Interviews with CMM and USAID officials and Team Leads ● USAID ● CMM ● Team leaders B-5 | CAF 2.0 EVALUATION Synthesis Approaches Evaluation Question Sub-questions Type of question Methodological Approach to analyze data Data collection methodology and tools Sources 3 What synthesis processes are teams most commonly using? (brainstorming, headlines, systems mapping, or hybrid) Descriptive question Describing what synthesis process teams used most frequently as reported in the assessment report or in interviews ● Desk Review ● Personal interviews with CMM officials who participated in the team and Team Leads − Conflict Assessment reports − Feedback reports − Team Leaders 1. What synthesis process is more useful or more appropriate in specific circumstances or models and why? Normative question What process yields better results; we need to define what an optimal result is (for example, size of the assessment team, utilization by Mission, analytical quality of the report in terms of depth of analysis; quality of data collected; easier to use, less cumbersome, practical) and compare findings against standard. Use of descriptive statistics; finding common trends among qualitative interviews ● Survey to USAID and CMM officials ● Personal interviews with team members and CMM officials participating in the assessment ● USAID ● CMM ● Team members 2. What is the perception of CAF teams regarding the synthesis process? Descriptive question Assessing perception of team members regarding the synthesis process: was it useful? Was it practical? Was it difficult to follow? What are the necessary skills ● Survey to USAID and CMM officials ● Personal interviews with team members and CMM officials participating in the assessment ● USAID ● CMM ● Team members CAF 2.0 EVALUATION | B-6 the team needs to have to use the synthesis process? Descriptive statistics; comparisons across qualitative interviews 3. What other synthesis process may be included/how do other assessment frameworks conduct synthesis? This is a Recommendation ● Personal interviews with team members and CMM officials participating in the assessment − USAID − CMM − Team members Mission Utilization of Assessment Reports (outputs) Evaluation Question Sub-questions Type of question Methodological Approach to analyze data Data collection methodology and tools Sources 4. To what extent have Missions utilized findings, conclusions and recommendations of Conflict Assessments in their programs and operations? Descriptive question Finding out what have Missions done with assessment recommendations. To what extent did they find the assessment useful, relevant, and timely? Have they used the assessment for their new CDCS? Projects? Activities? Have they inserted conflict dimensions into on￾going activities? Have they changed their activities or cancelled activities due to results of the assessment? Descriptive statistics, comparative analysis ● Survey of USAID and CMM officials ● Personal Interviews with USAID and CMM officials ● USAID ● CMM B-7 | CAF 2.0 EVALUATION 1. What made these outputs achievable in some instances and not others (timelines, funding, staffing level, Mission capability to support follow-on activities, follow on support from CMM (if appropriate))? Descriptive question What factors enable USAID to adopt the assessment conclusions and recommendations? How long does it take Missions to adopt recommendations? Comparative analysis; descriptive statistics ● Survey of USAID and CMM officials ● Personal Interviews with USAID and CMM officials ● USAID ● CMM 2. What were the challenges that prevented or limited the use of CAF findings to inform follow-on activities or outputs used by the Mission? Descriptive question; finding obstacles. Descriptive statistics and comparative analysis. ● Survey of USAID and CMM officials ● Personal Interviews with USAID and CMM officials ● USAID ● CMM 3. What could improve the extent to which Missions adopt recommendations? This is a Recommendation ● Personal Interviews with USAID and CMM officials ● USAID ● CMM C-1 | CAF 2.0 EVALUATION ANNEX C: LIST OF DOCUMENTS REVIEWED CAF COUNTRY LIST OF DOCUMENTS REVIEWED Cambodia • USAID/Cambodia Conflict Assessment Statement of Work, October 2012 • Cambodia Conflict Assessment debrief presentation, USAID/DCHA/CMM, Amex International, Inc., February 2013 • Cambodia CAF Recommendations, February 2013 • USAID/Cambodia Conflict Assessment Report, March 15, 2013 Georgia • USAID Georgia Conflict Assessment and Program Design TDY Scope of Work, September 5, 2014 • Georgia: Conflict Assessment Update, Debrief presentation, October 2014 • Situational Update on Georgia’s Conflicts, USAID Office of Conflict Management and Mitigation, November 2014 Guatemala • Guatemala Conflict Vulnerability Assessment: Draft Work Plan, April 2015 • Guatemala Conflict Vulnerability Assessment: Conflict Assessment Feedback Paper, September 2015 • Guatemala Conflict Vulnerability Assessment: Desk Study, June 2015 • Guatemala Conflict Vulnerability Assessment: In-brief Presentation, July 8 – July 23, 2015 • Guatemala Conflict Vulnerability Assessment: Debrief Presentation, July 8 – July 23, 2015 • Legacies of Exclusion: Social Conflict and Violence in Communities and Homes in Guatemala’s Western Highlands, Guatemala Conflict Assessment Final Report, Public Version, October 2015 • USAID Guatemala Conflict Assessment Country-Specific Technical Criteria, April 2015 Iraq • Iraq Program Vulnerability Assessment Statement of Work, USAID/Iraq • USAID/Iraq Program Vulnerability Assessment Fieldwork Paper • Iraq Vulnerability Roundtable: Response Factors Presentation • Iraq Vulnerability Roundtable: Response Factors Presentation, Part 2 • Iraq Program Vulnerability Analysis – Summary of Issues Paper In-brief Presentation • Iraq Program Vulnerability Assessment: Issues Paper (Desk Review) Final Draft, January 2012 • USAID/Iraq Program Vulnerability Assessment Final Report, May 2012 Kenya • Kenya Conflict Vulnerability Assessment: Conflict Assessment Feedback Paper, October 2016 • Kenya USG Program Review Assessment: Literature Review, April 2016 • Kenya USG Strategy Review Assessment: Work Plan, March 2016 • Kenya Conflict Vulnerability Assessment 2016 Country-Specific Technical Directions • Kenya Conflict Vulnerability Assessment Debrief Presentation, May 2016 • Kenya Conflict Vulnerability Assessment: Internal Paper, October 2016 CAF 2.0 EVALUATION | C-2 Kosovo • Kosovo Conflict Assessment: Breaking Down Invisible Walls, debrief Presentation, August 9, 2012 • Kosovo Conflict Assessment: Breaking Down Invisible Walls Report, November 2012 • Statement of Work: Kosovo Conflict Assessment, July 9, 2012 • Kosovo Conflict Assessment Workplan Liberia • Liberia Conflict Vulnerability Assessment: Feedback Paper, May 2016 • Liberia Conflict Vulnerability Assessment: Revised Desk Review Report, February 2016 • Liberia Conflict Vulnerability Assessment: Revised Work Plan, December 2015 • Liberia Conflict Vulnerability Assessment Debrief Presentation, March 11, 2016 • Liberia Conflict Vulnerability Assessment: Final Report, May 2016 • USAID/Liberia Conflict Vulnerability Assessment, Country-Specific Technical Directions, January – April 2016 • New Deal for Engagement in Fragile States Presentation slides, February 2016 Morocco • Morocco Development Context Assessment: Revised Desk Study, March 2017 • Morocco Development Context Assessment: Feedback Paper, August 2018 • Morocco Development Context Assessment: Executive Summary • Morocco Development Context Assessment: Final Report, December 2017 • USAID Morocco Development Context Assessment, Country-Specific Technical Direction • USAID/Morocco: Development Context Assessment Out-brief • Morocco Development Context Assessment: Draft Field Work Plan, March 2017 Niger • Niger Conflict/Crime Assessment Scope of Work • Niger: Conflict and Crime Assessment Executive Summary, July 14, 2014 • Niger Conflict – Crime Assessment orientation Workshop Presentation, November 11, 2013 Nigeria • Cross-Sectoral Conflict Assessment, Description of Statement of Work • Cross-Sector Conflict Assessment: Issues Paper, USAID/Nigeria, August 2014 • Cross-Sector Conflict Assessment: Field Work Plan, USAID/Nigeria, June 2014 • Nigeria Cross-Sectoral Conflict Assessment Out-brief Presentation, USAID/Nigeria, July 1, 2014 • Nigeria Cross-Sectoral Conflict Assessment: Final Report, August 2014 • Nigeria Cross-Sectoral Conflict Assessment: CAF 2.0 Feedback Paper, September 2014 Senegal • Senegal Conflict Vulnerability Assessment: Final Report, December 2017 C-3 | CAF 2.0 EVALUATION • Senegal Conflict Vulnerability Assessment: Conflict Assessment Feedback Paper, June 2017 • Senegal Conflict Vulnerability Assessment: Desk Review, March 2017 • Senegal Conflict Vulnerability Assessment: Work Plan, March 2017 • Senegal: Conflict Vulnerability Assessment, Mission Debrief Presentation, March 20 – April 7, 2017 • Section C: Description/Specifications/Statement of Work South Sudan • Agenda: Interview Protocol Development and Research Questions • USAID/South Sudan Conflict Assessment Final Report, February 5, 2013 • Carrie A. Gruenloh, Southern Sudan Conflict Meta-Analysis, Bureau for Democracy, Conflict, and Humanitarian Assistance Office of Conflict Management and Mitigation, December 2010 • South Sudan: Conflict Assessment Analysis, Synthesis, and Presentation of Findings, Schedule Southern Thailand • Southern Thailand Stocktaking and Monitoring Trip Debrief Presentation, September -October 2013 • 2013 Southern Thailand Stocktaking, USAID/Regional Development Mission Asia & USAID/CMM Sri Lanka • Sri Lanka Strategic Assessment Presentation, July – August 2013 • US Embassy Strategic Assessment Report, September 2013 Tanzania • Tanzania Peace and Stability Assessment: Field Work Plan, January 2015 • Tanzania Peace and Stability Assessment: Desk Study, December 2014 • Tanzania Peace and Stability Assessment: Conflict Assessment Feedback Paper, April 2015 • Tanzania Peace and Stability Assessment: In-brief Presentation, January 7, 2015 • Tanzania Peace and Stability Assessment: Out-brief Presentation, January 27, 2015 • Tanzania Peace and Stability Assessment: Final External Report, April 2015 • USAID/Tanzania Peace and Stability Assessment, Statement of Work Uganda • Uganda Conflict Assessment Update: Literature Review, 2015 (For internal use only) • Uganda Conflict Assessment Update: Literature Review, 2015, External • USAID/Uganda Conflict Assessment Update, Mission Out-brief, 2015 • Uganda Conflict Assessment: Report, 2015 Zimbabwe • Zimbabwe: Conflict Vulnerability Assessment: Conflict Assessment Feedback Paper, November 2015 • Zimbabwe: Conflict Vulnerability Assessment: Field Work Plan, July 2015 • Zimbabwe: Conflict Vulnerability Assessment In-Brief Presentation, August 24 – September 10, 2015 • Zimbabwe: Conflict Vulnerability Assessment Out-Brief Presentation, September 9, 2015 • Zimbabwe: Resilience and the Politics of Despair Zimbabwe Conflict Vulnerability Assessment Draft Report, October 2015 • Section C - Statement of Work D-1 | CAF 2.0 EVALUATION ANNEX D: DATA COLLECTION TOOLS SURVEY QUESTIONNAIRE Introductory Letter: Dear Survey Participant, USAID/CMM has commissioned Democracy International to conduct an evaluation of the Conflict Assessment Framework 2.0 (CAF) methodology application since 2012. CMM has provided your name as a representative from CMM, USAID, or an assessment team who participated in COUNTRY NAME Conflict Assessment. The data collection will help CMM understand the utility of different CAF models and the most effective strategies for applying the methodology to generate relevant outputs for Missions. Please take 30 minutes to complete this survey by Friday, January 25, 2019. Using this survey, the evaluation team will gather perceptions on the application of different conflict assessment models, the adherence to the CAF process as established in the Application Guide, including synthesis, the appropriateness of work processes and timelines, the relevance of the assessment findings to desired Mission outputs, and the Mission’s utilization of the assessment’s conclusions and recommendations. All responses are confidential and will not be shared beyond the evaluation team. There will be no individual attribution. Information will be aggregated and used to identify trends for presentation in the evaluation report. Since each assessment experience is unique, if you have participated in more than one conflict assessment, kindly complete one survey for each assessment you participated in. Thank you very much for your cooperation and taking the time to complete the survey. If you have any questions about the survey, please contact Abigail Cohen, acohen@democracyinternational.com. Sincerely, Democracy International CAF 2.0 EVALUATION | D-2 1. Please identify your gender: 1) Female 2) Male 2. Please identify the country where the conflict assessment was conducted (open field). 3. Please identify the year the conflict assessment was conducted (open field). 4. How did you engage with the conflict assessment? 1) Mission point of contact 2) Report Reviewer 3) Team member o Team Lead o Local Conflict Specialist o Mission Staff o CMM Staff o Other USAID o Other agency 4) User of Assessment Report 5) Other Please specify (open field) 5. What type (model) of conflict assessment was this? 1) Whole country 2) Regional assessment (within country) 3) Desk only assessment 4) Issue area assessment (sector, theme or event driven) Please specify issue (open field) 5) Multi-country—regional assessment 6. Was the conflict assessment conducted in collaboration with another USG Agency or another donor? 1) Yes 2) No If Yes, please specify which one (open field). 7. During planning or the assessment itself, was the CAF methodology or process adapted to accommodate the model (see question 5 above). 1) Yes 2) No If Yes, what were the adaptations? (open field) 8. Do you believe those adaptations improved the assessment? 1) Yes D-3 | CAF 2.0 EVALUATION 2) No If Yes, why? (open field) If No, why not? (open field) 9. Did the team make any of the following unplanned adjustments while collecting data in the field (select all that apply): 1) Scheduling new interviews 2) Traveling to new locations 3) Scheduling unplanned regrouping opportunities among team members 4) Other adjustments to pivot or accommodate new information gained in the field. Please specify (open field) 5) No, we did not have the flexibility to make unplanned adjustments. Please explain why not (open field) 10. If you selected any of the responses for the previous question, do you consider these unplanned adjustments useful? 1) Yes 2) No If yes, why? (open field) If no, why not? (open field) 11. What were the most significant operational challenges the team faced conducting the assessment? Check all that apply: 1) Budget 2) Timeline 3) Training 4) Staffing 5) Logistics 6) Other, please specify (open field) 12. Which operational aspects worked well during the assessment? Check all that apply: 1) Budget 2) Timeline 3) Training 4) Staffing 5) Logistics 6) Other, please specify (open field) 13. What was the objective of the Conflict Assessment? Check all that apply: 1) Informing CDCS process 2) Informing new programming (Missions or CMM supported) 3) Tracking changes in conflict dynamics to inform existing programming 4) Providing additional understanding of cross sectoral linkages that may have implications for conflict dynamics or conflict sensitivities CAF 2.0 EVALUATION | D-4 5) Creating general literacy within the Mission to inform thinking about conflict and conflict sensitive programming 6) Sensitizing the Mission to relationships between stakeholders and how to manage these relationships given conflict dynamics 7) Informing contingency scenario planning and adaptive management processes 8) Generating conversation and advocacy in Washington and supporting Mission budget requests around priority conflict related needs 9) Do not know 10)Other, please specify (open field) 14. On a 1 to 5 scale, where 1 is not at all useful and 5 is very useful, how useful do you think the assessment report was for: 1) Informing the understanding of the conflict dynamics in the targeted areas 1) Not at all useful 2) A little useful 3) Somewhat useful 4) Useful 5) Very useful 6) Do not know 2) Identifying useful and relevant programmatic recommendations to respond to conflict dynamics 1) Not at all useful 2) A little useful 3) Somewhat useful 4) Useful 5) Very useful 6) Do not know 3) Identifying appropriate policy recommendations 1) Not at all useful 2) A little useful 3) Somewhat useful 4) Useful 5) Very useful 6) Do not know 7) Not applicable 4) Providing information relevant to the intended purpose of the CAF (see question 13 above) 1) Not at all useful 2) A little useful 3) Somewhat useful D-5 | CAF 2.0 EVALUATION 4) Useful 5) Very useful 6) Do not know 15. Please select one of the options provided below for the following statement: Recommendations from the CAF Assessment were implemented. 1) Strongly disagree 2) Disagree 3) Neither agree nor disagree 4) Agree 5) Strongly agree 6) Do not know 16. What factors contributed to the Mission utilizing the recommendations? Please check all that apply. 1) Timing of the assessment 2) Quality and relevance of the recommendations 3) Ability to leverage / adjust portfolio 4) New budget / funding opportunities 5) Changes in country context 6) Commitment / buy-in of Mission leadership 7) Commitment / buy-in of Mission Staff 8) Availability of follow on support from CMM/other offices if needed. 9) Other, please specify (open field) 17. What factors contributed to the Mission not adopting the recommendations? Please check all that apply: 1) Timing of the assessment 2) Quality and relevance of the recommendations 3) Changes in portfolio 4) Budget constraints 5) Changes in country context 6) Changes in Mission leadership 7) Changes in Mission Staff 8) Lack of buy-in from Mission leadership 9) Lack of buy-in from Mission staff 10)Availability of follow on support from CMM/other offices if needed. 11)Other, please specify (open field) 18. Did the Mission request follow-on services from CMM? 1) Yes 2) No 3) Do not know 19. If yes, which services did they request? Check all that apply: CAF 2.0 EVALUATION | D-6 1) Design a new project/activity 2) Training for Mission staff 3) Adaptation of Mission systems or process to engender conflict sensitivity 4) Other, please specify (open field) 20. If you selected any of the responses for the previous question, did CMM provide the requested services? 1) Yes 2) No 3) Do not know If no, please describe briefly why not (open field) ONLY FOR THOSE WHO PARTICIPATED IN THE ASSESSMENT TEAM: 21. Which aspects of the assessment process did you undertake? Check all that apply: 1) Preparation (Desk Review, Desk Study, Interviews) 2) Team Orientation 3) Field work 4) Synthesis and Response 5) Report Writing 22. What were the biggest methodological challenges inherent in following the framework? Check all that apply: 1) Identifying grievances 2) Identifying resiliencies 3) Identifying key mobilizers 4) Identifying trajectories (trends and triggers) 5) Identifying theories of change 6) Application of programmatic filters (criteria of effectiveness, RPP Matrix, bright spots) 7) Developing recommendations 23. Why was this challenging? Check all that apply: 1) Data was not available 2) Did not understand the concepts 3) Lack of team capacity 4) Insufficient time 5) Other Please specify (open field) 24. What aspects of the CAF methodology and process were the most challenging to adequately address in the timeline allotted for the assessment? Check all that apply: 1) Time available to prepare logistics and security D-7 | CAF 2.0 EVALUATION 2) Time available for data collection (desk review) prior to departure 3) Time available for data collection (in country) 4) Time available for synthesis (in country) 5) Time available for recommendation generation (in country) 6) Time available for report writing process 7) Time available for follow on actions 8) None, there was sufficient time allocated for all aspects of the assessment 9) Other Please specify (open field) 25. What aspects of the CAF methodology and process had sufficient time allocated? Check all that apply: 1) Time available to prepare logistics and security 2) Time available for data collection (desk review) prior to departure 3) Time available for data collection (in country) 4) Time available for synthesis (in country) 5) Time available for recommendation generation (in country) 6) Time available for report writing process 7) Time available for follow on actions 8) None, there was sufficient time allocated for all aspects of the assessment 9) Other Please specify (open field) 26. Are there any changes or adaptations that you would recommend to improve the utility of the CAF framework methodology? 1) Yes 2) No If yes, please describe briefly what changes or adaptations you recommend. (open field) 27. What synthesis process did your team use for analyzing the data collected during field work? 1) Systems mapping 2) Headlines 3) Hybrid of headlines and systems 4) Brainstorming 5) Other Please specify (open field) 6) Don’t know 28. On a scale of 1 to 5, where 1 is not at all effective and 5 is very effective, how would you rate the effectiveness of the synthesis process used by your team for identifying conflict dynamics? 1) Not at all effective 2) Somewhat effective 3) Neither effective nor ineffective CAF 2.0 EVALUATION | D-8 4) Effective 5) Very effective 6) Synthesis process was not used Please briefly explain your answer (open field) 29. Would you recommend any other method or approach for the synthesis process? Brief description (open field) 30. On a scale of 1 to 5, where 1 is not at all effective and 5 is very effective, how effective was the team in developing recommendations that were both linked to the conflict dynamics, the Mission portfolio, and assessment objective (CDCS, issue, project development, etc.)? 1) Not at all effective 2) Somewhat effective 3) Neither effective nor ineffective 4) Effective 5) Very effective Please explain your answer (open field) 31. Is there anything else that you would like to communicate to the evaluation team that could help make the CAF a more useful tool and or help to improve assessment methodologies? (open field) Thank you for your time. D-9 | CAF 2.0 EVALUATION CONFLICT ASSESSMENT TEAM MEMBER PROTOCOL Thank you for your willingness to participate in the evaluation of the Conflict Assessment Framework 2.0 (CAF 2.0). Evaluation findings and recommendations will inform CMM efforts to continuously adapt and improve the CAF methodology and processes to generate usable recommendations that add value to Mission strategies and programs. Key informant interviews are an important part of our data collection process. Your candid input will be valuable in informing the evaluation findings. Please be assured that all discussions remain confidential. The following is a general outline to guide our discussion. At the end, you are welcome to convey any additional information or reflections that may not have been captured during the course of the discussion. Note to interviewer: Before moving into the questions, confirm / elaborate on the following based on information that we have already collected: • Conflict assessments that are being discussed, including country(ies) and year(s); • The informant’s role(s) in the conflict assessment(s) Model and Objectives: Note to interviewer: Before moving into the questions, ask the information to clarify and elaborate on the assessment model (whole country, regional within country, thematic, etc.) and objectives (inform programming, CDCS, etc.) Example: the Kenya CDCS used a thematic model focused on risk of electoral violence. The team collected data in regions of the country perceived to be potential hotspots based on past experience. The objective of the assessment was to inform conflict sensitive Mission programming particularly around upcoming elections. 1. Why was this model selected for the conflict assessment? What were the opportunities and challenges that you encountered using this model? 2. From your perspective, was the model appropriate for meeting the assessment objective? Why or why not? In retrospect, were there other models that could have been used to meet the assessment objective? How did the objective impact the data collection process and recommendations? Was there a relationship between the model selected and the utilization of recommendations by the Mission? Process: Note to interviewer: Before moving into the questions, make sure that informant understands the distinction between processes linked to the Application Guide (Preparation, Team Orientation, Field work, Synthesis and Response, and Report Writing) versus methodology (CAF 2.0 – addressed below). 3. In terms of the assessment process, from preparation to final submission of the report, what was the most difficult or challenging part of conducting the assessment? Why? Which parts of the process worked well? Why? In retrospect, what would you have done differently or kept the same? CAF 2.0 EVALUATION | D-10 4. Do you believe the team had sufficient time, resources, and logistical support to conduct the assessment? What could have been done differently to improve? What should stay the same for future assessments? CAF 2.0 Methodology: Note to interviewer: • If necessary, refresh on elements of the CAF 2.0 methodology (grievances, resiliencies, key mobilizers, trajectories, theories of change, application of programmatic and operational filters, and recommendations) • If interviewing a CMM person, probe more deeply into challenges and opportunities of CAF methodology. 5. Which aspects of the CAF 2.0 methodology were most / least useful for understanding conflict dynamics and articulating practical recommendations for the Mission? Why? How did this impact the team’s work? Do you have any suggestions or reflections relating to the CAF 2.0 methodology? Synthesis: Note to interviewer: • If necessary, refresh on timing of synthesis and different synthesis approaches (brainstorming, headlines, systems mapping, hybrid approaches) • If interviewing a CMM person, probe more deeply into challenges and opportunities of different synthesis approaches. 6. Which synthesis process did your team use for analyzing the data collected during field work? From your perspective, was the process effective for synthesizing data and informing recommendations? Why or why not? Could the synthesis process be improved? Are there other synthesis approaches that could be applied? Utilization by Mission: Note to interviewer: If the informant is a Mission-based team member who may also have insight into details of utilization, move to supplemental questions below. 7. From your perspective, was the assessment received well by Mission staff and leadership? Are you aware of any recommendations that were implemented or other follow-up actions on the part of the Mission? What was the driving force for or against implementation of recommendations? Other: 8. Reflecting on your participation in the conflict assessment -, is there anything else that you would like to share with the evaluation team? Thank you. D-11 | CAF 2.0 EVALUATION Utilization by Mission – Supplemental questions for team members who also have details of utilization: 9. Was the assessment well received by Mission (and Embassy if appropriate) staff and leadership? Were the recommendations appropriate and usable in relation to the assessment objectives and intended use? Why or why not? 10. Are you aware of any recommendations that were implemented or other follow-up actions on the part of the Mission (and broader Embassy if appropriate)? What was the driving force for or against implementation of recommendations? 11. To what extent has the Mission (and broader Embassy if appropriate) used the assessment findings to adapt their operations, change an existing activity or design a new one? How long has it taken the Mission to make the necessary adaptations? 12. From your perspective, what could improve the utility of conflict assessments for Missions (and broader Embassy if appropriate)? 13. If the Mission requested follow-on services from CMM, did this additional support help the Mission (and broader Embassy if appropriate) to accomplish their objectives? Why or why not? Are there other support services that CMM could offer that would help Missions implement conflict assessment recommendations and advance their conflict sensitivity objectives? CAF 2.0 EVALUATION | D-12 CONFLICT ASSESSMENT MISSION ENGAGEMENT (NON-TEAM MEMBER) PROTOCOL Thank you for your willingness to participate in the evaluation of the Conflict Assessment Framework 2.0 (CAF 2.0). Evaluation findings and recommendations will inform CMM efforts to continuously adapt and improve the CAF methodology and processes to generate usable recommendations that add value to Mission strategies and programs. Key informant interviews are an important part of our data collection process. Your candid input will be valuable in informing the evaluation findings. Please be assured that all discussions remain confidential. The following is a general outline to guide our discussion. At the end, you are welcome to convey any additional information or reflections that may not have been captured during the course of the discussion. Note to interviewer: Before moving into the questions, confirm / elaborate on the following based on information that we have already collected: • Conflict assessments that are being discussed, including country(ies) and year(s); • The informant’s relationship with the conflict assessment(s) Model and Objectives: Note to interviewer: Before moving into the questions, ask the informant if they are aware of the assessment model (whole country, regional within country, thematic, etc.) and objectives (inform programming, CDCS, etc.) • Example: the Kenya CDCS used a thematic model focused on risk of electoral violence. The team collected data in regions of the country perceived to be potential hotspots based on past experience. The objective of the assessment was to inform conflict sensitive Mission programming particularly around upcoming elections. If the informant does not have this information, provide them with the model and objective as stated in the desk review documents. 14. Do you know why this particular model was selected? From your perspective, was the model used appropriate for meeting the assessment objective? Why or why not? How did this impact the recommendations? In retrospect, were there other models that could have been used to meet the assessment objective? Was there a relationship between the model selected and the utilization of recommendations by the Mission? Process: 15. To what extent were you involved in the preparation, implementation, and review of the conflict assessment? Were there aspects of the preparation, implementation, and review process that worked well? Aspects that could have been improved? Utilization by Mission: 16. Was the assessment well received by Mission (and Embassy if appropriate) staff and leadership? Were the recommendations appropriate and usable in relation to the assessment objectives and intended use? Why or why not? D-13 | CAF 2.0 EVALUATION 17. Are you aware of any recommendations that were implemented or other follow-up actions on the part of the Mission (and broader Embassy if appropriate)? What was the driving force for or against implementation of recommendations? 18. To what extent has the Mission (and broader Embassy if appropriate) used the assessment findings to adapt their operations, change an existing activity or design a new one? How long has it taken the Mission to make the necessary adaptations? 19. From your perspective, what could improve the utility of conflict assessments for Missions (and broader Embassy if appropriate)? 20. If the Mission requested follow-on services from CMM, did this additional support help the Mission (and broader Embassy if appropriate) to accomplish their objectives? Why or why not? Are there other support services that CMM could offer that would help Missions implement conflict assessment recommendations and advance their conflict sensitivity objectives? Other: 21. Reflecting on the conflict assessment, is there anything else that you would like to share with the evaluation team? Thank you. E-1 | CAF 2.0 EVALUATION ANNEX E: LIST OF STAKEHOLDERS CONSULTED CAF COUNTRY STAKEHOLDER ROLE Cambodia End user Team member Georgia End user and Team member Team member Guatemala End user Team member Team member Iraq End User Team Member Team Member Kenya Team member(s) and end user Team member/end user Kosovo Team member Team member/end user Liberia End user Team members Team member Morocco End user Team members/end users Team members Team members Niger End user Nigeria Team members End user Team member Senegal End users/Team members Team member Team member South Sudan End user Team members Southern Thailand End user Team members Sri Lanka Team member Tanzania End user Team members Uganda Team members Zimbabwe Team members F-1 | CAF 2.0 EVALUATION ANNEX F: CONFLICT OF INTEREST FORMS CAF 2.0 EVALUATION | F-2 F-3 | CAF 2.0 EVALUATION