This publication was produced for review by the United States Agency for International Development (USAID). It was prepared by EnCompass LLC and Social Impact for the Mozambique Final Program Evaluation (Contract No. AID-OAA￾I-15-00021/Task Order No. 7200AA19F00017). The views expressed in this publication do not necessarily reflect the views of USAID or the United States Government. MOZAMBIQUE FINAL PROGRAM EVALUATION Evaluation Report December 2021 PHOTO CREDIT: OTI TABLE OF CONTENTS Executive Summary ............................................................................................................................................................ i Background...................................................................................................................................................................... i Findings............................................................................................................................................................................. i Recommendations ....................................................................................................................................................... iii Community Targeting ................................................................................................................................................. iii Beneficiary Targeting................................................................................................................................................... iii Sequencing and Layering of Programming.............................................................................................................. iii Programmatic Recommendations............................................................................................................................ iii Evaluation Background .....................................................................................................................................................1 Purpose............................................................................................................................................................................1 Tuko Pamoja Program Overview..............................................................................................................................1 Evaluation Questions....................................................................................................................................................1 Evaluation Design...............................................................................................................................................................2 Methodology ..................................................................................................................................................................2 EQ1 - What are the primary outcomes of programming under our PO1 youth objective?...........................2 SO 1.1 “Youth leaders have increased capacity to promote VE resilience among their peers” ...............2 SO 1.2 “Youth’s feelings of economic marginalization is reduced” ..................................................................6 PO 1 “Vulnerable youth resist VE influence more effectively”..........................................................................7 Unintended Consequences.........................................................................................................................................8 Conclusions..................................................................................................................................................................10 EQ2 - What impact did TP sequencing and layering have within geographic zones? .....................................11 Sequencing with the Same Communities..............................................................................................................11 Sequencing with the Same Individuals....................................................................................................................12 Cost/Benefit Considerations for Activity Sequencing/Layering .......................................................................12 Conclusions..................................................................................................................................................................12 EQ3 - To what extent do the evaluation findings at SO levels 1.1 & 1.2 support the respective hypotheses (ToC)? ................................................................................................................................................................................13 Respondent perceptions of the project logic.......................................................................................................13 SO 1.1 “Youth leaders have increased capacity to promote VE resilience among their peers” .............14 SO 1.2 “Youth’s feelings of economic marginalization is reduced” ................................................................14 Synergies between SO1.1 and SO1.2.....................................................................................................................14 Conclusions..................................................................................................................................................................14 EQ4 - Based on program learning, what changes would you recommend to make the ToC more accurately address the problem set?...............................................................................................................................................15 Recommendations...........................................................................................................................................................16 Community Targeting ................................................................................................................................................16 Beneficiary Targeting..................................................................................................................................................16 Sequencing and Layering of Programming.............................................................................................................16 Programmatic Recommendations...........................................................................................................................17 Annex I: Model ToC .......................................................................................................................................................18 Annex II: Activity Sampling............................................................................................................................................19 Annex III: Data Collection Approches Quality Assurance PROCESSES AND Limitations...........................20 Annex IV: Beneficiary Quotes on Reduced VE Recruitment................................................................................23 Annex V: Disagreement with TP TOC Assumptions, by Respondent Type and Strategic Objective ........25 Annex VI: Scope of Work .............................................................................................................................................26 Annex VII: Data Collection Tools...............................................................................................................................34 ACRONYMS COVID-19 Coronavirus Disease 2019 DAI Development Alternatives Inc. DCF Data Collection Firm FGD Focus Group Discussion IDP Internally Displaced Persons IRB Institutional Review Board ISIS-M Islamic State of Iraq and Syria-Mozambique KII Key Informant Interview MEL Monitoring, Evaluation, and Learning NAM Nampula NGO Non-governmental Organization OTI Office of Transition Initiatives P/CVE Preventing/Countering Violent Extremism PEM Pemba PO RFP Program Objective Request for Proposals QA Quality Assurance SI SO Social Impact Sub-Objective ToC Theory of Change TP Tuko Pamoja TVET USAID Technical and Vocational Education and Training United States Agency for International Development VE Violent Extremism USAID.GOV Mozambique Final Program Evaluation Report | i EXECUTIVE SUMMARY BACKGROUND Tuko Pamoja (TP) is a $16.3 million, 27-month program implemented by Development Alternatives Inc. (DAI) and funded by United States Agency for International Development’s (USAID) Office of Transition Initiatives (OTI). Following a strategy update in January 2021, Tuko Pamoja (TP) sought to help vulnerable youth resist violent extremist (VE) influence through the two strategic objectives (SOs). SO1.1 sought to capacitate youth leaders to promote violent extremism (VE) resilience among peers, and SO1.2 sought to reduce youth’s feelings of economic marginalization. TP was implemented through a series of 83 grants disbursed to local organizations. In June 2021, OTI contracted Encompass and its sub-contractor Social Impact (SI) to conduct an external evaluation of TP. The evaluation was designed to answer four evaluation questions (EQs) (detailed in the Evaluation Background section). Given the scale and heterogeneity of programming, the evaluation focused on a sub-portfolio of 24 activities covering two of the three geographic zones. Working with local data collection subcontractor Forcier, the evaluation team interviewed 158 respondents through 59 key informant interviews and 19 focus group discussions. Data was collected from USAID, TP, grantee, local government, beneficiary, and community member respondents. The primary limitations of the evaluation included challenges in the implementation of field interviews and selection bias stemming from a lack of beneficiary contact information and differential refusals (see Evaluation Methodology section). FINDINGS EQ 1 - WHAT ARE THE PRIMARY OUTCOMES OF PROGRAMMING UNDER THE PO1 YOUTH OBJECTIVE? ● Three in five SO1.1 beneficiary respondents cited evidence of increased youth leadership, including “opening up” of minds, increased self-efficacy, the creation of peer networks, increased engagement with community and local government leaders, and improved status of youth. ● Among SO1.1 activities that supported broader community objectives, outcomes included de￾stigmatization of VE discussion, improved community awareness of VE (e.g., correction of misperceptions about the drivers of the insurgency, knowledge of VE recruitment tactics), increased community vigilance regarding outsiders, and increased community cohesion. ● To varying degrees, SO1.2 activities led to improved acquisition of economic skills and transfer of economic assets. These investments led to noticeable economic gains among beneficiaries of intensive programming, with some respondents reporting substantial quality of life improvements. However, less intensive activities had mixed success at generating economic opportunities. In response to increased awareness of economic opportunities and/or access to opportunities facilitated by TP, one in four SO1.2 beneficiary respondents cited evidence of reduced feelings of economic marginalization, including increased confidence, ability to generate income for their family, community pride, and status. ● One in three TP beneficiary respondents provided evidence of having reduced vulnerability to VE stemming from their participation in TP activities. This was driven by improved community and individual understanding of vulnerability, increased community vigilance toward outsiders, increased knowledge about reporting mechanisms, and anecdotal references to reduced VE recruitment. ● The evaluation found evidence that TP increased tensions and feelings of marginalization in isolated instances. At the community level, this included feelings of exclusion from omission from programming and government favoritism in the allocation of some TP benefits, as well as possible consequences of ii | Mozambique Final Program Evaluation Report USAID.GOV increased community vigilance toward outsiders. At the individual level, this included youth being omitted from receiving support due to contested eligibility criteria, unfulfilled expectations among some SO1.2 participants, and isolated cases of conflict over the inadequacy of assets disbursed by TP. EQ2 - WHAT IMPACT DID TP SEQUENCING & LAYERING HAVE WITHIN GEOGRAPHIC ZONES? ● The evaluation found a strong positive correlation between the number of activities implemented in a community and the prevalence of key positive TP outcomes. Furthermore, in instances where TP provided support to the same individuals across multiple activities, the evaluation found evidence that, on average, sequenced programming led to more significant knowledge and behavioral changes. Successful programming at the community level was driven by (i) implementation of multiple, complementary activities, (ii) large investments, and (iii) low baseline level of community economic development. ● The success of the sequencing/layering approach must be assessed in conjunction with its costs. In some cases, TP invested upwards of ~$10,000 per beneficiary. This level of investment limits the number of beneficiaries that can be supported and, while in certain instances TP activities had substantial multiplier effects, in others, the benefits seem to have accrued more directly to individuals. EQ3 - TO WHAT EXTENT DO EVALUATION FINDINGS AT SO LEVELS 1.1 & 1.2 SUPPORT THE RESPECTIVE HYPOTHESES? ● There was substantial agreement among the respondents on most assumptions underpinning TP’s theory of change (ToC). Respondents were most likely to disagree that youth lacked skills/knowledge to resist VE (citing the primacy of economic determinants), that there are legitimate economic opportunities available to youth (citing barriers to employment), and that youth are influential in their communities (citing youth influence with peers but not the broader community). ● The SO1.1 logic was largely validated. Most youth beneficiaries increased their VE knowledge, many gained leadership skills, and some promoted VE resilience among peers. The evaluation found a few anecdotal examples of youth leaders claiming to sway peers from (re)joining the insurgency. ● The SO1.2 logic was also largely validated, with certain activities leading to economic outcomes and subsequently reduced perceptions of marginalization. These cases were concentrated among more intensive programming (i.e., large per beneficiary investments) and in areas with lower levels of economic development. While some beneficiaries of smaller activities did register improved economic and marginalization outcomes, beneficiaries that were not able to benefit economically were much less likely to cite reduced marginalization (in fact as noted in EQ1, marginalization for some of these beneficiaries could have increased). EQ4 - BASED ON PROGRAM LEARNING, WHAT CHANGES WOULD YOU RECOMMEND TO MAKE THE TOC MORE ACCURATELY ADDRESS THE PROBLEM SET? ● There was very little agreement among USAID and TP respondents on how best to revise the TP ToC to achieve P/CVE objectives. Conversely, there was significant agreement among beneficiaries, with requests for economic support (training and asset provision) most common. ● We provide our recommendations on how to revise the ToC and programmatic approach below. USAID.GOV Mozambique Final Program Evaluation Report | iii RECOMMENDATIONS COMMUNITY TARGETING Vulnerability and resilience factors are individualized, products of interactions between complex systems (e.g., psychological, interpersonal, socio-economic), and clustered at the community level. To leverage shared resilience factors and target localized vulnerabilities, SI recommends that USAID focus programming at the community level. We present a model ToC for such a strategy in Annex I. Develop a community targeting approach that considers strategic objectives and associated tradeoffs. There are fundamental differences in programming in urban and rural communities, including but not limited to the size of the beneficiary pool, operational cost, economic/social assets, and potential partners. If the objective is to affect broad-based knowledge or attitudinal changes, there is a need for a regional focus. If the objective is to optimize the number of beneficiaries, leverage network spread of information, or build upon existing assets (partners, economy, etc.), USAID should consider an urban focus. Finally, if the objective is to optimize substantive community effects, we recommend a rural focus. Incorporate community assessments throughout the lifecycle of program and activity design, not just at the design phase. Engaging with communities should be part of a continuous feedback loop that informs USAID programming to be adaptive and context specific. The most important topics are community resilience/vulnerability factors, government support/opposition to planned activities, the existence of other programming/non-governmental organizations, and rapid market assessments. BENEFICIARY TARGETING Use design considerations that mitigate the occurrence of elite capture, perceptions of favoritism, and the unintended creation of conflict in beneficiary targeting. Strategies to help refine mitigation measures could include: (i) verify the appropriateness of selection criteria with community stakeholders, (ii) publicize selection criteria for public activities, (iii) allow local government to suggest candidates, but conduct open calls for applications, (iv) as appropriate, select beneficiaries through public lottery, (v) verify adherence to selection criteria before the activity starts, and (vi) as appropriate, provide consolatory activities/support to eligible but not selected applicants. SEQUENCING AND LAYERING OF PROGRAMMING Investments can be amplified through repeat and sustained programming with the same communities, individuals, and grantees. These efforts build trust, allow for programmatic learning/adaptation, and increase the likelihood of outcome achievement. SI recommends that USAID design complementary activities in response to community resilience/vulnerability factors. To the extent possible, and only when USAID understands the local context, this should take the form of fewer, larger activities. However, in conducting repeated programming, particularly with the same individuals, it is important to periodically assess the cost-efficiency of investments. PROGRAMMATIC RECOMMENDATIONS We provide specific recommendations on achieving three key TP outcomes (Improved VE knowledge, Improved Youth Leadership & Engagement, and Reduced Economic Marginalization) under shorter and longer time horizons in the body of the report (Programmatic Recommendations). USAID.GOV Mozambique Final Program Evaluation Report | 1 EVALUATION BACKGROUND PURPOSE Tuko Pamoja (TP) is a $16.3 million, 27-month program implemented by Development Alternatives Inc. (DAI) and funded by United States Agency for International Development’s (USAID) Office of Transition Initiatives (OTI). TP is OTI’s first experience programming in the complex insurgency in Northern Mozambique. In June 2021, OTI contracted Encompass and its sub-contractor Social Impact (SI) to conduct an external evaluation of TP. With TP concluding in January 2022, OTI sought to derive lessons from the portfolio of investments to inform future OTI and/or Mission programming. The primary audiences for this evaluation are OTI and USAID/Mozambique. TUKO PAMOJA PROGRAM OVERVIEW TP is a flexible mechanism through which OTI sought to better understand the Islamic State of Iraq and Syria-Mozambique (ISIS-M) insurgency and strengthen community resilience to violent extremism (VE) in Northern Mozambique. After the first year of programming, during which TP primarily sought to conduct research into the drivers of conflict, OTI updated the Theory of Change (ToC) to include a program objective (PO) designed to help vulnerable youth resist VE influence more effectively.0F The updated ToC was to be achieved through the two strategic objectives below. The second key programmatic revision was the focusing of programming in three geographic zones, coastal Cabo Delgado, Pemba, and Nampula.1 · PO 1 – Vulnerable youth resist VE influence more effectively. · SO 1.1 – Youth leaders have increased capacity to promote VE resilience among their peers. · SO 1.2 – Youth feelings of economic marginalization reduced. TP was implemented through a series of 83 grants disbursed to local organizations. Given the scale and heterogeneity of programming, this evaluation focused on a sub-portfolio of 24 activities covering two of the three geographic zones (Pemba and Nampula). Fifteen grantees implemented the activities under the evaluation sub-portfolio for a total budget of $1,510,123. 2 The number of grants implemented per organization ranged from one to three, while grantee budgets ranging from ~$25,000 to ~$338,000. The sub-portfolio included activities that funded: preventing and countering violent extremism (P/CVE) media (8 activities), vocational training (7 activities), leadership/life skills training (5 activities), VE awareness training (5 activities), network creation and/or strengthening (4 activities), and asset transfer (4 activities). EVALUATION QUESTIONS The evaluation was designed to answer four evaluation questions: 1. What are the primary outcomes of programming under our PO1 youth objective? 2. What impact did TP sequencing and layering have within geographic zones? 3. To what extent do evaluation findings at SO levels 1.1 & 1.2 support the respective hypotheses (ToC)? 4. Based on program learning, what changes would you recommend to make the ToC more accurately address the problem set? 1 OTI also added a second PO on positive community/government engagement, terminated the research/learning PO and ended humanitarian assistance-like activities. TP continued learning and research through its action research model, a rapid research mechanism and Cabo Ligado. These elements are outside of the scope of this evaluation. 2 SO1.1 = $948,227, SO1.2 = $561,896 2 | Mozambique Final Program Evaluation Report USAID.GOV EVALUATION DESIGN METHODOLOGY The evaluation team worked with OTI to sample a sub-portfolio of activities that represented both SOs, included at least two communities per geographic zone, and prioritized larger and sequenced activities. The full activity sampling approach is described in Annex II. Due to the coronavirus disease (COVID-19) pandemic and security situation in Northern Mozambique, the evaluation team was not able to conduct fieldwork. To limit selection bias and ensure that the study solicited in-person perspectives, the evaluation team subcontracted Forcier, a data collection firm (DCF) with significant experience conducting research in Northern Mozambique, to administer in-country key informant interviews (KIIs) and focus group discussions (FGDs). The study used a mixed-methods approach comprised of document review, portfolio KIIs (USAID, TP), district KIIs (field advisors, grantees, local government), beneficiary FGDs, beneficiary KIIs, and community FGDs. In all, the evaluation interviewed 158 respondents across 59 KIIs and 19 FGDs. The primary limitations of the study stem from four factors. First, the large number of investments under review inhibited substantive investigation of any specific activity. Second, the Forcier team encountered significant challenges during fieldwork, leading to the termination of two field researchers and variable quality of transcripts. Third, there was evidence of selection bias stemming from the use of beneficiary lists as sampling frames (i.e., some lists had no contact information, only youth with phone numbers could be contacted) and variable refusal rates. Fourth, given the sensitive nature of the subject matter, there was likely some degree of response bias. See Annex II and Annex III for a detailed discussion of sampling approaches and sample sizes for each data collection approach, study limitations, and field quality assurance (QA) processes. EQ1 - WHAT ARE THE PRIMARY OUTCOMES OF PROGRAMMING UNDER OUR PO1 YOUTH OBJECTIVE? In answering this question, we look at the achievement of objectives from the TP ToC as well as the emergence of additional outcomes identified by respondents.3.In the section below, we present results for the two SOs separately before concluding with a discussion of unintended consequences. SO 1.1 “YOUTH LEADERS HAVE INCREASED CAPACITY TO PROMOTE VE RESILIENCE AMONG THEIR PEERS” Of the 24 sampled activities, 16 were programmed against SO1.1. Across this sub-portfolio TP grantees implemented many interventions that did not focus on building the capacities of youth leaders (e.g., community awareness-raising, information dissemination). There were two primary reasons for this deviation between strategy and practice. First, as previously noted, TP underwent a significant strategy revision in January 2021, focusing programming on fewer geographic and programmatic areas. 4 Of the 16 SO1.1, 8 were designed before the revision. Second, some SO1.1 activities developed after the strategy revision did not adhere to the ToC. In the words of one TP respondent, “This (SO) became a catch-all for youth programming as long as it was not under the other SOs.” According to OTI, this was a conscious decision due to the pending termination of the program and the insufficient time to develop and execute 3 Beneficiary respondents were asked to describe how their lives had been affected by TP. Due to this open format, the study cannot generalize the pervasiveness of specific outcomes. 4 The strategy change is a common practice in OTI's approach and such approach is not unique to Mozambique USAID.GOV Mozambique Final Program Evaluation Report | 3 a new strategy. Given the variety of SO1.1 activities under review, findings are consolidated under the three primary programming streams. YOUTH LEADERSHIP TP promoted youth leadership in several ways, targeting individuals with existing community platforms (e.g., journalists, musicians) as well as unaffiliated youth. The scope and intensity of these activities varied greatly, ranging from single discussion groups to repeat programming with the same individuals over the course of more than one year. Across the full set of SO1.1 beneficiary interviews, 59% of respondents explicitly noted positive youth leadership outcomes.2F 5 This phenomenon played out in various ways, including increased confidence, engagement, and open-mindedness of youth. In the words of a grantee, “there was a personal and psychological transformation.” This, in turn, translated into a greater sense of agency, where KII respondents provided examples of youth pursuing teaching, volunteering, and the creation of their own youth associations due to their involvement in TP. In the words of two PEM001 beneficiaries, this included having “learned and discovered that our young people have many talents” and that they should “not just depend on the government's or others' for help.” Trainings were conducted in small groups and oftentimes had an explicit objective to strengthen or create community networks. While the evaluation cannot assess the strength or durability of these networks, there is suggestive evidence that the YEP Youth association created through a chain of TP activities (PEM001/028/039) was operational at the time of interviews. In the words of a grantee, “the creation of this trust within the group are experiences that fortify the person, and that (they) can then take for life and do similar activities with other young people.” Finally, multiple respondents cited youth leaders as having become role models for peers. In the words of one PEM001 beneficiary, “we discovered that a lot of young people became attached to us and saw us as a mirror.” There was also evidence of increased youth engagement with community leaders. According to a TP respondent, “youth are marginalized. Not just economically but within their own communities. Authorities treat them like potential recruitment subjects… Youth are supposed to show respect and are treated with suspicion.” This low baseline level of status, paired with the centralized nature of Mozambican governance, results in limited interactions between youth and community leaders during which youth have a voice or agency. TP facilitated spaces for dialogue, leading to what one TP respondent called “one of most important results… Not only did these engagements with authority figures further increase the confidence of certain youth leaders, they increased community capacity to receive youth leaders.” While the extent of youth leader engagement with community leaders varied, there were demonstrable community-level outcomes of some engagements (e.g., the NAM014 the Tuna Festival). Finally, the evaluation found evidence that youth leadership investments led to the increased status of beneficiaries among local government, community leaders, and the broader community. Two examples of these include the youth-led community clean-up efforts in Isla Mozambique under NAM006 and the creation of music to disseminate peaceful messages/alternative narratives under NAM004, where, according to a grantee, “The community saw their children becoming somewhat famous and doing something good.” 5 The prevalence of this outcomes was most common for NAM004, NAM016, PEM016, PEM018, and PEM046, where all respondents reported positive effects. The lowest prevalence was noted for NAM013 and NAM017. 4 | Mozambique Final Program Evaluation Report USAID.GOV IMPROVED KNOWLEDGE OF VIOLENT EXTREMISM TP produced and disseminated a wide variety of media content, including radio programming,3F 6 music videos, and social media campaigns. Key lessons learned in this process included the effectiveness of radio programming, a trusted and accessible medium in target communities, paired with culturally resonant, in￾person forums (e.g., listening clubs and discussion groups). Conversely, TP’s online and social media campaigns proved to have limited reach. In the words of one TP respondent, “we were surprised by the lack of progress. We overestimated how online people were.” Across all sampled activities, 77% of beneficiary respondents reported having a better understanding of the insurgency due to their participation in TP. Importantly, this sentiment was similarly prevalent among SO1.1 and SO1.2 respondents (80% and 74%, respectively).4F 7 We describe the primary manifestations of improved VE knowledge below. Due to the government of Mozambique’s discouragement of open discussion around the insurgency, respondents reported substantial constraints in their ability to discuss P/CVE topics prior to the project. TP’s media and in-person activities contributed to the de-stigmatization of the subject. In the words of one Field Advisor (FA) respondent, “the population wasn't talking about VE. There was a stigma. This subject was a taboo to us… TP made us be able to talk about it, and to know more about the truth.” This perception was also noted by beneficiaries, such as a youth journalist who said that “we were able to show that [the insurgency] is actually real and it is not something that should be hidden. It is something that should be told so that the community can protect itself… I am not afraid to talk about Al-Shabaab, I speak without any fear and I have seen some young people also speak without fear." Respondents at all levels noted the inadequacy of credible information about the insurgency and the widespread misperceptions about the conflict before the project. In the aggregate, it is clear that TP increased coverage of the conflict in local media. Furthermore, there is substantial evidence that de￾stigmatization of the subject, the creation of media content, and complementary in-person activities (e.g., trainings and P/CVE discussions) improved awareness of VE in target communities. A common example cited by respondents was the correction of misperceptions around the cause of the insurgency (i.e., that the conflict was orchestrated by the government, organized crime, or religious groups). This sentiment was substantiated by a wide cross-section of stakeholders, including youth beneficiaries (e.g., “I've changed. I thought the ones [supporting VE] were religious, but I realized they're mainly doing so to get a job”), grantees (e.g., “the radio programs contributed [to people being] aware that VE was not created by the government.”), religious leaders (e.g., “people used to associate this with politics, and we needed to convince them it was about VE.”), and internally displaced persons (IDPs) (e.g., “I thought the government was to blame for the war, but after receiving the training I understood that it was people without ethnicity that cause these wars.”). One of the primary mechanisms through which TP sought to increase resilience was through the reduction of youth vulnerability to ISIS-M recruitment. This, in turn, was promoted through helping youth and community members better understand ISIS-M recruitment tactics and the consequences of joining the insurgency (personal, familial, community). Youth knowledge about the insurgency in general, as well as understanding of recruitment tactics in particular, was reportedly limited in the early days of the project. 6 The radio soap opera intervention (PEM016) was credited with getting information to remote areas, being highly engaging (e.g., community members referring to one another using character names), and affecting positive changes (e.g., promoting more sympathy toward government through the portrayal of a police officer character). 7 The prevalence of VE knowledge increase was most common for NAM004, NAM016, PEM015, PEM016, PEM018, and PEM046, where all respondents reported positive effects. The lowest prevalence was noted for NAM013 and PEM029 (an SO1.2 activity). USAID.GOV Mozambique Final Program Evaluation Report | 5 In the words of one government respondent, “some of the people that were getting involved with extremist groups didn’t know they were being recruited… None of the young men that went to Cabo Delgado knew what was about to happen. Some left here thinking they would do fishing activities. Some others thought they could find better educational conditions.” An example of improved understanding of recruiting tactics among youth was noted by a NAM003 beneficiary who said, "VEs look for unmarried women, promising marriage and a good life. I´ve learned how to interpret what people from VE say to these women, as I´m one of them." Finally, there is some evidence that certain beneficiaries were better able to critically assess ISIS-M communications. In the words of one youth journalist, “I´ve learned a lot about how to interpret the VE´s speech.” COMMUNITY COHESION Many TP activities explicitly sought to create linkages between community members, straddling age, gender, ethnic, and functional lines. Examples of these interactions included ad hoc meetings, promotion of shared spaces (e.g., cultural centers), dialogue between youth and religious leaders, and exchanges between community members and local government. Importantly, many of these activities included the participation of marginalized groups (e.g., inclusion of internally displaced youth in the YEP Youth Association). In the words of a government respondent serving this community, there were positive community effects of this approach: “At Nicuapa, their coexistence became relatively peaceful, despite ethnic differences... Even for those indirectly involved, such as welcoming families and the community in general, it was beneficial.” Furthermore, women, who were frequently cited as lacking full participation in public gatherings, were given a platform in TP activities. In the words of PEM001 respondents, “women were heard and this is not a common thing around here,” and “there is a greater involvement of women because prior to it women were not involved in community affairs”. Finally, there was some evidence of improved intergenerational linkages. A FA noted that “parents/guardians… learned how to talk to their children and help guide them to lead their lives in the direction of a good future, as well as to welcome back those that left the insurgency and not discriminate them upon their return…. As long as they would behave well, they were being re-integrated in the community.” These increased interactions were credited by certain respondents with precipitating higher level outcomes. Across all SO1.1 beneficiary respondents, 26% mentioned one or more manifestations of improved community cohesion as the result of TP programming. For example, one beneficiary respondent noted, “we have learned about unity, staying together and uniting to face evil people. After the event… contacts were exchanged between people, and a WhatsApp group was created.” Another example was the tuna festival, which was cited by multiple respondents as having increased community pride. Cohesion also became apparent in the identification and resolution of community problems. This sentiment was highlighted by FAs (e.g., “people found out they had someone to address their issues. They were able to talk about it in order to achieve a different type of answer.”) as well as beneficiaries (e.g., “(people) started to express themselves, problems were raised and solved without the need for money” and “the community has already started to solve its problem without the help of projects or the government.”) Finally, the evaluation found limited evidence that increases in VE knowledge paired with community cohesion resulted in increased community vigilance in Nachiropa and Angoche. In the words of one FA, “the community began to pay attention to who and what enters the community, they already knew the risks of receiving strange things or people, they became more vigilant, and this somehow kept them safe.” This sentiment was echoed by a grantee, who claimed that “the population became more vigilant… The community understood they had to be alert with those who were strange inside their community and may be a part of VE, searching for young people to join VE.” Finally, according to a grantee, “(NAM017) fishermen… would go out to their communities and spread the word about… how to prevent VE, 6 | Mozambique Final Program Evaluation Report USAID.GOV emphasizing the identification of strangers…to communicate to the government… about strange people who are in the area and this was not only something done by young fishermen… it increased awareness of various indirect beneficiaries of prevention strategy, or prevention tactics against violent extremism." SO 1.2 “YOUTH’S FEELINGS OF ECONOMIC MARGINALIZATION IS REDUCED” Recognizing that relative deprivation is a key determinant of VE vulnerability, OTI/TP developed an SO intended to reduce youths’ feeling of economic marginalization. However, given the low level of economic development in targeted zones as well as budget and time constraints, OTI recognized that economic development was outside the manageable interest of TP. Instead, the program facilitated and made youth aware of economic opportunities to reduce feelings of economic marginalization. Of the 24 sampled activities, 8 were implemented against SO1.2. In the section below, we detail the emergence of economic effects, starting at the output level (provision of trainings and assets), before moving to higher level outcomes (income and economic marginalization). Finally, we touch briefly on community-level effects. DIRECT BENEFICIARIES Across the sub-portfolio, TP provided vocational training in numerous fields (e.g., fish processing, construction, electrical, carpentry, bricklaying, plumbing, masonry)8F 8 and to varied beneficiary groups (e.g., fishermen associations, IDPs, women, unaffiliated youth). Across the set of SO1.2 beneficiary respondents, 61% provided evidence of increased economic skills (e.g., a grantee said that TP taught “that fish can be salted, the fish can be processed so that it doesn't rot, and be able to sell the fish at a relatively higher price”). Often, in conjunction with these trainings, TP facilitated the transfer of economic assets (e.g., sewing kits/machines, fishing equipment). In one case (NAM001), a significant investment was made to provide cold storage facilities outfitted with solar powered batteries to enable the storage and processing of fish. Through TP and beneficiary lobbying, the government transferred ownership of plots of land to the clubs to allow for facility construction. Through this process, a grantee noted that “the ownership of the processing centers now belongs to all of them and that this promotes the sustainability of clubs.” TP support was credited by respondents for leading to a greater “sense of agency” and increased confidence in their ability to earn income. This, in turn, led to positive economic effects. Across the set of SO1.2 beneficiary respondents, 54% cited some improvement in their economic situation stemming from the project.9 For example, all three respondents in the PEM029 FGD reported having started cutting and sewing businesses due to TP support, with one also citing that they now “also sell food products.” This development was also pronounced among beneficiaries of NAM001 youth clubs, with multiple respondents reporting substantial benefits stemming from their TP-mediated opening of microenterprises (e.g., fish selling, meals selling, baking [cakes and bread], vending).10 One such respondent said, “I had the opportunity to buy a stove and the money I was able to apply in my business. … [I’m] able to buy notebooks, school uniforms, besides having fishing as my main activity, the information has helped me to change my opinion about insurgency,” However, not all TP SO1.2 activities reduced beneficiaries’ perceptions of economic marginalization. For example, none of the six NAM002 and two-thirds of the 8 There is suggestive evidence that TP economic activities were responsive to market demands, as a different USAID project methodically assessed youth needs and supported similar economic activities. 9 This was most common among PEM029 and NAM001 and least common among NAM002 respondents. 10 While it was not an objective of TP, the evaluation found limited evidence of broader economic effects of some activities (e.g., Lúrio fishing clubs, Nacala fishermen’s association, and YEP micro-businesses). Positive effects were mostly driven by low levels of economic development and intensity of programming. USAID.GOV Mozambique Final Program Evaluation Report | 7 nine NAM012 beneficiaries interviewed for this evaluation were able to find any employment after completion of vocational trainings.11 Finally, the evaluation team found evidence of positive effects of SO1.2 programming on community/government interactions. As noted earlier, Mozambican citizens routinely blame the government for economic hardship and hold the belief that it is the government’s obligation to provide jobs. According to some respondents, TP programming reduced this sense of dependence and ameliorated the extent of the blame on the government for the conflict and economic situation. One NAM019 beneficiary said, “it changed my opinion because before I thought that this war was brought about by the government, but after the project was empowered, my way of seeing things completely changed.” Furthermore, a grantee said that, “they had the opportunity to discuss the different topics that can be used to earn a fixed income other than through government work.” This reduced sense of grievance with and dependence on the government could be seen as a reduction in economic marginalization. Across the set of SO1.2 beneficiaries, 24% provided some evidence of reduced feelings of economic marginalization.11F 12 This included beneficiaries being prouder of themselves and their community, demonstrating that youth have capacity/skills, and in the words of a grantee, “psychological support to overcome the trauma experienced in conflict.” Finally, there was some evidence that TP led to reduced perceptions that the government should resolve economic problems. COMMUNITY OUTCOMES The evaluation team also found evidence of positive community effects under SO1.2. These were sometimes the direct product of TP approaches, such as increased collaboration of community members toward a shared goal (e.g., according to a beneficiary, “the most impactful activity in our community was (NAM001) facilitating the involvement of young people in income generating activities… many young people participated in this activity, whether women or men, they came together for the development of the community.” However, at other times community effects were positive externalities of SO1.2 economic impacts. 13 For example, respondents from NAM001/009 noted that the TP batteries that supported fish processing centers also powered lights for community use. In the words of a grantee, “when night falls (people) sit around to play games. They are no longer at home just having children because there is no light.” In the aggregate, 7% of SO1.2 beneficiary respondents cited improved community cohesion. PO 1 “VULNERABLE YOUTH RESIST VE INFLUENCE MORE EFFECTIVELY” The evaluation found evidence that TP programming led to reduction in VE vulnerability. Across the full sample, 32% of beneficiary respondents provided evidence of having reduced vulnerability due to TP programming.13F 14 This was driven primarily by improved community and individual understanding of vulnerability, increased community vigilance toward outsiders, increased knowledge about reporting mechanisms, and improved economic standing. Furthermore, 9 beneficiary respondents (10% of the 11 These activities were implemented in urban areas, compared to other activities mentioned in this section. 12 The prevalence of this outcome was highest among NAM001 beneficiaries (43%) and the lowest among NAM12 and PEM029, where no respondents made mention of this outcome in the interview. 13 Across the SO1.2 sample, 20% of respondents provided evidence of community economic benefits. This was most frequently cited by NAM001 (33%) and least cited by NAM003 and PEM029 respondents (0%). 14 The prevalence was somewhat higher among SO1.1 respondents (35% vs 28%). 8 | Mozambique Final Program Evaluation Report USAID.GOV sample) provided anecdotal references to reduced VE recruitment.15 These came in the form of deciding not to join the insurgency personally, convincing peers not to join the insurgency, and convincing peers not to return to the insurgency. We provide all quotes in Annex IV, but demonstrate one excerpt from three key outcome streams below: Youth Leadership “They (came) back to the community with the intention of recruiting others due to the supposed benefits, but I talked to them and I managed to change their minds and they didn't go anymore.” (NAM003 Beneficiary) VE Knowledge “My opinion about the insurgency has changed because before I could be easily enticed with promises of work in Cabo Delgado but not anymore. When this insurgency thing started, people were recruited to go to work and in a short time they came back rich. At this point everybody wanted to go to Cabo Delgado, even me, but nowadays, thanks to the project, I don't think about going to Cabo Delgado anymore.” (NAM012 Beneficiary) Economic Marginalization “First, thanks God for not being killed in Cabo Delgado. With what I learned; I no longer think about joining VE in Cabo Delgado because I already have an occupation thanks to the project.” (NAM014 Beneficiary) It is not, however, possible to attribute these outcomes solely to TP. While the program was doubtless an important contributing factor, TP operated in a fluid context influenced by other interventions and developments. For example, the beneficiary quoted in the VE Knowledge section above also stated that “even before the project arrived, we already heard everything that was happening in Cabo Delgado through the press (radio and television). We saw on the internet people being decapitated, then we got a notion of what it was about, and this contributed to change our opinion.” Finally, the evaluation found some corroborating evidence of TP-mediated reductions in recruitment from a government respondent from Nacala-A-Velha. This individual credited NAM017 with reducing VE recruitment in the community, saying that, due the project, “fishermen discussed in depth the different manners of recruiting used by extremist groups. They were even able to recognize their vulnerability. Fishermen now notice that, without awareness, they would continue joining extremist groups, prone to false promises of fishing and getting money….. [these activities] indirectly benefited our community. We have recently registered a reduction in the admission of young people to violent groups.” UNINTENDED CONSEQUENCES The adaptive nature of TP programming, paired with the complex operating environment resulted in both positive and negative unintended outcomes. We highlight the most important consequences in this section. INCREASED COMMUNITY TENSIONS While most respondents cited improvements in community cohesion, there were four ways in which community tensions may have been exacerbated by TP. First, multiple respondents noted disappointment stemming from certain communities’ exclusion due to TP’s targeting strategy, which could potentially have created tensions with beneficiaries. Second, increased community vigilance noted earlier could create conditions where mistrust of outsiders leads to intra-group conflict. Third, targeted programming at IDPs in a context of generalized need can create tensions between displaced and permanent residents. One FA 15 Preventing VE recruitment is a difficult outcome to achieve and document through interviews. While the absolute number of cases was limited, given these constraints, the evaluation team views the relatively high prevalence rate as a favorable finding for TP. The prevalence was substantially higher among SO1.1 respondents (15% vs 4%). USAID.GOV Mozambique Final Program Evaluation Report | 9 respondent reported that a “community started denying the displaced people access to water and wood to build houses” in response to TP programming.16 INCREASED MARGINALIZATION As with many interventions, community demand exceeded TP’s capacity to provide services. This oversubscription necessitated the use of targeting criteria and, by extension, the exclusion of certain community members. Many activities were targeted based on vulnerability to ISIS-M influence. In some instances, this was determined on somewhat objective measures (e.g., youth caught on their way to Cabo Delgado), whereas in others vulnerability was determined by local government counterparts. The evaluation found three issues with this approach. First, multiple community respondents in Angoche and Nachiropa reported that selection favored families that were connected to the project and the local government (i.e., the ruling party).15F 17 Second, there were reports that IDPs were excluded due to the lack of fixed addresses and low community standing. And third, multiple respondents spoke about insufficient community communication of selection criteria in advance of determining participation. This sentiment was summarized by a grantee, who said, “some young people began to feel excluded because the number of beneficiaries was limited, and this exclusion could constitute a criterion for revolt.” However, there was also some limited evidence of increased perceptions of marginalization stemming from participation in TP. Multiple community respondents from Angoche noted problems with the way in which the NAM012 activity disbursed equipment and materials following a sewing training, leading to conflict among beneficiaries. One respondent said that “it can somehow affect the probability of these young people joining the insurgency because they feel deceived... They should have said at the beginning that it would be one machine for five people, and not at the end of the training, because this started problems among themselves regarding those machines, because of them feels like the owner of the machine.” Another respondent elaborated, saying “Even those who have had access to this opportunity feel outrage and there are possibilities of them joining Al-Shabaab, since they have not been given the promised (sewing) kits… Not to mention that (trainees of some) courses that have not received any kits. They were trained and then abandoned… The project has affected the likelihood of other beneficiaries joining the insurgency as (they) feel unsatisfied with the project for not having delivered as promised. A new war of the trainees of the sewing training is emerging because, with one machine for five people, it is not possible for each of them to work and earn income to support themselves.” While in absolute terms, there were few mentions of this phenomenon, unfulfilled raising of economic expectations can create adverse psychological effects, particularly in a sensitive context like Northern Mozambique. Even though 16 In conducting portfolio interviews, the evaluation team learned about a TP activity (PEM021) that organized the painting of P/CVE murals on public buildings. While the design of the murals was made in close collaboration with community members, a portfolio respondent noted that the murals “stereotyped who would participate in VE.” In their view, some messages could be interpreted as “advocat(ing) violence against groups in the community.” While, according to the TP and OTI staff, community representatives thought the images were positive in the design process, OTI and TP recognized that they could be seen as exclusionary to some groups and the murals were replaced with paintings depicting positive messages about community solidarity within 11 days. However, during the intervening period, it is possible that some community members may have felt marginalized in response to the murals. Since this activity was not implemented under PO1, the evaluation did not investigate this investment and has no evidence on how murals were perceived by community members. 17 “There has been a negative impact, but it's not just a matter of (TP), any project that appears here in the district, many times we from other neighborhoods only hear about it, we never directly benefit from it, when we ask, they say nothing, but when it comes to the election campaign to vote they look for everyone. So when a project arrives the same has to be done. And that for the community is a negative impact.” 10 | Mozambique Final Program Evaluation Report USAID.GOV TP instructed grantees not to guarantee jobs, according to a TP respondent “if you train youth and promise them jobs, it furthers cycles of frustration." CONCLUSIONS SO1.1 YOUTH LEADERS ● Three in five SO1.1 beneficiary respondents cited evidence of increased youth leadership, including “opening up” of minds, increased self-efficacy, the creation of peer networks, increased engagement with community and local government leaders, and improved status of youth. ● Due to changes in the TP strategy, many SO1.1 activities supported broader community objectives. Additional outcomes realized through these investments included de-stigmatization of VE discussion, improved community awareness of VE (e.g., correction of misperceptions about the drivers of the insurgency, knowledge of VE recruitment tactics), increased community vigilance regarding outsiders, and increased community cohesion. SO1.2 ECONOMIC MARGINALIZATION ● To varying degrees across the SO1.2 sub-portfolio, TP was able to improve the acquisition of life skills, vocational skills, and the transfer of economic assets. These investments led to noticeable economic gains among beneficiaries of intensive programming, with some respondents reporting substantial quality of life improvements. However, less intensive activities had much more mixed success at generating economic opportunities. ● One in four SO1.2 beneficiary respondents cited evidence of reduced economic marginalization, including increased confidence, ability to generate income for their family, community pride, and status. Additionally, the evaluation found evidence of reduced government dependence. PO1 VE RESILIENCE ● One in three TP beneficiary respondents provided evidence of having reduced vulnerability to VE stemming from their participation in the program. This was driven by improved community and individual understanding of vulnerability, increased community vigilance toward outsiders, increased knowledge about reporting mechanisms, and anecdotal references to reduced VE recruitment. ● This outcomes was more likely to be referenced by SO1.1 beneficiaries, though the majority of respondents seemed to point primarily to the economic underpinnings of VE vulnerability. UNINTENDED CONSEQUENCES ● The evaluation found evidence that in some instances the project increased tensions and feelings of marginalization at the community and individual levels. At the community level, this was driven by feelings of exclusion from government favoritism in the allocation of some TP benefits, community retribution against IDPs, and unintended effects of increased community vigilance toward outsiders. At the individual level, this included youth not receiving support due to contested eligibility criteria, lack of economic benefits among some SO1.2 participants, and isolated cases of conflict over TP disbursed assets. USAID.GOV Mozambique Final Program Evaluation Report | 11 EQ2 - WHAT IMPACT DID TP SEQUENCING AND LAYERING HAVE WITHIN GEOGRAPHIC ZONES? In an effort to better focus TP resources, the 2021 strategy deployed a “sequencing and layering” approach to increase programmatic intensity within select geographic locations.16F 18In this section we address two manifestations of this strategy: sequencing within the same communities, and among the same beneficiaries. SEQUENCING WITH THE SAME COMMUNITIES The evaluation focused on TP activities in five communities.19 We found that complementary programming within the same communities was the most effective approach in the context of limited time, budgets, and staff resources. In Table 1 below, we present the distribution of activities, evaluation respondents, and prevalence of key TP outcomes by community. We see a strong correlation between programmatic density and the frequency with which respondents provide evidence of various outcomes. In fact, except for increased community vigilance and reduced recruitment (both low-prevalence outcomes), every other objective was most pervasive in the communities with six or more activities: Lúrio and Pemba. It is important to keep in mind that sampling was conducted on an activity basis, not community representativeness. As such, the number and type of respondents vary across the five communities. Table 1: Number of Activities, Beneficiary Respondents, and Key TP Outcomes by Community Lúrio Pemba Angoche Nacala-A-Velha Nachiropa # of Respondents 19 25 19 10 17 # of Activities 6 8 4 3 2 # of SO1.1 Activities 4 8 2 3 1 # of SO1.2 Activities 2 0 2 0 1 SO1.1 Youth Leadership 33% 65% 26% 40% 8% SO1.1 Youth VE Knowledge 83% 77% 74% 70% 77% SO1.2 Economic Skills 63% 15% 47% 0% 6% SO1.2 Economic Impact 79% 27% 16% 0% 8% SO1.2 Community Economic Impact 29% 0% 11% 0% 0% PO Economic Marginalization 38% 4% 11% 10% 8% PO Community Cohesion 8% 27% 16% 10% 15% PO Community Vigilance 0% 0% 5% 0% 8% PO Reduced Vulnerability 33% 46% 26% 10% 23% PO Reduced Recruitment 8% 8% 11% 10% 15% * Green highlighting denotes the highest prevalence community for each outcome area The strength of the results in Lúrio and Pemba is at least partially a function of the relatively large number and variety of TP programming. For example, in Lúrio, TP created eight youth clubs, provided equipment and training for fish processing, constructed cold storage facilities, provided financial literacy training, disseminated radio programming (radio dramas, radio programs targeting youth, radio debates/lectures), and organized community events (listening clubs, community awareness meetings, and a Tuna Festival). In Pemba, TP conducted participatory mapping of drivers of community violence, developed a P/CVE social media platform, provided equipment to a cultural center, created a teacher peer network, provided youth 18 “TP will also narrow its geographic focus in each zone… to maximize limited funding and focus staff time and resources. This change allows TP to be intentional about how activities are layered, sequenced, and/or clustered, taking a portfolio perspective to better achieve the program goal and objectives.” TP Program Document (4/21) 19 The evaluation also conducted two interviews in Memba, which was added during field work. 12 | Mozambique Final Program Evaluation Report USAID.GOV leadership training, supported the registration of the YEP! youth association, provided support and seed funding for community projects/businesses, and financed a resilience messaging campaign targeted at IDPs. In both cases, TP provided substantial support to youth groups as a core programmatic element, augmented by a range of complementary community interventions.20 SEQUENCING WITH THE SAME INDIVIDUALS The sub-portfolio under review contained four programmatic chains that engaged the same beneficiaries through repeat activities: support to YEP! Youth in Pemba (PEM001/PEM028/PEM039), youth journalists in Angoche (NAM007/NAM016), the Nachiropa Fishermen’s Association (NAM003/NAM017), and youth in Angoche (NAM002/NAM012). There was strong evidence that many participants from the first three groups underwent significant knowledge and behavioral changes, driven by repeated and oftentimes intensive engagement. Even in the case of the youth in Angoche, where beneficiaries were less likely to report positive outcomes of the economic activity (NAM012), the organization of community dialogue sessions (NAM002) was cited as having improved knowledge and perceptions about the insurgency. COST/BENEFIT CONSIDERATIONS FOR ACTIVITY SEQUENCING/LAYERING It is clear that communities where TP supported multiple activities and repeatedly engaged beneficiaries reported the most substantial gains vis-a-vis TP objectives, however, given the scale of need and the limited resources at TP’s disposal, it is helpful to consider cost/benefit tradeoffs. For example, the YEP! activities accounted for ~$338,000 in programmatic costs. The first activity involved a larger cohort of beneficiaries while the latter two supported the same cohort of 20 youth, giving an estimated per/beneficiary cost of ~$10,000. Youth journalist activities were comparable, with a per/beneficiary cost of ~$8,400. It is not surprising that an investment of this size would lead to outcomes, particularly if it includes transfer of economic assets. However, given the inability to scale this support widely, one benchmark of success could be the extent to which investments permeated through the community, affecting other individuals. At the conceptual level, evidence for this multiplier effect is strongest among the youth journalists who have a large reach through their radio platform. Not only did these youth produce content with direct TP funding, but they also received training that could benefit them and their audience throughout their careers. While the evaluation could not explore the full scale or effects of these externalities, except for the Angoche youth chain, there was evidence that geographic sequencing and layering contributed to positive community P/CVE outcomes (e.g., members of the fishermen’s association trying to convince peers not to travel to Cabo Delgado to fish, YEP! beneficiaries employing community youth). Finally, future USAID programming addressing the economic determinants of VE resilience may want to consider the sustainability of economic investments. For example, respondents questioned whether YEP microenterprises will persist and noted poorly functioning batteries that inhibited the functioning of fish processing plants (“the freezer reduces its working capacity and the fish ends up spoiling.”). CONCLUSIONS ● The evaluation found a strong positive correlation between the number of activities implemented in a community and the prevalence of intended TP outcomes. Successful programming at the community level seems to have been driven by three factors: (i) implementation 20 While it was outside of the evaluation’s scope, multiple portfolio respondents also cited community-level impacts in Ibo and Montepuez, attributing success to sequencing/layering of activities in locations with low levels of development. One respondent noted that, “we picked areas where there were no organizations or opportunities.” USAID.GOV Mozambique Final Program Evaluation Report | 13 of multiple, complementary activities, (ii) large investments, and (iii) relatively low community economic and living conditions. ● There was strong evidence that sequencing multiple activities with the same individual beneficiaries led to knowledge and behavioral changes Due to TP’s mass media activities, it is possible that many beneficiaries of discrete activities were exposed to more than one intervention; however, the number of individuals that participated in multiple, intensive activities was limited. ● There are substantial resource implications and tradeoffs to this community- and beneficiary-intensive approach to layering activities. In some cases, TP invested upwards of ~$10,000 per beneficiary. At this level of investment, the total number of beneficiaries that can be supported is limited. While in certain instances TP activities had substantial multiplier effects (e.g., youth radio journalists), in others, the benefits seem to have accrued more directly to individuals. EQ3 - TO WHAT EXTENT DO THE EVALUATION FINDINGS AT SO LEVELS 1.1 & 1.2 SUPPORT THE RESPECTIVE HYPOTHESES (TOC)? In answering this question, we first assess key assumptions embedded in the ToC before exploring the emergence of outcomes following the program logic. RESPONDENT PERCEPTIONS OF THE PROJECT LOGIC Respondents were asked about the extent of their agreement with six assumptions from the TP ToC.18F 21 Table 2 provides the aggregate prevalence of disagreement while Annex V disaggregates responses by respondent type. Overall levels of disagreement were relatively low, with the most disagreement registered for the assumption that “youth lack the skills and knowledge to effectively resist VE.” Youth beneficiaries and community-level respondents were the most likely to disagree with this statement, most frequently citing the primacy of economic determinants of VE support over skills and knowledge. The second assumption with the highest disagreement was that “there are legitimate economic opportunities available for youth, but they don’t know about them.” Importantly, youth beneficiaries were the least likely to disagree with this statement, citing restrictions to accessing jobs by government and private interests. 19F 22 The third most frequently contested assumption was that “youth are influential in their communities and with their peers.” Respondents noted that youth were influential among peers but, due to low social status and the hierarchical nature of Mozambican society, not with adults, community leaders, and local government. Table 2: Percentage of respondents who disagreed with SO statements Statement Disagreement SO 1.1 Youth are influential in their communities and with their peers 13% Youth lack the skills and knowledge to effectively resist VE extremism 21% SO 1.2 There are legitimate economic opportunities available youth, but they don’t know about them. 17% Lack of economic opportunities increases youth’s vulnerability to joining/supporting VE. 7% Some possible legitimate income generating activities need TP support to be initially viable. 4% IS-M is providing economic incentives to entice vulnerable youth to join/support VE. 9% 21 Respondents were asked to register the extent of their agreement using a 4-point scale (1 strongly disagree, 4 strongly agree). We collapse ‘strongly disagree’ and ‘disagree’ statements in the reporting. 22 Youth cited “preferential treatments”, opportunities available only for “privileged people”, or requirements for formal credentials like certificates and diplomas 14 | Mozambique Final Program Evaluation Report USAID.GOV SO 1.1 “YOUTH LEADERS HAVE INCREASED CAPACITY TO PROMOTE VE RESILIENCE AMONG THEIR PEERS” The SO 1.1 ToC states that (IF) Tuko Pamoja provides youth leaders with the P/CVE skills and knowledge, (THEN) youth leaders have increased capacity to promote VE resilience among their peers. The evaluation found robust evidence that TP increased VE knowledge among youth leaders. This is most directly substantiated through the frequency with which beneficiaries spoke about TP informing them about the causes of the insurgency and helping them understand the consequences of joining ISIS-M. The study found moderate evidence of improved P/CVE skills among these beneficiaries, including improved self-confidence, open mindedness, agency, communication (e.g., with local leaders), and organization. This variability was mostly a function of how intensively each youth participated in TP activities. The evaluation found some evidence of increased youth leader capacity to promote resilience among peers. Most formally, this included the building of youth networks. These clubs galvanized youth communication, engagement, and a sense of community. However, the broader community reach and long-term viability of these networks without future support or funding is uncertain. Finally, the evaluation did find evidence that certain youth leaders tried to sway peers from joining the insurgency and a limited number of cases where youth claimed to have either prevented peers from joining/re-joining ISIS-M. SO 1.2 “YOUTH’S FEELINGS OF ECONOMIC MARGINALIZATION IS REDUCED” The SO 1.2 ToC states that (IF) Tuko Pamoja makes vulnerable youth aware of economic opportunities and facilitates the viability of these opportunities, (THEN) youth’s feelings of economic marginalization will be reduced. The evaluation found evidence that TP made youth more aware of economic opportunities through joint discussions and trainings. However, TP’s ability to facilitate viable economic opportunities was mixed. The evaluation found robust evidence that beneficiaries experienced non-trivial economic gains in cases where the program invested significant time and resources. Less intensive training activities, however, were not as effective in connecting youth to economic opportunities. While some respondents cited having been able to leverage skills and assets toward productive economic activities, others reported no economic benefits stemming from their participation in TP. Finally, the evaluation found mixed evidence that improving awareness and/or access to economic opportunities reduced feelings of economic marginalization. Beneficiaries receiving substantial support and those that were successfully able to leverage support into economic opportunities were the most likely to report improved economic status. SYNERGIES BETWEEN SO1.1 AND SO1.2 Many beneficiaries received support from both SO1.1 and SO1.2 activities. The evaluation found robust evidence that the two approaches are highly complementary. For example, using economic incentives (SO1.2) is highly motivating and encourages participation and openness to P/CVE conversations among individuals who might otherwise abstain from SO1.1 activities. Similarly, while improved economic outcomes could directly lead to reductions in vulnerability, the inclusion of SO1.1 components in SO1.2 programming is critical to leveraging these investments toward P/CVE objectives. CONCLUSIONS ● There was substantial agreement among respondents on most assumptions underpinning TP’s ToC. Respondents were most likely to disagree that youth lacked skills/knowledge to resist VE (citing the primacy of economic determinants), that there are legitimate economic opportunities available to USAID.GOV Mozambique Final Program Evaluation Report | 15 youth (with youth citing barriers to employment), and that youth are influential in their communities (with respondents agreeing about youth influence with peers but not the broader community). ● The SO1.1 logic was largely validated through the evaluation. Most youth beneficiaries increased their VE knowledge, many gained leadership skills, and some promoted VE resilience among peers. The evaluation found a few anecdotal examples of youth leaders claiming to sway peers from (re)joining the insurgency. ● The SO1.2 logic was also largely validated, with certain activities leading to economic outcomes (e.g., employment, income) and subsequently reduced perceptions of marginalization. However, these cases were concentrated among relatively intensive programming with large per beneficiary investments. While some beneficiaries of smaller activities (e.g., vocational training) did register improved and perceived economic and marginalization outcomes, individuals that were not able to benefit economically from their participation were much less likely to cite feeling reduced marginalization (in fact as noted in EQ1, marginalization for some of these beneficiaries could have increased). ● There was evidence of significant synergies between SO1.1 and SO1.2 programming, substantiating future investments that pair economic and knowledge/skills objectives. EQ4 - BASED ON PROGRAM LEARNING, WHAT CHANGES WOULD YOU RECOMMEND TO MAKE THE TOC MORE ACCURATELY ADDRESS THE PROBLEM SET? In this section we explore stakeholder perceptions of how to best revise the TP strategy and highlight one key challenge with the ToC. To streamline the report, we provide our recommendations on revising the ToC in the Recommendations section. STAKEHOLDER PERCEPTIONS All respondents were asked how they would revise the TP strategy to make it more effective if the program were extended by one year. Among USAID and TP respondents, there was very little agreement on how best to accomplish P/CVE objectives. Recommendations included: shifting the programmatic lens from a primarily P/CVE perspective to more focus on stabilization, broadening the beneficiary base through expanded definitions of vulnerability, more explicit focus on the economy, partnering more with the private sector, working more through local leaders and less though youth leaders, working with the Government of Mozambique to curb human rights abuses and extortion, and providing psychosocial support to survivors of trauma. Conversely, there was significant agreement among beneficiaries regarding priorities. Requests for economic support (training and asset provision) were the most common, particularly among SO1.2 beneficiaries. Additional trainings were also commonly requested among SO1.1 beneficiaries. This correlation between receipt of support and requests for support suggest that beneficiaries were largely satisfied with TP support and wanted more engagement. DEFINING TERMS Through interviews with TP stakeholders, it became clear that there was a need to further refine and break down key terms and elements of the strategy. In addition to sequencing/layering elaborated under EQ2, multiple respondents cited the need for elaborating and better contextualizing TP’s definitions of resilience and vulnerability as the program evolved. For example, one portfolio respondent stated that, “we defined [resilience] as mitigating impact, reducing vulnerability, and promoting recovery, but we didn’t define reduced vulnerability. Defining vulnerability in such a complex and fluid environment is not easy." While it may not be possible to comprehensively define all ToC terms, the program would have benefitted from continuing to refine definitions on a more regular basis so that TP could have benefitted from a stronger programmatic focus. 16 | Mozambique Final Program Evaluation Report USAID.GOV RECOMMENDATIONS In this section we provide recommendations, focusing on programmatic targeting and the usage of complementary programming. We conclude with recommendations for achievement of TP outcomes in the future, segmented by shorter and longer-term interventions. COMMUNITY TARGETING Vulnerability and resilience factors are individualized, products of interactions between complex systems (e.g., psychological, interpersonal, socio-economic), and clustered at the community level. To leverage shared resilience factors and target localized vulnerabilities, SI recommends that USAID focus programming at the community level. We present a model ToC for such a strategy in Annex I. Develop a community targeting approach that considers strategic objectives and associated tradeoffs. There are fundamental differences in programming in urban and rural communities, including but not limited to the size of the potential beneficiary pool, operational cost, economic/social assets, and potential partners. If the objective is to affect broad-based knowledge or attitudinal changes, there is a need for a regional focus. If the objective is to optimize the number of beneficiaries, leverage network spread of information, or build upon existing assets (partners, economy, etc.), USAID should have an urban focus. Finally, if the objective is to optimize substantive community effects, we recommend a rural focus. Incorporate community assessments throughout the lifecycle of program and activity design, not just at the design phase. Engaging with communities should be part of a continuous feedback loop that informs USAID programming to be adaptive and context specific. The most important topics are community resilience/vulnerability factors, local government support/opposition to planned activities, the existence of other programming/non-governmental organizations, and rapid market assessments. BENEFICIARY TARGETING Use design considerations that mitigate the occurrence of elite capture, perceptions of favoritism, and the unintended creation of conflict in beneficiary targeting. Strategies to help refine mitigation measures could include: (i) verify the appropriateness of selection criteria with community stakeholders, (ii) publicize selection criteria for public activities, (iii) allow local government to suggest candidates, but conduct open calls for applications, (iv) as appropriate, select beneficiaries through public lottery, (v) verify adherence to selection criteria before the activity starts, and (vi) as appropriate, provide consolatory activities/support to eligible but not selected applicants. SEQUENCING AND LAYERING OF PROGRAMMING Investments can be amplified through repeat and sustained programming with the same communities, individuals, and grantees. These efforts build trust, allow for programmatic learning/adaptation, and increase the likelihood of outcome achievement. SI recommends that USAID design complementary activities in response to community resilience/vulnerability factors. To the extent possible, and only when USAID understands the local context, this should take the form of fewer, larger activities. However, in conducting repeated programming, particularly with the same individuals, it is important to periodically assess the cost-efficiency of investments. USAID.GOV Mozambique Final Program Evaluation Report | 17 PROGRAMMATIC RECOMMENDATIONS Table 3: Recommendations for Achieving TP Outcomes, by Time Horizon Outcome Short-term Long-term Increased VE Knowledge On average, beneficiaries of SO1.1 programming were more likely to report improved resilience to VE. Given that these investments are less resource intensive, more scalable, and affect greater numbers of beneficiaries, we recommend prioritizing communications, youth leadership, and community building interventions for future P/CVE programming. Particularly if the objective is to increase VE knowledge, the strongest cost/benefit argument is for communications interventions, drawing on and expanding on the work done by TP (particularly around radio soap operas). Pairing mass media with the leveraging for local, influential voices (e.g., local, business, religious leaders) promote P/CVE messages would create positive complementarities. Given the dynamic nature of the conflict and the pervasiveness of misinformation, USAID should identify and proactively counter particularly dangerous, salient, and/or widespread messages. This approach would be assisted through monitoring media content (traditional and digital). Voices of influential persons/organizations can be further strengthened by building linkages between leaders within and across communities. This could be done by bringing leaders together to discuss approaches and supporting the development of joint strategies and/or messages. Increased Youth Leadership & Engagement USAID should identify and support youth journalists in the production of P/CVE media content. This approach is likely to make messages more resonant with youth while simultaneously promoting youth leadership and leveraging journalists’ peer networks. To engage youth and build their sense of community belonging, USAID should support youth-led community initiatives (e.g., providing seed funding for community/volunteer projects) and organizing community discussions where youth can discuss issues with adults and local leaders. Participation of females and marginalized populations should be promoted (e.g., IDPs). USAID should identify youth-serving organizations in target communities. Where viable organizations exist (e.g., community centers, youth clubs), they could be strengthened to increase the type and/or quality of offerings or to serve a broader beneficiary base (e.g., IDPs). Where there are no organizations, these could be created (e.g., informal networks, associations). Building mutual understanding and leadership could be supported through the organization of a youth congress at the regional level. Youth representatives could relay peer priorities/questions to other youth leaders. Reduced Economic Marginal￾ization Provision of skills in marketable trades was shown to reduce economic marginalization among some beneficiaries. In locations where public or private TVET providers operate, USAID should facilitate youth enrollment (e.g., stipends) ideally with the intentional pairing of youth from different communities, and religious/ethnic groups. Where there are no TVET providers, USAID should conduct rapid market assessment to determine demand and the provision of trainings (possibly supplemented by productive assets like sewing kits) to youth. Reducing economic marginalization was most effective through organizations (e.g., youth clubs, fishermen’s associations). We recommend strengthening/supporting promising organizations and facilitating the creation of organizations where there it makes economic sense. Support TVET providers where they exist (e.g., curriculum modernization, asset transfer, building repair). Where they do not, explore feasibility of facilitating creation of regional centers in cooperation with government. 18 | Mozambique Final Program Evaluation Report USAID.GOV ANNEX I: MODEL TOC Illustrative Community resilience to VE Improved understanding of VE USAID.GOV Mozambique Final Program Evaluation Report | 19 ANNEX II: ACTIVITY SAMPLING We selected 24 activities across four districts through a six-step, participatory approach: 1- Preliminary activity analysis based on the programmatic information from the OTI Activity Database using filtering criteria. 2- Cross checking the preliminary activity analysis with the TP Mindmap provided by OTI and reflective of the universe of activities that fell within SO1.1 and SO1.2 in the zones of interest. 3- Six Consultative interviews with key representatives of OTI and TP. 4- Primary selection of activities based on eight variables including: whether the activity was referenced as a priority in USAID discussions, whether it was referenced as problematic for selection by OTI, whether it was referenced as a priority in OTI discussions, whether the activity was implemented as part of a sequenced/layered activity chain, the total budget (estimated), the total number of beneficiaries (planned), the total time (activity start-end dates), and whether the activity was likely to have sufficient implementation progress by the time of data collection (defined as being beyond the “pending” stage of implementation). Priority was given to activities highlighted by OTI and those that were in sequencing/layering chains. Further activities were added to reflect secondary selection variables. 5- A design workshop with OTI to discuss the sampling strategy and agree on a final sample of priority activities. 6- The evaluation team created a tool that aggregated selected activities across district to facilitate in￾person data collection. The below table presents the full activity sampling frame, the subset of activities that were selected: SO PEMBA NAMPULA Selected Omitted Selected Omitted 1.1 PEM001 PEM041 NAM007 NAM005 PEM014 PEM038 NAM015 NAM010 PEM028 PEM022 NAM019 NAM011 PEM039 PEM036 NAM004 NAM018 PEM018 PEM044 NAM013 PEM016 NAM016 PEM015 NAM014 PEM046 PEM018 NAM017 1.2 PEM029 PEM022 NAM001 PEM036 NAM012 PEM044 NAM009 NAM003 NAM008 NAM002 NAM006 20 | Mozambique Final Program Evaluation Report USAID.GOV ANNEX III: DATA COLLECTION APPROCHES QUALITY ASSURANCE PROCESSES AND LIMITATIONS DATA COLLECTION APPROACHES: Document review: The evaluation team reviewed and analyzed programmatic documents21F 23 at the country, zone, and activity levels, including independent access to the OTI Activity Database. Portfolio Key Informant Interviews: The evaluation team leader conducted 10 telephone based, KIIs with OTI, USAID/Mozambique, and DAI/TP respondents. Respondent sampling was done in cooperation with USAID to select the most informed and central TP stakeholders. District Key Informant Interviews: The Forcier team conducted 23 KIIs with TP’s Field Advisors, TP grantees, and local government. Respondent sampling was done to select all Field Advisors within the study’s geographic areas, all grantees implementing activities within the sub-portfolio under review, and the most important government respondents involved in the implementation of TP activities. Beneficiary Focus Group Discussions: The Forcer team conducted 19 beneficiary FGDs with a total of 88 respondents. Sampling was conducted at two levels. First, within each district, activities were sampled based on size, importance,22F 24 and the existence of discrete beneficiaries. Within sampled activities, respondents were selected from beneficiary lists provided by TP. Random stratified sampling within lists was attempted but due to challenges in soliciting respondents, the Forcier team relied on convenience sampling to fill some FGDs (i.e., interviewing who was willing and available). Beneficiary Key Informant Interviews: In some instances, the Forcier team was not able to successfully schedule and/or complete beneficiary FGDs. Given the short time allocated for fieldwork, the evaluation team shifted some beneficiary FGDs to KIIs to ensure representation. Community Focus Group Discussion: The Forcier team conducted one FGD per community to solicit perceptions of non-beneficiaries. The sampling strategy was to get a broadly representative cross￾section of the community (e.g., age, sex, socio-economic status). Total number of KII and FGD respondents, by Interview Type Respondent Type KII FGD TOTAL Portfolio (USAID/TP) 10 0 10 Field Advisor 3 0 3 Grantee 14 0 14 Local Government 7 0 7 Beneficiary 25 88 113 Community Member 0 10 10 TOTAL 59 98 158 23 Documents provided by OTI included strategy documents, final activity evaluation reports, activity notes, etc. 24 Criteria included whether activities were part of programmatic sequences, were perceived as important/successful by portfolio respondents and were significant in dollar and/or beneficiary terms. To balance the sample, activities that were determined not to be successful were also included. USAID.GOV Mozambique Final Program Evaluation Report | 21 QA PROCESSES Procurement SI competitively procured a DCF subcontractor, selecting an organization with strong experience in Mozambique, having researchers from the sampled zones, and experienced in conducting FGDs and KIIs in Mozambique. The QA practices for the procurement phase are outlined below: ● Request for Proposals (RFP): SI prepared an RFP for OTI/Mozambique Final Program Evaluation in Mozambique in-country data collection, which was reviewed by EnCompass and approved for release on 07/14/2021 on the SI website. SI also sent the RFP directly to any local firms operating in Mozambique based on SI’s general knowledge of research firms in Mozambique. The offerors submitted questions and answers on RFP, and SI responded to them prior to the submissions. ● Selection Committee: the selection committee members independently reviewed and scored the technical proposals based on the information provided on Personnel, Technical Approach, and Past Performance. The cost proposals are scored based on completeness, accuracy & transparency, and reasonableness. ● Concluding the award: upon addressing SI’s requests for clarifications on the proposal, the selection committee concluded that Forcier’s proposal was stronger than the other four proposals in terms of (a) QA; (b) The management plan; (c) Sampling approach; (d) past performance in Mozambique; (e) experience conducting in-person data collection during COVID with well￾designed protocols; and (d) in-country network. Based on the proposal, Forcier has a similar skillset as SI does that can enhance the rigor of the product. Fieldwork Preparation: ● Instrument translation: Ensuring independence between the translation and back-translation staff, the instruments were translated into (and back-translated from) Portuguese and Emakua. Both translated and back-translated versions were vetted by SI study team and Forcier team. ● Training the supervisors and enumerators: 5 days of training three research teams (includes piloting and testing the materials) Data Collection: ● Supervisors reviewed all Portuguese notes submitted each day for quality and detail of responses, verified that the data entered in the notes are complete and accurate, and matches the details found on the fieldwork assignment forms. ● The call-backs were conducted with respondents to confirm the interview took place and edit all completed interviews if necessary. ● The research team reviewed the transcripts within three days after conducting the interviews. ● Supervisors facilitated debrief meetings with all the researchers after every data collection day (depending on internet connectivity) to provide feedback on interview performance, allow researchers to voice questions and concerns, and plan for the next day. ● Supervisors and researchers communicated any persistent data quality issues to the study coordinators. ● Troubleshoot on-the-ground problems with researchers: Even though we did multi-day training and piloting (five days of training, practicing, and piloting for three teams), the DCF partner encountered significant challenges in the field, including the termination of two researchers, multiple call-backs to fill in the missing information. We started with five researchers and one supervisor during the training sessions. To address the QA issues, we ended the fieldwork with three researchers and four supervisors. While supervisors conducted call-backs and reviewed all audio recordings, the quality of interviews was variable. This may be caused as a result of the 22 | Mozambique Final Program Evaluation Report USAID.GOV complex nature of the study with respondents having frequently participated in multiple activities; the respondents were not always able to differentiate between the activities. ● The study team performed QA on English notes, checking for accuracy and completeness. ● When any data quality issues are identified, the English notes be compared with the Portuguese notes to check the accuracy of translations. The research team consulted the supervisors to determine whether the issue is related to translation from local languages or to problems encountered during the interview itself. In some cases, the interview was replaced or had to be re-conducted. LIMITATIONS Breadth vs Depth: The heterogeneity of sampled activities, paired with limited evaluation resources necessitated the development of general instruments that were administered to all respondents, irrespective of activity. While instruments incorporated skip logic to account for variation, the inability to customize instruments to activities limited the depth of information that could be solicited. Challenges with fieldwork: Even with the substantial training and preparatory work (see Annex VII), the Forcier team encountered significant challenges during fieldwork. Some field researchers exhibited poor performance (e.g., challenges adhering to skip logic and interview probes, omission of questions in some interviews, variable quality in documentation) leading to the termination of two personnel. Forcier supervisors were remobilized to the field to conduct in-person oversight and conducted independent call￾backs to respondents whose transcripts were unclear or incomplete. Selection Bias: As with any qualitative study of large, complex programming, the relatively small sample size imposes limitations on the degree to which portfolio, or even activity, generalizations can be made. Furthermore, respondent selection was affected by the fact that, (i) sampling was contingent on having contact information, (ii) not all beneficiary lists contained contact information, and (iii) non-response and refusals were common and unevenly distributed across activities and respondent types. Finally, despite repeated attempts, Forcier encountered difficulty in scheduling interviews with local government officials. Response Bias: Respondents likely adjusted their answers due to the inherent sensitivity of the subject matter and perceptions of what researchers, and by extension USAID, might want to hear. This phenomenon could have led respondents to obscure or overstate VE knowledge. To mitigate these risks, the evaluation team triangulated findings to identify discrepancies in responses, used probes ask for specific examples responses, and clarified the independence of the study in interview informed consent. USAID.GOV Mozambique Final Program Evaluation Report | 23 ANNEX IV: BENEFICIARY QUOTES ON REDUCED VE RECRUITMENT We present select quotes relating TP participation to reduced recruitment. To protect respondent confidentiality, we remove all references to specific activities, locations, names, and other potentially re￾identifying information. Youth Leadership ● "Logging and the production of charcoal are the main sources of income in my community, so once a stranger arrived saying that he wanted young people to explore wood, the young people accepted and were taken to Cabo Delgado, but when they arrived, they stayed a week without working but they received money, so, as they listened to community radio, they heard about insurgents on the radio and then discovered that they were possibly recruited to be part of this group, so they managed to come back to the community with the intention of recruiting others due to the supposed benefits, but I talked to them and I managed to change their minds and they didn't go anymore.” ● “I shared with a lot of people: my family, my kids, my friends in the village, and with a lot of people who are clients of my [business]. It was good, because people thanked me a lot for sharing the information, as they could also prevent themselves from the attacks. I even saved a man who had a false job promise in [location], but he did not go after my explanation about Al-Shabab recruitment methods, as he realized that the job offer was a lie. Now he is grateful to me because in fact he even heard about the Al-Shabaab from a lot of people.” ● “We were able to rescue three young people who were going to Cabo Delgado” Economic Marginalization ● “First, thanks God for not being killed in Cabo Delgado, with what I learned, I no longer think about joining VE… because I already have an occupation thanks to the project…we have managed to change the way young people think about joining VE… knowing the extreme consequences to which the ones there are subjected.” ● “This information changed the view of their spouses about the insurgency, in that [location] district had the most people recruited for the insurgency. In the beginning they built mosques, then later after the followers disappeared there was the first attack in Cabo Delgado. A few days after the attack, those people returned with a lot of money and started businesses and things returned to normal. They went to Cabo Delgado and each time would return with more money and grew their businesses and the attacks continued happening. It became so normal that we in the community knew what was going on, and recruitment was happening even in the streets. By then some people left the district for fear of retaliation. When TP appeared in the district, some of those people helped with the awareness raising meetings. Since then, their view about VE has changed because they saw that what they were doing wasn't good and wasn't a solution, and now we don't see recruitment for VE in Cabo Delgado.” ● “I´m a single woman and seeing other women married, I thought about joining the VE because of the promises they make. So, thanks to TP project, those thoughts are gone, I now know I don´t want to get involved.” 24 | Mozambique Final Program Evaluation Report USAID.GOV VE Knowledge ● “My opinion about the insurgency has changed because before I could be easily enticed with promises of work in Cabo Delgado but not anymore. When this insurgency thing started, people were recruited to go to work and in a short time they came back rich. At this point everybody wanted to go to Cabo Delgado, even me, but nowadays, thanks to the project, I don't think about going to Cabo Delgado anymore. And even before the project arrived, we already heard everything that was happening in Cabo Delgado through the press (radio and television). We saw on the internet people being decapitated, then we got a notion of what it was about and this contributed to change our opinion.” ● “[The TP activity] changed their views yes, from the day one of the training since today, we haven't had a person who travelled away from home. Previously, not a single month would go by without us hearing ‘this guy's son or nephew has departed [to Cabo Delgado]’. Now we are communicating more with each other, when someone leaves we know where this person is going.” USAID.GOV Mozambique Final Program Evaluation Report | 25 ANNEX V: DISAGREEMENT WITH TP TOC ASSUMPTIONS, BY RESPONDENT TYPE AND STRATEGIC OBJECTIVE In the table below we present the proportion of respondents who either disagreed or strongly disagreed with each assumption in the PO1 ToC. Findings are disaggregated by respondent type. This analysis provides insight on the extent to which key assumptions underpinning the program logic were shared by TP stakeholders. 26 | Mozambique Final Program Evaluation Report USAID.GOV ANNEX VI: SCOPE OF WORK Technical Approach Evaluation Approach The situation in Mozambique’s Cabo Delgado, and OTI’s response through Tuko Pamoja, is continually evolving. To ensure the evaluation’s success, the team will emphasize co-creation with the OTI team to identify and build on strengths and enhance ownership of the evaluation process. This collaborative is especially useful in sensitive programming environments—such as addressing P/CVE in Cabo Delgado. The evaluation will use approaches that capture both intended and unintended outcomes through a combination of interview tools based on SO 1.1 and SO 1.2, and more open-ended outcome harvesting techniques. This combination will elicit the breadth of changes seen because of the program while also clearly assessing what seems to be working and what is not, to guide future P/CVE programming. Evaluation Methodology Evaluation Start-Up Activity Kickoff and Orientation. The evaluation team will begin by facilitating a virtual kick-off meeting with the OTI program team and Monitoring, Evaluation, and Learning (MEL) Management team to develop a shared understanding of the evaluation’s goals, as well as the program team’s concerns, hopes, and expectations of the process. We will clarify timing, roles, communication structures and other logistical considerations. We will request OTI provide relevant documents, set timing for any OTI orientations to take place, ensure access to the OTI Activity Database, and outline next steps for setting up design interviews and preparations for fieldwork. Having a team with a strong understanding of OTI’s operating approaches, core principles, and MEL structures will be essential to an essential evaluation. Senior team members new (or returning) to OTI will participate in an OTI orientation throughout the design phase. Evaluation Planning and Design. A thoughtful evaluation design to guide data collection and reporting is critical. After an initial review of relevant documents, we propose a half-day participatory evaluation design workshop with OTI’s program team who can provide input to inform the evaluation design. The goals of this workshop are to get an in-depth understanding of the SOs under consideration; explore options for fieldwork and geographies; and discuss possible intended, unintended and incremental outcomes of the program. We will also review and hone evaluation questions, explore the implementation context, prioritize areas of focus, and get input on data collection strategies. The workshop is envisioned to be conducted in English and has been budgeted accordingly. The workshop has been budgeted to take place virtually to maximize attendance considering ongoing COVID travel restrictions and in consideration of budget. USAID.GOV Mozambique Final Program Evaluation Report | 27 Before or after the participatory design workshop, we will conduct an additional 5-10 design interviews with the program team and other relevant OTI staff, implementing partners and other key stakeholders identified in collaboration with OTI who are well placed to inform our evaluation methodology. Evaluation Methodology Report and Document Review. With the inputs gathered during design processes, we will develop an evaluation methodology report for OTI’s review and finalize based on OTI inputs. The methodology report will include a review of existing documentation and identify gaps to be filled through data collection. It will also outline the evaluation purpose and questions, sampling strategies and final proposed sample, data collection methods and logistics and draft data collection instruments. The report will also facilitate a common understanding of the evaluation activities among various stakeholders. There will also be an option to revisit the activity budget at this stage to account for any substantial changes in scope or approach. Institutional Review Board. The sensitive nature of OTI’s programming in Mozambique and the ongoing context of the COVID-19 pandemic mean extra caution is needed for any primary data collection, especially in-person. EnCompass has an institutional review board (IRB) to oversee evaluation activities for compliance with standards pertaining to the protection of human subjects. EnCompass’ IRB will review the evaluation plan, data collection tools, and the report to ensure the evaluation process aligns with established ethical research standards. The Project Manager, with oversight from the Project Director, will oversee the process of IRB review and will ensure that the review does not delay evaluation activities. Illustrative Evaluation Design A detailed evaluation design will be developed in collaboration with the OTI team and other stakeholders. We anticipate that our evaluation will follow a qualitative approach that will focus on answering the draft evaluation questions: What are the primary outcomes of programming under our PO1 youth objective? Did outcomes vary by zone (PEM and Nampula)? What are the unintended outcomes of our programming under our PO1 youth objective? Did TP sequencing and layering within geographic zones help us improve the likelihood of achieving outcomes? Do the evaluation findings at SO levels 1.1 & 1.2 support the respective hypotheses (ToC), and validate program assumptions? The team will address what elements of the programming have been most effective in supporting P/CVE and how this might affect next steps for OTI, USAID Mission, or other United States Government programming. Data collection methods 28 | Mozambique Final Program Evaluation Report USAID.GOV We anticipate that our evaluation activities will include three major components: document review, KIIs, and FGDs. Document review. The evaluation team will conduct a thorough review of relevant OTI activity documentation including reports from the OTI Activity Database; progress reports prepared by implementing partners; and other survey or research conducted in Mozambique by OTI or other stakeholders including cluster evaluations. Documents to be reviewed will largely be provided by TP and OTI, though the evaluation team may also include external or publicly available literature and databases. The team will also review any documents recommended during fieldwork and will incorporate them into findings. Key Informant Interviews. KIIs will focus on lines in inquiry tied to SO 1.1 and SO 1.2, the sequencing and layering of activities, and unexpected/unanticipated outcomes. Focus Group Discussions. FGDs will bring respondents together to discuss feedback on SO 1.1 and SO 1.2 to validate or invalidate expected outcomes and to capture a broad range of unexpected outcomes. Focus groups will also capture data on perceptions around the sequencing and laying within geographic zones, and the most effective approaches and sequences. Focus groups will be gender disaggregated and include at least one FGD for male youth and one for female youth in each location. FGDs will be facilitated to encourage the greatest interaction among participants. Site selection and sampling Tuko Pamoja has a wide variety of stakeholders who could speak to the evaluation questions. Design interviews with OTI staff will help the evaluation team select respondents best suited to fill gaps in data. It is critical for the evaluation to collect data in the two primary regions where activities are taking place: Nampula and Pemba. The evaluation team plans for data collection to take place in person (pending continued improvement in COVID-19 and no major security restrictions), conducting interviews and FGDs in two communities in each location (four communities total). The team has budgeted to conduct approximately 12-15 KIIs and 3-4 FGDs in each community, or roughly 3-4 interviews or 2 FGDs per data collection day. This will result in a total of 48-60 KIIs and 12-16 FGDs. Interviews and FGDs will each have appropriate gender representation. Respondents will include: Community members and program participants OTI program staff (HQ and local) Local community leaders and government officials Other stakeholders We present two data collection scenarios below: USAID.GOV Mozambique Final Program Evaluation Report | 29 Preferred Approach: Data collection in local communities will be conducted by a local firm familiar with the context and with past experiences in qualitative data collection in Mozambique. The local DCF will recruit data collectors and work with the Project Director and Evaluation Specialist to train data collectors according to quality standards. The DCF will then complete logistical planning and execute data collection and quality control activities according to protocols developed by SI. The local firm will be responsible for transcribing and translating all the KIIs/FGDs notes. The local firm will provide the ET with a clean, labeled dataset and regular updates about data collection progress. In this scenario, the Project Director and Evaluation Specialist would still conduct interviews with OTI staff, IP staff, and relevant high-level stakeholders following the training for data collectors. This approach is recommended due to the complex operating environment and the lack of strong evaluation capacity in the targeted regions of Mozambique. Collaborating with a DCF in Mozambique is more likely to produce higher quality interviews, higher quality notes, fewer delays, and better security for all parties. Alternate Approach: Under this scenario, data in local communities will be conducted by local data collectors familiar with the context. Two data collectors will receive training in qualitative data collection and complete piloting with the Data Collection Manager and the Team Leader. The team will spend five days collecting data in each location, with one day for travel between communities each week. The Team Leader would conduct the data collection training alongside a local Data Collection Manager who will provide ongoing QA. After the training, the TL will conduct interviews with OTI staff and relevant high-level stakeholders. At this time, we are budgeting for in-person KIIs and FGDs, though some KIIs may be conducted virtually depending on respondent preference and availability and any applicable local restrictions related to COVID￾19. Data Synthesis, Analysis, and Reporting Regular Communication with OTI. The evaluation team will regularly meet with the OTI team to discuss the evaluation status, next steps and proactively address any issues. The calls may be conducted weekly or biweekly, depending on the stage of the evaluation. In addition, the SI evaluation team will provide weekly bulleted emails to the OTI team that summarize completed and pending actions. Data Processing. After the data collection is complete, the evaluation team will organize and consolidate data in one encrypted database for secure data analysis and synthesis. Recorded interviews will be transcribed and translated into English for analysis and QA. Analysis. The team will develop a codebook using a combination of pre-set codes mapped to Evaluation Questions, Strategic Objectives, and emergent codes derived from document review, remote KIIs, and review of community FGD and KII notes. The team will use Dedoose software to analyze qualitative data, following a rigorous procedure for data cleaning, coding, inter-coder reliability checks, and QA procedures. All transcripts and notes will be uploaded to Dedoose with each KII and FGD linked to a set of descriptors based on the team’s codebook. Use of 30 | Mozambique Final Program Evaluation Report USAID.GOV descriptors in Dedoose enables a mixed-methods approach to the study and allowed the ET to analyze data based on a set of specific characteristics. Interpretation workshop. After the evaluation team has done initial analysis, we will conduct a 4- hour interpretation workshop with OTI staff including program team and implementing partners to share, ground-truth and contextualize the emerging findings with the partners who deeply understand the activities and the context. This session can also identify additional areas of analyses and priority themes for reporting and further analysis. The evaluation team and stakeholders can also use this session to collaboratively develop initial findings, brainstorm conclusions, and reflect on recommendations. Prior to the interpretation workshop, the evaluation team will prepare a document presenting emerging findings with supporting data for the review of the session participants. We have budgeted for the interpretation workshop to take place virtually (may take place over two days as two 2-hour workshops) and to take place in English. Draft and Final Report. Following the interpretation workshop, the evaluation team will develop the first draft of the evaluation report with findings, conclusions, and recommendations for OTI’s review. The evaluation team will ensure the timeline allows for two rounds of revisions. The evaluation team will deliver the final report after incorporating feedback from OTI. We have also budgeted to develop an evaluation summary/memo with key findings and lessons for translation in Portuguese for wider dissemination. Result Presentation. After the first draft of the report has been finalized, and before the submission of the second draft of the report, the evaluation team will hold a Result Presentation. The ET will present key findings, conclusions and facilitate a discussion to identify the next steps for the implementation of recommendations. The overall objective of this session is to maximize the utility of the second draft of the report for OTI and USAID. We have budgeted for one 2- hour validation session. Team Composition Role Relevant skills Responsibilities Project Director & Team Lead ▪ Understanding of OTI programs ▪ Extensive experience in leading and managing evaluations in conflict-affected areas on sensitive topics ▪ Skilled in qualitative data analysis and proposed methodologies (KIIs, FGDs, outcome harvesting) ▪ Experienced evaluator with technical skills in qualitative analysis and participatory methods ▪ Responsible for quality and timeliness of deliverables Lead all evaluation activities such as design, data collection, and report writing Supervise all aspects of the evaluation Coordinate IRB process Lead various meetings and workshops by co-developing the design of the meetings, reviewing and providing scripts for the meetings, and facilitating Participate in primary data collection Work closely with the DCF USAID.GOV Mozambique Final Program Evaluation Report | 31 Role Relevant skills Responsibilities Provides senior-level technical and managerial support to ensure overall quality Supports Project Manager in budget and workplan management and personnel management as necessary Evaluation Specialist/Project Manager (SI) Previous experience managing OTI evaluations Extensive experience managing evaluations in complex environments on politically sensitive topics Support qualitative data analysis through codebook development, preparing templates for analysis, and providing QA for qualitative coders Co-facilitate interpretation and validation workshops Manage evaluation activities and day-to-day aspects of the budget and work planning Coordinate IRB process Set agendas and prepares scripts for team meetings, client meetings, and participatory workshops Recruit and manage local DCF, serving as liaison between the Team Leader and the DCF to ensure delivery of high￾quality data Provide internal QA review Research/Project Assistant (SI) Experience coding qualitative data using analysis software Experience supporting all logistical and administrative tasks needed for this evaluation Conduct coding using Dedoose of qualitative data Provide logistical and administrative support to the evaluation team, including scheduling interviews, providing technology solutions/SharePoint and email access, and notetaking Recruit and contract local DCF Manage consulting contracting, invoicing, and travel and logistics support Data Collection Firm Operate in Mozambique Technical ability to provide the service with high quality Past performance showing similar data collection services provided in the country Knowledgeable of local context in Nampula and Pemba Execute the data collection (KIIs and FGDs) Recruit and train the data collectors with support from the Team Leader Transcription and translation of the KIIs/FGDs notes Completing and submitting a clean, labeled dataset Subject Matter Expert ▪ Understanding of OTI programs ▪ Experience in conducting research/evaluations in conflict￾affected areas on sensitive topics ▪ Extensive knowledge on P/CVE, preferably in Mozambique Provides subject matter expertise to team members over the lifetime of the evaluation Contribute to the development of the evaluation plan, approaches, and methods Contribute to analysis and the development of evaluation report 32 | Mozambique Final Program Evaluation Report USAID.GOV Role Relevant skills Responsibilities ▪ Skilled in qualitative data analysis and proposed methodologies (KIIs, FGDs, outcome harvesting) Tentative Workplan The below workplan prioritizes a start to data collection that will not risk the departure of key OTI respondents, while meeting OTI’s desire to conclude primary activities before the end of November. Critical junctures for timeline consideration include: ● Timeline to recruit evaluation team members ● Timeline to identify local data collectors or local DCF and complete procurement procedures ● IRB review and approval of data collection tools and procedures ● Timeframe for data collection (depending on team structure) and timeframe for receipt of transcribed, translated data The team’s priority is to deliver useful results to OTI in a timeline that will support use and further decision-making. The below timeline proposes holding an interpretation workshop after preparing a preliminary findings brief, prior to the draft report. This will ensure that the 2nd draft of the report (envisioned near the middle of November) is close to final and can be used for decision making. The final version of the report and the evaluation summary will be submitted in December to allow sufficient time for thorough copy editing, formatting compliance, and finalization without delaying critical information for OTI stakeholders. Task Due Date Finalization of SOW and budget June 29, 2021 Recruitment completed and core evaluation team onboarded July 9, 2021 Kick-off meeting with core team members July 12, 2021 Design interviews, document review, and orientations July 12-20, 2021 Draft evaluation methodology report July 23, 2021 OTI review July 23-30, 2021 Final evaluation methodology report August 6, 2021 Preparation for fieldwork completed including finalization of data collection guides, contracting of local data collectors or DCF, IRB review August 20, 2021 Piloting, in-brief presentation, and data collection August 23 – September 24, 2021 Data analysis and preliminary findings (including interpretation workshop to be held during the late part of analysis) September 27 – October 8, 2021 Draft report v1 October 22, 2021 OTI review of draft report October 22 – October 29 USAID.GOV Mozambique Final Program Evaluation Report | 33 Task Due Date Results Presentation November 9, 2021 Draft report v2 November 12, 2021 OTI presentation (OTI only) November 15, 2021 OTI review of draft report v2 November 15 – November 26 Final report and evaluation summary December 10, 2021 34 | Mozambique Final Program Evaluation Report USAID.GOV ANNEX VII: DATA COLLECTION TOOLS CONSENT FORM FOR KEY INFORMANT INTERVIEWS: Hello. My name is [name] and I am here with my colleagues [names] on behalf of Social Impact. We are independent evaluators, conducting an evaluation of the Mozambique Final Program Evaluation, funded by the United States Agency for International Development (USAID) and implemented by the Office of Transitions Initiatives (OTI). The information we collect will provide USAID/OTI with lessons from the portfolio of investment to inform future OTI and/or Mission programming. We have been contracted by EnCompass to conduct an independent evaluation of this project. We are not affiliated with and do not represent either USAID or OTI. We have invited you to participate in a KII, lasting no more than 60 minutes, to discuss the impact of the Mozambique Final Program. You were selected given your involvement with the program or its alumnae. If you choose to participate you would be one of approximately [#] stakeholders participating as part of this evaluation. Your participation in this study is completely voluntary, and you are under no obligation to participate. You may leave the discussion at any time for any reason, or can decline to answer any of our questions, without penalty. There are no direct benefits or risks to participating, other than the time you could spend doing other things. The information you share with us will be aggregated and analyzed along with information shared by others and will be presented in a report in consolidated format. The report will eventually be made public on USAID’s website but no individual names or information that could identify you will be provided in the report. Anonymized transcripts will be provided to USAID/OTI, but will not be made public. We ask that all focus group participants respect each other’s privacy and agree not to share anything that is discussed with anyone outside of this group once this conversation ends. Remember that you are free to refuse to answer any questions. With your permission, we would like to record this discussion strictly for note-taking purposes and making sure that we have captured the discussion notes accurately. Only the evaluation team members will have access to it. After our notetaking and analysis is complete, we will delete the recording. If you would not like to be recorded, we can proceed with the discussion without the recorder. If you have any questions or concerns about your rights as a participant, you may contact [name and affiliation of local contact] at [number and/or email] or the Social Impact IRB at irb@socialimpact.com or +1 703 465 1884. I can leave a copy of this form with you if you would like. Do you have any questions? [ ] Yes [ ] No Do you agree to participate in this study? [ ] Yes [ ] No Agreed to recording? [ ] Yes [ ] No USAID.GOV Mozambique Final Program Evaluation Report | 35 CONSENT FORM FOR FOCUS GROUP DISCUSSIONS: Hello. My name is [name] and I am here with my colleagues [names] on behalf of Social Impact. We are independent evaluators, conducting an evaluation of the Mozambique Final Program Evaluation, funded by the United States Agency for International Development (USAID) and implemented by the Office of Transitions Initiatives (OTI). The information we collect will provide USAID/OTI with lessons from the portfolio of investment to inform future OTI and/or Mission programming. We have been contracted by EnCompass to conduct an independent evaluation of this project. We are not affiliated with and do not represent either USAID or OTI. We have invited you to participate in a FGD, lasting no more than 90 minutes, to discuss the impact of the Mozambique Final Program. You were selected given your involvement with the program or its alumnae. If you choose to participate you would be one of approximately [#] stakeholders participating as part of this evaluation. Your participation in this study is completely voluntary, and you are under no obligation to participate. You may leave the discussion at any time for any reason, or can decline to answer any of our questions, without penalty. There are no direct benefits or risks to participating, other than the time you could spend doing other things. The information you share with us will be aggregated and analyzed along with information shared by others and will be presented in a report in consolidated format. The report will eventually be made public on USAID’s website but no individual names or information that could identify you will be provided in the report. Anonymized transcripts will be provided to USAID/OTI, but will not be made public. We ask that all focus group participants respect each other’s privacy and agree not to share anything that is discussed with anyone outside of this group once this conversation ends. Remember that you are free to refuse to answer any questions. With your permission, we would like to record this discussion strictly for note-taking purposes and making sure that we have captured the discussion notes accurately. Only the evaluation team members will have access to it. After our notetaking and analysis is complete, we will delete the recording. If you would not like to be recorded, we can proceed with the discussion without the recorder. If you have any questions or concerns about your rights as a participant, you may contact [name and affiliation of local contact] at [number and/or email] or the Social Impact IRB at irb@socialimpact.com or +1 703 465 1884. I can leave a copy of this form with you if you would like. Do you have any questions? [ ] Yes [ ] No Do you agree to participate in this study? [ ] Yes [ ] No Agreed to recording? [ ] Yes [ ] No 36 | Mozambique Final Program Evaluation Report USAID.GOV Tool: KII With DAI and USAID (Portfolio-level) # Question Probe EQ We will start by asking you about what you think are the most important outcomes of the TP activities implemented under PO1. SO1.1 states, “Youth leaders have increased capacity to promote VE resilience.” 1 What do you think were the most important outcomes/results of programming among program participants (in other words direct beneficiaries)? Heterogeneity Enabling/Inhibiting factors EQ1 2 What do you think were the most important outcomes/results of programming at the community level (in other words indirect beneficiaries)? Heterogeneity Enabling/Inhibiting factors EQ1 3 Which activities in this SO do you think were most successful at affecting community-level outcomes? Why EQ1 4 To what extent were there important unintended or surprising outcomes (both positive and negative) of programming in this SO? Elaborate EQ1 5 To what extent do you think TP succeeded in achieving this objective (SO1.1)? What were some specific examples? Why EQ1 6 What were the primary challenges in achieving this objective? 7 To what extent did success vary between Pemba and Nampula zones? EQ1 SO1.2 states, “Youth’s feelings of economic marginalization are reduced” 8 What do you think were the most important outcomes/results of programming among program participants (in other words direct beneficiaries)? EQ1 9 What do you think were the most important outcomes/results of programming at the community level (in other words indirect beneficiaries)? 10 Which activities in this SO do you think were most successful at affecting community-level outcomes? Why EQ1 11 To what extent were there important unintended or surprising outcomes (both positive and negative) of programming in this SO? Elaborate EQ1 12 To what extent do you think TP succeeded in achieving this objective (SO1.2)? What were some specific examples? EQ1 13 What were the primary challenges in achieving this objective? 14 To what extent did success vary between Pemba and Nampula zones? Why EQ1 USAID.GOV Mozambique Final Program Evaluation Report | 37 We will now talk about activity sequencing/layering. (DISPLAY THE SIMPLIFIED MIND-MAP) 15 Looking at these groupings, which activity sequences do you think were most effective in achieving outcomes? Why EQ2 16 Which activity chains do you think were least effective in achieving outcomes? Why EQ2 17 How successful do you think the sequencing strategy has been in achieving SO or PO objectives? Why EQ2 I will read you a series of statements about the project logic. Please tell me the extent to which you agree with these statements using a 4-point scale. 1 is strongly disagree, 2 is disagree, 3 is agree, 4 is strongly agree. You can also respond “I don’t know” if you feel you don’t have enough information. 18 Youth are influential in their communities and with their peers. (If 1 or 2) - why? EQ3 19 Youth lack the capacity to affect positive change around P/CVE. (If 1 or 2) - why? EQ3 20 Youth lack skills and knowledge to P/CVE. (If 1 or 2) - why? EQ3 21 Proving youth leaders with P/CVE skills and knowledge helps these leaders promote VE resilience among their peers. (If 1 or 2) - why? EQ3 22 There are legitimate economic opportunities available for youth, but they don’t know about them. (If 1 or 2) - why? EQ3 23 Perceived lack of economic opportunities increases youth’s economic marginalization and vulnerability to joining or supporting VE. (If 1 or 2) - why? EQ3 24 Some possible legitimate income generating activities need TP support to be initially viable. (If 1 or 2) - why? EQ3 25 IS-M is providing economic incentives to entice vulnerable youth to voluntarily join/support VE. (If 1 or 2) - why? EQ3 26 Making youth aware of economic opportunities or facilitating the viability of these opportunities can reduce youth’s feeling of economic marginalization? (If 1 or 2) - why? EQ3 27 If TP were extended another year, how would you revise the PO1 TOC to more accurately address the problem set? SO1.1 SO1.2 Why EQ4 Looking back at TP, from inception to today, what do you think are the most important lessons USAID should learn as they relate to: 28 Future USAID programming in countering CVE 29 Future USAID programming in Northern Mozambique Is there anything important we didn’t ask you about or anything else you wanted to say? 38 | Mozambique Final Program Evaluation Report USAID.GOV Tool: KII With field advisors, grantees, and government officials (community-level) # Question Probe EQ 1 Please tell us about your involvement with TP. Length Specific role TP implemented x activities in this area. For this study, we are focusing on activities xyz. 2 Could you please tell us which of these activities you were personally involved in, which you are aware of, and which you are unaware of? I will now ask you three questions about each of the activities you were involved in or aware of: 3 ACTIVITY 1: In what specific ways did this project benefit people who were directly involved with it? (i.e., direct beneficiaries)? Elaborate EQ1 4 ACTIVITY 1: To what extent did the project have impacts beyond those directly involved? In other words among other members of the community (i.e., indirect beneficiaries). Elaborate Among whom EQ1 5 ACTIVITY 1: Were there any unexpected or surprising impacts from this activity? These could be good things or bad things. Elaborate Among whom EQ1 REPEAT 3 QUESTIONS FOR ALL OTHER ACTIVITIES MENTIONED BY RESPONDENT IN Q2 Some TP activities were designed as stand-alone projects while others were intended as connected sequences of projects. We will now ask for your insights about (INSERT NUMBER) of these project groups. 6 CHAIN 1: To what extent did these projects build on one another? How Why not EQ2 7 CHAIN 1: To what extent do you think this grouping of projects led to more or better impacts? EQ2 REPEAT QUESTIONS FOR ALL OTHER CHAINS MENTIONED BY RESPONDENTS IN Q1 8 (IF MORE THAN ONE PROJECT CHAIN) Which of these project groups was the most impactful for participants (i.e., direct beneficiaries)? Why 9 (IF MORE THAN ONE PROJECT CHAIN) Which of the project groups do you think was the most impactful for the community (i.e., indirect beneficiaries)? Why We began by exploring individual projects. We then spoke about project groupings. We will now ask you some questions about the full set of activities implemented by TP in this area. USAID.GOV Mozambique Final Program Evaluation Report | 39 10 One of the most important goals of TP was to increase the capacity of youth leaders to promote resilience to violent extremism among their peers (in other words reduce the level of youth sympathy for the insurgency, provision of material support to insurgents, and/or likelihood of joining the insurgency). What would success look like for this goal in your community? Examples EQ1 11 To what extent did TP help achieve any of these objectives? Elaborate EQ1 12 Which of these successes do you think was most important? Elaborate 13 Another of the most important goals of TP was to reduce youth’s feelings of economic marginalization (in other words youth feelings of being excluded from jobs). What would success look like for this goal in your community? Examples EQ1 14 To what extent did TP help achieve any of these objectives? Elaborate EQ1 15 Which of these successes do you think was most important? Elaborate Finally, we would like to ask your opinion about TP’s project logic. I will read you a series of statements. Please tell me the extent to which you agree with these statements using a 4-point scale. 1 is strongly disagree, 2 is disagree, 3 is agree, 4 is strongly agree. You can also respond “I don’t know” if you feel you don’t have enough information. 16 Youth are influential in their communities and with their peers (If 1 or 2) - why? EQ3 17 Youth lack the skills and knowledge to effectively resist VE (If 1 or 2) - why? EQ3 18 Youth lack the skills to help peers resist VE (If 1 or 2) - why? EQ3 19 Proving youth leaders with skills and knowledge helps these leaders promote VE resilience among their peers. (If 1 or 2) - why? EQ3 20 There are legitimate economic opportunities available for youth, but they don’t know about them. (If 1 or 2) - why? EQ3 21 Lack of economic opportunities increases youth’s vulnerability to joining or supporting VE. (If 1 or 2) - why? EQ3 22 Some possible legitimate income generating activities need TP support to be initially viable. (If 1 or 2) - why? EQ3 23 IS-M is providing economic incentives to entice vulnerable youth to voluntarily join/support VE. (If 1 or 2) - why? EQ3 24 Youth are influential in their communities and with their peers (If 1 or 2) - why? EQ3 25 If TP were extended another year or two, how would you recommend revising the project to better help youth resist VE influence (in other words reduce the level of youth sympathy for the insurgency, provision of material support to insurgents, and/or likelihood of joining the insurgency)? SO1.1 SO1.2 Why EQ4 40 | Mozambique Final Program Evaluation Report USAID.GOV Looking back at TP, from inception to today, what do you think are the most important lessons USAID should learn as they relate to: 26 Future USAID programming in countering VE Is there anything important we didn’t ask you about or anything else you wanted to say? Tool: KII With Beneficiaries (Activity-Level) # Question Probe EQ 1 Please tell us about your involvement with Activity X. How long Specific role I will now ask you a series of questions about each activity you were involved in: (IF SO1.1) 2 To what extent did this project increase your knowledge about the insurgency? What specifically did you learn? Elaborate EQ1 3 Was this information helpful to you? If so, how? If not, why not? 4 Did you try to share this information with your peers? If so, how? What worked/didn’t 5 (IF TRIED TO SHARE) To what extent do you think this information changed your peers’ views of the insurgents? Elaborate Among whom 6 (IF TRIED TO SHARE) To what extent do you think this information will affect the likelihood that your peers might support the insurgency (for example with money or information)? Elaborate Among whom 7 (IF TRIED TO SHARE) To what extent do you think this information will affect the likelihood that your peers might join the insurgency? Elaborate Among whom 8 To what extent did this project have other impacts on your life? These could be little things or important things. They could be good or bad. EQ1 9 To what extent did this project have other impacts on your community? These could be little things or important things. They could be good or bad. Elaborate Among whom (IF SO1.2) USAID.GOV Mozambique Final Program Evaluation Report | 41 10 Were you able to access any new opportunities because of the project? If so, what were they If not, why? 11 Did the project affect your ability to have an income now and in the future? If so, how If not, why 12 Do you think having a job makes youth less likely to join the insurgency? If so, why If not, why 13 To what extent did this project change your views of the insurgency? How 14 To what extent do you think this project affected other beneficiaries’ support the insurgency (for example with money or information)? Why 15 To what extent do you think this project affected the likelihood that other beneficiaries might join the insurgency? Why 16 To what extent did this project have any other effects on your life? These could be little things or important things. They could be good or bad. 17 To what extent did this project have other impacts on your community? These could be little things or important things. They could be good or bad. Elaborate Among whom IF PROJECT THEY PARTICIPATED IN WAS PART OF A CHAIN 18 To what extent do you think grouping or sequencing these activities led to more or better impacts? I will now ask you about other TP projects that were implemented in your community. 19 Could you please tell us which of these activities you were personally involved in, which you are aware of, and which you are unaware of? (READ COMMUNITY LIST) 20 ACTIVITY 1: In what specific ways did this project benefit people who were directly involved with it? (i.e., direct beneficiaries)? Elaborate Among whom EQ1 21 ACTIVITY 1: To what extent did the project have impacts beyond those directly involved? In other words among other members of the community (i.e., indirect beneficiaries). Elaborate Among whom EQ1 22 ACTIVITY 1: Elaborate Among whom EQ1 42 | Mozambique Final Program Evaluation Report USAID.GOV Were there any unexpected or surprising impacts from this activity? These could be good things or bad things. REPEAT FOR ALL ACTIVITIES RESPONDENT PARTICIPATED IN OR WAS AWARE OF 23 Which project do you think was the most impactful for participants (i.e., direct beneficiaries)? Why 24 Which project do you think was the most impactful for the community (i.e., indirect beneficiaries)? Why Finally, we would like to ask your opinion about TP’s project logic. I will read you a series of statements. Please tell me the extent to which you agree with these statements using a 4-point scale. 1 is strongly disagree, 2 is disagree, 3 is agree, 4 is strongly agree. You can also respond “I don’t know” if you feel you don’t have enough information. 25 Youth are influential in their communities and with their peers (If 1 or 2) - why? 26 Youth lack the skills and knowledge to effectively resist VE (If 1 or 2) - why? 27 Youth lack the skills to help peers resist VE (If 1 or 2) - why? 28 Proving youth leaders with skills and knowledge helps these leaders promote VE resilience among their peers. (If 1 or 2) - why? 29 There are legitimate economic opportunities available for youth, but they don’t know about them. (If 1 or 2) - why? 30 Lack of economic opportunities increases youth’s vulnerability to joining or supporting VE. (If 1 or 2) - why? 31 Some possible legitimate income generating activities need TP support to be initially viable. (If 1 or 2) - why? 32 IS-M is providing economic incentives to entice vulnerable youth to voluntarily join/support VE. (If 1 or 2) - why? 33 Youth are influential in their communities and with their peers (If 1 or 2) - why? 34 If TP were extended another year or two, what are some things that TP could do to better help youth resist VE influence in your community (specifically sympathizing with, supporting, or joining the insurgency)? SO1.1 SO1.2 Why Is there anything important we didn’t ask you about or anything else you wanted to say? USAID.GOV Mozambique Final Program Evaluation Report | 43 Tool: FGD with Beneficiaries (Activity-level) # Question Probe EQ 1 Please tell us your name and how you were involved with Activity x. (IF SO1.1) 2 What were the most important things you learned from the project? Elaborate EQ1 3 To what extent did this information change your views of the insurgency? How 4 To what extent do you think this information changed the level of support for the insurgency among your peers (for example being sympathetic or providing money or information)? Why Why not 5 To what extent do you think this information has changed the likelihood that your peers might join the insurgency? Why Why not 6 Did this project have other impacts on your life? These could be little things or important things. They could be good or bad. 7 Did this project have other impacts on your community? These could be little things or important things. They could be good or bad. IF SO1.2 8 Were you able to access any new opportunities because of the project? If so, what were they If not, why? 9 10 Do you think having a job makes youth less likely to join the insurgency? Why/why not EQ1 11 To what extent did this project change your views of the insurgency? EQ1 12 To what extent do you think this project affected other beneficiaries’ support to the insurgency (for example with money or information)? EQ1 13 To what extent do you think this project affected the likelihood that other beneficiaries might join the insurgency? EQ1 14 To what extent did this project have other impacts on your life? 44 | Mozambique Final Program Evaluation Report USAID.GOV These could be little things or important things. They could be good or bad. 15 To what extent did this project have other impacts on your community? These could be little things or important things. They could be good or bad. IF ACTIVITY WAS PART OF A CHAIN 16 To what extent do you think grouping or sequencing these activities led to more or better impacts? I will now ask you about other TP projects that were implemented in your community. TP implemented x activities in this area. (READ LIST OF PROJECTS) 17 Which projects do you think were the most impactful for participants (i.e., direct beneficiaries)? Why 18 Which projects do you think were the most impactful for the community (i.e., indirect beneficiaries)? Why 19 If TP were extended another year or two, what are some things that TP could do to better help youth resist VE influence in your community (specifically sympathizing with, supporting, or joining the insurgency)? SO1.1 SO1.2 Why Finally, we would like to ask your opinion about TP’s project logic. I will read you a series of statements. Please tell me whether you agree or disagree with the statements. 20 Youth are influential in their communities and with their peers (If disagree) - why? 21 Youth lack the skills and knowledge to effectively resist VE (If disagree) - why? 22 Youth lack the skills to help peers resist VE (If disagree) - why? 23 Proving youth leaders with skills and knowledge helps these leaders promote VE resilience among their peers. (If disagree) - why? 24 There are legitimate economic opportunities available for youth, but they don’t know about them. (If disagree) - why? 25 Lack of economic opportunities increases youth’s vulnerability to joining or supporting VE. (If disagree) - why? 26 Some possible legitimate income generating activities need TP support to be initially viable. (If disagree) - why? 27 IS-M is providing economic incentives to entice vulnerable youth to voluntarily join/support VE. (If disagree) - why? Is there anything important we didn’t ask you about or anything else you wanted to say?