FINAL PERFORMANCE EVALUATION TUNISIA ACCOUNTABILITY, DECENTRALIZATION, AND EFFECTIVE MUNICIPALITIES (TADAEEM) June 1, 2022 This publication was produced for review by the United States Agency for International Development (USAID). It was prepared by International Business & Technical Consultants, Inc. (IBTCI), under the Monitoring and Evaluation in Tunisia and Libya (METAL) Activity (AID-280-TO-17-00001). MONITORING & EVALUATION FOR TUNISIA AND LIBYA (METAL) FINAL PERFORMANCE EVALUATION TUNISIA ACCOUNTABILITY, DECENTRALIZATION, AND EFFECTIVE MUNICIPALITIES (TADAEEM) AUTHORS: Keri Culver Amal Lajmi Jihed Haj Salem Aida Kraiem Zoubeir Daly JUNE 1, 2020 International Business & Technical Consultants, Inc. (IBTCI) In the US: 8618 Westwood Center Drive Suite 400 Vienna, VA 22182 USA +1.703.749.0100 In Tunisia: 1st floor, Blue Bloc Vital Cube Building 4 L’Ile de Tasmanie Street Les Berges du Lac 2 1053 Tunis, Tunisia AID-280-TO-17-00001 DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of IBTCI and do not necessarily reflect the views of USAID or the United States Government. i ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 ABSTRACT This mixed-methods final performance evaluation evaluates the Tunisia Accountability, Decentralization, and Effective Municipalities (TADAEEM) Activity, implemented by Deloitte. The evaluation questions are: (1) To what extent was TADAEEM aligned with citizen participation and service delivery in the 2018 Code des Collectivités Locales (CCL), including pivoting as necessary? (2) To what extent did TADAEEM contribute to the operationalization of the decentralization development plan? In what ways might TADAEEM have done better with this changing context? (3) To what extent and how have TADAEEM’s individual activities contributed to achieving improvements to each of the four objectives? What were the successes and what could we have done better? (4) What were the internal and external factors that affected the timeliness and achievement of TADAEEM’s objectives and what measures did TADAEEM and its implementing partners take to mitigate those factors? (5) How effective was TADAEEM in integrating women, youth, and other marginalized groups into its activities and supporting elected women and career women staff in municipalities? (6) What measures has TADAEEM taken to ensure the sustainability of improvements in citizens’ participation in municipal decision-making, municipal service delivery, national and subnational government service delivery mechanisms, and coordination between national and subnational governments to respond to national and local emergencies and other crises? The evaluation concluded that TADAEEM’s important outcomes in improving citizen participation and municipal service delivery put in place building blocks for future interventions. However, design decisions impacted TADAEEM’s achievement of outcomes, as did challenges in relationship management, and the slow pace of decentralization as implemented by the Government of Tunisia. The report concludes with a set of recommendations linked to evaluation conclusions in each section, including a section on monitoring and evaluation that was not part of the evaluation questions. The recommendations are divided into five categories: working with the central government, building on what TADAEEM has put in place, USAID internal issues, strengthening weaknesses in programming, and designing future programming. TABLE OF CONTENTS ABSTRACT ACRONYMS iii 1. EXECUTIVE SUMMARY 1 1.1 Methodology 1 1.2 Evaluation Purpose and Questions 1 1.3 Findings and Conclusions 2 1.4 Recommendations 4 2. EVALUATION PURPOSE AND QUESTIONS 1 2.1 Evaluation Purpose, Uses, and Users 1 2.2 Evaluation Questions 1 3. BACKGROUND 2 3.1 Country Context 2 3.2 Activity 3 4. METHODOLOGY AND LIMITATIONS 3 5. FINDINGS AND CONCLUSIONS 7 5.1 EQ1 and EQ2: Alignment with and operationalization of decentralization 7 5.2 EQ3: Effectiveness in TADAEEM Objectives 9 5.3 EQ4: Challenges and mitigations 31 5.4 EQ5: Inclusion 36 5.5 EQ6: Sustainability 38 5.6 Monitoring and evaluation 40 6. RECOMMENDATIONS 42 7. KEY TAKEAWAYS 45 ANNEX A: SCOPE OF WORK 47 ANNEX B: METHODOLOGY AND LIMITATIONS 57 ANNEX C: DATA COLLECTION TOOLS 65 ANNEX D: DOCUMENTS REVIEWED 80 ANNEX E: RESPONDENTS INTERVIEWED 83 ANNEX F: EVALUATION TEAM 83 ANNEX G: CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT 85 ANNEX H: EVALUATION MATRIX 86 ANNEX I: TADAEEM RESULTS FRAMEWORKS – 2017 (left) and 2021 (right) 89 ANNEX J: MEMORANDUM OF INTENT: MALE AND TADAEEM 90 i ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 MAPS Map 1: TADAEEM’s SDIPs by Municipality 24 LIST OF TABLES Table 1: Activity Summary Information 8 Table 2: Evaluation respondents (see Annex E for detail) 10 Table 3: Final evaluation site visit sample 10 Table 4: PAI attendance rates, before and after TADAEEM, and as percent of population 19 Table 5: Municipal Service Delivery Equipment Provided 23 Table 6: Indicators from 2018 and 2021 TADAEEM AMELPs 49 LIST OF FIGURES Figure 1: Did TADAEEM contribute to increased attendance and participation of CSOs and citizens i municipal meetings? 17 Figure 2: Did TADAEEM help improve citizen participation in your municipality? 20 Figure 3: Did TADAEEM help your municipality improve service delivery? 24 Figure 4: Were you satisfied with the support provided by TADAEEM? 25 Figure 5. Comparison of Citizen Satisfaction by Municipal Service in 2019 and 2020 CPS 26 Figure 6. 2021 Citizen Satisfaction by Service in Urban and Rural Areas 26 Figure 7. Citizens’ Perceptions of their Municipalities’ Responsiveness to their Service Needs in CPS 2021 28 Figure 8: Did TADAEEM help your municipality work better with regional and national government? 30 Figure 9: Local coordination committees established (at left) and operating regularly (at right) 32 Figure 10: Critical gaps at municipal level: Survey responses from municipalities 34 Figure 11: How did TADAEEM mitigate challenges in your municipality? 37 Figure 12: Were you satisfied with TADAEEM’s support in your municipality? 37 n ii ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 ACRONYMS AMELP Activity Monitoring, Evaluation, and Learning Plan CCL Code des Collectivités Locales CDCS Country Development Cooperation Strategy COR Contracting Officer’s Representative COVID-19 Coronavirus Disease 2019 CPS Citizen Perceptions Survey CSO Civil Society Organization DG Democracy and Governance DO Development Objectives EQ Evaluation Question ET Evaluation Team FCR Findings, Conclusions, and Recommendations GIZ Gesellschaft für Internationale Zusammernarbeit, the German agency for international development GOT Government of Tunisia IBTCI International Business & Technical Consultants, Inc. IR Intermediate Result IT Information technology KII Key Informant Interview LED Light-emitting diodes MALE Ministère des Affaires Locales et de l’Environnement (Ministry of Local Affairs and the Environment) MEL Monitoring, Evaluation, and Learning METAL Monitoring and Evaluation for Tunisia and Libya MOI Memorandum of Intent O1 Objective 1 O2 Objective 2 O3 Objective 3 PAI Plan Annuel d’Investissement (Annual Investment Plan) PMP Performance Management Plan PWD Persons with Disabilities RF Results Framework iii ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 SDIP Service Development Improvement Plan SOW Scope of Work TADAEEM Tunisia Accountability, Decentralization, and Effective Municipalities TL Team Leader USAID United States Agency for International Development USG United States Government WB World Bank iv ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 1. EXECUTIVE SUMMARY The Tunisia Accountability, Decentralization, and Effective Municipalities (TADAEEM) Activity was implemented by Deloitte and aimed to bridge the gap between the Government of Tunisia (GOT)’s decentralization process and bottom-up municipal development to produce tangible change and improve the relationship between Tunisians and their government institutions. The Activity ran for four years with a six-month extension, with an award of $49,157,614. TADAEEM’s Objective 1 (O1) aimed to increase citizen participation and oversight at the local level, while Objective 2 (O2) involved improving service delivery mechanisms in five key service areas (public lighting, waste management, road maintenance, and others). Under Objective 3 (O3), TADAEEM worked on service delivery mechanisms at regional and national levels to ensure intergovernmental links. TADAEEM directly contributed to USAID/Tunisia’s Development Objective (DO) 2: Social Cohesion Promoted through Democratic Consolidation. 1.1 Methodology The mixed-methods design was approved in late March 2022, and fieldwork commenced at that time, alongside the Ramadan holiday period. Fieldwork concluded on May 17, 2022. Qualitative methods included a document review and interviews; quantitative methods included an online survey of municipal Activity partners and civil society organizations (CSOs) from among the 33 municipalities where TADAEEM operated. 1.2 Evaluation Purpose and Questions The purpose of the evaluation is to identify the extent to which TADAEEM increased government responsiveness to citizen needs at national and local levels, and expanded citizen engagement for accountable governance. The evaluation questions (EQs) are as follows: EQ 1: To what extent was TADAEEM aligned with citizen participation and service delivery in the 2018 Code des Collectivités Locales (CCL), including pivoting as necessary? EQ 2: To what extent did TADAEEM contribute to the operationalization of the decentralization development plan? In what ways might TADAEEM have done better with this changing context? EQ 3: To what extent and how have TADAEEM’s individual activities contributed to achieving results? What were successes and what could have been better? a) Citizen participation in and oversight of decision-making processes at the subnational level b) Municipal institutional and service delivery performance c) Regional and national service delivery mechanisms to respond to citizen needs d) Improved coordination and communication of municipalities, regional governments, and central government in responding to national and local emergencies and other crises EQ 4: What were the internal (staffing, procurement, organizational structure, etc.) and external factors (lack of local technical capacity, national and subnational resources, political will, corruption, etc.) that affected the timeliness and achievement of TADAEEM’s objectives and what measures did TADAEEM and its implementing partners take to mitigate those factors? EQ 5: How effective was TADAEEM in integrating women, youth, and other marginalized groups into its activities and supporting elected women and career women staff in municipalities? EQ 6: What measures has TADAEEM taken to ensure the sustainability of improvements in citizens’ participation in municipal decision-making, municipal service delivery, national and subnational government service delivery mechanisms, and coordination between national and subnational governments to respond to national and local emergencies and other crises? 1 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Primary audiences for this evaluation are the USAID/Tunisia Democracy and Governance Office, USAID/Tunisia, and the Tunisian government. Other audiences include civil society organizations (CSOs) engaged with municipalities, implementing partners, other donors, and national and subnational government authorities engaged in decentralization. Recommendations will be used to inform USAID/Tunisia local governance design, GOT decentralization efforts, and the Mission Country Development Cooperation Strategy (CDCS). 1.3 Findings and Conclusions Evaluation Questions 1 and 2: Alignment with GOT decentralization and legislation Evaluation findings show that TADAEEM objectives and activities tackling service delivery difficulties and improving citizen participation in decision-making processes (O1 and O2) were well-aligned with CCL and regulations in force. A Memorandum of Intent (MOI) formalized and operationalized how TADAEEM could be most relevant and contribute to decentralization. However, the GOT’s slow process of decentralization hindered TADAEEM’s ability to collaborate with some regional counterparts and potential counterparts. At the same time, there were potential entry points for TADAEEM among the more innovative clauses of the CCL that were not explored. Most challenging for TADAEEM was a difficult relationship with GOT interlocutors, which had critical effects throughout the life of the Activity, with the exception of the COVID-19 period. TADAEEM focused on its municipal level interventions, with little attention to the GOT’s own priorities. This affected ownership and sustainability. Evaluation Question 3a: Citizen participation and oversight TADAEEM carried out thoughtful work at the ground level with innovative and inclusive initiatives to improve CSO and citizen participation in and (to a lesser extent) oversight over local government. TADAEEM succeeded in increasing citizen participation rates, particularly through municipal annual planning processes, though the percentage of citizens who participate remains low by GOT standards. TADAEEM also fostered trust through a variety of outreach and consultation initiatives that enabled citizens to voice their concerns and needs in terms of municipal service improvement. However, TADAEEM failed to build effective and long-lasting mechanisms to ensure sustained citizen engagement in and oversight of their municipalities. Evaluation Question 3b TADAEEM’s approach assisted partner municipalities in strengthening service delivery capacities and meeting citizen needs. Between 2019 and 2021, TADAEEM’s 33 municipalities carried out 64 service improvements that also improved municipalities’ service delivery capacity, with wide variation across intervened sites and services. Where citizen engagement was included, and later, a gender-sensitive participatory approach, it enhanced the strength of the process and the results. Citizen engagement motivated improved attention to citizen needs among municipal authorities. However, TADAEEM did not support partners to develop systematic citizen oversight of implementation and results. Despite service delivery improvements, citizen satisfaction increased only very slightly, in part because Tunisians’ priorities center on economic issues and job creation more than on municipal public services. TADAEEM did not deeply explore the ways it might have supported municipalities on these issues as opposed to the set of services where it did intervene. Evaluation Question 3c Work on O3 demonstrated TADAEEM’s and USAID’s commitment to national-level goals. While this was not the primary focus of the Activity, positive outcomes, products, and relationships were built. Unfortunately, in part due to the July 25 presidential action in Tunisia (2021), the promise of most of the 2 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 efforts has not been realized. The lack of deconcentration to regions also limited how well TADAEEM could work with regions and governorates. Evaluation Question 3d The Ministère des Affaires Locales et de l’Environnement (Ministry of Local Affairs and the Environment or MALE) successfully coordinated the COVID response and donor contributions to it, and TADAEEM used its $5m budget amendment to procure equipment and supplies quickly and successfully for 151 municipalities, and provided videoconferencing equipment nearly nationwide. However, cross-GOT crisis coordination committees have not taken root in municipalities. Evaluation Question 4: Challenges and TADAEEM’s efforts at mitigation Main challenges included the following: Evaluation Question 5 TADAEEM’s technical leadership did not put sufficient focus on gender mainstreaming from the start of the Activity or ensure that teams were committed to mainstreaming as an integral goal of programming. There were no specific and mainstreamed strategies to encourage participation by youth or people with disabilities (PWD) – despite some high-profile but one-off achievements. When TADAEEM field managers, municipal staff, and local CSOs were engaged together on gender, bringing women to the table for municipal decision-making, for example, it was notably more successful, in the latter months of the Activity. Evaluation Question 6 Citizen participation was not well-sustained, in part due to COVID-19 disruptions to gathering, and in part due to the lack of systematic mechanisms, a more inclusion-focused approach, and stronger collaboration with CSOs. Service delivery outputs showed more evidence of sustainability in equipment and technical know-how but TADAEEM-produced technical guides await dissemination. Several targeted efforts to strengthen regional and national service delivery mechanisms, such as the e-Construction platform, waste management, and other service strategies and standards, have not survived the end of the Activity, or the significant ministry reshuffling following Tunisia’s July 25, 2021, presidential action. 3 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Videoconferencing equipment was put in place and has been used but sustained improvements to communications and crisis coordination are not in evidence. Additional findings: Monitoring and Evaluation Monitoring indicators were diluted throughout the life of the project, leaving little evidence of outcomes with the exception of service delivery outcomes at the municipal level. A planned midterm evaluation that might have provided helpful inputs was not undertaken, representing an important missed opportunity. 1.4 Recommendations Recommendations are divided into five categories: working with the central government, USAID internal recommendations, strengthening programming weaknesses, building on what TADAEEM has put in place, and future programming recommendations. Each of these is developed more fully in the Recommendations chapter. Working with the central government • USAID should take into consideration central government strategy for reforms at the state level, and through institutions that support subnational governments. USAID should map and include these institutional stakeholders during design, and formally agree on milestones for continued cooperation. The success of activities relies on central GOT commitment. Recommendations for USAID • USAID/Tunisia’s activities should be coherent and collaborative across the portfolio, taking best advantage of shared geographies, research opportunities, and models of integrated interventions that the GOT could scale up holistically. Activities that are part of the same CDCS should be working together toward integrated goals. • USAID/Tunisia needs better activity management and can count on its Monitoring, Evaluation, and Learning (MEL) platform for more support. Activity Monitoring, Evaluation, and Learning Plans (AMELPs) should attend more to outcomes than to simple outputs and should monitor context – as is done in the USAID/Tunisia Ma3an Activity. A midterm or developmental evaluation would help an activity with high contextual uncertainty such as TADAEEM. Strengthening weak areas • USAID’s future interventions in this space should partner better with local and national CSOs, including local media in the south of the country. At the national level, CSOs are active on decentralization and could be a conduit providing citizen insight into any future work with the GOT. • Future investment in citizen participation efforts should be more inclusive, and this should be reflected in output and outcome indicators. Given the historical neglect of rural Tunisia, inclusion should include to these populations as well. Activities should also be required to report not just numbers of participants but to unpack engagement and oversight. Building on TADAEEM’s beginnings • USAID and implementers should keep gender and youth at the forefront of their programming in local governance. Scale up innovative TADAEEM pilots or test them in other environments, like those with more traditional gender role adherence, to identify local best practices. • USAID should consider local revenue collection, financial management, and investment, beyond technical assistance to include concrete projects like TADAEEM’s property survey pilot, oriented to resolve municipalities’ financial needs and build self-sustainable revenue systems. • USAID and its implementers should include the range of TADAEEM service delivery products when designing new activities to ensure they are propagated to municipalities nationwide. This 4 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 may include using peer-to-peer learning where TADAEEM municipalities with successful outcomes learn to train other municipal leaders elsewhere, supporting both sustainability and scale-up. Future programming considerations • USAID should consider supporting the planned deconcentration to district/governorate-level roles, and election of bodies at those levels. The emergent quality of programming in this area – given that (like municipalities during TADAEEM) any elected bodies will be completely new – will require patience on the part of USAID and implementers to identify and begin to address capacity￾building and service delivery needs. • USAID should use an approach that situates municipalities within their systems, bringing together conclusions from multiple EQs. Though the dynamics of decentralization are stalled, when USAID decides to intervene again, that approach will need to consider the web of GOT actors at different levels, including municipal peers, CSOs at all levels, other donors, and the private sector, and calls for broader inclusion. Capacity-building must also be systematic. 5 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 2. EVALUATION PURPOSE AND QUESTIONS 2.1 Evaluation Purpose, Uses, and Users The United States Agency for International Development (USAID)/Tunisia requested that the Monitoring and Evaluation for Tunisia and Libya (METAL) Activity of International Business & Technical Consultants, Inc. (IBTCI), design and conduct an independent final evaluation of the TADAEEM Activity, implemented by Deloitte. The Scope of Work (SOW) for this external, final performance evaluation is included as Annex A. The purpose of the evaluation is to identify the extent to which TADAEEM increased government responsiveness to citizen needs at national and local levels, and expanded citizen engagement for accountable governance. The evaluation identifies successes, challenges, and lessons learned that affected implementation. The primary audiences for this evaluation are the USAID/Tunisia Democracy and Governance Office, the USAID/Tunisia Mission, and the Tunisian government. Other audiences include CSOs, implementing partners, and other donors engaged in decentralization. Evaluation recommendations will inform design of USAID/Tunisia local governance initiatives, and Government of Tunisia (GOT) decentralization efforts. 2.2 Evaluation Questions This final evaluation focused on the Evaluation Questions (EQs) below: EQ 1: To what extent was TADAEEM aligned with citizen participation and service delivery in the 2018 Code des Collectivités Locales (CCL), including pivoting as necessary? EQ 2: To what extent did TADAEEM contribute to the operationalization of the decentralization development plan? In what ways might TADAEEM have done better with this changing context? EQ 3: To what extent and how have TADAEEM’s individual activities contributed to achieving results? What were successes and what could have been better? a) Citizen participation in and oversight of decision-making processes at the subnational level b) Municipal institutional and service delivery performance c) Regional and national service delivery mechanisms to respond to citizen needs d) Improved coordination and communication of municipalities, regional governments, and central government in responding to national and local emergencies and other crises EQ 4: What were the internal (staffing, procurement, organizational structure, etc.) and external factors (lack of local technical capacity, national and subnational resources, political will, corruption, etc.) that affected the timeliness and achievement of TADAEEM’s objectives and what measures did TADAEEM and its implementing partners take to mitigate those factors? EQ 5: How effective was TADAEEM in integrating women, youth, and other marginalized groups into its activities and supporting elected women and career women staff in municipalities? EQ 6: What measures has TADAEEM taken to ensure the sustainability of improvements in citizens’ participation in municipal decision-making, municipal service delivery, national and subnational government service delivery mechanisms, and coordination between national and subnational governments to respond to national and local emergencies and other crises? 1 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 3. BACKGROUND 3.1 Country Context After decades of authoritarian and centralized rule, the 2011 revolution, and the 2014 Constitution, Tunisia initiated a process toward democratic decentralization, taking important steps toward a more decentralized state to strengthen local authorities by handing over appropriate functions and budgetary resources to regions and municipalities. The 2014 Constitution states that “the State undertakes to strengthen decentralization and to implement it throughout the national territory, within the framework of the unity of the State.” Importantly, municipal and regional councils are to be directly elected. The Constitution and subsequent electoral law call for gender parity and representation of youth and PWD on municipal councils. An additional – and central – precept of the 2014 Constitution is the commitment to participatory citizen engagement at the municipal level. The new Constitution of 2014 thus enshrined democratic processes of local governance and reinforced the constitutional foundations and principles of decentralization. Not all of this has been put into law yet. However, since 2016, the entire country has finally been “communalized,” meaning that areas previously left out of the spatial municipal demarcations were incorporated. Also at this time, the GOT established the Ministry of Local Affairs and Environment (MALE).1 Allocating local affairs to a full-fledged ministry was unprecedented in Tunisia, where municipalities had always been under the supervision of the Ministry of the Interior. The adoption of the Local Government Law [Code des Collectivités locales (CCL)] in April 2018 and the municipal elections in the following month were a start to the decentralization of power, which would allow for more independent local development, empower local decision-making, encourage public participation, contribute toward more equitable distribution of resources and improve service delivery.2 The CCL also established three institutions to strengthen the decentralization process: the High Authority of Local Finances,3 the National Training Committee in support of local authorities,4 and the support, equalization, and solidarity fund for local authorities (later codified in the 2021 Finance Law). The promise of TADAEEM was to contribute to these developments, alongside other national and international actors. Successes have been mixed, however. Citizen expectations for improvements in their daily lives continue to be high while trust and satisfaction with government performance have declined. Research conducted as part of TADAEEM found that municipalities do not have a strong understanding of the current state of their services and are often reluctant to engage citizens in decision-making, fearing criticism. Staff levels are critically low in many municipalities, and their resource levels are too low to put into practice service delivery goals. Most municipalities also lack the capacity to fully realize their own taxation. Citizens are generally not aware of the constraints or challenges facing municipalities and are not confident that making their needs known will lead to change. Moreover, partisanship in the context of national politics has also affected the municipal councils. Low participation in later elections revealed apathy and lack of citizen trust toward the political class: “The average Tunisian citizen still fails to see how electing municipal, and later regional, councils falls in the context of increasing local authority and leading to participatory local democracy.”5 A more complete literature review is available in the evaluation’s Inception Report as Annex 11. 1 Decentralisation operationalisation process: Update of strategic orientations to 2039 and action plan 2018–2021. 2 https://www.giz.de/en/worldwide/73233.html 3 Governmental decree 2019-351, 10 April 2019, appointing president and members of the higher authority of local finances. 4 Government decree 2019-350, 10 April 2019, appointing the chairman and members of the National Commission for the Training of Local Councils. 5 Nouira, Asma. “Tunisia’s Local Elections: Entrenching Democratic Practices,” Arab Reform Initiative, July 12, 2012, p. 5. Retrieved: Tunisia’s Local Elections: Entrenching Democratic Practices – Arab Reform Initiative (arab-reform.net) 2 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 3.2 Activity Table 1: Activity Summary Information Activity Name Tunisia Accountability, Decentralization, and Effective Municipalities (TADAEEM) Implementer Deloitte Contract # Contract under AID-OAA-I-14-00065/AID-664-TO-17-00002 Total Estimated Ceiling $49,157,614 million Life of Activity September 2017–March 2022 Active Geographic Regions 33 municipalities in 10 governorates across Tunisia Development Objective (DO) DO2: Social Cohesion promoted through Democratic Consolidation USAID Office USAID/Tunisia USAID’s municipal development activity, TADAEEM, sought to bridge the gap between the GOT’s long￾term decentralization process and bottom-up municipal development to generate tangible change, while improving the relationship between Tunisians and their government. It directly contributes to USAID/Tunisia’s DO 2: Social Cohesion Promoted through Democratic Consolidation. TADAEEM had three initial objectives and added a fourth one in Year 4, as follows: 1. Objective 1: Citizens’ participation in and oversight of key governmental functions improved 2. Objective 2: Municipal institutional service delivery performance improved 3. Objective 3: Regional and national service delivery mechanisms improved to respond to needs of citizens 4. Objective 4: Improved coordination and communication of municipalities, regional governments, and central government in responding to national and local emergencies and other crises6 TADAEEM’s overall development hypothesis reads: If TADAEEM works with municipalities to improve service delivery and engage citizens in selecting solutions, the relationship between citizens and their local governments will improve. If TADAEEM improves the relationship between Tunisians and their civic and government institutions, the result will be an increase in the legitimacy of the Tunisian political system. TADAEEM’s Results Framework (RF) was linked to the USAID CDCS DO2: Social Cohesion Promoted through Democratic Consolidation and its two sub-intermediate results (IRs): Participatory Systems Improved, and Enhanced Responsiveness of Government Institutions. In 2020, the final RF from the TADAEEM work plan presented significant changes to intermediate results and sub-IRs, as well as to its objective-level development hypotheses, and to its indicators. The final RF is shown in Annex I for reference. 4. METHODOLOGY AND LIMITATIONS The evaluation was conducted from February 28, 2022, to May 18, 2022. The evaluation team (ET) designed a mixed-methods evaluation, comprising face-to-face and virtually collected qualitative and quantitative data. The data collection included key informant interviews (KIIs) combined with site visits, a survey of municipal actors and CSO representatives, and an extensive document review. The evaluation 6 With the COVID-19 pandemic, USAID used TADAEEM’s networks to provide 151 municipalities with sanitizing and other materials, through a 2020 contract modification. The Activity MEL Plan indicators were also updated at that time. 3 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 was carried out by a local team that included the Team Lead, two field researchers, and two subject matter experts, working in parallel to conduct data collection during the period of March 31–May 18, 2022. Site visits and Key Informant Interviews KIIs were one of the main methods of data collection for this evaluation. The ET interviewed 84 respondents from different categories (as in Table 2 below). Of 84 interviews, 45 were conducted face￾to-face (54 percent) during visits to the project implementation sites in eight municipalities, and in Tunis. Table 2: Evaluation respondents (see Annex E for detail) Planned7 Women Men Total USAID 4 2 4 6 Implementing partner - Tunis 6 6 10 16 Implementing partner - Field 8 5 5 10 GOT - National 8 5 8 13 GOT - Subnational (elected) 8 3 6 9 GOT - Subnational (administrative) 8 2 7 9 CSOs - Tunis 3 5 2 7 CSOs - Subnational 8 4 7 11 Other donors 4 1 2 3 Total 57 33 51 84 The site visits allowed the ET to reach former members of the TADAEEM field or “hub” teams, and simplified access to most subnational elected and administrative officials and representatives of local CSOs. Interviews with USAID, Tunis TADAEEM teams (including expats), and other donors, were conducted virtually using Google Meet and other communication software. Eight municipalities for the site visits were selected to maximize heterogeneity of the sites. One substitution to the original list of selected municipalities had to be made because the political situation did not allow data collection from government officials after a regional delegate in Gabes took over a mayor’s functions due to unresolved disputes on the municipal council. The ET replaced Gabes with Metouia, a similar municipality in the same region. In addition to the eight municipalities, the ET also interviewed the mayor of Tunis and her technical team, since they were involved in a TADAEEM activity to map the city with a Geographical Information System (GIS). The final sample was as follows the population, and the criteria we used to determine the sample communities: Table 3: Final evaluation site visit sample Municipality Hub Population (Jan 2020) Regional Development Score (2018) Performance (2019) El Kef Kef 65,905 Low 85 Dahmani Kef 27,677 Low 59 El Alaa Kairouan 28,257 Very low 75 El Nathour Kairouan 32,088 Very low 64 Tozeur Tozeur 50,744 High 70 Kebili Tozeur 46,311 Low 75 Gabes Gabes 106,438 High 78 Metouia Gabes 13,289 High8 919 Sakiet Ezzit Gabes 66,833 Very high 53 Tunis Tunis 610,995 Very high 81 7 The ET always knew these planned interview numbers were illustrative rather than firmly fixed but considered these minimum figures for confidence in our findings. As with evaluation procedures elsewhere, the ET looked for saturation – that is, the lack of divergent responses – as a key to terminating our interviews in a given category. 8 This figure was not available for only Metouia, rather for its region, Gabes. 9 This figure was available for Metouia only for 2018, rather than 2019 as for the other municipalities. 4 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Surveys The team also carried out an online survey of municipal actors (mayors and others on municipal councils and municipal staff) and one of CSO representatives. The purpose of the survey was to understand municipal actors’ and CSO representatives’ experiences with TADAEEM across the 33 municipalities, including questions of satisfaction, achievements under each objective, sustainability, and challenges and mitigation. Using contact information supplied by TADAEEM, the ET wrote to TADAEEM’s 60 municipal contacts from the 33 treated municipalities. There were ultimately 26 responses from 19 of the treated municipalities, for a response rate of 43 percent. A METAL staffer on the ET reached out to non￾respondents by phone and extended the deadline to attempt to increase response rate, with some effect. However, the ET suspects that Ramadan affected potential responses, and there was also the possibility that municipal councils and administration members have left their positions – which the ET did hear of on occasion while doing fieldwork. Almost all municipal respondents were between 30 and 55 years of age. Only three among the 26 municipal respondents were women. Because the sample is not representative, the ET uses the number of responses, rather than percentages, in most of the reporting on survey results, so as not to imply proportions that might be read to apply more broadly. TADAEEM provided a list of 320 CSO contacts, which included all CSOs known to TADAEEM. Some of these were direct recipients of Activity assistance or participants, but the great majority were not. This was confirmed in survey responses. When asked whether they had been involved with TADAEEM, 25 of 37 CSO respondents had not, leaving only 12 complete responses (7 women, 5 men). As a result, despite attempts to re-contact and an extension of the closing date, the response rate was only 3.75 percent. Almost all CSO respondents were between 30 and 55 years of age. Seven were women. Seven said they were somewhat involved with TADAEEM, and five said they were very involved. Documents and other secondary data The evaluation also relied on TADAEEM quantitative indicator data and three waves of TADAEEM’s Citizen Perceptions Survey (CPS), along with a review of activity documentation (please see Annex D) and external documents, particularly decentralization documentation from the GOT. The EQs were mapped against data sources, collection methods, and analysis methods, as shown in Annex H. Data Analysis Quantitative survey data was summarized, tabulated, and presented for review. Data are disaggregated by location, gender, and minority group as appropriate, though sample sizes and self-selection into the survey mean that these data are not representative. The ET assembled findings across sources by EQ, and analyzed these jointly during a half-day face-to-face session, complementing team member fieldwork summaries. This was assembled in an Excel spreadsheet with the totality of responses per EQ and theme, allowing the ET to review findings holistically. Working jointly provided a check on confirmation biases. The ET developed conclusions that are well-founded in the data and traceable for the reader. For more information on data analysis, please see Annex B with the detailed methodology. Dissemination and Utilization USAID has noted ways in which the report and its evidence will be disseminated within the Agency and externally in Tunisia. First, the evaluation will serve as an input to the formulation of the new CDCS. The team’s preliminary presentation provided earlier information for that process. Additionally, the team has prepared an Executive Summary and four-page briefer for various audiences. These will be translated into Arabic and French, including any data visualizations that are useful to include. The Final Report and annexes will be available on the Development Experience Clearinghouse (DEC) upon approval, and the ET will 5 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 collaborate with USAID to offer any other presentations they deem useful, such as with GOT, CSOs, or other audiences. Limitations and Mitigation Strategies Biases that might have affected the evaluation and the team’s mitigation measures were predicted and in the Inception report and planned for in fieldwork. More information on the biases is found in Annex B. Recall bias. Among interviews with TADAEEM staff, the ET observed very emphatic and repeated responses about certain less positive internal characteristics of the Activity, and about the extent of results, at times using the same or very similar language. This seems to indicate that respondents shared these opinions among themselves. The elapsed time since the end of respondents’ employment with TADAEEM, along with the strength of these considerations, seemed to shape recall around those issues, while other memories were less clear. These issues are comprehensively covered in this report but recall bias around them likely limited the findings on other nuanced but less emotionally charged issues. Social desirability bias. Some TADAEEM respondents were unerringly positive, particularly at the start of the evaluation, perhaps due to social desirability bias. At the same time, many were unable to provide much detail, which may be related to recall bias. In later interviews, with more information under its belt, the ET was able to plumb contentious issues raised by others with gentle probes around the details, which elicited more nuanced responses and allowed it to understand the broad TADAEEM team and its strengths and challenges better. The analysis used triangulation to see challenges and strengths from different angles, helping the team to interpret evidence more objectively. Self-selection bias. The survey response rates were lower than hoped, particularly for CSOs, and given the Ramadan season as well as the elapsed time since the end of TADAEEM, the ET cannot estimate why some potential respondents opted in while others did not. In survey research more generally, repeated observations indicate that people opt in to surveys when they have stronger, more extreme views, and since the results are more positive for municipalities and less so for CSOs, that was a confirmatory finding with field interview data that suggested the promise of work with CSOs was not fully realized. Still, it is important to note that the survey data are not representative, and to ensure this was not misinterpreted by readers, the ET reported only numbers of respondents, rather than percentages, which would have indicated proportionality among the larger population. Selection bias. The ET used a purposive sample and supplemented planned interviews by asking respondents who else should be interviewed. In two cases, this resulted in notably negative responses against the original interviewee’s more positive responses, indicating that asking for recommendations gave these two interviewees a chance to put forward names of people who could be more candid. Evaluation timing. The ET was concerned about response rate overall because of the Ramadan and Eid holidays, but found that people in municipalities were generally willing to interview, though at a slower pace and with more rescheduling than is common. The team used a USAID-signed letter and gentle persistence to encourage participation. There were cases where the team pursued multiple individuals in given line ministries and among other donors, with minimal success. In addition, the TADAEEM Activity closed and team members had departed. The ET worked with Deloitte to track team members as a priority and used USAID-approved additional time to good advantage. There may be limitations to the validity of given data points from among the ET’s sources because of recall, social desirability, and other biases and timing issues. However, the ET exceeded the number of planned interviews and was able to triangulate from a range of different perspectives. In addition, the ET has presented preponderant evidence in this report, rather than single anecdotes, to avoid over￾interpreting the inputs of any given respondent. In the case of the survey, while the ET cannot speak representatively about TADAEEM’s interventions based on these data, there are strong trends for questions of satisfaction, sustainability, and challenges and mitigations that are usefully shared in this report. In any case, TADAEEM was a demand-driven Activity at the municipal level: there was no expectation that 6 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 all municipalities would receive the same treatments, work on the same axes, or reach the same conclusions about the intervention. As such the survey data remains valuable. 5. FINDINGS AND CONCLUSIONS 5.1 EQ1 and EQ2: Alignment with and operationalization of decentralization EQ 1: To what extent was TADAEEM aligned with citizen participation and service delivery in the 2018 CCL, including pivoting as necessary? EQ 2: To what extent did TADAEEM contribute to the operationalization of the decentralization development plan? In what ways might TADAEEM have done better with this changing context? Findings EQ 1 and 2 ask about the relevance of TADAEEM in its context. The Organization for Economic Cooperation and Development Assistance Committee (OECD/DAC) defines relevance in development programming as “The extent to which the intervention objectives and design respond to beneficiaries, global, country, and partner/institution needs, policies, and priorities, and continue to do so if circumstances change.” In Tunisia, TADAEEM’s relevance also meant the degree to which the Activity was able to capitalize on progress in Tunisia’s legal and de facto decentralization at the time of the award. The CCL is a critical piece of the country’s overall 27-year decentralization plan, and the major step that had been taken from that plan. Operationalization, in 2017–2018 when TADAEEM began, meant in large part working in concert with the CCL. For that reason, the two questions go hand in hand. TADAEEM was designed in 2015–2016 (prior to the adoption of the CCL in May 2018) to directly contribute to USAID/Tunisia’s DO2: Social Cohesion Promoted through Democratic Consolidation. The ET queried USAID, GOT, and other donors on the design and inception phases, and the degree to which the GOT was consulted and involved. According to USAID team members, the nascent USAID Mission at the time was led by a Senior Development Specialist, who restricted the design team to speaking only with the Ministry of International Cooperation while they prepared the design, which was being undertaken to respond to a large democracy and governance (DG) earmark from the U.S. Congress. Because of the active local governance sector among donors, one USAID person involved reported that the design team reached out instead to other donors, who were closely involved with the MALE, as a proxy for direct contact. The resulting design, if more demand-driven at the municipal level than some USAID activities in this sector, was still within the boundaries of USAID’s decentralization efforts worldwide, according to a USAID desk officer. Deloitte US entered the ensuing procurement process and won the award. Two key decentralization players – one in a high-level GOT role and the other an activist in the sector who became TADAEEM’s Technical Director – played important roles in the development of both TADAEEM and that of the CCL. Activity leadership reported that this meant their work was closely aligned with the GOT’s goals on decentralization. TADAEEM’s two key components were Objective 1 (O1), service delivery, which is discussed in the CCL (articles 235–242) and Objective 2 (O2), citizen participation, which appears in CCL articles 29–37. The CCL obliges municipalities to respect citizen participation in annual meetings, Plan Annuel d’Investissement (PAI) and budget discussions, and democratic referendums. Article 240 of the CCL discusses the public services that municipalities should provide, such as roads and parks, public lighting, waste collection, and municipal markets, inter alia. Article 30 further obliged municipalities to keep a registry of local CSOs with updated contacts and to take into account citizens’ suggestions and complaints. Working per the design, TADAEEM was indeed closely aligned with these facets of the CCL. Less of a “pivot,” then, when the CCL became law and municipal councils were newly elected, TADAEEM was able to simply “ramp up” what it had already been planning to do. It worked directly with municipalities without central government interlocution. That is, 7 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 TADAEEM was in contact with the appropriate MALE teams, and even signed a Memorandum of Intent (MOI) with them, but did not share decision-making power with the central GOT. Evaluation respondents from participating municipalities reported that they were quite happy to be the focus of this “bottom-up” mode of addressing decentralization. When asking about the degree of coherence of TADAEEM’s design with GOT decentralization goals, the ET regularly heard that the Activity’s Service Development Improvement Plan (SDIP) process operationalized the goals of the CCL and the Constitution with respect to both participation and service delivery improvement. The great majority of GOT respondents disagreed vehemently with this format for the activity. They felt that TADAEEM should have supported the central government’s own decentralization reforms directly, and that the Activity should have worked with municipalities only through the MALE – a more “top-down” approach. Some within TADAEEM agreed, because working with the MALE interlocutors in the absence of that direct support had been so contentious. Others felt TADAEEM aligned well enough with the early draft of the CCL and with the Constitution, but that the final CCL brought paradigms and principles such as peer-to-peer municipal support, power devolution and power sharing, and issues around local finance – which TADAEEM did not broadly undertake. Nor did TADAEEM’s most visible interventions support the central government with its own efforts on decentralization – though its collaborative Objective 3 (O3) work was designed for that purpose. While there is further discussion on O3 in the response to EQ3c, it is important to mention here that the GOT’s progress on its own decentralization calendar has been uneven, in part due to reluctance on the issue of decentralization. The GOT has not yet deconcentrated functions to regional authorities, as planned, such that some TADAEEM collaborations proposed between different levels of government were less effective, or even impossible, as a result. Another critical example is that the regulations needed to operationalize the CCL more fully are still not passed in the legislature, nearly four years later. These did not prevent TADAEEM from operating, but they curtail some activities. The MALE and TADAEEM signed a non-binding Memorandum of Intent (MOI) in August 2018, declaring mutual commitment to “improve local level service delivery and facilitate the long-term transition to effective decentralized governance” based on a defined “scope and methodology of cooperation” and a “firm belief in working together” to achieve shared objectives. This document, included as Annex J, is focused almost entirely on municipal-level interventions, with the exception of a bullet on supporting MALE structures “to in turn support local communities” and the promise to set up a sort of steering committee at the national level. TADAEEM’s Attachment to the MOI also includes limited language about coordinating across GOT levels. It’s easy to see how readers from one perspective or another could find what they needed to justify their arguments, but in broad terms, the MALE agreed to the MOI’s stated municipal focus only to later disagree with it. This change of opinion is natural, given frequent “change of GOT vision and agenda” on decentralization. There were six different ministers over the life of TADAEEM: it would have been difficult for TADAEEM to align with the preferences of each, since the objectives and overall design were codified in a contract. The 27-year decentralization plan was in place but missing deadlines, Still, on the central goals of citizen participation and service delivery improvement at municipal level, TADAEEM did align consistently with CCL principles as well as USAID’s Democracy and Governance (DG) goals, as is detailed in the sections on EQ3a and 3b, below, which reflect the relative successes of TADAEEM’s Objectives 1 and 2. Conclusions TADAEEM objectives and activities tackling service delivery difficulties and improving citizens’ participation in decision-making processes (O1 and O2) were well-aligned with CCL and regulations in force (see also the responses to EQ3a and 3b, below, for supporting data).The slow process of decentralization and stalled steps in its national plan hindered TADAEEM’s ability to collaborate with some regional counterparts and potential counterparts. Some central government actors were reluctant about the active 8 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 implementation of these reforms, which puts TADAEEM in a position of not being able to do all it promised for collaborations between government levels – municipal, regional, and national. Though TADAEEM was designed prior to the passage of the CCL, there were discussions with MALE and needs assessments – including the first wave of the Activity’s Citizen Perception Survey (CPS) – which defined gaps and prepared the ground for the Activity. This and the eventual MOI operationalized how TADAEEM could be most relevant and contribute to decentralization. Though the CCL is a complex legal text, it presents opportunities for activities like TADAEEM, even before passage of operationalizing regulations. There do exist other areas with which TADAEEM could have aligned (or a future project could align), like the new paradigms and principles of intercommunity collaboration, and power devolution and sharing. Another critical area is that of local finance issues, which TADAEEM touched but did not focus upon. USAID activities can certainly do more, up to and including support to the state reforms needed for decentralization – there may even be benefits to working both top-down and bottom-up at once – but it is not a priori a requirement and would involve a trade-off like any design decision. The municipal focus was codified in a project MOI with the relevant ministry, but this did not forestall conflict over this issue. There was a fundamental disagreement between MALE and TADAEEM management teams, despite the language of the MOI, on the primary unit of the intervention: the municipality. Other important differences existed at different times over the life of the Activity, and the relationship was weak on other fronts. But if TADAEEM had worked more at the central level on decentralization-related reforms, this might not have been such a long-term problem for the Activity. There might also have been more “ownership” of the O3 products (discussed in detail in the response to EQ3c) and more use. 5.2 EQ3: Effectiveness in TADAEEM Objectives To what extent and how have TADAEEM’s individual activities contributed to achieving results? What were successes and what could have been better? a) Citizen participation in and oversight of decision-making processes at the subnational level b) Municipal institutional and service delivery performance c) Regional and national service delivery mechanisms to respond to citizen needs d) Improved coordination and communication of municipalities, regional governments, and central government in responding to national and local emergencies and other crises EQ3a: Citizen participation in and oversight of decision-making processes at the subnational level Findings TADAEEM’s Objective 1 called for interventions that would increase citizen participation in municipal decision-making and public planning processes. The most common process where TADAEEM worked on this with municipalities was the Plan Annuel d’Investissement (PAI – Annual Investment Plan) process. According to the final Activity report, TADAEEM organized 44 PAI trainings in 21 municipalities for 655 participants, of which 44 percent were CSO members. TADAEEM also convened 11 PAI citizen mobilization workshops in 10 municipalities to coordinate outreach activities between municipal authorities and CSOs. In other partner municipalities, TADAEEM did strong ground-level work to facilitate collaboration between CSOs and municipal authorities in implementing the PAI process. As an example, TADAEEM collaborated with 15 CSOs and the municipality of Kairouan to design and implement a citizen mobilization plan for the local PAI. In total, CSOs and municipal authorities jointly conducted 27 PAI citizen outreach and mobilization activities across 19 partner municipalities with the support of TADAEEM. 9 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 TADAEEM also encouraged partner municipalities and CSOs to use collaborative and grounded ways to sensitize citizens about their roles in local governance and mobilize them to participate in the PAI sessions. In Kebili, Tozeur, and Gafsa, for example, the WeYouth CSO trained 30 women ambassadors with the support of a TADAEEM grant to provide a safe space to 110 female participants to discuss their roles and responsibilities in their municipalities by inviting them to prepare a shared lunch in their homes. Another case in point was the production and distribution of the first Braille PAI information sheets in Gabes municipality in partnership with the Blind High School in Gabes. In addition to these citizen outreach and mobilization efforts, TADAEEM provided direct assistance to municipal authorities and CSOs in organizing plenary PAI meetings in 24 partner municipalities. These plenary meetings used the neighborhood I model to mobilize citizens from different sub-areas within partner municipalities, as well as participatory and inclusive methods to engage citizens from various social groups. Municipal authorities and local CSOs interviewed for this evaluation consistently mentioned the positive outcomes of TADAEEM’s support at the PAI sessions in increasing citizen participation in their municipalities. The evaluation survey results also corroborate this finding. As shown in the Figure 1 below, 30 out of 38 CSO (8) and municipal representatives (22) shared the opinion that TADAEEM interventions increased citizen and CSO participation in their municipal meetings. Figure 1: Did TADAEEM contribute to increased attendance and participation of CSOs and citizens in municipal meetings? According to the final Activity report, in the 23 partner municipalities where TADAEEM assisted municipal authorities in the organization of 2019/2020 PAI plenary meetings, the number of citizens participating in these meetings grew by 76 percent compared to before TADAEEM. However, while this percentage rise in citizen participation at the PAI sessions is remarkable, it also obscures several notable limitations. First, as shown in Table 4 below, PAI meeting participation rates were very low or even zero in several municipalities prior to TADAEEM interventions. As a result, any increase in the number of individuals participating in the PAI meetings supported by TADAEEM meant a significant percentage rise in attendance rates; TADAEEM respondents recognized this was a “low bar” as well but did not find a better way to report on participation, and related indicators were removed from their reporting. The only data TADAEEM reported on participation appear in the municipal summaries, which show PAI participation from 2018 and 2019 PAI processes (informing 2019 and 2020 PAIs). There are no later data from TADAEEM to show whether participation rebounded after COVID-19 restrictions, because they had no pertinent indicator on the topic. TADAEEM’s CPS data is also instructive with regard to participation in the municipal contexts. CPS respondents – who are representative of municipal populations – who confirmed attending at least one 10 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Municipality Before After Population % Metouia 0 270 13,289 2% Ghannouche 0 161 31,871 0.5% Bouhajla 43 184 68,539 0.3% Tataouine 50 445 91,300 0.5% Gabes 27 186 106,438 0.2% El Fahs 62 110 36,252 0.3% Haffouz 35 85 26,474 0.3% Ennadhour 89 98 32,088 0.3% Zaghouan 159 189 41,264 0.5% Dahmani 91 169 27,677 0.6% Kebili 20 554 46,310 1.2% Tozeur 20 160 50,744 0.3% Sidi Alouane 11 31 27,350 0.1% Ksour Essef 0 25 36,936 0.1% Oueslatia 53 130 27,785 0.5% Makther 73 146 29,107 0.5% Tajerouine 57 87 25,445 0.3% Gafsa 30 115 119,438 0.1% Kairouan 25 163 181,197 0.1% Agareb 78 223 43,841 0.5% Siliana 54 102 49,410 0.2% Sers 30 158 17,586 0.9% El Kef 40 512 65,905 0.8% Total 1,047 4,303 1,139,519 0.4% municipal session increased by only 3 percent between 2019 (5 percent) and 2021 (8 percent) in TADAEEM partner municipalities. This is because the rates of citizen participation attained during the 2019 and 2020 PAI processes in TADAEEM partner municipalities remain low by GOT standards. Per the GOT’s Annual Municipal Performance Indicator, Tunisian municipalities earn 8 points in their total performance score when they mobilize more than 1 percent of their population at PAI sessions. Table 4 below shows that only Metouia and Kebili met that standard in 2019/2020 PAI sessions – which, not coincidentally, parallels the engaged and empowered responses from participating CSOs and others in those two sites compared to elsewhere in the evaluation site visits. In nine TADAEEM municipalities this percentage was between 0.5 and 0.8 and it was below 0.5 in 12 TADAEEM municipalities. While there is no genuine counterfactual showing impact attributable to TADAEEM, the data show population mobilization in TADAEEM sites was not exemplary, even if it had improved from before the Activity. Table 4: PAI attendance rates, before and after TADAEEM, and as percent of population Nearly half of the 92 percent of CPS respondents who acknowledged not attending a single municipal session in 2021 said their municipalities did not inform them about these sessions, and a little under a third of the 92 percent declared they were not interested in attending. These data points show that there are limits to TADAEEM’s citizen outreach and mobilization efforts, even when they are collaborative and grounded. In evaluation interviews, TADAEEM field teams said that social structures and civil society dynamics varied among partner municipalities, which also limited citizen mobilization at times. This also helps to explain why PAI attendance rates differ so widely among TADAEEM partner municipalities. The table above shows that in municipalities with more urban and close-knit social communities and vibrant civil society space, such as Metouia, Kebili, and El Kef, citizen mobilization and participation rates during 11 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 PAI sessions were higher. In municipalities with a more rural or metropolitan social structures and a minimal CSO presence, such as Ksour Essef or Sidi Alouane, these rates were much lower. Outside of the PAI process, TADAEEM also assisted partner municipalities in strengthening CSO and citizen engagement in improving public municipal services – so, it linked O1 and O2. TADAEEM collaborated with municipal authorities and CSOs as part of its SDIP methodology to involve local communities in selecting priority areas for service improvements. TADAEEM supported 54 municipal service data collection and diagnostics in 21 municipalities in collaboration with local CSOs, university students, and citizens, as part of this process. These data collection and diagnostic activities used a variety of citizen input approaches, such as participation surveys, scorecards, planning workshops, and community events. As a result, 18 TADAEEM partner municipalities conducted 33 in-person and 6 online surveys to gather citizen feedback and recommendations on municipal services. TADAEEM also supported 11 SDIP workshops in 20 partner municipalities, bringing together 130 officials, CSOs, and residents to examine and approve their SDIPs. TADAEEM also held six workshops in four partner municipalities, engaging municipal staff, CSOs, and citizens in particular green park rehabilitation initiatives. The Tounissiet CSO received a TADAEEM grant to train 15 female leaders from Mareth, Ghannouch, and Gabes on how to utilize gender-responsive scorecards to evaluate municipal services via a gender lens. The 15 women leaders arranged training workshops for 345 individuals to offer proposals for improving priority services for women, then combined their findings into three policy papers and submitted them to municipal officials. Following the installation of 255 light-emitting diode (LED) lights in Agareb, TADAEEM assisted local CSOs, artists, and activists in organizing a local LED festival involving 85 people and municipal staff to enhance community awareness about the value and maintenance of LED lighting. In addition, TADAEEM trained municipal staff and CSOs to improve municipal communications and responsiveness to residents. It also helped municipalities set up open phone lines, manage their Facebook sites, and employ information technologies (IT) to connect with citizens more efficiently while also handling their complaint reports. As part of these efforts, TADAEEM held 12 communication trainings for 84 municipal workers and 6 other trainings for 82 CSO activists throughout 18 partner municipalities. TADAEEM also provided IT equipment to 33 partner municipalities to enable continued engagement with citizens throughout the COVID-19 outbreak, and trained 1288 municipal staff on maintaining online interactions with citizens during and after COVID-19. The evaluation survey results provide important stakeholder feedback on the effectiveness of these activities. As seen in the figure at left, 22 of 38 CSO and municipal respondents agreed to some extent that TADAEEM interventions helped improve citizen participation in their municipalities, while 10 fully agreed, and 6 disagreed. Interestingly, CSO respondents tended to be less appreciative of TADAEEM contributions than were respondents from partner municipalities. Two important findings derived from evaluation interviews provide clarity to this picture. First, TADAEEM collaborated with dozens of local CSOs to improve citizen participation in partner municipalities, but awarded activity grants to only seven partner CSOs, two of which have a more national than local constituency. Despite TADAEEM’s announcement of two proposal solicitations for local CSOs, during which TADAEEM field staff made concerted attempts to co-design activity concepts with CSOs and municipalities, the solicitations were canceled without providing 12 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 justifications to CSO stakeholders. As a result, a number of CSOs and other local civil society players have been deterred from collaborating with TADAEEM. During evaluation interviews, several stakeholders characterized these aborted activity grants to local CSOs as a squandered opportunity for TADAEEM to build more effective and sustainable mechanisms for citizen participation in partner municipalities. Although local CSOs generally lack the financial resources and sophisticated technical expertise needed to conduct complicated initiatives, they do often have thorough awareness of the dynamics and needs of their communities. Local CSOs also have greater local interest and mobilization capacity to steer participation in and oversight over local government institutions because they are embedded in their communities. More crucially, the evaluation interviews revealed that despite the atmosphere of trust and patterns of citizen engagement created during the yearly PAI sessions, TADAEEM did not establish any durable mechanisms to enable continued citizen participation in partner municipalities – although it is necessary to note that COVID restrictions were also responsible for reductions in participation in 2020 and, to a much lesser extent, in 2021. Though oversight was part of the goal of Objective 1, TADAEEM did not help municipalities establish effective tools or systems for citizens to exercise oversight over local spending or holding municipalities accountable for executing investment plans adopted during participatory PAI meetings. According to interviews, this has contributed to increased citizen discontent and disengagement in a number of TADAEEM partner municipalities, as municipal officials have been unable to deliver on promises made to citizens during PAIs and other participatory planning activities. However, the limitations of TADAEEM interventions to improve citizen participation must be seen in light of context as well. Municipal election voter turnout rates in 2018 (the first elections for municipal councils) were 23 percent lower than in the 2014 legislative elections. Only 1.9m citizens voted in 2018 (around 36 percent) and 3.5m abstained, while in the 2014 legislative elections, participation was 65 percent – almost 3.6m votes. The 2018 elections came in the context of a governance crisis aggravated by economic decline. Two years later, the COVID-19 outbreak worsened governance and economic crises in Tunisia, fueling political polarization and citizen distrust toward government institutions. As a result, the country witnessed a significant uptake in social protests and labor strikes. These factors negatively affected the legitimacy, stability, and politicization of local government institutions, as detailed in the response to EQ4, below. Conclusions Under Objective 1, TADAEEM made ample efforts and carried out thoughtful work at ground level with innovative and inclusive initiatives to improve CSO and citizen participation in and (to lesser extent) oversight over local government institutions. TADAEEM succeeded in increasing citizens’ participation rates in municipal decision-making and planning as compared to before TADAEEM, particularly through the PAI process, although the percentage of citizens who participate remains low by GOT standards and varied greatly between partner municipalities. TADAEEM has also fostered an atmosphere of trust and patterns of citizen engagement in partner municipalities through outreach and consultation initiatives that enabled citizens to voice their concerns and needs in terms of municipal service improvement. On the other hand, TADAEEM failed to build effective and long-lasting mechanisms to ensure sustained citizen engagement in and oversight of their municipalities. Despite some promising success stories, TADAEEM was also unable to build long-term and durable collaborations between partner municipalities and local CSOs. In this regard, the absence of activity grants to local CSOs was a significant internal constraint. In addition, the national context of citizen distrust toward government institutions, driven by the governance crisis, economic decline, and political polarization, imposed strong external constraints on TADAEEM’s O1 programming. EQ3b: Municipal institutional and service delivery performance Findings 13 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 The elected municipal councils in the 2018 elections inherited both a centralized legacy and high expectations from citizens to improve municipal performance. The councils had to learn quickly to incorporate and implement new decentralization legislations and policies. To support partner municipalities (both elected councils and civil servants in the administrative staff) to improve their service delivery capabilities and cater to citizen needs, TADAEEM implemented a Service Delivery Improvement Plan (SDIP) approach in integrated four phases: 1. Service Prioritization: TADAEEM facilitated workshops with municipal staff, CSOs, and citizens to review CPS data and identify citizens’ satisfaction levels with municipal services and improvement needs, then determine which services partner municipalities should prioritize in implementing TADAEEM-supported SDIPs. Per final reporting, TADAEEM: • Convened 13 SDIP citizen engagement workshops with 49 municipal staff, 43 CSOs members, and 239 citizens in 11 partner municipalities • Organized 5 CSO service prioritization workshops with 66 citizens and 30 CSOs members in 5 partner municipalities 2. Root Causes Analysis: To uncover issues with prioritized services, TADAEEM experts guided municipal staff to review the entire management and delivery process of these services. Then, each issue was assessed and graded according to its impact on citizens, operational efficiency, and the municipality’s capacity to handle it. Finally, the root causes of the prioritized issues were identified jointly by TADAEEM experts and municipal staff. 3. SDIP: TADAEEM staff and partner municipalities used the root causes analysis to identify three interventions that could be completed in approximately six months. On this basis, TADAEEM and municipal partners developed SDIPs with detailed activities, responsibilities, and performance targets. 4. Implementation: TADAEEM provided hands-on, on-the-job training to municipal staff, citizens, CSOs, and institutional partners to implement SDIPs and measure improvements. TADAEEM also enhanced partner municipality capacities to improve their service management systems by collecting performance data and mapping services with open-source Geographic Information Systems (GIS). TADAEEM convened 55 service delivery trainings for 401 municipal staff and 78 citizens in 25 partner municipalities. TADAEEM also provided 41 trainings on service data collection for 172 municipal staff and 160 citizens in 21 municipalities. In addition, TADAEEM supported SDIP implementation by providing partner municipalities with service equipment, as detailed in Table 5 below. Table 5: Municipal Service Delivery Equipment Provided Service Equipment Total Number of Municipalities Waste Management $2,148,518 33 Roads $2,139,211 33 Public Lighting $584,919 18 Parks and Green Spaces $421,288 33 Municipal Building Solar PV $119,850 6 Markets $12,283 2 Traffic Planning $11,420 2 Rainwater Drainage $1,489 1 TOTAL $5,438,978 33 Source: TADAEEM Final Report As shown in Map 1 below, and reported in their final monitoring data, TADAEEM supported the implementation of 64 SDIPs for five prioritized municipal public services across 33 partner municipalities between 2019 and 2021. These SDIPs yielded varying results, according to TADAEEM and municipality performance statistics. The following highlights improvements in service delivery achieved through SDIPs for three main prioritized services: 14 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 • Solid waste management: The SDIPs for waste management increased municipal waste collection by 18 percent across 24 partner municipalities. Partner municipalities also increased the garbage collection coverage by an average 11 percent; Ghomrassen had the biggest improvement at 38 percent and lowered its cost per ton of waste collected by 20 percent. SDIPs also aided in the eradication of 407 informal dumpsites throughout partner municipalities, eliminating 64 in Kef municipality alone. • Public lighting: The SDIPs for public lighting assisted 15 municipalities to install nearly 3,000 LED bulbs, replacing 11 percent of municipal public lighting networks with LEDs and improving their lifespan by 417 percent. This resulted in a 234 percent increase in brightness in partner municipalities. SDIP pilot areas reduced energy usage by 27 percent on average, with Gabes and Gafsa municipalities cutting energy consumption by 44 percent. This has resulted in a 31 percent average savings for the municipalities per year, nearly USD 6,000 in Gafsa municipality alone (using the exchange rate in October 2021, as of TADAEEM’s final report). • Road maintenance: SDIPs for road maintenance enabled 12 partner municipalities to map 486.2 km of municipal roads in GIS by type and condition to promptly identify issues and save bigger repairs. TADAEEM also built the capacity of 147 municipal workers who completed 2,969 meters of repairs during on-the-job trainings. TADAEEM trained 37 municipalities on a MALE-endorsed Road Coordination Manual to help them better coordinate utility providers’ operations on municipal roadways. Map 1: TADAEEM’s SDIPs by Municipality According to the evaluation survey results in Figure 3 below, 18 respondents from municipalities (16) and CSOs (2) recognize that the SDIPs helped improve service delivery to a large extent, 17 to some extent, and only 3 said there were no improvements. Municipal officials and CSO respondents spoke most positively about the SDIP implementation approach during the evaluation interviews. Source: TADAEEM Final Report 15 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Figure 3: Did TADAEEM help your municipality improve service delivery? Partner municipalities particularly praised the blending of service provision with fit-for-purpose support and on-the-job trainings in the SDIPs, which provided costly equipment and much-needed technical expertise while improving the capacities of their personnel via practical learning. Their appreciation reflects staffing and financial gaps in municipalities: just 11.8 percent of total municipal workforce are professionals or technical managers, and in 2019 the financial resources of local governments accounted only for 2.8 percent of Gross Domestic Product (GDP) – only 1.8 percent of which was for municipalities.10 Another area municipal and CSO respondents praised was how the Activity incorporated citizen perspectives and input through the SDIP process. They reported that citizen engagement in prioritizing and diagnosing targeted services increased citizen participation in their municipalities and motivated authorities to be more attentive to citizen concerns and needs. Survey results shown in Figure 4 below, reinforce these findings. Of 26 municipal respondents, 21 were highly satisfied with TADAEEM’s support to their municipalities, five are moderately satisfied, and none dissatisfied. CSO respondents’ answers were overall less positive, which aligns with their lesser involvement with the benefits of TADAEEM. However, CSOs members also emphasized that civil society and citizen engagement in SDIPs was merely consultative and should have enabled durable, systematic oversight over SDIP implementation and results. Moreover, interview responses from many sites said participation had waned since TADAEEM ended. 10 Haute Instance de la Finance Locale, « Premier Rapport Annuel: 2019 », December 3, 2020, p. 27. Retrieved from: HIFL￾RA2019 FR-03-12-20_Fn.pdf 16 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Figure 4: Were you satisfied with the support provided by TADAEEM? Despite improvements in municipalities’ service delivery capabilities achieved through TADAEEM’s SDIPs, the CPS data displayed in Figure 5, below, shows that citizen satisfaction increased by only 2 percent between 2019 (46 percent) and 2021 (48 percent) across partner municipalities, and across all five services prioritized by SDIPs. The highest increase in satisfaction was recorded with waste management (3 percent) and public lighting (5 percent). Satisfaction with the state of parks and green spaces and roads declined by one percent. By any reckoning, a two percent increase in citizen satisfaction with these services is a poor result for a decentralization project of TADAEEM’s magnitude and expenditure. However, the results need to be unpacked in light of various internal and external factors. Figure 5: Comparison of Citizen Satisfaction by Municipal Service in 2019 and 2020 CPS Source: TADAEEM CPS Report CPS statistics show that citizen satisfaction with municipal services is consistently greater in urban than in rural areas. From 2018 to 2021, satisfaction in urban areas grew by 8 percent, from 44 to 52 percent, but in rural areas satisfaction remained unchanged at 28 percent. Furthermore, as shown in Figure 6, below, urban respondents were more satisfied with all municipal services, with the exception of parks and green areas, than rural respondents. 17 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Figure 6: 2021 Citizen Satisfaction by Service in Urban and Rural Areas Source: TADAEEM CPS Report Nearly a third – 32 percent – of the Tunisian population was living in rural areas outside municipal constituencies until 2016. The new municipal territorial structure, which went into effect in May 2016, either annexed rural areas to existing municipalities or constituted them as separate municipalities. CPS respondents were also mostly urban, at 62 percent, with 32 percent rural respondents. Although TADAEEM did not engage with any of the newly constituted municipalities, some partner municipalities saw significant territorial growth to encompass neighboring rural areas. Municipal services have historically been concentrated in urban areas, explaining the 24 percent variance in satisfaction ratings between urban and rural respondents for the services targeted by TADAEEM. Moreover, the urban/rural split is mirrored in the variance in satisfaction ratings with municipal services within municipalities. For example, 62 percent of urban respondents in El Alaa municipality saw improvements with public lighting in 2021, compared with only 4 percent of rural respondents. In Bouhajla municipality, half of urban respondents were satisfied with solid waste management, but just 16 percent of rural respondents. Moreover, TADAEEM piloted the majority of the SDIPs inside urban centers in partner municipalities. This not only explains the disparity in satisfaction ratings between rural and urban respondents, but it also suggests that SDIPs may have unintentionally exacerbated established disparities in access to municipal services between urban and rural communities. This also explains, however, why TADAEEM’s SDIPs were more effective in raising satisfaction ratings in small urban municipalities than in large metropolitan municipalities. For example, in Dahmani and Metouia, both low-population-density urban towns, satisfaction with public lighting improved by 11 percent. However, satisfaction with public lighting declined by 4 percent and 7 percent, respectively, between 2019 and 2021 in Gafsa and Gabes, two metropolitan municipalities with extensive public lighting networks. Beyond the urban-rural gap, Tunisians’ low levels of trust in government influenced their satisfaction with public services from all levels of government, which was exacerbated by the COVID-19 crisis. Despite being the most democratic country in the region, trust in the government in Tunisia has plummeted. The Arab Barometer polls show how dissatisfied Tunisians are with government performance, and its decline during the pandemic, with 17 percent satisfied in July 2020, and 15 percent in March 2021.11 The CPS data 11 Arab Barometer. “Taking Tunisians’ Pulse: Key findings from AB6 survey 2020-2021,” January 10, 2022. Link: Taking Tunisians’ Pulse : Key findings from AB6 survey 2020-2021 – Arab Barometer 18 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 are consistent with these results. Figure 7 below shows that rural respondents were much less satisfied with municipalities’ responsiveness; this figure worsened overall over time, as well. Some 62 percent of respondents in 2021 considered that their municipalities were not responsive to citizens’ needs with regard to public services prioritized by TADAEEM’s SDIPs, up from 52 percent in 2019. Figure 7: Citizen Perceptions of their Municipalities’ Responsiveness to Service Needs, CPS 2021 Source: TADAEEM CPS Report However, Tunisians’ top priorities in 2020 and 2021 were economic issues and COVID-19 as opposed to municipal public services, according to Arab Barometer surveys. During a COVID-19 wave of infections in October 2020, 40 percent of Tunisians were concerned about the spread of COVID and 40 percent were most concerned about the economy. But before that COVID wave, in surveys conducted in July 2020 and March 2021, 51 percent of Tunisians saw the economy as the country’s main concern, while 15 percent cited COVID-19.12 These findings – in part from TADAEEM’s own CPS – suggest that although TADAEEM’s SDIPs offered considerable assistance to partner municipalities to improve the quality and efficacy of their public services, they did not enable municipalities to respond to Tunisians’ highest priority needs: economic challenges and job growth. Outside of service delivery, TADAEEM piloted only one initiative to assist partner municipalities in enhancing their own income generation, which would empower them to become agents of local economic growth. In that pilot, in 2019, TADAEEM aided Kebili municipality in conducting a property survey to produce a more accurate estimate of property tax income. The amended property database was uploaded to the Budgetary Resource Management site of the Ministry of Communication Technology, where the municipal accountant confirmed the new properties and municipal assets. To scale-up this initiative, TADAEEM awarded a grant to undertake property tax surveys in El Alaa, El Jem, El Kef, El Nadhour, Haffouz, Metouia, Om Larayes, Siliana, Tataouine, and Tozeur. Initial evaluations revealed that 42,870 new and obsolete properties needed to be examined throughout the 11 partner municipalities. In total, this survey resulted in the addition of 65,162 properties to municipal property databases. The 11 partner municipalities increased their property tax database records by 38 percent for buildings, 11 percent for vacant land, and 43 percent for commercial and industrial properties. The Secretary General of Kebili municipality estimated the updated property database has resulted in an additional 32 percent more tax revenues. In January 2022, municipalities that conducted the surveys received 50 percent of their estimated revenues as an advance from the GOT, which the municipalities can invest while 12 Ibid. 19 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 collecting the remaining revenues. TADAEEM, on the other hand, did not capitalize on the promising outcomes of this initiative to improve the capacity of partner municipalities in investing their own generated income to promote employment creation and economic development in their communities. Conclusions TADAEEM’s integrated SDIP approach assisted partner municipalities in strengthening their service delivery capacities and meeting citizen needs. Between 2019 and 2021, TADAEEM supported the implementation of 64 SDIPs for five prioritized municipal public services across 33 partner municipalities. These SDIPs have resulted in significant improvements in the service delivery capacity of partner municipalities, though the extent of improvement varies greatly depending on the service or services and the municipality. Given the size of the award, the ET is concerned that this level of key Activity outcomes is low. The service provision and hands-on trainings during TADAEEM SDIPs were highly useful and improved municipal capacities, often in a sustainable way. Where citizen engagement was included, and later a gender-sensitive participatory approach, it enhanced the strength of the SDIP process and of the results. Citizen engagement in prioritizing and diagnosing service improvement needs increased citizen participation and motivated improved attention to citizen needs among municipal authorities. However, civil society and citizen engagement in SDIP processes was merely consultative and should have enabled citizen oversight over the implementation and results of the SDIPs. Despite improvements in municipalities’ service delivery capabilities achieved through SDIPs, citizen satisfaction increased only very slightly, in an overall context of low levels of trust in government, which were in turn exacerbated by the COVID-19 crisis. Satisfaction was higher in urban areas than in rural areas, in parallel with implementation. Tunisians’ priorities center on economic issues and job creation more than on municipal public services. TADAEEM did not deeply explore the ways it might have supported municipalities on these issues as opposed to the set of services – valuable to be sure, but less valued by Tunisians – where they did intervene. EQ3c: Regional and national service delivery mechanisms to respond to citizen needs Findings For this objective, TADAEEM worked across GOT levels to build sustainable and service-oriented coalitions, such as national strategies, platforms, and standards. This was despite the contextual challenge that the GOT has still not deconcentrated power to regions or governorates as called for in its decentralization plan, limiting opportunities for cooperation. At the same time, the GOT’s July 25, 2021, actions changed the institutional landscape considerably. Still, the TADAEEM Final Report cites 11 successful efforts, as follows: Demolition and construction waste Roads service policy Waste management service Household waste management strategy Municipal markets service Public lighting standards Parks and green spaces service Public lighting service Public–private partnerships Roads intergovernmental coordination Planning and budgeting 20 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Figure 8: Did TADAEEM help your municipality work better with regional and national government? Many interviewees referenced these works in discussions about TADAEEM’s O3 accomplishments, which link thematically to a set of technical implementers’ guides for municipal practitioners. The ET heard differing reports on the degree of involvement of the GOT in their production, from one report of “tight coordination with five ministries” and regional offices, to others who cited “no involvement of ministry technical services” – with these conflicting reports coming from the same Ministry, and from respondents with direct knowledge. Similarly, there were reports of both “long and important ministry vetting” and “ministries completely cut out of vetting.” Figure 8 below shows survey responses to the question of whether TADAEEM helped their municipality work better with regional and national government, with the majority of responses saying TADAEEM had done so, at least some. Given the 11 distinct efforts, it is possible that some of these were more collaborative than others. The ET heard similar acrimony among GOT partners regarding O3 as that described around EQ1 and 2. GOT actors wanted to drive efforts like this and were not content with TADAEEM’s management. Moreover, the guides, strategies, and documents have not been disseminated. Responsibility for the lack of dissemination is complex: the GOT’s post-July 25 ministry reshuffle has meant the MALE is reconfigured beneath the Ministry of Interior, and the guides have not been pushed to municipalities by the MALE staff who remain. At the same time, TADAEEM’s failure to translate these to Arabic makes them less useful. And the poor relationship between TADAEEM and the MALE set the effort on a shaky path from the start, with the result being that the GOT has less ownership of these guides and other materials. Another effort was an e-Construction platform that would allow citizens to apply for and receive construction permits online. The MALE, Ministry of Equipment, the Order of Architects of Tunisia, and GIZ worked together – according to reports across the respondents – to develop, test, and roll out the system. Despite notable progress and successful testing, and TADAEEM’s support with computers for municipalities, the platform has not been implemented. TADAEEM worked across actors to coordinate between utility companies and municipalities to map and provide a framework for day-to-day collaboration between public utilities and municipalities. This was applied a handful of times and generated very positive reports from municipalities. Most respondents (from municipalities) in the three pilot sites did say, however, that implementation had stalled when TADAEEM ended. Conclusions Work on Objective 3 demonstrated TADAEEM’s and USAID’s commitment to national-level goals. While this was not the primary focus of the Activity, there were positive outcomes, products, and relationships built. Unfortunately, in part due to the July 25 presidential action in Tunisia (2021), the promise of most of the efforts has not been realized. The lack of deconcentration to regions also limited how well TADAEEM could work with regions and governorates. Considering the at times antagonistic relationships with GOT, the failure of these efforts to be implanted in GOT structures and used across the country is not entirely surprising. Survey respondents said TADAEEM supported their work with GOT bodies, at least “some” 30 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 EQ3d: Improved coordination and communication of municipalities, regional governments, and central government in responding to national and local emergencies and other crises Findings Respondents from around the country and in different roles agreed that the MALE coordinated donors very well, including USAID’s TADAEEM. TADAEEM’s $5m in additional funding was combined with that of other donors to get critical materials and equipment to municipalities. The Nabeul governorate provided warehouse space, and TADAEEM quickly procured sanitation supplies and equipment that 151 municipalities then retrieved from that location. Given low resource allocations from the GOT to municipalities, this support was widely appreciated. However, those same resource levels mean that municipalities have not often been able to replace these materials once expended. TADAEEM also procured videoconferencing equipment for 327 municipalities, 17 governorates, and the MALE and its training center. Having added the new Objective 4 to their scope on communication and coordination during emergencies, this was quite different from their earlier scope, but it worked well – according to respondents – in improving communication and emergency coordination during COVID. The equipment has been used for communications and training, including some training on the guides mentioned under EQ3.c above. Twenty municipal survey respondents (of 26), and six from CSOs (of 12), agreed that TADAEEM had improved emergency coordination and communication during COVID. Nevertheless, municipalities are sanguine about any durable effects of these efforts. TADAEEM helped some municipalities establish a local coordination committee (below left, Figure 9) but respondents report that such committees are not always continuously operating (below right). Figure 9: Local coordination committees established (at left) and operating regularly (at right) Conclusions MALE successfully coordinated the COVID response and donor contributions to it, and TADAEEM used its $5m budget amendment to procure equipment and supplies quickly and successfully for 151 municipalities, which otherwise would not have had access. Low resources continue to limit municipalities’ abilities to respond to emergencies. There were some improvements in communications and coordination, largely from TADAEEM’s provision of videoconferencing equipment nearly nationwide. However, cross-GOT crisis coordination committees have not taken root in municipalities. 5.3 EQ4: Challenges and mitigations 31 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 What were the internal (staffing, procurement, organizational structure, etc.) and external factors (lack of local technical capacity, national and subnational resources, political will, corruption, etc.) that affected the timeliness and achievement of TADAEEM’s objectives and what measures did TADAEEM and its implementing partners take to mitigate those factors? Findings Many of the most important challenges in this section have both internal and external elements, so the ET has not separated the following challenges on that variable. This section also does not lend itself to conclusions, per se, though recommendations from among these lessons will be included in this report. Difficult relationships between GOT and TADAEEM. This began even before award, when the USAID design team was restricted to working only with the Ministry for International Cooperation, rather than the more appropriate MALE. MALE and other GOT actors reported to the ET that USAID and then TADAEEM had not sufficiently consulted with – and worked through – the MALE to implement the project. During the first 18 months, there was significant turnover in both the Ministry (6 Ministers, and multiple Ministry re-shuffles) and TADAEEM (two Chiefs of Party), which damaged continuity and involved directional changes. Relationships remained strained throughout these changes. O3 was positioned to work more directly with the GOT at the national level, and a former MALE staffer was hired at a high technical level in the Activity. Both were supposed to have the effect of smoothing the relationship. But USAID, GOT, and TADAEEM respondents reported that the issue was rooted in the design of the Activity, the USAID design, which focused not on national-level interventions or leadership but on a demand-driven municipal model. Respondents said this was probably the most challenging aspect of the work, because it contradicted the wishes of their most important partner and discouraged national ownership of TADAEEM’s products, even those from O3. At the same time, the timing of the award was prior to the passage of the CCL and municipal elections, which translated into an inception phase of more than a year, after which TADAEEM teams dispersed to the municipalities to work with newly elected leaders. Imbalance between the center in Tunis and the hubs. Various respondents throughout the evaluation stated that TADAEEM’s management model resulted in an imbalance between the center in Tunis and the hubs, from which teams worked day-in and day-out with municipal partners. Hiring was reported to be constant in Tunis, with large technical and administrative teams and high turnover, while the hubs had difficulty recruiting for positions in the field, with the right skillsets. The staff roster of TADAEEM, which Deloitte provided to the ET, runs to 206 names. Some are team members who left before the end of the Activity, but others who left in the first 18 months are not on the list, so a full list would be even longer. The majority of TADAEEM technical leaders and hub teams remarked in evaluation interviews that they felt hiring was excessive, and disconnected from needs. Hub teams reported that the recruitment focus was on international service experts who made only cursory trips to the field, instead of Tunisian technical specialists who could work more long-term with capacity support to municipalities. Problematic financial management. Some municipalities and a regional line ministry representative cited unfulfilled promises. Significant financial challenges were also blamed for TADAEEM letting 100 staffers go late in the Activity, but without warning. Two respondents said they were surprised that financial operations were managed on simple Excel sheets that were often in disarray, meaning that TADAEEM leadership did not know their financial situation from day to day. Numerous respondents mentioned over￾hiring without a plan for recruitment or needs analyses as among the reasons of increasing operational costs – while at the same time the hub respondents said they had to consistently ask for people who could train municipalities in the field on the technical aspects of services. In some cases, what was reported to be months-long consultant clearances backstopped by Deloitte US were part of the problem in getting the right people on staff on time, along with overreliance on consultants from Deloitte Tunisia, who tended to be international experts with less or no concrete experience in Tunisia. 32 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Design challenges were also brought up in evaluation interviews. One was that working on O1 and O2 was separate, according to TADAEEM respondents at the central and hub levels, resulting in slow progress on citizen participation goals. Another set of challenges emerged around the grants program, which was slow to start, issuing the first grant over halfway into Year Two. Application materials were offered in English only; then, after local CSOs had gone through a laborious application process, grants for local CSOs were simply cancelled. Of 64 grants in TADAEEM’s database, 57 went to municipalities, 2 to national CSOs, and 5 to local CSOs. TADAEEM relied on CSOs as part of their strategy for citizen participation, according to three respondents from the hub teams. While it is well within TADAEEM’s rights and responsibilities to decide not to grant funds if CSOs lack the required systems, no explanation was given for the decision. This resulted in antipathy toward TADAEEM among eight local CSO representatives interviewed. Contextual challenges. From the beginning, TADAEEM operated in a demanding institutional environment characterized by conflicts and tensions. The sociopolitical context was characterized by economic crisis and enduring citizen protests around unmet expectations, which worsened during COVID. Respondents described how the GOT’s appetite for decentralization waxed and waned with different administrations and ministry configurations. The decentralization process was often stalled, such as deconcentration of power to regions and elections at the regional level, which still have not occurred. Respondents in municipalities told the ET that municipal councils mirrored their party relationships at the national level, with strong partisan reactions. Since governors are still appointed by the Prime Minister, working relationships with municipal councils led by opposing or independent parties were often strained. Municipal administration members, CSO representatives, and even some municipal council members reported that newly elected officials were more loyal to their parties than to citizens, to the detriment of improving participation and service delivery. Such political conflict in Kebili prevented Gafsa and Tozeur from receiving funds at one point, and similar in-fighting affected the evaluation itself, in Gabes. One respondent lamented that “People are against everything, even when it is for their own good.” This person went on to explain that no matter the value of a proposal, if it came from a member of an opposition party, a municipal councilor would oppose it on those grounds. Some local habitual behaviors that affected the project include the use of informal trash dumping areas, and the burning of trash – including in the project-provided bins – were part of the day-to-day challenges of building and maintaining the level of service in the municipalities. Respondents at the municipal level cited citizens’ lack of awareness as the cause of some destruction of park grounds and equipment provided by TADAEEM. Setbacks like these created skepticism among citizens who had participated to bring these activities to fruition. Capacity gaps. National data report that only 11.8 percent of municipal staff are technical or professional (see the response to EQ3b), and TADAEEM’s reports describe critical capacity gaps. Evaluation evidence supports this assessment, as in Figure 10 at right where municipal respondents report the gaps that affected their success with TADAEEM’s interventions. Some themes, which TADAEEM respondents called “back-of-the-house” skills – budgeting, accounting and finance, strategic planning, and municipal tax collection – were not developed broadly during TADAEEM. As in the response to EQ3b on service delivery, TADAEEM built capacity around technical tasks among their five operational axes. But they said they found they should address back-of-the-house capacity as well. Respondents from Tozeur reported being affected by this gap, along with Gafsa and Kebili; but staffing shortages Figure 10: Critical gaps at municipal level: Survey responses from municipalities 33 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 nationwide suggest that these problems make all municipal service delivery weaker. TADAEEM did not cover those needs across all 33 municipalities, according to respondents, and the gaps have not been filled to date. The COVID-19 pandemic. The pandemic created a sudden need for remote work, which TADAEEM worked to meet by (among other efforts) providing videoconferencing equipment throughout the country. However, as the world has learned in the last two years, training quality and effectiveness can be impaired in virtual environments if the training is not well planned and executed. In general, COVID-related delays meant that many activities were on hold for several months. A change in direction from USAID. USAID’s Contracting Officer’s Representative (COR) left in 2019 and was replaced a few months later. This happens regularly worldwide, but the ET heard repeatedly about the upheaval this change caused, when the new COR had very different ideas about the value of TADAEEM’s model. Every respondent who observed interactions between this new COR and TADAEEM described those interactions as antagonistic. These TADAEEM, GOT, and external respondents perceived that the COR publicly denigrated project staff, expressed the belief that donors “laughed at USAID” because TADAEEM focused on waste management, and publicly used language that these respondents deemed inappropriate. On the other hand, two respondents from USAID shared the opinion that Deloitte attempted to go around the COR’s leadership using legal maneuvering in Washington. Among respondents who discussed these issues, there was unanimous agreement that this situation was highly detrimental to the Activity and damaged relations further with the GOT. Two GOT respondents reported being caught in the middle of this fight, while four observers noted that GOT had at times used the discord to push for greater focus on national level interventions. According to respondents, the disagreement ended when a large-scale amendment to the scope of the Deloitte contract was signed for Year Four of project implementation. Mitigation While the ET asked the question on challenges and mitigation of all or nearly all respondents, there is not always a clear one-to-one relationship of challenge to mitigation in each case. Nor were all mitigations successful – though of course, some were. Given turnover in the GOT and in TADAEEM, TADAEEM “reset” relationships as new actors came into these critical roles. There were also efforts to support the GOT more directly with O3 activities, to assuage the national government’s dissatisfaction at not being the conduit to the work with municipalities. This was not entirely successful, and the dissatisfaction was still strongly reported during evaluation interviews. The differences of opinion between hubs and the central office of TADAEEM were strongly present in evaluation interviews, suggesting that Tunis’ efforts to mitigate the problem by meeting hub teams’ needs were also not entirely successful. The ET heard from nearly all hub respondents that the Activity itself was “centralized” and as such was out of step with the decentralization mandate. This played out in staffing as noted in the challenges section, where hub teams felt they didn’t have the right team members in place in the regions to support municipalities throughout the SDIP processes. They also felt that too many resources were taken up in Tunis, while (across hub team members’ perceptions) the key work of the Activity was in municipalities. The ET cannot comment on how funds were distributed for the Activity, as such a review was outside the scope of the evaluation. The team also cannot say if this the configuration of funds was per the design and contract or was negotiated and adapted over the life of the Activity. However, there is impressionistic interview data from three respondents that indicates it was a function of the size of the funding pipeline, which forced Deloitte to push more expensive recruitment – expat consultants and staff, Tunis-based staff – while hiring at local level was widely reported to be difficult and was, logically, less costly because Tunisian field salaries would have been lower. The ET sees a paradox here in that the Tunis team could not, and indeed would not, be able to please both the hub teams – 34 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 where essential work was, indeed, underway – and the GOT, where the “ownership” of TADAEEM’s service delivery reforms would ultimately land. No mention was made in any interviews of efforts to mitigate TADAEEM’s financial management challenges, with the exception of ending the contracts of dozens of employees from one day to the next. Since that happened quite late in the Activity, the ET surmises that financial management challenges persisted. Related difficulties in clearance processes through Deloitte US appeared also to persist. In terms of design challenges, an important mitigation was linking the O1 and O2 processes clearly in the SDIP, which occurred as a result of internal learning discussions. This made the two sets of interventions each work better – O1 was more grounded when there were concrete services to discuss, while O2 was highly convincing for municipalities when citizens were consulted on priority efforts and participated in decision-making. These did not have to be separate interventions, and in fact they were better together. With respect to grants, it was reported that TADAEEM’s grants applications were ultimately translated to French and Arabic, with the clause included that in any dispute, the English version would prevail. On the question of not awarding CSO grants, no changes were made to that policy over the life of the Activity, and while some CSOs continued to work with TADAEEM on citizen participation and other goals, while others did not. In both cases, many CSO respondents remembered that acrimony toward the Activity years later in the evaluation interviews. Little could be done in many cases with context issues that were challenging for TADAEEM since there was reticence and even antagonism to the goals of decentralization among GOT actors – even their counterparts at the MALE, depending on changing ministry leadership. It could be said that TADAEEM’s mitigation was working diligently on SDIPs in its purview to show the potential for improved municipal services, which GOT respondents did in fact cite in their interviews. On the other hand, TADAEEM did mitigate the partisanship in some municipal councils by building common ground with councils around service delivery and citizen engagement, which made collaboration more likely by appealing to the need to have useful examples on which to campaign. Municipalities’ capacity gaps, particularly on technical topics, were a key component of TADAEEM’s O2 interventions. TADAEEM also provided some trainings and technical accompaniment for financial and strategic planning functions, though this was not at the forefront of training efforts. Municipal survey respondents selected four key ways TADAEEM mitigated challenges, as follows: Figure 11: How did TADAEEM mitigate challenges in your municipality? Responses from CSOs were similar, but without the technical assistance (which was only provided to municipalities). And it is important to note the high level of satisfaction among municipal respondents, the great majority of whom were satisfied with TADAEEM’s support, as shown in Figure 12 below: 35 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Figure 12: Were you satisfied with TADAEEM’s support in your municipality? Respondents spoke of some efforts at cross-municipality learning, such as cascaded and peer-to-peer training as part of an elected women’s leadership intervention. El Kef and Ennadhour respondents described exchanges with three non-TADAEEM municipalities on best practices in road management. Where respondents mentioned these efforts, they were appreciated. Mitigation of COVID involved TADAEEM’s pivot to procurement of sanitation equipment and supplies, as well as videoconferencing equipment. As discussed in EQ3d, this effort was widely lauded and coordinated across donors by the MALE. With respect to the contentious relationship with the second USAID COR, TADAEEM respondents said that the COR repeated that he did not understand the mission or methods of the Activity, so they met with him, prepared response papers for him, and ultimately arranged a Deloitte/USAID Washington/Mission/Activity conference on the issue, but that the COR’s stance remained unchanged. As a result, they reported that they did as he asked and changed the Activity fundamentally. Unfortunately there is evidence from various respondents that the poor relationship continued. 5.4 EQ5: Inclusion How effective was TADAEEM in integrating women, youth, and other marginalized groups into its activities and supporting elected women and career women staff in municipalities? Findings The 2014 Constitution and CCL promoted representation of women, youth, and PWD on municipal councils and into participatory processes. At the same time, women’s participation has historically been limited, particularly outside major cities, where traditional cultural norms tend to discourage their more mainstream participation. Still, the results of the 2018 municipal elections saw the beginning of women’s participation on municipal councils around the country. The ET found that the great majority of respondents did not mention women, youth, or PWD spontaneously when asked about TADAEEM’s achievements or about their own proudest accomplishments. In large part, the work carried out on inclusion was carried out by grantees, especially for gender and youth integration efforts. Notably these grants were carried out in only six municipalities, rather than across TADAEEM’s 33 partner municipalities. Objectives 1 and 2, at the heart of TADAEEM’s programming, were designed to increase citizen participation more broadly, but TADAEEM did not report on mainstreaming this commitment to inclusion across their activities because it lacked pertinent indicators on the topic. The CPS was designed to be representative of women at the municipal level, but TADAEEM saw no statistically significant differences in men’s and women’s expressed needs – so it did not pursue the issue further. Nevertheless, focus groups TADAEEM carried out later with women and 36 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 men in municipalities revealed important distinctions that would have helped the Activity to make gender a more central proposition of citizen engagement. Gender integration. Evaluation data show geographical differences on the integration of gender and prioritization of gender outcomes. In five municipalities (of eight), respondents reported more positively on TADAEEM’s progress working with women, and in three, respondents said this was not a focus. Some municipalities were more disposed to work on gender-sensitive efforts, while in others using the word “gender” was met with community-level resistance, making it more difficult to engage directly. Municipalities where active CSOs worked on gender had natural allies for improving women’s participation in the process of supporting citizen participation, but this was not common. TADAEEM highlighted the work of a female Secretary General and that of a female Mayor to bring awareness to women’s leadership roles. Respondents in TADAEEM field offices also differed in the degree to which they spoke about gender: when asked about integrating women and their needs into municipal functioning, three responded that TADAEEM was there to help “citizens,” of which some were women. Given the importance placed on gender in Activity documentation, these responses seemed at odds. Of 2,127 persons trained, 550 (26 percent) were women. In municipalities where respondents spoke positively of women’s participation, both CSOs and municipal respondents spoke of efforts to bring women’s concerns into decision-making processes, usually mentioning the PAI. TADAEEM field teams from these areas reported similarly. These positive efforts tended to be clustered in the latter 18 months of the Activity, when a new Gender Specialist started. Several respondents reported on her focus on a gender-sensitive, participatory approach, saying she was able to convince colleagues at the municipal level that using such an approach would benefit their own work on Objectives 1 and 2. She was also able to carry out some stand-alone activities, such as one gender mainstreaming webinar with GOT and municipal groups. Grants recipients also carried out five municipal outreach activities, and 10 “kitchen dialogues.” Another effort involved four online capacity-building sessions with 34 women elected to 17 municipal councils. The trainings were very well-received and participants stayed in touch for over six months after their sessions ended. TADAEEM reported gender outcomes that were primarily the work of two grant recipients (WeYouth and Tounissiet) who worked in six municipalities in total, rather than from gender mainstreaming across the 33 Activity municipalities. TADAEEM respondents themselves reported mixed success in gender mainstreaming, both across teams and over time. Several women respondents reported that the mostly male technical team stated they did not want to discuss or work on gender, and that they ignored inputs from women team members. The first Gender Specialist did not respond to the ET’s request for an interview, but others based in TADAEEM’s Tunis office at different times and in different roles confirmed that the technical team rebuffed that Specialist’s efforts to incorporate gender in municipal-level activities in the early years of the project. People with disabilities. TADAEEM provided grants to a small number of disability-focused CSOs, who translated PAI information sheets in Braille in one municipality, provided sign language interpretation so deaf people could participate in a plenary session in another, and built ramps to allow access to public buildings in a third. In the latter municipality, 11 municipal staff and 15 PWD were trained in improving participation in local government deliberations. While these efforts are isolated, they are also laudable and novel in the municipalities where they occurred. Still, as with gender, the ET did not find any TADAEEM efforts that would lead to broader and more systematic inclusion of PWD. An example is that parks and green spaces supported by the project did not always permit access, according to TADAEEM staff. Integration of youth. The evaluation found little evidence of active CSOs in municipalities working with youth specifically, though there were two respondents who mentioned youth participation in citizen forums. TADAEEM partnered with local institutions to bring youth into the technical aspects of implementation on a few occasions, such as helping map lighting networks, conducting a property survey, 37 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 and developing databases. The ET heard from municipal partners that youth participated in some green space focus groups. However, as with PWD, there was no strategy for comprehensive youth inclusion. Conclusions TADAEEM technical leadership did not put sufficient focus on gender mainstreaming from the start of the Activity or ensure that teams were committed to mainstreaming as an integral goal of programming. There were no specific and mainstreamed strategies to encourage participation of youth or PWD – despite some high-profile but one-off achievements. When the TADAEEM hub or field manager, municipal staff, and local CSOs were engaged together on gender, bringing women to the table for municipal decision-making was notably more successful. This was more prevalent in the last 18 months of the project, a relatively late start for an Activity in which integration is a cross-cutting issue. The existence of CSOs working on gender in a municipality and female leadership in the municipal council were correlated with better outcomes on women’s participation. Given these lackluster outcomes, TADAEEM technical leaders were insufficiently attuned to this issue. This is out of step with TADAEEM’s design, best practices in citizen participation, and USAID’s and Deloitte’s own gender policies and strategies. Progress monitoring in these areas was an opportunity missed, including better use of the costly CPS data. 5.5 EQ6: Sustainability What measures has TADAEEM taken to ensure the sustainability of improvements in citizens’ participation in municipal decision-making, municipal service delivery, national and subnational government service delivery mechanisms, and coordination between national and subnational governments to respond to national and local emergencies and other crises? Findings The ET heard from TADAEEM respondents that sustainability was part of the team’s thinking from the inception phase, with the idea of capitalizing on good practices from each service improved in a municipality. By building on success, they reported, the 33 sites could become peer mentors for others – with the right materials and guides in place. TADAEEM could not have predicted either the upheaval of COVID or the current uncertain political context that has made the sustainability of its efforts much more challenging, since the MALE was subsumed under the MOI. Sustainability in citizens’ participation in municipal decision-making. Perspectives were mixed regarding the sustainability of citizen participation in municipal decision-making. In one municipality, respondents told the ET that participation in PAI processes is down, and that needs assessments and planning processes have not survived. Participation was surely affected by COVID restrictions, but since TADAEEM did not maintain rigorous data on citizen participation, it is not possible to say with certainty how participation evolved over the life of the Activity, and how it might have rebounded after restrictions were lifted. Though the PAI allowed citizens to discuss tax issues and assess waste management, among other topics, the frequency and effectiveness of the meetings varied. Other municipal representatives said they still included citizens in their decision-making process thanks to the trainings led by TADAEEM, but CSO respondents tended to say this was not common. Responses clearly indicate that during TADAEEM there were successes in bringing citizens, CSOs, and municipalities together to debate local issues and work together. However, the ET found little evidence that participation was regular or widespread. Respondents also did not mention any oversight processes when asked about which TADAEEM activities were sustainable. Sustainability in municipal service delivery. TADAEEM stakeholders readily identified the sustainability of some improvements to citizen services. Provision of equipment was most frequently mentioned as the most 38 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 sustainable of TADAEEM’s interventions, because it minimized costs, improved green spaces, and included capacity-building for maintenance. The positive and longer-term outcomes of street lighting, waste management, and road management equipment were recurrent themes in many interviews. Several respondents said that persistent low levels of municipal funding meant they would never have been able to improve these services on their own, nor maintain them. Improving household waste management was mentioned frequently by municipal respondents, who cited the enduring cost savings from minimizing routes and using more efficient equipment. However, respondents also cautioned that the same low funding levels would make it more difficult to maintain equipment and methods they learned. Still, many reported being hopeful because they felt they’d learned how to diagnose, plan, and make changes based on evidence. However, there were calls to work more with municipal technical and administrative staff more than with municipal councils, since the latter leave after serving their term. A substantial segment of municipal respondents said that diagnostics and field research would continue to be helpful for municipalities to identify potential future projects without the help of external actors. This was because TADAEEM created databases and trained technical teams in their use. Having data at hand can help them apply for future funds for further interventions. When asked about sustainability, TADAEEM teams spoke of the series of 30 guides they produced on six technical topics for the use of municipalities nationwide. However, other stakeholders – CSOs, GOT, and some from USAID – noted that while the guides were available on a GOT website, they were not translated to Arabic and were not disseminated widely. Some GOT respondents were dissatisfied with the guides, which they felt were not properly aligned with GOT standards, not written clearly enough for municipal teams to implement, and not vetted and agreed upon by GOT line ministries. Respondents reported a small number of trainings for municipalities on one or more guides, using the videoconferencing equipment TADAEEM provided. But it was in no way broad or comprehensive – not across the topics or across municipalities where they might be useful. Given the subsuming of the Ministry of Local Affairs within the Ministry of Interior following the July 25 crisis, the existence of an institutional home to eventually propagate the guides is in question. Sustainability in national and subnational government service delivery mechanisms. Objective three activities worked to unite and simplify guidelines, procedures, systems, and teams across levels of government. In theory there would be increased transparency and efficiency while also helping ensure sustainability. One example was the collaboration between TADAEEM, MALE, the Ministry of Equipment, GIZ, and the Order of Architects of Tunisia to create an e-construction platform to digitize and standardize the construction permit process. However, this platform is not up and running, despite great efforts to launch it and the initial success of beta testing. Similarly, TADAEEM worked with stakeholders at different levels to agree upon a national waste management strategy, one for municipal rights-of-way, and standards for public lighting. Unfortunately, there is no evidence that these are in place and functioning. Production of technical guides for municipal service delivery was another way to extend the reach of the TADAEEM’s benefits, but as noted above, dissemination, training, and translation were not undertaken. TADAEEM intended for peer-to-peer sharing to be part of sustainability planning, and in one effort with support to women elected leaders it appears training was “cascaded,” but this was not the case for service delivery mechanisms. Returning back to findings shared earlier in the report helps to make sense of these disappointing results. TADAEEM and the GOT had a difficult relationship and TADAEEM’s work in the municipalities was not funneled through the MALE. GOT coordination of TADAEEM’s (and other donors’) efforts in decentralization was also lacking. Without a stronger and functional relationship with GOT partners, TADAEEM was unable to position their efforts for sustainability. Sustainability in coordination between national and subnational governments to respond to national and local emergencies and other crises. With the pandemic, the MALE coordinated a multi-donor rapid response. TADAEEM delivered critical equipment across 151 municipalities. Videoconferencing equipment was put in place in all municipalities to facilitate virtual meetings and ultimately some trainings, as described above, 39 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 and was much appreciated across municipal respondents. When asked about crisis coordination and communication, 20 municipality survey respondents said TADAEEM contributed to this, most often by establishing a coordinating committee. A small number of interviewees agreed, but the bulk of responses were about the in-kind donation of masks, disinfectants, sprayers, and other physical resources – rather than the coordination or communication aspects. Only five of the survey respondents (among 26 municipal representatives and 12 from CSOs) said the coordinating committee was still meeting in their area. Conclusions O1: Sustainable citizen participation depends on continued efforts within municipalities, and there were as many who said they had preserved participatory approaches as those who said these efforts had waned post-TADAEEM. There is no specific evidence of any oversight mechanisms outlasting TADAEEM. There was also evidence of an important missed opportunity to use a gender-sensitive participatory approach systematically as a way to engage with and empower citizens, as reported in the section on inclusion. O2: Service delivery outputs have been delivered and equipment and technical know-how have continued in many sites, providing the strongest evidence of sustainability under Objective 2 – but with important gaps in management and technical capacities and materials. Another critical gap is the status of the technical guides – available (in French) on a website but no longer the subject of any training, however minimal, or dissemination. TADAEEM municipalities may need continued support to use them, because of their technical level, and any wider use would certainly require that support, which has not been institutionalized in any Ministry or the relevant training center. O3: Efforts to strengthen regional and national service delivery mechanisms, such as the e-Construction platform, waste management, and other service strategies and standards, have not survived the end of the Activity. Eleven such efforts are reported in the TADAEEM final report and technical documentation exists, but the GOT has not adopted or put any of the efforts into regular use. O4: Videoconferencing equipment was put in place and has been used. Sustained improvements to communications and crisis coordination are not in evidence. 5.6 Monitoring and evaluation Interviews and document review suggest that monitoring, evaluation, and learning in TADAEEM had strengths and important weaknesses that affected implementation and results and warrant a brief discussion in the evaluation report. During the life of TADAEEM, few indicators that were included in the first performance management plan remained by the end of the project. With the COVID crisis interrupting their efforts, Activities worldwide were poised to miss their targets, however understandably; still, the changes to the Performance Management Plan (PMP)13 predate COVID in most cases and show both shifts in focus and low expectations, with eight outputs and only two outcomes.14 Table 6 shows the early and later AMELP composition. The 2019 AMELP contained multiple changes to the list of indicators, before COVID, that represent a dilution in expectations. 13 The PMP was the standard term for activities’ monitoring and evaluation plans when TADAEEM began. This was later changed to an Activity Monitoring, Evaluation, and Learning Plan (AMELP) per USAID guidance. We use AMELP throughout to avoid confusion with the Mission’s PMP. 14 Two standard indicators in the PMP are marked as outcomes but are outputs per the 2021 Performance Progress Report list of standard indicators. The TADAEEM PMP also does not alter them in substantial ways so that they reflect outcomes. 40 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Table 6: Indicators from 2018 and 2021 TADAEEM AMELPs 2018 (13 indicators) 2021 (10 indicators) Output Outcome Output Outcome Objective 1 3 3 4 0 Objective 2 3 2 1 2 Objective 3 2 1 3 0 Cross-cutting 2 0 2 0 Totals 715 6 8 2 Three of the initial indicator changes (in the 2019 AMELP) came as the result of a Data Quality Assessment in 2019. The first deepened the definition and utility of a crucial indicator on the SDIP – including detail on progress through the SDIP. A second, standard, indicator was defined better for TADAEEM’s particular interventions, but by the final report had reverted to its original form – the one that the data quality assessment (DQA) critiqued. The DQA found serious data quality issues with the third indicator – also standard – which was an outcome indicator tracking the establishment of formal mechanisms for participation. It was subsequently dropped, leaving no outcome measures of Objective 1. USAID did not consistently have experienced MEL support in-house for Tunisia during the life of TADAEEM. A USAID Monitoring and Evaluation staffer did review changes to the 2018 AMELP, according to email traffic from within the Agency shared with the ET, but the AMELP had already been approved at that time. Further changes emerged between 2019 and 2021. Two more outcome-level indicators were dropped – municipal own revenue and average municipality performance scores – for lack of data, at least in part due to the COVID pandemic.16 Another that counted citizens who participated in municipal decision-making processes had been lowered from an outcome (percentage) to an output (number) in 2019, and then was dropped altogether by 2021. Other changes, as indicated by the table above, show how the power of the PMP to show TADAEEM’s progress on important outcomes was diminished with the changes in 2019. The first TADAEEM Results Framework (RF) and AMELP operationalized the intention to institutionalize systems for all three objectives, while the 2019 version only aimed at institutionalizing some service delivery aspects. There is no measurement of citizen participation, much less engagement or oversight, and no effort to capture the participation of women, youth, or PWD more systematically. The changes also meant little continuity through the life of the project – only 2 indicators remain from the first iteration. Data against indicators in the TADAEEM Final Report show lukewarm progress even against the service delivery outcome targets, and a misrepresentation of citizen service delivery satisfaction from the CPS: TADAEEM reports achieving 84 percent of the target, but the actual achievement is only 20 percent – 2 of 10 percentage points’ targeted improvement against baseline. TADAEEM’s targets were low in the 2019 PMP version, such as the crucial SDIP indicator with a life-of￾project target of 64 (with an achievement of 62), indicating not quite two processes per treated municipality over four years – though each process was set to last one year. COVID interrupted TADAEEM’s work in municipalities and required remote work strategies for over a year, likely affecting outcomes. Still, the targets that predate COVID indicate TADAEEM’s intentions were low given their own one-year time frame for SDIPs in 33 municipalities. Further, the ET noted that the Ma3an Activity, 15 Some indicators serve more than one indicator but are not double-counted in the totals. 16 Please see Annex I for the Results Framework, taken from the Final Report, where the two indicators are shown in red. 41 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 which has a similar-sized award, is working in exactly the same number of municipalities, indicating USAID may have taken on board the calculation of what can be done with an award of that size. The ET feels it would be wise to revisit those figures for future awards and expectations around outcomes. One of the indicators added in 2019 was to count anti-corruption efforts “proposed, adopted, or implemented.” Final reporting indicates that the e-Construction platform was counted for each of these stages, as if for a total of three such efforts. Lackluster results against such unambitious targets is problematic, particularly given the level of investment. As noted above, given the COVID crisis it was not unexpected that activities may not meet targets, and may in fact change tack to undertake other, more pressing efforts. The CPS was an ambitious and well-designed effort to gather information on a key indicator of citizen satisfaction with service delivery – one of the two indicators that was present throughout the life of the Activity. Two key challenges arose, outside of the COVID crisis. First, citizens’ preferences about services were not always highest where TADAEEM could intervene. Only a small set of government services are devolved to the municipalities, and while some were of interest to citizens, their higher priorities were outside the municipalities’ scope. That meant that the sample sizes, certainly adequate for municipal-level disaggregation about priorities, were actually used to capture much smaller differences – say, between citizen preferences around waste management and road improvements. Second, satisfaction measures are notoriously challenging to measure in a dynamic environment that included health and economic crises and widespread, durable protest movements.17 Improvements were very modest – two percentage points overall, though higher for some services. A final concern within MEL is that a midterm evaluation – however challenging during COVID – is likely to have uncovered at least some of the difficulties documented in this final evaluation: poor relationship with GOT partners at the national level, scant systematization of interventions for sustainability, and low attention to systematically improving inclusion of women, youth, and PWD. METAL proposed such an evaluation in June 2019, before the arrival of the second COR. Despite indications that an evaluation would be useful, in November 2019 USAID informed METAL that no such evaluation would take place. 6. RECOMMENDATIONS The recommendations here are directly linked to evaluation conclusions in each section, including that of EQ4 (for which “conclusions” per se are not appropriate) and of the monitoring and evaluation section, which was added in response to serious challenges that are, however, not part of the evaluation questions. The recommendations are divided into five categories: working with the central government, for USAID internally, for strengthening weaknesses in the programming, for building on what TADAEEM has put in place, and for future programming. Working with the central government • USAID should take into consideration central government strategy for reforms at the state level, and through institutions that support subnational governments. Through these institutions USAID can also support municipalities by doing the following: o Strengthening the local finance agency o Helping the training institute o Studying prospects for transfer and sharing of powers o Drafting and implementing regulations o Support the administrative court to implement a posteriori review o Facilitating financial transfers along with authority 17 Satisfaction surveys, like any subjective scale, are highly susceptible to the influence of exogenous political and economic conditions, among other potential influences. 42 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 • USAID should map and include these institutional stakeholders during design, and agree on milestones in a Memorandum of Intent for continued cooperation. The success of activities relies on central GOT commitment. • Consider how and where change management is necessary as part of needs assessments. Both central ministries and municipalities – as well as other subnational bodies – may need support with change itself. • There will be turnover in ministries, so as part of any Memorandum and other initial negotiations, develop succession planning that identifies institutional anchors and civil servants to provide continuity. Recommendations for USAID • USAID should build the budget for future activities that reflect shared priorities and realistic expectations. Budgets cannot simply be buckets for earmarked money, and it may be that one Activity for a significant earmark is not enough – or not manageable if USAID/Tunisia is not sufficiently staffed. o Identify USAID priorities and comparative advantages over what is already operating. Review current METAL donor mapping exercise to see what other strong teams are doing in decentralization. o Guide implementing partners to propose how they will ensure strategic and cost-effective recruitment, Tunisian leadership, and an appropriate balance between the field and the center office. Then follow up to make sure it happens. • USAID/Tunisia’s activities should be coherent and collaborative across the portfolio, taking best advantage of shared geographies, research opportunities, and models of integrated interventions that the GOT could scale up holistically. Activities that are part of the same CDCS should be working together toward integrated goals. • At the same time, USAID, GOT, and donors should work on the overall coherence of programming in the decentralization space. This will require stronger coordination – which, having been ably carried out by the MALE during the COVID crisis, should be in the hands of the GOT. But it may need support to take on this demanding role. • USAID/Tunisia requires better activity management tools and should make better use of its MEL platform to improve activity management. PMP/AMELPs for new activities must attend more to outcomes than to simple outputs, and should also be monitoring context – as is currently being done in the USAID/Tunisia Ma3an Activity. There also needs to be evaluation at least at midpoint, or – even better – a developmental evaluation for an activity with such high contextual uncertainty. • Monitor turnover in USAID to ensure program continuity. In the event change is warranted, this should be documented by an external evaluation or an internal working group, rather than around the preferences of a single person, such as a COR. Strengthen weak areas • USAID’s future interventions in this space should partner better with local and national CSOs. At the municipal level, implementers should be using CSOs’ understanding of community dynamics, working through their networks, and helping them build or improve oversight and social accountability mechanisms and ongoing citizen engagement. There is also the imperative of leaving installed capacity among such organizations, who are invested in improving municipalities. Rather than simply calling off all grants to CSOs, USAID should support this element of self-reliance with 43 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 grants and scaffold their organizational development. Tunisia has a strong local media presence in the south of the country which could also be considered in this vein. At the national level, there are CSOs working actively on decentralization with whom future interventions might work, which could be a conduit providing citizen insight into any future work with GOT on these issues. • USAID should insist that implementers have a plan for the politics inherent in this work, including regular political economy analysis at whatever level of government where they are working. This is the qualitative side of the context indicator recommendation above: given the partisan nature of the elected political bodies with whom USAID’s implementers will intervene, regular critical thinking about these issues is paramount. And it must be fine-grained enough – municipality by municipality, party by party, governorate by governorate – to be useful. • Future investment in citizen participation efforts should more inclusive – building, perhaps, on the gender-sensitive approach used in the latter part of TADAEEM. If an Activity goal is that women, youth, and PWD are more involved, then indicators should reflect that. Given the historical neglect of Tunisian citizens in rural areas, inclusion efforts should also examine ways to increase outreach to these populations as well, in addition to urban-centered interventions. Activities should also be required to report not just numbers of participants but about actual engagement and oversight mechanisms – otherwise the words in the scope are merely ornamental. And engagement with citizens will require, as noted above, deeper work with CSOs that pool their interests. Building on TADAEEM’s beginnings • USAID and implementers should keep gender and youth at the forefront of their programming decisions in the decentralization space. Innovative TADAEEM pilots could be scaled up or tested in other environments, like those with more traditional gender role adherence, to identify best practices for mainstreaming gender into citizen participation. Such approaches would help bridge and provide feedback between citizen engagement and service delivery improvements, as was in use in the latter months of TADAEEM. The pilot work of some gender-focused grants should be tested elsewhere. • Future programs should make use of the municipal-level diagnostics as well as the CPS data from TADAEEM to identify the kinds of service delivery a new project might target, including plumbing the data for increased attention to gender-sensitive municipal priorities. • USAID should consider for future interventions the great need at the municipal level for improving local revenue collection, financial management, and investment. This should go beyond technical assistance to include concrete projects like TADAEEM’s property survey pilot in Kebili. These interventions should be oriented to resolve municipalities’ financial needs and build self-sustainable revenue systems. • USAID and its implementers should take the range of TADAEEM service delivery products – guides, presentations, templates, strategies, the e-construction platform, inter alia – and include these in the design process to ensure the work is used for Tunisian municipalities nationwide. This may include using intercommunality, as described in the CCL, as a form of peer-to-peer learning where TADAEEM municipalities with successful outcomes are mentored to train other municipal leaders elsewhere. Such efforts can help GOT scale up USAID activities as well, because of cost￾effectiveness and cascading effects. • Take advantage of relationships from TADAEEM, too. CSO representatives who were trained could also be re-engaged in knowledge sharing and transfer like this. And, the Public–Private Partnership Guide produced by the Activity but as yet unused should be employed to bring together support to municipalities and the top priority for Tunisians: improving economic opportunities and creating jobs. 44 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Future programming considerations • USAID should consider supporting the planned deconcentration to district/governorate-level roles, and election of bodies at those levels. The emergent quality of programming in this area – given that (like municipalities during TADAEEM) any elected bodies will be completely new – will require patience on the part of USAID and implementers to identify and begin to address capacity￾building and service delivery needs. • Use a systems approach to local government change that situates municipalities within their dynamic system and works more effectively within this complexity. This recommendation brings together conclusions from multiple EQs in an attempt to show how early design choices and the poor relationship with the GOT at national level could be improved in future programming. Though the dynamics of decentralization are stalled while the nation resolves primary issues around July 25, when USAID decides to intervene again, that systems approach will need to take into account the web of national, regional, and municipal actors, including peers; institutions detailed in the first recommendation above; the CSOs at all levels as recommended under the weaker areas listed above; and the parameters for broader inclusion. • Think about capacity-building as well from a systems approach – taking the lessons of successful TADAEEM hands-on training to back-of-the-house functions more intensively. While there is space to induct new municipal councils into their roles, the balance of training should tilt toward civil servants who are more likely in place for their careers. • Consider strategic opportunities for digitization in service of citizens and businesses, including launching and monitoring underused systems. Digitization activities could include the e￾Construction platform, mobile apps for citizens, e-payment, e-requisitions, and the national electronic procurement system. 7. KEY TAKEAWAYS The unusual contextual conditions – USAID minimal presence in Tunisia and inability to interact with critical GOT institutions during design, plus an unprecedented earmark for DG work in Tunisia – that had outsized negative effects on TADAEEM do not exist anymore. As such, USAID/Tunisia’s design decisions – while still bounded by contextual constraints – will be better able to program around needs assessment, stakeholder mapping, and other on-the-ground considerations, informed with inputs from this evaluation, METAL’s Subnational Governance Assessment and donor mapping exercises, and other data, rather than around the earmark. TADAEEM’s achievements, while not as robust as might have been expected, were affected by COVID￾19 and by difficult relationships and turnover among the main counterparts. Even so, there are valuable elements of their programming, as well as lessons learned, that provide a sound basis for building the next generation of USAID/Tunisia interventions in the governance space. The SDIP process, as it was enhanced by a gender-sensitive participatory approach, is one such building block, and the technical guides and materials are a second set of products that should be taken forward in new programming. Gender, youth and disability cross-cutting themes were not mainstreamed into TADAEEM’s programming, though the last 18 months of programming saw an improvement with a gender-sensitive approach applied in some municipalities. Though rural communities are more marginalized, this was not a focus of TADAEEM – and should be considered in future programming. Critically important for USAID DG designers is the need for coherence and integration across the Mission CDCS, so as to meet citizens where they are most engaged – around economic development and job creation. It is clear that these priorities already exist for USAID in Tunisia, and it will enhance programming 45 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 in DG to link interventions across these thematic areas. Stronger donor coordination – preferably led by the GOT – is also vital, because of the intense European interest in Tunisia and local governance. The innovative structure of TADAEEM – working primarily from the “bottom-up” with municipalities, is not likely a stable, acceptable format for Tunisia. The resolution of the July 25 presidential decree is yet to appear in Tunisia, and it is premature to plan precisely how USAID could intervene in DG until some of the current constitutional and parliamentary conditions are more settled. Similarly, a key facet of decentralization – deconcentration to elected bodies at a regional level between the state and municipalities – is unresolved, but could be a useful entry point for USAID interventions. There is currently too much uncertainty to allow for the systems approach the ET recommends. More profound engagement with CSOs in future programming – at national, regional, and local levels – will balance the recommended increase in engagement and leadership from the central government. It is also necessary for sustainability of citizen participation and oversight functions. Monitoring indicators and supervision of those indicators and their data need to improve, if USAID is to take best advantage of programming to achieve development objectives. 46 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 ANNEXES ANNEX A: SCOPE OF WORK Note: The evaluation questions presented in the evaluation report to which this SOW is annexed were discussed and approved as part of the inception phase of the evaluation. Email approval from USAID/Tunisia is available upon request. PURPOSE OF THE EVALUATION This statement of work (SOW) for the Monitoring and Evaluation for Tunisia and Libya contract (METAL) outlines USAID/Tunisia’s need for a final performance evaluation of the Tunisia Accountability, Decentralization, and Effective Municipalities (TADAEEM). The purpose of the evaluation is to identify the extent to which TADAEEM increased government responsiveness to citizen needs at national and local levels, and expanded citizen engagement for accountable governance. The evaluation will identify successes, challenges, and lessons learned that affected implementation of the activity. The evaluation recommendations will be used to inform the design of future USAID/Tunisia local governance initiatives, and Government of Tunisia decentralization efforts. The primary audiences of this evaluation are the USAID/Tunisia Democracy and Governance Office, broader USAID/Tunisia Mission, and Tunisian National and Subnational level governments. Other audiences include CSOs engaged with municipalities, USAID/Tunisia implementing partners, and other donors working with municipalities and relevant national and subnational government authorities engaged in decentralization. SUMMARY INFORMATION Activity Name Tunisia Accountability, Decentralization, and Effective Municipalities (TADAEEM) USAID Office USAID/Tunisia Implementer(s) Deloitte Contract # Contract under AID-OAA-I-14-00065/AID-664-TO-17-00002. Total Estimated Ceiling of the Evaluated Activity $ 49,157,614 Life of Activity September 2017–March 2022 Active Geographic Regions 33 municipalities across Tunisia Development Objective(s) DO2: Social Cohesion promoted through Democratic Consolidation Required evaluation? Yes External or internal External evaluation? 47 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 BACKGROUND The 2011 revolution was a powerful expression of Tunisians’ desire to have their voices heard in the economic and political life of their country. Following that event, the Government of Tunisia embarked on a series of efforts to improve the quality of governance, advance administrative reforms, devolve government authorities, fight corruption, increase social inclusion, and reduce regional disparities. USAID/Tunisia supports these efforts through economic and governance assistance programs that promote both economic participation and democratic consolidation. In supporting the Government of Tunisia’s (GOT) decentralization agenda, USAID/Tunisia launched the Tunisia Accountability, Decentralization, and Effective Municipalities (TADAEEM) activity to assist municipalities in achieving their new mandates as prescribed in the Tunisian constitution and the Code des Collectivités Locales of 2018 (i.e., local government law). While Tunisians relished the freedoms that resulted from the 2011 revolution and held multiple rounds of free and fair elections, they became increasingly frustrated with persistent corruption, deep inequality, and lack of economic progress since 2011. In response to widespread political protests against the government during the summer of 2021, on 25 July President Saied invoked Article 80 of the Tunisian constitution to dismiss the Prime Minister and freeze Parliament and announced his intention of introducing constitutional amendments to restructure the country’s system of governance. Central to those anticipated amendments is an inverted pyramid of vertically integrated councils. Per the decree, there would be 264 local councils, whose members are directly elected and are principally responsible for planning development projects; 24 regional councils, whose membership consists of one representative of each local council within the region and who are responsible for overseeing approved local projects; and a single national assembly, whose membership is likewise drawn from the local councils but whose responsibility is national legislation. This new structure conflicts with the structure currently in place and described by the extant local government law, the Code des Collectivités Locales. The impact of the pandemic and a surge in COVID-19 cases over the summer has increased unemployment and pushed an already strained economy toward crisis. Since his suspension of Parliament and dismissal of his government in July 2021 and the publication of his decree calling for a new structure of councils, President Saied on 11 October introduced a new government, headed by the first female Prime Minister in Tunisia’s history, Najla Bouden. Significantly, he did not simultaneously name a new minister for Local Affairs, and placed responsibilities for local affairs under the Ministry of Interior. Description of the Problem and Context TADAEEM’s primary objective is to improve the relationship between Tunisians and their civic and government institutions, focusing particularly on underserved populations. The activity seeks to bridge the gap between the Government of Tunisia’s (GOT’s) long-term decentralization process and bottom-up municipal development to rapidly produce tangible change while improving the relationship between Tunisians and their government institutions. Description of the Intervention to be Evaluated and Theory of Change TADAEEM’s three key objectives include improving: citizen participation in and oversight of key government functions; municipal institutional and service delivery performance; and regional and national coordination and service delivery mechanisms to respond to needs of citizens via local government structures. In year 4, TADAEEM added a fourth objective, “Improved coordination and communication performance of municipalities, regions, and Ministry in responding to national and local emergencies and other crises. TADAEEM works with 33 municipalities spread across Tunisia and also worked at the central level to assist the previous Ministry of Local Affairs with strategic planning and communications with subnational entities. 48 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 TADAEEM directly contributes to USAID/Tunisia’s Development Objective (DO) 2: Social Cohesion Promoted through Democratic Consolidation and Mission’s Intermediate Result (IR) 2.1 and 2.2 by working to enhance the responsiveness of government intuitions and institutionalize participatory systems and to improve municipal citizen engagement in municipal planning and budgeting. Below is the Causal Logic Model linking the activity’s results to the USAID/Tunisia Development Objective 2, and goal. Reference: FY 21 TADAEEM AMELP 49 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 EVALUATION QUESTIONS The evaluation team will review and finalize questions in collaboration with USAID prior to finalizing the evaluation design.18 EQ 1: To what extent are TADAEEM’s objectives aligned with the Government of Tunisia’s (GOT) decentralization law (i.e., constitution and the Code des Collectivités Locales of 2018) and subsequent GOT decentralization strategy.19 EQ 2: To what extent and how have TADAEEM’s individual activities contributed to achieving the following improvements to: a) Citizen participation in and oversight of government decision-making processes at the subnational level b) Municipal institutional and service delivery performance c) Regional and national service delivery mechanisms to respond to the needs of citizen d) Improved coordination and communication of municipalities, regional governments, and central government in responding to, national and local emergencies and other crises EQ 2a: What were the internal (staffing, procurement, organizational structure, etc.) and external factors (lack of local technical capacity, national and subnational resources, political will, corruption, etc.) that affected the timeliness and achievement of TADAEEM’s objectives and what measures did TADAEEM and its implementing partners take to mitigate those factors? EQ 3: What are the key lessons learned and takeaways from TADAEEM’s experience in working with municipalities and other levels of government, and how can they inform other USAID/Tunisia activities that support Tunisian government institutions? EQ 5: How effective was TADAEEM in integrating women, youth, and other marginalized groups into its activities and supporting elected women and career women staff in municipalities. EQ 6: What measures has TADAEEM taken to ensure the sustainability of improvements in citizens’ participation in municipal decision-making, municipal service delivery, national and subnational government service delivery mechanisms, and coordination between national and subnational governments to respond to national and local emergencies and other crises? EVALUATION DESIGN AND METHODOLOGY This performance evaluation will utilize a mixed-methods research design, employing both quantitative and qualitative methods to strengthen the validity of the findings and provide room for data triangulation. METAL will describe and document the methodological approach that will be used, which follows USAID Evaluation best practices. The model will include an evaluation framework and assessment tools for each evaluation question, and highlight the conceptual model(s), specifying the measurement criteria to be used to respond to each question. It will discuss any risks and limitations that may undermine the reliability and validity of the evaluation results. In order to ensure the maximum value for learning and use, a description of the proposed evaluation methodology will include the following: ▪ Review of the existing baseline relevant in data analysis ▪ Methods of data collection 18 USAID approved slight updates to these EQs in the Inception Report. 19 The strategy presumes a steady progression in the passage of legislation, transfer of authorities, and eventual autonomy of municipal councils. 50 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 ▪ An evaluation design that shows how sampling will be done and appropriate sample sizes required to ensure scientific rigor (if applicable) ▪ Use of primary and secondary data in data analysis and a plan for analysis ▪ Evaluation Matrix (see template below) Evaluation Design Matrix Template Questions Suggested Data Sources Suggested Data Collection Methods Suggested Data Analysis Methods 1. [Evaluation Question] 2. [Evaluation Question] 3. [Evaluation Question] USAID expects that, at a minimum, the evaluation team will: ● Upon award, familiarize themselves with documentation about the TADAEEM activity and USAID’s current assistance in the Democracy and Governance area in Tunisia and the region. ● Review and assess the activity’s performance reports and performance data. ● Test survey instruments for clarity prior to use. ● Meet and interview USAID project implementing partner staff, beneficiaries, partners, other donors, and host government counterparts at appropriate levels. ● Interview USAID staff and a representative number of experts working in the sector. To the extent possible, the evaluation will be done in country, although some interviews may take place remotely due to logistical and other factors. The contractor will submit the preliminary evaluation design for review by USAID. The evaluation Contracting Officer’s Representative (COR) must approve the evaluation design prior to fieldwork commencing. DELIVERABLES AND REPORTING REQUIREMENTS 1. In-briefing The evaluation team will conduct an in-briefing with the DG Officers and local staff for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to adjust the SOW, if necessary. 2. Evaluation Inception Report The evaluation team must prepare an evaluation inception report that includes the following: 1. Draft evaluation team schedule, including key milestones and logistical arrangements 2. Members of the evaluation team, delineated by roles and responsibilities 3. Description of the proposed evaluation methodology – this section should include a detailed evaluation design matrix that links the evaluation questions from the SOW (in their finalized form) to data sources, methods, and the data analysis plan 4. List of potential key informants and proposed selection criteria and/or sampling plan, as applicable 5. Draft data collection tools 6. Limitations to the evaluation design 7. Dissemination plan (designed in collaboration with USAID) The data analysis plan should clearly describe the evaluation team’s approach for analyzing quantitative and qualitative data (as applicable), including proposed sample sizes, specific data analysis tools, and any 51 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 software proposed to be used, with an explanation of how/why these selections will be useful in answering the evaluation questions for this task. Qualitative data should be coded as part of the analysis approach, and the coding used should be included in the appendix of the final report. Gender, geographic, and role (beneficiary, implementer, government official, NGO, etc.) disaggregation must be included in the data analysis where applicable. All dissemination plans should be developed with USAID and include information on audiences, activities, and deliverables, including any data visualizations, multimedia products, or events to help communicate evaluation [findings/conclusions/recommendations]. See the Evaluation Toolkit for guidance on Developing an Evaluation Dissemination Plan. If applicable based on the Disclosure of Conflict of Interests Forms submitted with the awardee’s proposal, the evaluation design will include a conflict of interest mitigation plan. 3. Mid-term Briefing and Interim Meetings The evaluation team is expected to hold a midterm briefing with the USAID/Tunisia Democracy and Governance (DG) team, Program Office MEL staff, and other relevant USAID/Tunisia team members on the status of the evaluation, including potential challenges and emerging opportunities. The team will also provide the evaluation COR/AOR with periodic briefings and feedback on the team’s findings, as agreed upon during the in-briefing. If desired or necessary, weekly briefings by phone can be arranged. 4. Preliminary Presentation for Recommendations Development The evaluation team is expected to hold a preliminary presentation by virtual conferencing software to discuss the summary of findings, conclusions, and recommendations and present USAID with the key initial findings and recommendations either in a presentation or a short 2-3- page document. Any presentations or workshops will be scheduled as agreed upon during the in-briefing. 5. Draft Evaluation Report The draft evaluation report should be consistent with the guidance provided in Section IX, Final Report Format and must meet all evaluation report and quality requirements detailed in USAID Automated Directives (ADS) Chapter 201. The report will address each of the questions identified in the SOW. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, the USAID/Tunisia team will have 10 working days in which to review and comment on the initial draft, after which point the COR will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit a revised final draft report within 10 working days, and again the USAID/Tunisia team will review and send comments on this final draft report within ten working days of its submission. 6. Final Evaluation Report The evaluation team will be asked to take no more than 10 working days (or as agreed upon in the work plan) to respond to and incorporate the final draft evaluation report and presentation comments from the USAID/Tunisia team. 7. Submission of Final Evaluation Report to the Development Experience Clearinghouse Per USAID policy (ADS 201.3.5.18) the contractor must submit the evaluation final report and its summary or summaries to the Development Experience Clearinghouse (DEC) within three months of final approval by USAID. 8. Submission of Dataset(s) to the Development Data Library Per USAID’s Open Data policy (see ADS 579, USAID Development Data) the contractor must also submit to the COR and the Development Data Library (DDL), at www.usaid.gov/data, in a machine-readable, non-proprietary format, a copy of any dataset created or obtained in performance of this award, if 52 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 applicable. The dataset should be organized and documented for use by those not fully familiar with the intervention or evaluation. Please review ADS 579.3.2.2 Types of Data To Be Submitted to the DDL to determine applicability. 9. Other Deliverables a. Post-Evaluation Action Plan. Within 30 days of the approval of the evaluation report, the contractor must work with the USAID technical team to develop a “post-evaluation action plan” per the guidelines in ADS 201. EVALUATION TEAM COMPOSITION The Contractor must propose an appropriate team for this assignment and provide information about evaluation team members, including their curricula vitae. The Contractor must explain how each evaluation team member’s qualification and expertise meets the requirements in the evaluation SOW and will contribute to the achievement of a technically sound, objective, and high-quality evaluation. Submissions of writing samples or links to past evaluation reports and related deliverables composed by proposed team members are highly desirable. The evaluation team must be approved by the METAL COR. Any substitutes to the proposed key personnel must be vetted and approved by the COR before they begin work. USAID may request an interview with any of the proposed evaluation team members via conference call, Skype, or other means. Per ADS 201.3.5.14, all team members must provide to USAID a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest relative to the project or activity being evaluated (i.e., a conflict of interest form). DELIVERABLES AND TIMELINE The Contractor must provide the following deliverables. All written documentation for submission by the Contractor to USAID/Libya must be in English. This schedule is illustrative and will be updated (as necessary) in collaboration with USAID. Deliverables Due Date (Timeframe) 1 Inception report containing work plan and evaluation design Prior to the start of the document review and meetings, the Contractor must submit to USAID an Inception Report for USAID approval. The report must include: ▪ Complete list of documents planned for review ▪ Complete list of planned interviews and proposed ▪ individuals ▪ Proposed list of questions for the interviews ▪ Detailed description of the data collection plan and ▪ analysis methodology ▪ Literature review USAID will have one week to review and provide comments/feedback. Work plan and evaluation design period March 16, 2022 (February 28–March 16, 2022) USAID Review Period March 23, 2022 March 17–23, 2022) 2 PowerPoint presentation of TADAEEM Final Evaluation preliminary findings, conclusions, and recommendations Data collection, analysis and presentation preparation 53 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Deliverables Due Date (Timeframe) The Contractor must develop and deliver a PowerPoint April 22, 2022 presentation of the preliminary findings, conclusions, and recommendations of the TADAEEM Final Evaluation to USAID/Tunisia staff and other relevant stakeholders to be delivered to the USAID/Tunisia team. (March 24–April 22, 2022) 3 Draft TADAEEM Final Evaluation Report Refer to section “Final Report Format and Supporting Data” below. Should be submitted as an MS Word document. Draft report will include feedback from PowerPoint presentation of preliminary findings, conclusions, and recommendations. USAID will have 10 working days to review and provide comments/feedback. Draft Report Prep Period: May 10, 2022 (April 23–May 10, 2022) USAID Review Period May 24, 2022 (May 11–24, 2022) 4 Final TADAEEM Performance Evaluation Report Refer to section “Final Report Format” below. All comments provided by USAID should be addressed in the Final Report. The report should be submitted in MS Word and PDF formats. Final Report Prep Period: May 31, 2022 (May 24–31, 2022) FINAL REPORT FORMAT 1. Abstract 2. Executive Summary 3. Evaluation Purpose 4. Background on the Context and Activity Being Evaluated 5. Evaluation Questions 6. Methodology 7. Limitations to the Evaluation 8. Findings, Conclusions, and Recommendations 9. Annexes See the Evaluation Toolkit for the How-To Note on Preparing Evaluation Reports and ADS 201mah, USAID Evaluation Report Requirements. The evaluation abstract of no more than 250 words should describe what was evaluated, evaluation questions, methods, and key findings or conclusions. The executive summary should be 2-5 pages and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, and conclusions (plus recommendations and lessons learned, if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methods (e.g., in sampling; data availability; measurement; analysis; any potential bias such as sampling/selection, measurement, interviewer, response, etc.) and their implications for conclusions drawn from the evaluation findings. 54 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Annexes to the report must include: - Evaluation SOW (updated, not the original, if there were any modifications) - All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides - All sources of information or data, identified and listed - Statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team, if applicable - Signed disclosure of conflict-of-interest forms for all evaluation team members, either attesting to a lack of or describing existing conflicts of interest CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure quality. - Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. - Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. - The Executive Summary should present a concise and accurate statement of the most critical elements of the report. - Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. - Evaluation methodology should be explained in detail and sources of information or data properly identified. - Limitations to the evaluation should be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). - Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. - Conclusions should be specific and concise, and include an assessment of quality and strength of evidence to support them supported by strong quantitative and/or qualitative evidence. - If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. - If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. See ADS 201mah, USAID Evaluation Report Requirements for additional guidance. OTHER REQUIREMENTS All modifications to the required elements of the SOW of the contract/agreement, whether in evaluation questions, design and methodology, deliverables and reporting, evaluation team composition, schedule, and/or other requirements will be agreed upon in writing by the COR. Any revisions made will be noted in the SOW annexed to the final Evaluation Report. 55 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 LIST OF ANNEXES to the SOW Activity Monitoring, Evaluation, and Learning Plan (AMELP) Years 1, 2, 3, and 4 Work Plans Years 1, 2, and 3 Annual Reports Task Order and executed modifications Local Authorities Code Data Quality Assessments (DQAs) 56 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 ANNEX B: METHODOLOGY AND LIMITATIONS The ET assembled the necessary names and contact information, and created draft instrument guides, to facilitate a quick start to the evaluation. The reason for this quick start is that Ramadan starts on April 2. The ET also reached out to the TADAEEM point of contact as soon as USAID provided their contact details, to further the efforts for a quick start. The ET proposed a sample of sites for the IR and discussing logistics requirements with METAL. With approval from USAID for funding and for the IR, the team set out on fieldwork quickly. Data Collection As a mixed-methods study, this evaluation involved continuous collection of primary and secondary, qualitative and quantitative data. The team accessed documents and data from TADAEEM and the GOT, and from other sources such as the World Bank and CSOs. The team found data on municipal performance, to take best advantage of secondary evidence to help answer the EQs more substantially. Unfortunately, GOT data on municipal performance was last collected in 2019. The evaluation was conducted from February 28, 2022, to May 18, 2022. Primary data collection involved semi-structured interviews with partners and officials, to gather comparable information across actors and stakeholders, but also with space for open-ended, nuanced responses about successes, challenges, and sustainability. The survey instruments highlighted required questions that are the most critical for answering the EQs. As part of each interview, the ET queried respondents regarding other valuable interviewees, to effect a snowball sample as appropriate and helpful. This helped for triangulation not only because the recommended interviewees add to the evidence but also because those potential respondents were somewhat more likely to have divergent views. The team took notes in the language of their choice, and translated to English for the team to analyze. The team lead and METAL QA reviewed incoming data, refined instruments, and provided detailed team supervision throughout the fieldwork. The ET fielded the survey using Google Forms after an extensive process of translation, piloting and refinement. The survey was sent to approximately 60 municipal contacts of TADAEEM (using email addresses provided by TADAEEM) and to a long list of 320 CSOs also provided by TADAEEM. During fieldwork, the team found that not all CSOs listed for sampled sites were actually involved with TADAEEM. The ET extended the deadline until after the Eid holiday and followed up with phone calls to maximize response rate. During routine monitoring of the responses, the team found no anomalies that would require adjustments to the surveys. Sampling Approach: survey The survey sample was censal: that is, all 33 municipalities (mayors, municipal councils, and administrative staff), as well as representatives of CSOs from these 33 municipalities, were potential respondents. It was not representative, since there is no information on why those who opt out of completing the survey do so, or whether they share the opinions and perspectives of those who answer the survey. Still, the range of responses was valuable to inform USAID and the GOT of successes, challenges, and lessons learned to inform the design of future initiatives. Because sample is not representative, the ET uses the number of responses, rather than percentages, in most of the reporting on survey results, so as not to imply proportions that might be read to apply more broadly. Using contact information supplied by TADAEEM, the evaluation team contacted 60 potential respondents from among the 33 municipalities that participated, from 19 of the treated sites. Only three among the 26 municipal respondents were women. TADAEEM also supplied a list of 320 CSO representatives, though contacts with them in the field showed that the list was not the list of participating CSOs, but rather a list of all CSOs TADAEEM had identified. This was confirmed in the survey responses. When asked whether they had been involved with 57 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 TADAEEM, 25 of the 37 respondents said they had not, leaving only 12 complete responses. Seven women responded from among these 12 CSO respondents. Table B-1 shows their response to this question on whether they were somewhat or very involved. Table B-1: Survey samples, by how involved they were with TADAEEM Somewhat involved Very involved Total CSOs 7 5 12 Municipalities 10 16 26 Total 17 21 37 The CSOs and municipalities that sent responses were as follows: Table B-2: Survey responses, by municipality and respondent type Municipality CSOs Municipality 1-El Kef 1 0 13-Mahdia 1 0 15-El Jem 0 1 16-Sidi Alouane 0 1 17-Gabes 2 1 18-Mareth 0 1 20-Metouia 0 1 21-Sfax 1 0 22-Sakiet Ezzit 0 1 23-Agareb 0 3 24-Tataouine 1 2 25-Ghomrassen 0 1 26-Tozeur 1 0 27-Kebili 0 1 28-Gafsa 1 1 30-Ennadhour 0 1 32-Ettadhamen 0 1 33-Mnihla 0 2 4-Sers 0 2 5-Siliana 1 2 6-Makthar 0 2 7-Kairouan 3 1 9-El Alaa 0 1 The ages of the respondents are shown in Figure B-1, below: 58 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Figure B-1: Survey respondents, by age and type Survey respondents were overwhelmingly administrative staff, which provides some balance against the interviews that included more mayors and vice mayors or other council members. Figure B-2: Municipal survey respondents, by role in the municipality Sampling Approach: site visits The team conducted site visits to eight municipalities, with one substitution from the proposed sample in the Inception Report. This substitution was due to a political issue in which the Gabes delegate assumed the mayor’s functions due to unresolved disputes within the municipal council. The evaluation team substituted Metouia for Gabes. Metouia is also in the Gabes Hub. The final sample was as follows: 59 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Table B-3: Final site visit sample Municipality Hub Population (Jan 2020) IDR Score (2018) Performance (2019) El Kef Kef 65,905 Low 85 Dahmani Kef 27,677 Low 59 El Alaa Kairouan 28,257 Very low 75 El Nathour Kairouan 32,088 Very low 64 Tozeur Tozeur 50,744 High 70 Kebili Tozeur 46,311 Low 75 Gabes Gabes 106,438 High 78 Metouia Gabes 13,289 High20 9121 Sakiet Ezzit Gabes 66,833 Very high 53 For the KIIs, the sample was purposive, in accordance with the list of EQs and the stakeholders and actors who are best positioned to respond to related interviews. This included line ministry roles at the central, governorate and municipal levels with whom TADAEEM worked, and others with useful perspectives such as other donors. In municipalities, the team interviewed CSO partners from TADAEEM activities, elected members of municipal councils, civil servants, and civic leaders. In the site visits as well as in Tunis and with virtual means, the team interviewed 84 people, per Table B￾4 below. Most interviews were with individuals, though some small group interviews were conducted where appropriate, such as three CSOs in one municipality that all worked somewhat with TADAEEM, or multiple members of a GOT team. Table B-4: Final list of interviewees, by sex Women Men Total US Government 2 4 6 Implementing partner - Tunis 6 10 16 Implementing partner - Field 5 5 10 GOT - National 5 8 13 GOT - Subnational (elected) 3 6 9 GOT - Subnational (administrative) 2 7 9 CSOs - Tunis 5 2 7 CSOs - Subnational 4 7 11 Other donors 1 2 3 Total 33 51 84 Of the 84 interviews, 45 were face-to-face (54 percent), including nearly all of the subnational elected and administrative interviewees and CSOs, as well as the Deloitte field or “hub” teams. Virtual interviews generally included USAID and the Tunis TADAEEM teams, many of whom were expats, as well as other donors, and totaled 39 (46 percent of the total). The ET prefers not to identify respondents in any way that would identify where they are from (with the exception of TADAEEM staff and consultants), to protect confidentiality. Given that USAID approved the Inception Report with that clause, the evaluation team stressed this level confidentiality in informed 20 This figure was not available for only Metouia, rather for its region, Gabes. 21 This figure was only available for Metouia for 2018, rather than 2019 as for the other municipalities. 60 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 consent discussions. In this way, respondents are likely to be more candid with interviewers. As a result, the list of respondents in Annex E does not identify people by name. The evaluation also relied on TADAEEM quantitative indicator data and the three waves of the Citizen Perceptions Survey they undertook, along with a review of activity documentation (please see Annex D) and outside documents, particularly decentralization documentation from the GOT. The EQs were mapped against data sources, collection methods, and analysis methods, as shown in Annex H. Data Analysis Quantitative survey data was summarized, tabulated, and presented to the team for review. Data were disaggregated by location, gender, and minority group as appropriate, though it is important to note that sample sizes and self-selection into the survey do not permit representativeness from the survey data. The overarching analytical model is that of findings, conclusions and recommendations (FCR), a model in which triangulation across sources is paramount. The ET assembled findings across the sources and methods, and across the EQs, and analyzed these jointly as a team during a half-day session in person, which complements the preliminary presentation preparation in which team members prepared summaries from among their interviews. This takes best advantage of the combined knowledge of the team, and the individual knowledge of each member. This triangulation allows joint interpretation across the array of sources from multiple streams of data – interview notes, secondary data, documentary review, context analysis, and the expertise brought by the team members. The ET also made use of individual team member knowledge, to ground-truth particular interpretations, thereby providing a check on confirmation biases. By presenting ideas together for shared scrutiny, the team developed conclusions that are well￾founded in the data and traceable for the reader. Qualitative data from key informants was coded based on the EQs and emergent themes. This was assembled in an Excel spreadsheet that shows the totality of responses per EQ and theme, and allowed the ET to review these responses holistically. Dissemination and Utilization USAID has noted ways in which the report and its evidence will be disseminated within the Agency and externally in Tunisia. First, the Mission is currently at a midpoint in the formulation of its new Country Development Cooperation Strategy (CDCS), so the evaluation will serve as an input to that process. The team prepared a preliminary presentation of findings near the end of data collection so as to slot into the CDCS timeline earlier than the final report. USAID determined the attendees of the preliminary presentation session and any that follow. Additionally, the ET prepared an Executive Summary and related four-page briefer to be used with various audiences. To that end, once finalized, these will be translated into Arabic and French, including any data visualizations that are useful to include. The Final Report and annexes will be available on the DEC upon approval, and the ET will collaborate with USAID to offer any other presentations they deem useful, such as with GOT, CSOs, or other audiences. Further detail on the site sample The team conducted a purposive selection of municipalities for site visits based on the following criteria: 1. Geographic diversity TADAEEM implemented its activities in 33 municipalities across 10 governorates. To coordinate field interventions, TADAEEM established offices in four regional hubs: Kef, Kairouan, Tozeur, and Gabes. Each hub implemented activities in a different number of municipalities across multiple governorates. To ensure geographic diversity, the team selected eight municipalities across TADAEEM regional hubs: two in Kef hub, two in Kairouan hub, two in Gabes hub, and two in Tozeur hub. 61 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 2. Population size The average population size of Tunisia’s 350 municipalities was estimated at 33,452 in January 2020.22 Over 270 municipalities have a population size below 50,000, 37 municipalities have a population size between 50,000 and 100,000, and 16 municipalities have a population size over 100,000. The team selected four TADAEEM municipalities with a population size under 50,000, three with a population size between 50,000 and 100,000, and one with a population size over 100,000, roughly reflecting the overall breakdown. 3. Regional development indicator The regional development indicator (Indicateur de dévelopement régional, IDR)23 factorizes several variables of human capital, infrastructure, employment, and demographics, to measure levels of human development in Tunisian regions. The IDR scores range between 0 (very low) and 1 (very high). In 2018, IDR scores were indexed in four ranks: 1 = 0.543 – 0.752 (very high), 2 = 0.489 – 0.543 (high), 3 = 0.446 – 0.489 (low), 4 = 0.285 – 0.445 (very low). The IDR scores are calculated at the governorate and delegation levels. A delegation can contain more than one municipality within its administrative limits. However, the delegations where TADAEEM municipalities are located contain exclusively one municipality. This makes it possible to use the IDR as a proxy indicator for local development to categorize and sample TADAEEM municipalities. Among the municipalities selected for the sample, three have a very high IDR score, 10 have a high score, 13 have a low score, and five have a very low score. One has a very high IDR score, two municipalities have a high score, three have a low score, and two have a very low score. Unfortunately, these data have not been updated since 2018 because of COVID. The data would be useful for USAID decision-making on new activities, and could potentially be accessed from the GOT when it is ready. 4. Indicator of municipal performance This indicator measures the performance of Tunisian municipalities according to three metrics: enhanced service delivery, participation and transparency, and enhanced resources. The indicator is calculated based on an annual performance evaluation of Tunisian municipalities conducted by the Public Services Oversight Authority. Indicator data is available for 2017, 2018, and 2019. The annual performance evaluation of municipalities for 2020 was canceled due to the COVID-19 situation. In 2019, the average performance score of Tunisian municipalities was estimated at 72,98.24 The team selected four TADAEEM municipalities with a performance score above national average, and four municipalities with a score below average. Limitations and Mitigation Strategies Biases that might have affected the evaluation and the team’s mitigation measures were predicted and in the Inception report and planned for in fieldwork. Recall bias is a systematic error that occurs when participants do not remember previous events or experiences accurately, omit details, or change the relative importance. The ET conducted as many KIIs and FGDs as possible within the fieldwork period to triangulate responses to increase the validity of the evaluation findings. The team sought saturation (recurrent responses) on each EQ during data collection, 22 Statistics published by Ministry of Interior at: http://www.collectiviteslocales.gov.tn/-/municipalites_population2020/ 23 Institut Tunisien de la Compétitivité et des Etudes Quantitative (ITCQE), « Indicateur Régionale de Développement: méthodologie et résultats », May 2018. At: http://www.itceq.tn/files/developpement-regional/indicateur-dev-regional.pdf 24 Public Services Oversight Authority, « Performance Evaluation Report of Tunisian Municipalities: 2019 », October 2021. Retrieved from: evaluation2019_rapport.pdf (collectiviteslocales.gov.tn) 62 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 indicating that the major themes, their variations, and important details had been sufficiently captured, to craft lessons learned from among the cases. Among interviews with TADAEEM staff, the ET observed very emphatic and repeated responses about certain less positive internal characteristics of the Activity, and about the extent of results, at times using the same or very similar language. This seems to indicate that respondents shared these opinions among themselves. The elapsed time since the end of respondents’ employment with TADAEEM, along with the strength of these considerations, seemed to shape recall around those issues, while other memories were less clear. These issues are comprehensively covered in this report, but recall bias around them likely limited the findings on other nuanced but less emotionally charged issues. Social desirability bias is the risk that key informants may be motivated to provide responses that they “think the evaluators want to hear” or that they imagine would be influential in obtaining further donor support. The ET mitigated this risk by explaining the objectives and potential benefits associated with the evaluation exercise clearly at the beginning of each KII, and in the informed consent portion of the online survey. The ET also remained neutral in reactions to respondent statements, and used neutral prompts to garner deeper details. Some TADAEEM respondents were unerringly positive, particularly at the start of the evaluation, perhaps due to social desirability bias. At the same time, many were unable to provide much detail, which may be related to recall bias. In later interviews, with more information under the belt of the evaluation team, the team was able to plumb contentious issues raised by others with gentle probes around the details, which elicited more nuanced responses and allowed the ET to understand the broad Activity team and its strengths and challenges better. The team also used triangulation to see challenges and strengths from different angles, helping the team to interpret the evidence more objectively. Self-selection bias. The survey response rates were lower than hoped, particularly for CSOs, and given the Ramadan season as well as the elapsed time since the end of TADAEEM, the ET cannot estimate why some potential respondents opted in while others did not. In survey research more generally, repeated observations indicate that people opt in to surveys when they have stronger, more extreme views, and since the results are more positive for municipalities and less so for CSOs, that was a confirmatory finding with field interview data that suggested the promise of work with CSOs was not fully realized. Still, it is important to note that the survey data are not representative, and to ensure this was not misinterpreted by readers, the evaluation team reported only numbers of respondents, rather than percentages, which would have indicated proportionality among the larger population. Selection bias is an error in choosing the individuals or groups to take part in a study, often exposing the evaluation to critiques about the representativeness of the population selected. Qualitative research is rarely representative, in the statistical sense, and as such that is not what the ET’s sampling and selection sought. Instead, the ET selected purposively with the goal of seeing a range of responses from among well￾functioning and less well-functioning municipalities, and across other categories to help the analysis inform design and implementation in the future. Still, the possibility exists that the ET would select the sites that are easiest to reach, all urban, or some other skew based on logistics or a failure to identify the important variables from among the study population. Even with the best intentions, the ET might also be pointed primarily to most active, responsive, or engaged stakeholders, or steered away from those with more mixed or even negative responses. To mitigate this, evaluation sampling attempts to cut across key variables that may have influenced TADAEEM’s effectiveness, as described in the section on Sampling, above. The ET used a purposive sample and supplemented planned interviews by asking respondents who else the team should interview. In two cases, this resulted in notably negative responses against the original interviewee’s more positive responses, indicating that asking for recommendations gave these two interviewees a chance to put forward names of people who could be more candid. The ET also obtained 63 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 interviews that were not recommended by TADAEEM, USAID or GOT interlocutors, which expanded the understanding and perspectives of the evaluation’s qualitative dataset. Evaluation timing. The timing of the evaluation was sensitive. First, Ramadan was in course throughout the data collection phase. When respondents are fasting, some will not go to their offices regularly, if at all. To mitigate these challenges the ET worked from a longer list of proposed interviewees, and with greater flexibility about which official in a given bureau answered questions. The ET used the extensive email and phone lists, and asked TADAEEM for their contact lists as well, while also pursuing snowball sampling, to ensure all means at the team’s disposal were exhausted in reaching out to a wide range of potential respondents. The evaluation team found that people in municipalities were generally willing to interview, though at a slower pace and with more rescheduling than is common. There were cases where the team pursued multiple individuals in given line ministries and among other donors, with minimal success. Second, the TADAEEM Activity closed and team members had departed. The ET worked with Deloitte to track team members as a priority, and used USAID-approved additional time to good advantage. The ET brought a USAID-signed letter to field visit sites to further reach out to elected officials, civil servants and civil society representatives. Being there in person, and demonstrating USAID’s commitment to the evaluation, appeared to help response rate, even during Ramadan. However, there were cases of refused or avoided interviews, notably among other donors, and among regional and line ministries. The other timing challenge came with the closure of the TADAEEM Activity and the departure of many of its team members. The ET worked with USAID and the TADAEEM point of contact from Deloitte to track team members – particularly regional hub leads – to follow up with these crucial respondents. This was a priority for the ET and began as soon as USAID approved the IR. Overall effect of the limitations There may be limitations to the validity of given data points from among the ET’s sources because of recall, social desirability, and other biases and timing issues. However, the ET exceeded the number of planned interviews and was able to triangulate from a range of different perspectives. In addition, the ET has presented preponderant evidence in this report, rather than single anecdotes, to avoid over￾interpreting the inputs of any given respondent. In the case of the survey, while the ET cannot speak representatively about TADAEEM’s interventions based on these data, there are strong trends for questions of satisfaction, sustainability, and challenges and mitigations that are usefully shared in this report. In any case, TADAEEM was municipal demand-driven: there was no expectation that all municipalities would receive the same treatments, work on the same axes, or reach the same conclusions about the intervention. As such the survey data remains valuable. 64 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 ANNEX C: DATA COLLECTION TOOLS Interview guides Each interview will begin with a question taken from the Appreciative Inquiry literature, designed to put respondents at ease and give them space to speak about their own proudest achievement as partners in the implementation of TADAEEM. The interviews will continue with questions that derive from each Evaluation Question that pertains to their role. The evaluation team, as experienced interviewers, will have latitude to pursue topics apart from those set out in the interview guides. While they will attempt to cover each topic area for the role of each interviewee, the ET understands as well that not all interviewees in a category will have the same ability to cover each topic. Question order will proceed from more general to more specific, and, where applicable, from less to more sensitive, to take advantage of time to gain rapport and confidence from interviewees. Introduction Thank you for taking the time to meet. My name is […] and I’m an independent evaluator who has been asked to look at the TADAEEM project from 2017 to the present, to learn about your experiences and perspectives, and to make recommendations for possible future projects. My team members X and Y are here with me. [IF USAID IS ATTENDING THE INTERVIEW, ADD THIS] Also on the call is ZZZ from USAID’s program office. They do not work with the office that managed TADAEEM, but with an office that manages evaluations.] The interview should take around an hour or less. We will hold your answers and comments in strict confidence. We will not share your answers with USAID, the government, or TADAEEM. [The USAID personnel on the call have the same requirement of confidentiality.] If there’s anything you’d like to say but are particularly concerned about its sensitivity, just let us know. We will take note of your concerns. It is very valuable to hear your perspectives – positive, negative, and mixed – to be able to offer useful recommendations for any future project. Your participation is voluntary – that is, you may decline to answer any question, or to participate at all. Do you have any questions? If not, may we begin? GOT Central Can you please tell me your name and title? How have you been involved with or known about the USAID TADAEEM activity, if at all? For how long? [Assuming they worked with TADAEEM:] Could you please tell me, what is your proudest accomplishment in working on decentralization, including the work you did with TADAEEM? In your opinion, how well did TADAEEM’s work align with the principles of citizen participation and improved service delivery that are found in the Code des Collectivités Locales? [PROBE: Did this change over time, and if so, how?] Did TADAEEM’s implementation help advance the decentralization goals of the GOT? If so, how, and if not, why not? [PROBE: Did this change over time, and if so, how?] 65 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 How did TADAEEM support regional and national service delivery, if at all? What might have been better? How did TADAEEM support equipment, coordination and communication during the COVID crisis, if at all? What might have been done better? [PROBE for municipal communication in particular, if they do not mention it.] The goal of decentralization is complex, with many challenges along the way. What were some of these challenges that TADAEEM faced, and how did they react to these challenges? What would you say is the likelihood of sustainability of the changes TADAEEM was able to support? [PROBE for national and subnational delivery mechanisms, coordination between government levels to respond to crises.] What would you do differently, if you were designing USAID’s next support package for decentralization and local governance efforts in Tunisia? What lessons did you learn from TADAEEM? GOT Governorate, Regional Can you please tell me your name and title? How have you been involved with or known about the USAID TADAEEM activity, if at all? For how long? [Assuming they had contact with TADAEEM:] Could you please tell me, what is your proudest accomplishment in working on decentralization, including the work you did with TADAEEM? In your opinion, how well did TADAEEM’s work align with the principles of citizen participation and improved service delivery that are found in the Code des Collectivités Locales? [PROBE: Did this change over time, and if so, how?] Did TADAEEM’s implementation help advance the decentralization goals of the GOT? If so, how, and if not, why not? [PROBE: Did this change over time, and if so, how?] How did TADAEEM support regional and national service delivery, if at all? What might have been better? How did TADAEEM support equipment, coordination and communication during the COVID crisis, if at all? What might have been done better? 66 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 The goal of decentralization is complex, with many challenges along the way. What were some of these challenges that TADAEEM faced, and how did they react to these challenges? Were you aware of any TADAEEM efforts to integrate women, youth or other marginalized groups into municipal processes? If so, what were they, and what did you know about them? Are they sustainable? What would you say is the likelihood of sustainability of the changes TADAEEM was able to support? [PROBE: for the function of national and subnational delivery mechanisms during and after TADAEEM, coordination between government levels to respond to crises.] [Feel free to change the wording if people do not understand “sustainability” – something like Do you think that TADAEEM helped change mechanisms of coordination at regional level and how? Did TADAEEM help coordination and delivery mechanisms to efficiently respond to crises?] What would you do differently, if you were designing USAID’s next support package for decentralization and local governance efforts in Tunisia? What lessons did you learn from TADAEEM? 67 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 GOT Municipality Can you please tell me your name and title? How have you been involved with or known about the USAID TADAEEM activity, if at all? For how long? [Assuming they had contact with TADAEEM:] Could you please tell me, what is your proudest accomplishment in the work you did with TADAEEM? How does your municipality work with CSOs and citizens? Has that changed since you began working with TADAEEM? In your opinion, how well did TADAEEM’s work align with the principles of citizen participation that are found in the Code des Collectivités Locales? [PROBE: Did this change over time, and if so, how?] In your opinion, how well did TADAEEM’s work align with the principles of improved service delivery that are found in the Code des Collectivités Locales? [PROBE: Did this change over time, and if so, how?] Did TADAEEM’s implementation help advance the decentralization goals of the GOT? If so, how, and if not, why not? [PROBE: Did this change over time, and if so, how?] How did TADAEEM support citizen participation in municipal government, if at all? How about women’s participation in particular? What might have been better? How did TADAEEM support municipal service delivery, if at all? What might have been better? Has TADAEEM had any effect on municipal communication and collaboration between your municipality and the central and/or regional governments? If so, could you please describe these effects? Has TADAEEM had any effect on the communication or collaboration between the municipal authority here and the municipal council and mayor? If so, could you please describe these effects? Has TADAEEM had any effect on communication and collaboration between your municipality and other municipalities? If so, could you please describe these effects? How did TADAEEM support coordination and communication during the COVID crisis, if at all? What might have been done better? How effective have the TADAEEM trainings been, if at all? Can you give examples? 68 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Has your opinion changed of the municipal council’s response to citizen needs since TADAEEM activities began? How and why? The goal of decentralization is complex, with many challenges along the way. What were some of these challenges that TADAEEM faced, and how did they react to these challenges? Were you aware of any TADAEEM efforts to integrate women, youth or other marginalized groups into municipal processes? If so, what were they, and what did you know about them? Are they sustainable? What would you say is the likelihood of sustainability of the changes TADAEEM was able to support? [PROBE: national and subnational delivery mechanisms, coordination between government levels to respond to crises.] Do you think that grants helped the municipality to face project funding and respond to citizens’ needs? How do you integrate grants in your budget? Are any other groups [donors or others] supporting your municipality with funds or activities related to citizen participation, service delivery, or crisis committees? If so, which groups, when and how does that support compare to the support from TADAEEM? If so, have donors or others coordinated their support to your municipality? What would you do differently, if you were designing USAID’s next support package for decentralization and local governance efforts in Tunisia? [PROBE for support to central government ministries, governorate/regions, municipalities, CSOs, grants, other] What lessons did you learn from TADAEEM? Civil Society (could be group interview) Can you please tell me your name and title? And tell me about your organization here. How have you been involved with or known about the USAID TADAEEM activity, if at all? For how long? [Assuming they had contact with TADAEEM:] Could you please tell me, what is your proudest accomplishment in the work that TADAEEM supported? How has your organization worked with the municipal council or the municipality, if at all? Has this changed since TADAEEM came? [PROBE for challenges between the two] What has been the response from the municipality to your engagement? How about the municipal council? 69 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 In your opinion, how well did TADAEEM’s work align with the principles of citizen participation and improved service delivery that are found in the Code des Collectivités Locales? [PROBE: Did this change over time, and if so, how?] How did TADAEEM support increased citizen participation and oversight of municipality, if at all? How about women’s participation in particular? What might have been better? How did TADAEEM support improved municipal service delivery, if at all? What might have been better? How did TADAEEM support coordination and communication during the COVID crisis, if at all? What might have been done better? The goal of decentralization is complex, with many challenges along the way. What were some of these challenges that TADAEEM faced, and how did they react to these challenges? Has the TADAEEM project had any effects on public perceptions of local government performance or legitimacy? If yes, what kinds of effects? Were you aware of any TADAEEM efforts to integrate women, youth or other marginalized groups into municipal processes? If so, what were they, and what did you know about them? Are they sustainable? What would you say are the long-term benefits of TADAEEM, if any? [PROBE: citizen participation, municipal service delivery, coordination between government levels to respond to crises.] What would you do differently, if you were designing USAID’s next support package for citizen participation and local governance in Tunisia? [PROBE for support to central government ministries, governorate/regions, municipalities, CSOs, grants, other] What lessons did you learn from TADAEEM? Participants/beneficiaries (could be group interview) Can you please tell me your name and age? Please tell us how you were involved with TADAEEM. [PROBE: adapt for citizen participation, services, women’s or marginal groups’ participation and activities] [Assuming they had contact with TADAEEM:] Could you please tell me, what is your proudest accomplishment in the work that TADAEEM supported? 70 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Do you think that TADAEEM increased citizen participation and oversight of municipality? If so, how? What might have been better? Do you think that TADAEEM supported municipal service delivery? If so, how? What might have been better? Can you think of any challenges that TADAEEM faced while working in your municipality? What were they? How did TADAEEM respond to these challenges? Were you aware of any TADAEEM efforts to integrate women, youth or other marginalized groups into municipal processes? If so, what were they, and what did you know about them? Are they sustainable? Do you think the changes TADAEEM was able to support will last? Why or why not? [PROBE: citizen participation, municipal service delivery] If you could tell the people who designed TADAEEM what to work on next, what would you recommend? What lessons did you learn from TADAEEM? Other donors Can you please tell me your name and title? Can you tell us about your interactions with TADAEEM, if any, and what you know about its implementation? [PROBE: how did this align with your agency’s efforts?] Did your agency coordinate with USAID, GOT, TADAEEM or others working on decentralization? If so, how? How efficient and effective was this coordination? In your opinion, how well did TADAEEM’s work align with the principles of citizen participation and improved service delivery that are found in the Code des Collectivités Locales? [PROBE: Did this change over time, and if so, how?] Did TADAEEM’s implementation help advance the decentralization goals of the GOT? If so, how, and if not, why not? [PROBE: Did this change over time, and if so, how?] How do you evaluate the experience of facing the COVID crisis and coordination between GOT and donors? Are coordination meetings still being maintained? 71 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 The goal of decentralization is complex, with many challenges along the way. Do you know of any challenges that TADAEEM faced, and how they responded to challenges? What would you say is the likelihood of sustainability of the changes TADAEEM was able to support? What would you do differently, if you were designing USAID’s next support package for decentralization and local governance efforts in Tunisia? What lessons did you learn from TADAEEM? TADAEEM (leadership) (could be group interview) Can you please tell me your name and title? How long were you with TADAEEM, starting when and ending when? Could you please tell me, what is your proudest accomplishment in the work you did with TADAEEM? [If respondent has experience from TADAEEM’s start-up:] How was TADAEEM designed to align with Tunisian decentralization systems and goals at the start of the project? How did TADAEEM adapt when the CCL was passed? What were the major successes of TADAEEM in the four objectives? [PROBE for each.] Please tell us about the contract amendment that added the fourth objective. When was that, and what were the conditions that led to the amendment? Can you also please talk about the MEL Plan indicators that changed – the how and why? What were the most important challenges you faced in the four objectives? [PROBE for each.] What did TADAEEM do to mitigate those challenges? [PROBE for each.] Can you identify any challenges or missed opportunities that you would go back to change if you could? Tell us about your efforts to integrate women, youth or marginalized groups into your activities. [PROBE for each. If they do not mention supporting elected women and career women staff in municipalities, probe on that specifically.] 72 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 Which TADAEEM efforts are most likely to be sustaining, and why? Which are least likely to sustain, and why? What would you do differently, if you were designing USAID’s next support package for decentralization and local governance efforts in Tunisia? What lessons did you learn from TADAEEM? TADAEEM (hub teams) Can you please tell me your name and title? How have you were you on the TADAEEM team? Has your role changed over that time? Could you please tell me, what is your proudest accomplishment in the work you did with TADAEEM? [If respondent has experience from TADAEEM’s start-up:] How was TADAEEM designed to align with Tunisian decentralization systems and goals at the start of the project? How did TADAEEM adapt when the CCL was passed? What were the major successes of TADAEEM in the four objectives? [PROBE for each.] What were the most important challenges you faced in the four objectives? [PROBE for each.] What did TADAEEM do to mitigate those challenges? [PROBE for each.] Can you identify any challenges or missed opportunities that you would go back to change if you could? Tell us about your efforts to integrate women, youth or marginalized groups into your activities. [PROBE for each. If they do not mention supporting elected women and career women staff in municipalities, probe on that specifically.] Which TADAEEM efforts are most likely to be sustaining, and why? Which are least likely to sustain, and why? What can you tell us about the indicators you reported on during TADAEEM? Which ones were helpful and which ones were not? What other ways to measure success of a project like TADAEEM can you think of? 73 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 What would you do differently, if you were designing USAID’s next support package for decentralization and local governance efforts in Tunisia? What lessons did you learn from TADAEEM? USAID Can you please tell me your name and title? How have you been involved with the TADAEEM activity? For how long? Has your role changed over time? If so, how? Could you please tell me, what is your proudest accomplishment in the work you did with TADAEEM? [If respondent has experience from TADAEEM’s start-up:] How was TADAEEM designed to align with Tunisian decentralization systems and goals at the start of the project? How did TADAEEM adapt when the CCL was passed? What has been USAID’s sense of TADAEEM’s added value? Do different USAID teams see TADAEEM’s successes and challenges differently? On what occasions were TADAEEM’s efforts thwarted by conditions – political, economic, social, or other – and how did you work with TADAEEM during this time? Were considerations around gender, youth and other marginalized groups central to your leadership of TADAEEM? Were these considerations served equally across the geographies of the implementation? Were there any donor coordination efforts across the donors that are implementing decentralization and local governance activities? Why or why not? If there were such efforts, can you talk about the results, please? What were your discussions with TADAEEM about sustainability? What efforts did TADAEEM make, in your opinion, to ensure that their work would be sustainable? I want to talk with you about the MEL Plan and specifically the indicators, including this major change in indicators that happened in 20__. Can you talk about the how and why of this change? What lessons did you learn from TADAEEM? 74 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 SURVEY INSTRUMENTS Survey questions: CSO 1. Were you involved with the project called TADAEEM? a. Yes, very much b. Yes, some c. No (or I don’t know) (SKIP OUT – thank them for their time) 2. Were you satisfied with the support TADAEEM provided? a. Yes, very much b. Yes, some c. No d. I don’t know 3. Why do you say so? (OPEN-ENDED) 4. Do you feel the TADAEEM project helped improve citizen participation in your municipality? a. Yes, very much b. Yes, some c. No d. I don’t know 5. Regarding citizen participation and oversight, did TADAEEM contribute to increase or improve citizens and CSOs attending and speaking up in municipal discussions? a. Yes b. No c. I don’t know 6. Did TADAEEM contribute to increase or improve citizens and CSOs participating in budgeting and annual investment plan (PAI)? a. Yes b. No c. I don’t know 7. Do you feel the TADAEEM project helped improve service delivery in your municipality? a. Yes, very much b. Yes, some c. No d. I don’t know 8. Do you feel the TADAEEM project helped your municipality work better with the regional and national governments? a. Yes, very much b. Yes, some c. No d. I don’t know 9. Did any of these positive factors affect how well the TADAEEM project performed in your municipality? (Please choose as many as apply) a. Having appropriate staffing in your municipality b. Having good procurement processes c. Having a functional organizational structure d. Having good technical capacity of your municipal team e. Having resources available to your municipality (national, regional, or other) f. Having strong political will g. Lacking corruption h. I don’t know 75 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 i. Other ________________ 10. Did any of these negative factors affect how well the TADAEEM project performed in your municipality? (Please choose as many as apply) a. Not having the right staffing in your municipality b. Not having the right procurement processes c. Not having the right organizational structure d. Not having the right technical capacity on your municipal team e. Not having resources available to your municipality (national, regional, or other) f. Not having strong political will g. Having any sort of corruption h. I don’t know i. Other ________________ 11. What did the TADAEEM team do to deal when there were challenges that were hard to overcome? (Please choose as many as apply) a. Sending someone to help with technical assistance b. Communicating to help resolve the issue c. Offering training or capacity-building sessions d. Bringing equipment e. I don’t know f. Other _____________ 12. Did TADAEEM contribute to better coordination between national and subnational governments to respond to national and local emergencies, compared to before TADAEEM? a. Yes b. No c. I don’t know 13. Did TADAEEM help your municipality to establish any new coordination mechanisms like the following: (MULTIPLE) a. Local committee b. Regional committee c. Other ______________ d. I don’t know e. None (SKIP NEXT) 14. Are any coordination meetings established with the help of TADAEEM still operating? a. Yes, regular meetings b. Yes, ad hoc meetings as needed c. Yes, but without meetings d. No e. I don’t know 15. Did your organization receive a grant from TADAEEM? a. Yes b. No (SKIP NEXT) c. I don’t know (SKIP NEXT) 16. What is your opinion of the outcome of the grant? If you had to give it a score, would you say the result was… a. Very positive b. Somewhat positive c. Negative d. Neutral e. I don’t know 17. What is your municipality? 76 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 18. What is your CSO’s goal in your municipality? (MULTIPLE) a. Citizen engagement in government b. Improved service delivery c. Participation of women, youth or other marginalized groups d. Other ___________________________________ 19. What is your gender? a. Female b. Male c. Other 20. What is your age? a. 15-29 b. 30-55 c. 55 or over Survey questions: MUNICIPAL OFFICIALS/MUNICIPAL COUNCIL/MAYOR 1. Were you involved with the project called TADAEEM? a. Yes, very much b. Yes, some c. No (or I don’t know) (SKIP OUT – thank them for their time) 2. Were you satisfied with the support TADAEEM provided? a. Yes, very much b. Yes, some c. No d. I don’t know 3. Why do you say so? (OPEN-ENDED) 4. Do you feel the TADAEEM project helped improve citizen participation in your municipality? a. Yes, very much b. Yes, some c. No d. I don’t know 5. Regarding citizen participation and oversight, did TADAEEM contribute to increase or improve citizens and CSOs attending and speaking up in municipal discussions? a. Yes b. No c. I don’t know 6. Did TADAEEM contribute to increase or improve citizens and CSOs participating in budgeting and annual investment plan (PAI)? a. Yes b. No c. I don’t know 7. Do you feel the TADAEEM project helped improve service delivery in your municipality? a. Yes, very much b. Yes, some c. No d. I don’t know 8. Do you feel the TADAEEM project helped your municipality work better with the regional and national governments? 77 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 a. Yes, very much b. Yes, some c. No d. I don’t know 9. Did any of these positive factors affect how well the TADAEEM project performed in your municipality? (Please choose as many as apply) a. Having appropriate staffing in your municipality b. Having good procurement processes c. Having a functional organizational structure d. Having good technical capacity of your municipal team e. Having resources available to your municipality (national, regional, or other) f. Having strong political will g. Lacking corruption h. I don’t know i. Other ________________ 10. Did any of these negative factors affect how well the TADAEEM project performed in your municipality? (Please choose as many as apply) a. Not having the right staffing in your municipality b. Not having the right procurement processes c. Not having the right organizational structure d. Not having the right technical capacity on your municipal team e. Not having resources available to your municipality (national, regional, or other) f. Not having strong political will g. Having any sort of corruption h. I don’t know i. Other ________________ 11. What did the TADAEEM team do to deal when there were challenges that were hard to overcome? (Please choose as many as apply) a. Sending someone to help with technical assistance b. Communicating to help resolve the issue c. Offering training or capacity-building sessions d. Bringing equipment e. I don’t know f. Other _____________ 12. Did TADAEEM contribute to better coordination between national and subnational governments to respond to national and local emergencies, compared to before TADAEEM? a. Yes b. No c. I don’t know 13. Did TADAEEM help your municipality to establish any new coordination mechanisms like the following: (MULTIPLE) a. Local committee b. Regional committee c. Other ______________ d. I don’t know e. None (SKIP NEXT) 14. Are any coordination meetings established with the help of TADAEEM still operating? a. Yes, with regular meetings b. Yes, with ad hoc meetings as needed c. Yes, but without meetings 78 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 d. No e. I don’t know 15. Did your municipality receive a grant from TADAEEM? a. Yes b. No (SKIP NEXT) c. I don’t know (SKIP NEXT) 16. What is your opinion of the outcome of the grant? If you had to give it a score, would you say the result was… a. Very positive b. Somewhat positive c. Negative d. Neutral e. I don’t know 17. What is your municipality? 18. The Government of Tunisia scores municipalities according to effectiveness, financial management, and other themes. Has your municipality’s score improved as a result of working with TADAEEM? a. Yes b. Partially c. No d. I don’t know 19. What is your gender? a. Female b. Male c. Other or prefer not to answer 20. What is your age? a. 18-29 b. 30-55 c. 55 or over 21. What is your role in your municipality? a. Mayor b. Council member c. Administrative staff d. Other _________ 79 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 ANNEX D: DOCUMENTS REVIEWED Literature review - GOT. (n.d.). Decentralisation operationalisation process: Update of the strategic orientations to 2039 and action plan 2018-2021. - GOT. (2016). Economic and Social Development Plan 2016-2020. - GOT. (2018). National Consultation on the Decentralization Track - Intermediate Report. May 2018 - GOT. (2021). Decentralisation as seen by the Municipalities: Summary report of the consultation on the decentralisation process. National Federation of Tunisian Municipalities (FNCT). May 2021. - GOT, Ministry of Local Affairs and Environment. (2018). Operational Guide of local authorities Evaluation. - TURKI, S.Y., & BEN JALLOUL, M –HAL. (2018). Vers une reconfiguration des territoires en Tunisie au temps de la décentralisation et de la communalisation intégrale- open Science 2018. - BEN JELLOUL, M. (2015). « Régionalisation et équité territoriale: Vers une nouvelle gouvernance territoriale en Tunisie », Maghreb-Machrek, 2015/4, n° 226. - DAFFLON, B., & GILBERT, G. (2018). L’économie politique et institutionnelle de la décentralisation en Tunisie: Etat des lieux, bilan et enjeux. Edition AFD septembre 2018. - BELLAMINE. Y. (2015). A quand un projet de loi de décentralisation (1ère Partie) – Nawat website. - FEMMEDINA. (2021). “An assessment of women’s participation in the Medina of Tunis” Retrieved from: https://www.citiesalliance.org/sites/default/files/2021 - TURKI, S. Y., & VERDEIL, E. (2013). “La décentralisation en Tunisie” - Lebanese Center for Policy Studies. - Institut Tunisien des Etudes Stratégiques ITES. (2014). Quelle décentralisation dans une Tunisie reconfigurée. - Carnegie. (2018). La décentralisation en Tunisie-Autonomiser les villes-engager les citoyens. - BEN NASER, T., Ministère des affaires locales. (2021). Les dessous d’une suppression – Nawat website. - YOUSFI, H. (2017). Redessiner les relations Etat/collectivités locales en Tunisie: Enjeux socio-culturels et institutionnels du projet de décentralisation – Agence Français de Développement Research paper. - International Crisis Group. (2019). Decentralisation in Tunisia: Consolidating Democracy without Weakening the State. MENA Region Report n° 198 (2019), Retrieved from https://www.crisisgroup.org/middle-east-north-africa/north-africa/tunisia/198- - United Cities and Local Governments (UCLG). (2007). « Tunisian Republic, 2007 ». Retrieved from https://www.gold.uclg.org/sites/default/files/Tunisia_0.pdf - TARCHOUNA, L. (2019). « The Experience of Decentralization in Tunisia since 2014 », Arab Reform Initiative, July 19, 2019, p. 4. Retrieved from: The Tunisian Experience of Decentralization Since 2014 – Arab Reform Initiative (arab-reform.net) - BLACKMAN, A., CLARK, J., & ŞAŞMAZ, A. (2017). « A. Generational divide in Tunisia’s 2018 municipal elections: Are youth candidates different? » Democracy International, 2017. Retrieved from http://democracyinternational.com/media/LECS_Policy%20Brief_Youth.pdf - TADAMUN. (2019). « Decentralization and Women’s Representation in Tunisia: The First Female Mayor of Tunis », June 24, 2019. Retrieved from: Decentralization and Women’s Representation in Tunisia: The First Female Mayor of Tunis - Tadamun - Instance Supérieure Indépendante pour lesElections. (2018). See: جائلنتا ائیةھلنا – Instance Supérieure Indépendante pour les Élections (isie.tn) - SLAIMIA, A. (2022). « What does the phenomenon of dissolution of municipal councils tell us in Tunisia? | Legal ؟تونس في ایدبلال ت جالم س لانحا ل ھاظ رة رنخبت ا اذما :from Retrieved 2022. 31, January ,Agenda Legal,» Agenda (legal-agenda.com) 80 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 - NOUIRA, A. (2012). « Tunisia’s Local Elections: Entrenching Democratic Practices », Arab Reform Initiative, July 12, 2012. Retrieved from: Tunisia’s Local Elections: Entrenching Democratic Practices – Arab Reform Initiative (arab-reform.net) - Haute Instance de la Finance Locale. (2020). « Premier Rapport Annuel : 2019 », December 3, 2020. Retrieved from : HIFL-RA2019 FR-03-12-20_Fn.pdf - ABDELLATIF, N. (2014). « Décentralisation et Fiscalité Locale en Tunisie », Ministry of Finance, 2014. Link : Microsoft PowerPoint - 4-Décentralisation de la Fiscalité_MF_011014.ppt [Mode de compatibilité] (finances.gov.tn) - World Bank. (2018). « Rapport de synthèse sur la performance de la gestion des finances publiques », 6 April 2018. - Tunisie Numerique. (2020). « Rapport : Les taxes municipales ne dépassent pas 1.7% du total des recettes fiscales », Tunisie Numérique, July 9, 2020. Retrieved from: Tunisie - Rapport : Les taxes municipales ne dépassent pas 1.7% du total des recettes fiscales… - Tunisie (tunisienumerique.com) - SADIKI, L. (2019). “Regional Development in Tunisia: The Consequences of Multiple Marginalization”, Brookings, January 2019. Link: Regional-development-in-Tunisia-the-consequences-of-multiple￾marginalization_English-Web.pdf (brookings.edu) Documents - TADAEEM Project Annual and Quarterly Progress Reports (all years, including Final report in 2021) - TADAEEM project: Citizen’s perceptions of municipal services baseline survey report (2018), first year study (2019) and presentation, and final study (2021) and presentation - TADAEEM Activity Performance Management Plan (all years) - TADAEEM Project Work Plans (all years) - TADAEEM Project: Municipal Grants Support 2019 (Ghannouche, Ksour Essef, El Alaa, Agareb, Tozeur, Makthar) / separate documents - TADAEEM Gender Report, 2021 - TADAEEM Project: Interventions by municipality / separate documents - TADAEEM Project: Biweekly bullets, success stories, agendas, and other digital documentation provided by METAL or directly by TADAEEM - TADAEEM Project: Political Economic [sic] Assessment - TADAEEM Project: Internal documents on a range of topics, from the 2018 elections to political economic [sic] analysis, sentiment analysis, concept notes and strategies for each service package, stakeholder mapping. These will be cited individually as and when they are used in the evaluation. - METAL/USAID. (2019). Data Quality Assessment, TADAEEM - METAL. (2022). Final TADAEEM SOW - METAL. (2022). Tunisia subnational governance assessment: findings and recommendations- report and presentation February 2022. - METAL. (2020). Mapping report of donor activities: Democracy, Rights and Government and Peace￾building. - USAID. (2022). Tunisia Gender Analysis - Preliminary Presentation and report. - USAID. (n.d.). Steps for central government decentralization. - USAID. (2016). Country Development Cooperation Strategy (CDCS) (last updated Sept 2021, per website). - USAID. (2009). Democratic Decentralization Programming Handbook. Data - Citizen Perceptions Survey data, 2018, 2019, 2021. 81 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 - METAL. (2021). Subnational Governance Assessment questionnaire data for Municipalities and Civil Society Organizations. - TADAEEM Service maps – February 2021. - TADAEEM indicator data – all years. - Statistics published by the Ministry of Interior (2020) Retrieved from: http://www.collectiviteslocales.gov.tn/-/municipalites_population2020/ - Institut Tunisien de la Compétitivité et des Etudes Quantitative (ITCQE) (2018) « Indicateur Régionale de Développement: Méthodologie et résultats », May 2018. Retrieved from : http://www.itceq.tn/files/developpement-regional/indicateur-dev-regional.pdf - Public Services Oversight Authority. (2019). « Performance Evaluation Report of Tunisian Municipalities: 2019 », October 2021. Retrieved from: evaluation2019_rapport.pdf (collectiviteslocales.gov.tn) Legal framework - GOT. (2016). Government Decree No. 2016-951 of 28 July 2016 on the establishment and organisation of the Ministry of Local Affairs. - GOT. Article 117(4) and Article 117(5), Electoral Law. - Governmental decree n°.2016-600 of 26 May 2016, relating to the creation of new municipalities in the governorates of Ariana, Ben Arous, Sidi Bouzid, Gabès, Médenine, Gafsa and Kébili. - Governmental decree n°.2016-601 of 26 May 2016, relating to the creation of new municipalities in the governorates of Ben Arous, Manouba, Bizerte, Nabeul, Zaghouan, Béja, Jendouba, Kef, Siliana, Kasserine, Sidi Bouzid, Kairouan, Sousse, Mahdia, Sfax, Gafsa, Kébili, Gabès, Médenine and Tataouine - Governmental Decree n°.2016-602 of 26 May 2016, modifying the territorial limits of certain municipalities. - Law 75-33 of 14 May 1975, promulgating the organic law on municipalities. - Organic Law n°. 2018-29 of 9 May 2018, on the Local Government Code. Tunisian Constitution (2014). 82 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 ANNEX E: RESPONDENTS INTERVIEWED Note: The list totals do not add exactly to 84 persons, as mentioned in the main body of the report, because one interviewed person held two roles. 83 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 ANNEX F: EVALUATION TEAM The evaluation team provided a complementary mix of evaluation and sectoral specialists. Team Leader Keri Culver has more than 20 years of evaluation experience and a strong background in democracy and governance programming, including monitoring, evaluating and learning. Aida Kraiem is a Tunisian specialist in decentralization and local governance, with significant experience in government and in examining donor programming, including the recently concluded Subnational Governance Assessment carried out by METAL. Zoubeir Daly is a specialist in decentralization and civil society, and brought field experience with CSO actors to the table. Jihed Haj Salem has worked in and researched donor programming in countering violent extremism and poverty, and is a quantitative data specialist as well. Amal Lajmi brings extensive experience with MEL and field research across sectors, and coordinated team fieldwork. METAL also provided the services of Sana Ben Salem, who has carried out multiple online surveys, and who served the team with refining the draft survey, launching the survey online, and tabulating responses for reporting. The management structure made use of the METAL team and resources, to ensure the ET was up to date on all recent research and evaluation that is pertinent to the TADAEEM evaluation; communicate through a central node; use their quality assurance and supervisory support; and take best advantage of their experience of seasoned MEL and sector professionals in designing, fielding, and reporting on the evaluation. Personnel Roles and Responsibilities ET Member Responsibilities Keri Culver o Lead the evaluation, e.g., desk review, data collection, analysis, report writing. o Manage the team, delegate responsibilities, guide team members, monitor progress, and provide quality assurance on deliverables. o Serve as liaison with METAL and USAID/Tunisia and lead presentations. Aida Kraiem o Provide context on decentralization and local government. o Identify appropriate interviewees and conduct interviews. o Develop instruments and protocols for fieldwork. o Take part in data collection in Tunis and in sampled municipalities. o Participate in analysis, report writing, and dissemination. Zoubeïr Daly o Provide context on decentralization and civil society. o Identify appropriate interviewees and conduct interviews. o Develop instruments and protocols for fieldwork. o Take part in data collection in Tunis and in sampled municipalities. o Participate in analysis, report writing, and dissemination. Amal Lajmi o Provide context on Tunisian development programming environment and MEL. o Identify and analyze appropriate secondary data sources and conduct interviews. o Develop instruments and protocols for fieldwork. o Take part in remote data collection. o Participate in analysis, report writing, and dissemination. Jihed Haj o Provide context on regions and municipalities, including vulnerability and poverty. Salem o Identify and analyze appropriate secondary data sources and conduct interviews. o Develop instruments and protocols for fieldwork. o Take part in data collection in Tunis and in sampled municipalities. o Participate in analysis, report writing, and dissemination. 84 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 ANNEX G: CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure quality. - Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. - Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. - The Executive Summary should present a concise and accurate statement of the most critical elements of the report. - Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. - Evaluation methodology should be explained in detail and sources of information or data properly identified. - Limitations to the evaluation should be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). - Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. - Conclusions should be specific and concise, and include an assessment of quality and strength of evidence to support them supported by strong quantitative and/or qualitative evidence. - If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. - If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. 85 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 ANNEX H: EVALUATION MATRIX EQ1 and EQ2 Data Sources Data collection methods Data Analyses EQ1. To what extent was TADAEEM aligned with citizen participation and service delivery in the 2018 CCL, including pivoting as necessary? EQ2. To what extent did TADAEEM contribute to the operationalization of the decentralization development plan? In what ways might TADAEEM have done better with this changing context? ⎜ GOT (central) Mondher Boussnina (IPAPD) / Amel Baoueb (Haute Instance de la Finance Locale – HIFL)/ Samia Lousseif (Directrice generale des progammes municipaux)/ Ridha Saadi (Dir general du CFAD); CPSCL ⎜ Local and regional government (Sample of mayors) ⎜ TADAEEM national and local teams ⎜ Literature review: Consultations and Assessments (government and National Federation of Tunisian Municipalities – FNCT)/ Donor reports/CCL/Legal framework/ Quelle decentralisation dans une Tunisie reconfiguree? Decree Ministère des affaires locales / MALE ⎜ Document review: Annual Reports/ AMELPS/ Work Plans/Research reports/ Success stories, intervention lists ⎜ USAID and other donors working in decentralization and local governance (World Bank, EU, GIZ, others) ⎜ Officials from governorates or regions ⎜ Desk Review, secondary data ⎜ Key Informant Interviews Triangulation across findings, conclusions recommendations (FCR) using: ⎜ Content analysis ⎜ Gap analysis ⎜ Gender lens 86 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 87 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 How effective was TADAEEM in integrating ⎜ Municipal councils and administrative leadership ⎜ Desk Review Triangulation women, youth, and other marginalized groups into its activities and supporting elected women and career women staff in municipalities? ⎜ Grants holders and other partners for specialized initiatives ⎜ Activity beneficiaries – women, youth, PWD ⎜ Key Informant Interviews across FCR using: ⎜ Content analysis ⎜ Gap analysis ⎜ Gender lens EQ6 Data Sources Data collection methods Data Analyses What measures has TADAEEM taken to ensure the sustainability of improvements in: citizens’ participation in municipal decision￾making, municipal service delivery, national and subnational government service delivery mechanisms, and coordination between national and subnational governments to respond to national and local emergencies and other crises? ⎜ Municipal councils and administrative leadership ⎜ Desk Review Triangulation ⎜ Civil society partners ⎜ Key Informant across FCR using: ⎜ Partner GOT agencies involved in crisis response – MOH, MOE Interviews ⎜ Online survey ⎜ Content analysis ⎜ TADAEEM leadership ⎜ Gap analysis ⎜ Officials from governorates or regions ⎜ Gender lens ⎜ Small sample from among the 151 supported during COVID ⎜ Frequencies, cross-tabs 88 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 ANNEX I: TADAEEM RESULTS FRAMEWORKS – 2017 (left) and 2021 (right) 89 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001 ANNEX J: MEMORANDUM OF INTENT: MALE AND TADAEEM [This is in PDF form, and will be added to the final PDF report] 90 ⎜ TADAEEM FINAL PERFORMANCE EVALUATION – METAL AID-280-TO-17-00001