DISCLAIMER: This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. N056 MALI JUSTICE PROJECT PERFORMANCE EVALUATION FINAL REPORT February 2023 Prepared under Contract No.: GS-10F-0033M / 7200AA18M00016, Tasking N056 DRG LEARNING, EVALUATION, AND RESEARCH ACTIVITY II (DRG-LER II) MALI JUSTICE PROJECT PERFORMANCE EVALUATION: FINAL REPORT February 2023 Tasking N056 Contract No. GS-10F-0033M / Order No. 7200AA18M00016 Submitted to: Matt Baker, COR Submitted by: NORC at the University of Chicago Attention: Matthew Parry, Program Manager Bethesda, MD 20814 Tel: 301- 634-9489; E-mail: parry-matthew@norc.org DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | i TABLE OF CONTENTS ACRONYMS................................................................................................................................. III EXECUTIVE SUMMARY.............................................................................................................. V OVERVIEW AND OBJECTIVES ................................................................................................. 1 CONTEXT ..................................................................................................................................................................... 1 GENDER MAINSTREAMING......................................................................................................................................... 1 PURPOSE ........................................................................................................................................................................ 1 EVALUATION QUESTIONS..................................................................................................................................... 1 EVALUATION DESIGN............................................................................................................... 2 DESK REVIEW AND FINALIZATION OF EVALUATION ACTIVITIES........................................................ 2 DESK REVIEW.................................................................................................................................................................... 2 CONFIRMATION OF ACTIVITIES THAT FALL WITHIN THE SCOPE OF THIS EVALUATION............ 2 DATA SOURCES.......................................................................................................................................................... 3 ACCESS TO JUSTICE (EQ1).......................................................................................................................................... 6 INSTITUTIONAL CAPACITY OF KEY JUSTICE SECTOR INSTITUTIONS (EQ2)....................................... 6 FRAMEWORK AND PROCESSES OF ACCESS TO JUSTICE (EQ3).................................................................. 7 CORRUPTION (EQ4)...................................................................................................................................................... 8 ANALYSIS....................................................................................................................................................................... 8 QUANTITATIVE DATA ANALYSIS............................................................................................................................. 9 QUALITATIVE DATA ANALYSIS................................................................................................................................. 9 GENDER AND SOCIAL INCLUSION (GESI)........................................................................................................... 9 LIMITATIONS ................................................................................................................................................................ 9 RESPONSE BIAS................................................................................................................................................................. 9 CAUSAL ATTRIBUTION..............................................................................................................................................10 FINDINGS.................................................................................................................................... 11 EQ1: MJPS’ CONTRIBUTION TO IMPROVED ACCESS TO JUSTICE....................................................... 11 SUPPORT TO CSOS FROM CDRAS ON GRANT MANAGEMENT, PROGRAM, AND TECHNICAL KNOWLEDGE .................................................................................................................................................................11 CDRA AND CSO-LED TRAINING TO COMMUNITY LEADERS ON DISPUTE RESOLUTION ..........15 ACCESS TO JUSTICE TRENDS FROM THE WJP’S ROL INDEX .....................................................................18 EQ2: MJP’S CONTRIBUTION TO IMPROVED INSTITUTIONAL CAPACITIES OF KEY JUSTICE SECTOR INSTITUTIONS......................................................................................................................................... 20 HUMAN RESOURCE MANAGEMENT SYSTEM AND TRAINING OF CENTRAL SERVICES STAFF AT MOJ, DNAJ, DNAJS ........................................................................................................................................................20 TECHNICAL ASSISTANCE ON THE DEVELOPMENT, IMPLEMENTATION, AND MONITORING OF ANNUAL PERFORMANCE PLANS ...........................................................................................................................21 TRAINING AND MENTORSHIP OF DNAJ STAFF ON MONITORING COURT PERFORMANCE.....22 EQ3: MJP’S CONTRIBUTION TO IMPROVED FRAMEWORKS AND PROCESSES FOR ACCESSING JUSTICE......................................................................................................................................................................... 24 REVISIONS TO THE PENAL CODE AND CRIMINAL PROCEDURE CODE...............................................24 REVISIONS OF CENTRAL SERVICES UNDER THE JUSTICE SECTOR ORIENTATION AND PROGRAMMING ACT..................................................................................................................................................26 LEGAL AID BILL ADVOCACY....................................................................................................................................27 TRADITIONAL JUSTICE BILL ADVOCACY...........................................................................................................29 EQ4: MJP’S IMPACT ON CORRUPTION............................................................................................................ 30 EQ5: SUSTAINABILITY............................................................................................................................................. 35 RECOMMENDATIONS.............................................................................................................. 36 ii | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV ANNEX I: REFERENCES ........................................................................................................... 38 ANNEX II: DESK REVIEW ........................................................................................................ 39 ANNEX III: KEY INFORMANT INTERVIEWS ....................................................................... 42 ANNEX IV: RESPONDENT LIST ............................................................................................. 53 ANNEX V: TWO-WAY FIXED EFFECTS ANALYSIS OF CORRUPTION OUTCOMES . 55 TABLE OF FIGURES Figure 1: Mali's Ranking on the WJP's Civil Justice Index (of 139 countries).................................................... 19 Figure 2: Mali's Ranking on the WJP's Criminal Justice Index (of 139 countries)............................................ 19 Figure 3: Proportion of Respondents Who Paid Bribes (Afrobarometer 2005-2020)................................... 32 Figure 4: Proportion of Respondents View Police as Corrupt (Afrobarometer 2005-2020) ....................... 32 Figure 5: Proportion of Respondents Who Paid Bribes by Region..................................................................... 33 Figure 6: Proportion of Respondents View Police as Corrupt by Region......................................................... 33 Figure 7: Proportion of Respondents Who Paid Bribes in Sikasso vs. Synthetic Sikasso............................... 35 Figure 8: Proportion of Respondents View Police as Corrupt in Sikasso vs. Synthetic Sikasso................... 35 Figure 9: Sikasso vs. rest of Mali overtime (police bribe, two-way fixed effects analysis)............................ 55 Figure 10: Sikasso vs. rest of Mali (police corruption, two-way fixed effects analysis)................................... 56 TABLE OF TABLES Table 1: MJP Evaluation Matrix....................................................................................................................................... 4 USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | iii ACRONYMS ADS Automated Directives System CDCS County Development Cooperation Strategy CDRA Community Dispute Resolution Advisors CSO Civil Society Organization CSR Corporate Social Responsibility CPL Standing Legislative Committee DCF Data Collection Firm DFM Directorate of Finance and Materials DNAPES National Directorate of Penitentiary Administration and Supervised Education DNAJ National Directorate for Justice Administration DNAJS National Directorate for Judicial Affairs and Seal DQA Data Quality Assurance EQ Evaluation Question ET Evaluation Team FCR Findings, Conclusions, and Recommendations FGC Female Genital Cutting GBV Gender-Based Violence GESI Gender and Social Inclusion GII Gender Inequality Index GOM Government of Mali GPEEC Gestion Prévisionnelle des Emplois, des Effectifs et des Compétences GRDI Gender Related Development Index HDI Human Development Index HiiL The Hague Institute for Innovation of Law HRIS Human Resource Information System ICRC International Committee on the Red Cross ICT Information, Communications and Technology IT Information Technology INFJ National Judicial Training Institute KII Key Informant Interview MINUSMA United Nations Multidimensional Integrated Stabilization Mission in Mali MJP Mali Justice Project MOJ Ministry of Justice MPFEF Ministry of Women’s Affairs NORC National Opinion Research Center OPG Office of the Prosecutor General PDG Peace, Democracy, and Governance PE Performance Evaluation PI Principal Investigator QIP Quick Impact Projects ROL Rule of Law SI Social Impact TA Technical Assistance TGI Tribunal de Grand Instance iv | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV UNODC United Nations Office on Drugs and Crime USAID United States Agency for International Development WJP World Justice Project USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | v EXECUTIVE SUMMARY The purpose of this study is to evaluate the performance of the Mali Justice Project (MJP), a $31.8 million program implemented from 2016-2022 in Kayes, Koulikoro, Sikasso, Segou, Mopti, Gao, Tombouctou and Bamako. The MJP program had three overarching objectives: 1. Enhance the Quality and Efficiency of Services Provided by Justice Sector Institutions. 2. Increase Immediate Access to Justice. 3. Increase Access to Justice and Reduce Corruption along the Sikasso-Koury and Sikasso￾Heremakono Trade Corridors. To achieve these objectives, MJP implemented a large number of sub-activities, ranging from grants for justice-sector CSOs to mediation training for community leaders to technical assistance and capacity building for key justice-sector institutions. To evaluate these activities, we conducted KIIs with project stakeholders and beneficiaries and examined trends in longitudinal quantitative data on corruption and access to criminal and civil justice. Because testimonials from project stakeholders and beneficiaries may be influenced by response bias, and because our quantitative analysis does not employ an experimental or quasi-experimental design, we caution that our results should not be interpreted as definitive evidence of impact. We organize our findings according to the five evaluation questions that guide this study: Evaluation Question 1: To what extent and in what ways did MJP improve justice service delivery? ● The main MJP activities that were designed to increase access to quality justice services included: i) providing grants, training, technical assistance, and other forms of support to CSOs in the justice sector, and ii) training community leaders on mediation and conflict resolution. ● Regarding MJP’s support to CSOs providing legal services, CSO officials report that MJP helped them expand their reach and services within the communities they serve. As a result of MJP’s support, CSOs were able to provide greater legal assistance, conduct media campaigns, and incorporate GESI and counter-human trafficking components into their services. However, a small number of CSOs also noted increased administrative workloads associated with MJP support that drew resources away from service provision. ● Regarding MJP’s support for training community leaders, numerous leaders reported that disputes within their communities had decreased since they began receiving training from MJP, a pattern they attribute to their improved conflict resolution and mediation skills, which in turn reduced conflicts within their communities. Community leaders expressed appreciation for MJP’s support and requested that this support continue in the future. ● Whether MJP’s support for CSOs and community leaders had an impact on access to justice in the aggregate remains unclear because our findings are based on beneficiary testimonials, which may be skewed by response or recall bias. At the aggregate level, secondary data from the World Justice Project indicate that access to quality civil and criminal justice in Mali is among the lowest in the world and has remained so for much of MJP’s tenure. But it is impossible to attribute these patterns to the presence or absence of MJP --- relative to the scale of the justice vi | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV sector in Mali, MJP is a small program, and it is likely that broader social, economic, and political forces outside the control of MJP drive aggregate, country-level trends. Evaluation Question 2: To what extent has MJP built the institutional capacities of key justice sector institutions? ● To increase the institutional capacities of key justice sector institutions, MJP provided human resources management training to key staff at the MOJ, DNAJ, and DNAJS. MJP also provided technical assistance to implement annual performance plans within the DNAJS. Lastly, MJP provided training to DNAJ staff on how to more effectively oversee courts. ● MOJ, DNAJ, and DNAJS officials who received MJP’s human resources management training report that this support strengthened their internal managerial capacities and significantly improved staff management, particularly in regards to forecasting and preventing staff shortages due to turnover and/or rotation. However, officials also noted the limitations of MJP’s support in light of the broader context of severe staff-shortages within justice sector institutions, shortages which are outside MJP’s control. ● DNAJS officials who received technical assistance on annual performance plans report that this support helped agency staff develop annual performance plans, identify annual goals and relevant performance metrics, and balance these objectives vis-à-vi agency budgets. Officials further report that these skills strengthened their strategic planning processes and helped staff identify and avoid financial risks. ● Regarding the impact of MJP’s support for DNAJ staff on how to monitor lower-level courts, officials report that MJP’s support helped them develop data collection tools to track staffing needs, which in turn helped managers more effectively assess the staffing needs at lower-level courts vis-à-vi caseloads and prevent staff shortages and/or unreasonable delays in access to justice. Evaluation Question 3: To what extent has MJP improved the framework and processes of access to justice? ● To improve the framework and processes of access to justice, MJP convened working groups and national validation workshops concerning four areas of legislative reform: revisions to the penal code and criminal procedure, revisions of central services under the Justice Sector Orientation Act, the Legal Aid Bill, and the Traditional Justice Bill. ● Across all four initiatives, officials expressed appreciation for MJP’s ability to “drive progress forward” by convening core stakeholders, hiring consultants to draft revised legal text, and advocating with government officials to build buy-in for reform. Officials also praised the highly collaborative process by which MJP advocated for these initiatives. While many of these legal reform initiatives are still in progress, officials report that they are further along than they otherwise would be in the absence of MJP. Evaluation Question 4: To what extent has MJP reduced the level of corruption in Mali? ● MJP’s anti-corruption activities were concentrated in the Sikasso region and organized around three primary activities: (i) support for Citizen Advocacy Offices and Watch Cells to reduce corruption along the Sikasso trade corridors; (ii) implementing a reporting system for roadside corruption; and (iii) education and public awareness to empower citizens and journalists to combat corruption. USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | vii ● Drawing on longitudinal survey data, we find that the proportion of Sikasso residents who paid bribes to the police increased by five percentage points between 2015 and 2018 before declining by three percentage points in 2020. We further find that the proportion of residents who believe that “most” or “all” of police are corrupt increased by five percentage points between 2015 and 2018 before declining by 20 percentage points in 2020. However, despite employing advanced analysis techniques including difference-in-differences and synthetic control, we are unable to attribute these patterns to MJP rather than broader social, political, or economic dynamics. However, this “absence of evidence” is not “evidence of absence”; we simply can’t differentiate changes related to MJP from changes related to broader socio-economic trends. Evaluation Question 5: To what extent has MJP successfully implemented sustainable practices for the Malian government and civil society that will continue after the project’s end in December 2022? ● Beneficiaries were not optimistic about the sustainability of MJP improvements. CSOs warned that their expanded activities could not continue without funding and technical assistance. This was especially true for their expanded legal aid services. Community leaders expressed that one-off trainings did not allow for continuous learning and adaptation; they also emphasized the need for refresher trainings in the future. For the MJP’s capacity building at key justice sector institutions, learning loss due to time and staff turnover were chief concerns. Based on these findings, we offer three sets of recommendations, all of which are aimed at USAID/Mali and its implementing partners (IPs): Recommendations for increasing access to justice ● In light of the encouraging albeit provisional evidence that MJP’s support for CSOs helped them expand their services within the communities they serve, we recommend that USAID/Mali and its IPs continue to invest in training and technical assistance for justice sector CSOs. Similarly, in light of the encouraging if provisional evidence that MJP’s support for training community leaders in mediation and dispute resolution reduced disputes in beneficiary communities, we recommend that USAID/Mali and its IPs continue to train community leaders on mediation and dispute resolution. For both activities, USAID/Mali and its IPs should prioritize repeated, regular trainings over one-off events, even if this means reaching fewer communities. o To increase the sustainability of these programs, USAID/Mali and its IPs should consider training CSOs in fundraising, and/or referring them to other donors and INGOs interested in their services, so they may continue to fund them. ● Alongside these efforts, USAID/Mali and its IPs should seek to generate more definitive evidence regarding the effectiveness of CSO support and community leader training. To this end, USAID/Mali and its IPs should design future programming in such a way as to allow for rigorous evaluation. This includes collecting before vs. after information on justice outcomes in beneficiary communities and employing experimental or quasi-experimental, comparison-group evaluation designs to generate credible evidence about the effectiveness of these programs in Mali. viii | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV Recommendations for building the institutional capacities of justice sector institutions ● Based on MOJ, DNAJ, and DNAJS officials’ positive testimonies regarding the impact of MJP’s technical assistance and training programs, we recommend that USAID/Mali continue to invest in this type of programming in the future. Recommendations for promoting legislative reforms in the justice sector ● In light of the high praise expressed by core legislative stakeholders for the inclusive, participatory, and determined manner in which MJP advocated for legislative reforms, we recommend that USAID/Mali use this approach as a model for future legislative reform programming and/or advocacy. USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 1 OVERVIEW AND OBJECTIVES CONTEXT The Mali Justice Project (MJP) aims to support the Government of Mali’s (GOM) goal of establishing more efficient, effective, and competent justice sector institutions that are accountable and responsive to citizens’ needs. Implemented by Checchi and Company Consulting, Inc. from 2015 to December 2022, the project has three overarching objectives: Component 1 (C1): Enhance the Quality and Efficiency of Services Provided by Justice Sector Institutions. Component 2 (C2): Increase Immediate Access to Justice. Component 3 (C3): Increase Access to Justice and Reduce Corruption along the Sikasso-Koury and Sikasso-Heremakono Trade Corridors. To achieve these objectives, MJP implemented a large number of sub-activities, ranging from grants for justice-sector CSOs to mediation training for community leaders to technical assistance and capacity building for key justice-sector institutions. GENDER MAINSTREAMING USAID/Mali completed a Gender Assessment and Addendum in 2012 and 2013 that outlined key priorities and gaps within the country, such as the prevalence of gender-based violence (GBV) and female genital cutting (FGC), low rates of women’s education, and low global rankings on indices such as the Human Development Index (HDI), the Gender Inequality Index (GII), and the Gender Related Development Index (GRDI). As a result of these inequities, USAID/Mali tasked MJP with incorporating gender throughout its programming. In particular, USAID/Mali tasked MJP with supporting these priorities through trainings on sensitive topics like GBV and human trafficking, developing platforms for increased female voice and agency, enabling access to justice for women, and encouraging linkages between stakeholders within the Ministry of Women’s Affairs (MPFEF) and other government ministries for the advancement of policies to empower women. PURPOSE The purpose of this Performance Evaluation (PE) is to evaluate the performance of MJP and inform future programming within the United States Agency of International Development (USAID) Mali’s Peace, Democracy and Governance (PDG) office. EVALUATION QUESTIONS This performance evaluation is guided by the following five evaluation questions (EQ): EQ1. To what extent and in what ways did MJP improve justice service delivery? EQ2. To what extent has MJP built institutional capacities of key justice sector institutions? EQ3. To what extent has MJP improved the framework and processes of access to justice? EQ4. To what extent has MJP reduced the level of corruption in Mali? 2 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV EQ5. To what extent has MJP successfully implemented sustainable practices for the Malian government and civil society that will continue after the project’s end in December 2022? EVALUATION DESIGN The evaluation began with a desk review of MJP documents to better understand MJP’s core sub￾activities. This review was followed by a series of discussions with USAID regarding which specific sub￾activities should fall within the scope of this evaluation. Recognizing that MJP is a large and expansive project, and that not all project activities are suitable for evaluation or would fit within this evaluation’s budget, the evaluation team (ET) met with its counterparts at USAID and MJP in December 2021 and January 2022 to identify which specific project activities should be the focus of this evaluation. The remainder of this section outlines the desk review and the decision process for determining which MJP sub-activities would be the focus of this evaluation. The section then discusses the data sources that will be used to evaluate each sub-activity. DESK REVIEW AND FINALIZATION OF EVALUATION ACTIVITIES DESK REVIEW During the start-up phase of this evaluation, the ET conducted a desk review of secondary materials. This desk review covered over 500 project files, reports, and contact lists and was designed to identify and finalize focal sub-activities, draft KII guides, and develop a preliminary list of KII respondents. The review included but was not limited to: USAID strategic planning documents (Regional and Country Development Cooperation Strategies and Performance Monitoring Plans); MJP’s Quarterly and Annual Reports; MJP’s Activity Monitoring and Evaluation Plan; and MJP activity, training, and sub-grantee reports. Annex I provides a summary of the desk review and its findings. The findings from this review were used to inform the design and analysis of primary data collected through KIIs. CONFIRMATION OF ACTIVITIES THAT FALL WITHIN THE SCOPE OF THIS EVALUATION Concurrent with the desk review, the ET held a series of meetings with its counterparts at USAID and MJP to discuss which specific MJP activities would be prioritized in this evaluation. Project activities that were robustly implemented and were most highly relevant to USAID’s broader learning objectives for justice sector reform programming were given special consideration. The outcome of these discussions was a refined set of priority project activities including: ● Technical Assistance (TA): One of the core features of MJP was its provision of TA to a broad spectrum of stakeholders pursuant to its objectives of capacity-building and greater access to justice. The ET will evaluate the extent to which TA improved justice service delivery (EQ1), built institutional capacity (EQ2), and led to sustainable change (EQ5). Upon discussions with USAID and at the request of MJP, the ET expanded this activity to also include TA provided to support legal/policy reform (EQ3). This inclusion recognizes the following limitations in assessing gains in legal/policy reform: USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 3 o Gains from such activities can be very difficult to measure due to the complexity of the legal/policy reform process and, in many cases, reform efforts take years to materialize. o The success and failure of legal/policy reforms often hinges on factors outside the control of MJP and the stakeholders it supports. ● Grantee Support and Funding: MJP provided significant financial and technical support to civil society organizations (CSOs) working to expand access to justice, address gender-based violence (GBV), counter human trafficking, and address other MJP priorities, including corruption (EQ4). ● Mediation and dispute resolution training for community leaders: Working through its Community Dispute Resolution Advisors (CDRAs) and local CSOs, MJP provided mediation and dispute resolution training for community leaders in Bamako, Kayes, Koulikoro, Mopti, Gao, Timbuktu, Menaka, Kidal, Dioila, Kangaba, and Taoudénit. ● Anti-corruption programming in Sikasso: MJP’s anti-corruption activities were concentrated in the Sikasso region and organized around three primary objectives: (i) reduced corruption along the Sikasso trade corridors; (ii) implementing a reporting system for roadside corruption; and (iii) education and public awareness to empower citizens to combat corruption. DATA SOURCES To structure the primary data collection activities, we organized MJP’s constituent activities and sub￾activities according to their intended impacts and our associated evaluation questions. Table 1 depicts this organization in the form of an evaluation design matrix. The table specifies the main MJP sub￾activities as they relate to the EQs, the associated KIIs and data sources used to evaluate each sub￾activity, and the evaluation topics as tailored to each sub-activity. KII protocols can be found in the Annex. 4 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV Table 1: MJP Evaluation Matrix MJP COMPONENT SUB-ACTIVITIES INTERVIEW TOPICS DATA SOURCES/ RESPONDENTS EQ1: To what extent and in what ways does MJP contribute to the improvement of justice service delivery? C2: Increase access to justice by supporting CSOs providing legal services Support to CSOs from Community Dispute Resolution Advisors (CDRAs) (MJP staff) on grant management, program, and technical knowledge Type of support received by MJP, how it has influenced their work and reach; sustainability of MJP-linked improvements; financial sustainability Eight KIIs with CSO management C2: Increase access to justice by supporting CSOs providing legal services Support to CSOs from Community Dispute Resolution Advisors (CDRAs) (MJP staff) on grant management, program, and technical knowledge Type of support provided to CSOs and community￾level leaders, how support changed capacity for CSOs to carry out their work; sustainability of MJP-linked improvements Four KIIs with CDRAs C2: Increase access to justice by supporting CSOs providing legal services CDRA and CSO-led training to community leaders on dispute resolution Content of training and mentorship received, how it has influenced justice services, practices, and reach within community 12 KIIs with community leaders, WJP Rule of Law Index EQ2: To what extent has MJP built the institutional capacities of key justice sector institutions? C1: Enhance the capacity of justice sector institutions through TA Development of a human resource management system and training of Central Services staff at MOJ, National Directorate for Justice Administration (DNAJ), and National Directorate for Judicial Affairs and Seal (DNAJS) on its use Types of human resource training received and how the training and human resource management system influenced operations, oversight, and provision of services Six KIIs with MOJ, DNAJ, DNAJS, and court staff C1: Enhance the capacity of justice sector institutions through TA Provision of TA on the development, implementation and monitoring of annual performance plans Role of practical exercises and mentorship in the budgeting, planning, and execution of performance management plans Six KIIs with MOJ, DNAJ, DFM, DNAPES, and court staff C1: Enhance the capacity of justice sector institutions through TA Training and mentorship of DNAJ staff on how to monitor the performance of courts Types of training and mentorship received, how its influenced operations and oversight of courts Six KIIs with DNAJ and court staff USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 5 MJP COMPONENT SUB-ACTIVITIES INTERVIEW TOPICS DATA SOURCES/ RESPONDENTS EQ3: To what extent has MJP improved the framework and processes of access to justice? C1: Enhance the capacity of justice sector institutions by supporting the review and revision of legal codes and text Revisions and validation of the Penal Code and development of the Criminal Procedure Code Intention of revisions, format of the review process, and anticipated effect of the revised bill on the provision of and access to justice Five KIIs with Working Group Participants C1: Enhance the capacity of justice sector institutions by supporting the review and revision of legal codes and text Revisions and validation of central services under the Justice Sector Orientation and Programming Act Intention of revisions, format of the review process, and anticipated effect of the revised bill on the provision of and access to justice Three KIIs with Working Group Participants C2: Increase access to justice through legal reform Support the Legal Aid Bill by building consensus and identifying potential funding streams for legal aid if enacted Intention of revisions including gaps in previous legislation, format of the review process, and anticipated effect of the revised bill on the provision of and access to justice Three KIIs with the Malian Bar Association, Employer Federation, and Avocats sans Frontières C2: Increase access to justice through legal reform Drafting of the Traditional Justice Bill Intention of bill, format of the review process, and anticipated effect of the bill on the provision of, and access to, justice Three KIIs with Working Group participants, KIIs with community leaders noted above EQ4: To what extent has MJP reduced the level of corruption in the country? C3: Reduce corruption along trade corridor users Support for Interregional Platform and Citizen Advocacy Offices aimed at reducing instances of corruption among trade corridors Secondary data will review the extent to which metrics on citizens’ perception of corruption have changed over the period of MJP Afrobarometer public opinion data EQ5: To what extent has MJP successfully implemented sustainable practices for the Malian government and civil society that will continue after the program’s end in December 2022? Cross-cutting across all the above sub-activities and data sources 6 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV ACCESS TO JUSTICE (EQ1) MJP had two main sub-activities designed to increase immediate access to justice (EQ1). The first was the provision of grants and technical training to CSOs on fundraising, grant management, and technical topics related to service delivery. Supported in Year Six by MJP’s CDRAs, this sub-activity was designed to increase the quality, depth, and reach of the services provided by these CSOs, thereby increasing access to justice for ordinary citizens. To evaluate this activity, we spoke with management staff at CSOs who received MJP support. Under C2, MJP worked with 16 CSOs and provided the ET with contacts at each of these institutions. We conducted KIIs with a total of eight randomly selected CSO contacts. The interviews covered the type of support received, how support influenced their operations and reach (EQ1), and the perceived sustainability of the MJP-linked improvements (EQ5). We also interviewed MJP’s CDRAs, who were responsible for providing direct, embedded support to CSOs. At the time of data collection, there were four active CDRAs, all of whom interviewed on topics including: how CDRAs anticipate their support will impact the CSOs’ operations and reach (EQ1), and the perceived sustainability of MJP-linked improvements (EQ5). The second of MJP sub-activities designed to increase immediate access to justice was the training of traditional authorities and religious leaders on conflict resolution and compliance with human rights standards. According to MJP’s records, CDRAs and CSOs jointly trained these community leaders. A subset of these leaders also received ongoing mentoring support by CDRAs to further develop their conflict resolution skills and knowledge of human rights. According to MJP’s records, a total of 50 traditional authorities and religious leaders received this support. We conducted KIIs with 12 randomly selected community leaders. The interviews inquired about the content of the training and support that they received and how this support subsequently impacted the conflict resolution and mediation services they provide in their home communities. We completed the CSO, CDRA, and community leader KIIs with a longitudinal analysis of the World Justice Project’s (WJP) Rule of Law Index, which includes two sub-indices of interest to this evaluation: the civil justice sub-index and the criminal justice sub-index. The civil justice sub-index captures the degree to which civil justice is accessible and affordable, free of discrimination, free of corruption, not subject to unreasonable delay, effectively enforced, and free of improper government influence. The criminal justice sub-index measures the degree to which the criminal justice system is timely and effective, impartial, free of corruption, free of improper government influence, and operates in adherence to due process and the rights of the accused. Although we cannot attribute trends in these indices specifically to MJP, we nevertheless document them to understand the broader context in which MJP has operated. INSTITUTIONAL CAPACITY OF KEY JUSTICE SECTOR INSTITUTIONS (EQ2) MJP has three core sub-activities aimed at strengthening the capacities of key justice sector institutions (EQ2). The first sub-activity is the modernization of human resource management including the installation of a Human Resource Information System (HRIS) and the Forward Planning of Staff, Jobs, and Skills (Gestion Prévisionnelle des Emplois, des Effectifs et des Compétences, or GPEEC). USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 7 MJP provided human resource management support to the Central Services divisions of the MOJ, the National Directorate for Justice Administration (DNAJ), the National Directorate for Judicial Affairs and Seal (DNAJS), and the National Judicial Training Institute (INFJ). This support included the design, development, procurement, and installation of the HRIS which included study visits to Belgium and the development of a working group composed of officials from DNAJ and National Directorate of Public Service and Personnel. Among other tasks, MJP updated job descriptions and developed a GPEEC Operational Strategy guide at the request of the heads of the Central Services and the courts. Trainings on both HRIS and GPEEC commenced in 2020 for heads of the Central Services and court officials. To evaluate this sub-activity, we conducted semi-structured KIIs with officials from the MOJ, DNAJ, and DNAJS. A list of these individuals was provided to the ET by MJP; from this list, the ET randomly sample six individuals. The interviews covered the services and training received and how support has influenced operations. The second sub-activity covered under EQ2 is advisory support to Program Managers of the Central Services and the courts in the development and execution of performance monitoring plans. Trainings included practical support on formulating strategic objectives, setting performance indicators, identifying relevant activities, budgeting of activities, and development and implementation of a work plan. The ET conducted KIIs with six individuals from MOJ, DNAJ, DFM, DNAPES, and the courts. These individuals were randomly sampled from a list of relevant individuals provided by MJP. The interviews inquired about the type of training and support received, how this support aligned with their institutional needs and priorities, and how it influenced their institutional capacity, including their ability to execute annual workplans. The third and final MJP sub-activity designed to improve institutional capacity was training and mentorship to DNAJ Managers on how to monitor the performance of courts. MJP assisted in the development of tools, including a guidebook to assess court performance, and provided on-the-job training during court performance monitoring missions. To evaluate this sub-activity, the ET conducted KIIs with DNAJ and court staff about the monitoring tools developed by MJP, the type of support received, and how this support impacted their institutional capacity to oversee the courts. FRAMEWORK AND PROCESSES OF ACCESS TO JUSTICE (EQ3) Components 1 and 2 of MJP aimed to strengthen the framework and processes of access to justice by reviewing, revising, and drafting legal codes and texts. According to its own accounting, MJP affected 7.2 policies per year on average, whether through introduction, adoption, revision, or repeal. Four of MJP’s distinct sub-activities can be linked to these achievements. The first is drafting of the Criminal Procedure Code to clarify the rules around investigation, detention, and processing, and revisions to the Penal Code. MJP supported several steps in the drafting and revision process including provision of a consultant, formulation of a working group to develop proposed changes to the Penal Code, and support to the Standing Legislative Committee (CPL) to advance the chances of enacting these changes into law. The ET interviewed three members of this working group and two members of the CPL. The second sub-activity, which falls under the Justice Sector Orientation and Programming Act of 2020- 2024, included the revision of texts about the operation of several of the Central Services, including 8 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV DNAJ, among others. The purpose of the proposed revisions is to strengthen the autonomy and human resources of the Central Services to complete their respective missions. MJP supported a working group established by the Minister of Justice and Human Rights. The ET interviewed six members of this working group. The third sub-activity designed to strengthen the framework and processes of access to justice was support for the revision of the 2001 Legal Aid Bill, which calls for those without means to receive legal aid but lacks a provision for funding such aid. To address this funding gap, MJP worked closely with the Malian Bar Association to (1) expand the definition of legal aid to include paralegal and other support and (2) encourage them to provide these services pro bono. Avocats sans Frontières, one of MJP’s CSO grant recipients, also played a role in the review and revision of the bill as one of the leading providers of paralegal support. In addition, MJP secured a pledge from the Conseil National du Patronat (Employer Federation) to work towards an additional stream of legal aid funding under their corporate social responsibility commitments. To evaluate this sub-activity, the ET conducted KIIs with individuals from each of the following institutions: Malian Bar Association, Avocats sans Frontières, and Conseil National du Patronat. The fourth and final of MJP sub-activities designed to strengthen the framework and processes of access to justice is the drafting of the Traditional Justice Bill. The Bill clarifies the role of traditional justice and its interaction with the formal justice system but is still awaiting approval by the Legislature. To advance the likelihood of passage, MJP in collaboration with the United Nations Mission in Mali conducted a mapping of the informal/formal justice sectors, hired a consultant to draft the bill, established a taskforce to review and promote the bill, provided technical and financial support for the CPL’s review of the bill, conducted a validation workshop, and developed a traditional authority training program. To evaluate these sub-activities, we interviewed two members of the taskforce and a representative from the United Nations Mission in Mali, drawing on the contacts provided by MJP. All interviews related to the four sub-activities discussed in this sub-section inquired about the intention of revisions (or drafting of a new policy), the format of the review process, and the anticipated effect of the revised legislation on the framework and processes of access to justice. CORRUPTION (EQ4) Component 3 of MJP attempted to reduce corruption along trade corridors. Due to resource constraints for this evaluation, we elected to answer EQ4 primarily through a pre vs. post, difference-in￾differences, and synthetic control analysis of public opinion data from the Afrobarometer. In particular, we compiled public opinion data from Afrobarometer survey from 2005, 2008, 2012, 2015, 2017, and 2020. These surveys measured two core outcomes of interest in this evaluation: whether respondents report paying a bribe to police in the past year and whether respondents believe “most” or “all” of police are corrupt. ANALYSIS We analyzed data through an iterative process during and after the data collection. We triangulated findings from the desk review documents, quantitative data, and the KII data to assess trends, consistency, affinity, and contradiction between data sources, aiming to better answer the EQs, generate more reliable findings, and reach well-founded conclusions. We also applied a gender equality and social USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 9 inclusion (GESI) lens to understand how the findings related to USAID/Mali’s gender mainstreaming priorities. QUANTITATIVE DATA ANALYSIS Our quantitative analysis of secondary data is focused on documenting overtime trends in key project outcomes, including corruption experiences, perceptions of the criminal justice system, and access to due process in civil and criminal justice matters. While trends cannot be interpreted as evidence of MJP’s causal impact, they do help to understand the broader context in which MJP operates. The one exception to our narrow focus on nationwide trends is our analysis of MJP’s anti-corruption activities, which were concentrated entirely in Sikasso, thereby opening the door for our analysis to use Mali’s remaining regions as a comparison group. To this end, we also conduct difference-in-differences and synthetic control analyses to estimate MJP’s impact on corruption in Sikasso. For full details, see the Findings section below. QUALITATIVE DATA ANALYSIS We complemented the longitudinal analysis of the quantitative indicators with in-depth, qualitative analysis of the key informant interviews described above. During data collection, the ET took detailed notes, and recorded (with consent) all discussions. After data collection, the ET utilized a structured coding approach to conduct content and thematic analysis using Excel software for identification of emergent themes and contextual factors upon which to draw findings and conclusions and to develop evidence-based recommendations. The ET utilized a Findings, Conclusions, and Recommendations Matrix to triangulate data across data sources and to ensure an evidence-based, triangulated, and systematic analysis process. As a result of small sample sizes, the ET reports findings for all MJP partner respondents together, rather than disaggregating the findings by type of partner (MOJ, DNAJ, etc.). Within sub activities, respondent groups tended to provide similar feedback. Thus, unless otherwise noted, each finding is consistent across respondent groups (CSOs, CDRAs, government respondents, etc.). GENDER AND SOCIAL INCLUSION (GESI) As part of the evaluation, we assessed the extent to which MJP implemented and/or adopted planned activities, such as gender-specific training, guidelines, indicators, policies, and strategies in line with USAID/Mali’s gender mainstreaming priorities. With a GESI-sensitive approach, we reviewed secondary data and aimed to maintain gender balance in the KIIs to the extent feasible. We used a GESI scorecard in compliance with USAID’s Automated Directives System (ADS) 205 standards. This allowed us to assess gender sensitivity throughout the evaluation process. LIMITATIONS RESPONSE BIAS Response bias occurs when respondents’ responses systematically diverge from the “true” or “accurate” response to a question. Once form of response bias is what social scientists refer to as “social desirability bias”, or the tendency of respondents to overreport information that they believe is socially- 10 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV desirable. For example, with the knowledge that evaluation results may inform the future funding decisions, respondents may provide evaluators with positive feedback about the project because they want activities to continue. Similarly, MJP staff may overemphasize MJP’s successes to reflect well on their own performance. Another form of response bias is recall bias, which occurs when respondents simply cannot accurately recall the information that is being asked, but still offer a response. Indeed, response bias can sometimes be unconscious: respondents may believe something had an effect when it did not, or they may recall something that did not actually happen. We attempted to mitigate this bias by asking for verifiable examples of all core claims in the KIIs, and by communicating clearly to respondents during the consent process that there are no direct benefits to participating in the interviews and that there will be no retaliation against them for their responses. Despite these efforts, we cannot rule out the risk of response bias. To the contrary, respondents frequently expressed their desire for continued MJP support, suggesting their responses may be influenced by this prospect and highlighting the importance of interpreting their testimony with caution. CAUSAL ATTRIBUTION The EQs are inherently causal in nature – they inquire about the causal impact of MJP activities on the quality of justice service delivery (EQ1), institutional capacity (EQ2), corruption (EQ4), and sustainable practices (EQ5). Answering causal questions requires identifying a comparison group that accurately captures the counterfactual – in this case, what justice service delivery, institutional capacity, and corruption would have been like in the absence of MJP program. But several factors prevent us from employing a comparison group-based research design. First, MJP is not a single intervention but rather a collection of myriad distinct interventions, ranging from legislative reform to TA to conflict mediation to support for CSOs to workshops for community leaders. Evaluating the impact of any one of these activities would require a unique identification strategy to identify a valid comparison group for the set of beneficiaries associated with the chosen intervention. Although it might be possible to do this for some of MJP’s constituent interventions, doing so for all or even several of MJP’s activities would far exceed the budget for this project. A second barrier to our ability to infer casual attribution is that MJP was often highly selective in which units it targeted for support. For example, MJP may have (understandably) focused its legislative reform efforts on the most promising or likely-to-pass legal reforms, or it may have targeted its TA towards agencies and bureaucrats most willing to engage. Because MJP was selective in targeting its interventions, it would be difficult or perhaps impossible to identify valid comparison groups. Instead of employing an evaluation design based on a comparison group, we intend to address the EQs using descriptive data about how beneficiaries perceived MJP’s services, what effect they think MJP had on access to justice and what evidence (examples) they can provide, and how sustainable they feel those impacts will be. We further triangulate these findings through a pre vs. post trends analysis of Afrobarometer data and other secondary data sources on justice sector outcomes. But these methods have significant limitations that need to be kept in mind when interpreting the results of this study. USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 11 FINDINGS EQ1: MJPS’ CONTRIBUTION TO IMPROVED ACCESS TO JUSTICE To improve access to justice, MJP provided grants and technical assistance to CSOs in the justice sector. MJP also provided mediation and dispute resolution training to community leaders. This section evaluates each of these interventions in turn. SUPPORT TO CSOS FROM CDRAS ON GRANT MANAGEMENT, PROGRAM, AND TECHNICAL KNOWLEDGE MJP worked to build CSO capacity among those that interface between justice sector actors and users through law clinics, legal aid groups, and gender mainstreaming activities embedded within these clinics and legal aid groups. To facilitate this, MJP developed a list of lawyers from the Young Lawyer Association who were willing to provide pro bono judicial and legal assistance to law clinics and law assistance offices of partner CSOs. MJP also awarded grants to eleven CSOs for additional technical and programmatic assistance. According to CSO respondents, the primary form of support that MJP￾affiliated CDRAs provided to CSOs was support for mediation and paralegals (36 responses, 8/8 respondents).1 One CSO respondent noted, “our role is to build the capacity of paralegals in our intervention zones to enable them to intervene in the communities through talk shows and radio programs, but also to facilitate access to justice for victims of [GBV], i.e., to bring them closer to the actors who provide care and also to set up consultation frameworks for all the actors who intervene in this field.” A CDRA respondent discussed the training they provided for CSOs’ paralegals: “the paralegals are regularly trained, or I talk on the phone with those who are not in Timbuktu to find out about the difficulties they are facing and to be able to report them to my expertise, because there are cases, like the ones I just mentioned, that require our intervention for legal assistance.” MJP also supported gender and social inclusion (GESI) interventions (30 responses, 6/8 respondents), various forms of TA (28 responses, 5/8 respondents), and trainings (24 responses, 5/8 respondents). Grant management was not a primary means of activity support, as it was noted only in three responses. Respondents also noted that capacity-building support was helpful for CSOs (14 responses, 4/8 respondents) and for stakeholder coordination (13 responses, 4/8 respondents), but most importantly for citizen engagement and impact (20 responses, 5/8 respondents). One CDRA respondent noted, “MJP provides communities with legal assistance through lawyers, and this is also very important. Women victims of sexual violence receive a small amount of money for medical care and what follows because when there is a rape for example, the 72 hours are very important for the medical care of the young lady to avoid unwanted pregnancy or [sexually transmitted diseases].” MJP-supported CSOs’ citizen engagement activities included not only legal assistance, but also media campaigns, direct outreach, and training sessions for CSOs on sensitive topics such as GBV and human trafficking. As one CSO respondent summarized, “in our intervention zones…we have conducted many information campaigns…the team has contracted with local radio stations and during this contract they have conveyed a lot of information. But before that, there were training sessions because in order to convey information on trafficking, one must first be equipped with the necessary tools.” 1 In this section, “responses” refers to the number of mentions in our interviews with CSO Officials (N=8). It does not refer to the number of respondents. 12 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV CSO respondents also indicated that their reach as an organization increased while working with MJP (nine responses, 5/8 respondents), with one CSO respondent stating: “Our activities have increased because there are a lot of things we have developed. We have BAPCs [Citizen Advocacy and Assistance Offices] instead of [Citizen Advocacy Offices]. We have been able to launch administrative documents for a more national holistic vision. We have taken over the protection of children. We have expanded the GBV component through the BAPC, especially in terms of the management of GBV survivors in addition to prevention activities. Nowadays, BAPC agents have a national coverage because instead of four numbers to call, there are toll-free numbers on which we are accessible, especially with the call center system that has been set up. With this call center system, we have multiplied by ten the number of people who request our support and also the number of people who benefit from our treatments…From January to June 2022, we have processed more than 2000 requests.” Another CSO respondent highlighted the increased diversification of clientele that occurred while working with MJP: “It is not the same recipients [as before]. We have many cases. We have not only the victims of [GBV]; we also have cases of rape. We have cases of cross-border exploitation that we receive. Not many because they are not very well known. But we have a diversified clientele according to the forms of violence that we know. There are also cases of disenfranchisement of inheritance issues that we deal with. As I said, it is really a wide range of violent cases.” Yet another CSO respondent noted, “the work has also increased because it has allowed us to leave Bamako to go into these circles and work in 51 communes as I mentioned and it is true that in the past years, we were in the field but did not have as much work volume, of communes to survey.” However, a few respondents indicated that their administrative workloads increased because of pressure from MJP to deliver or increase administrative processes, rather than allowing them to focus on their reach as an organization. As one CSO respondent described, “the workload of the organization has really increased, because the MJP tools require a lot of effort, we have a lot of forms to fill out in relation to the case information at the paralegal level. There is also a database that is centralized at MJP level and that is very small…There are quarterly narrative reports.” CSOs’ ability to provide legal support also increased and improved because of MJP’s assistance, according to several respondents. Some CSO respondents reported that MJP-led collaboration with other CSOs strengthened their legal support services by providing opportunities for knowledge exchange, expansions of services into areas they had been unable to before due to lack of support, or opportunities for programming sustainability. Several CSO respondents noted in particular the importance of the Hakew Sabatili network, which started in Year 2 of the project as a framework and formally recognized cadre of nine CSOs for “advocacy for the rights of women and children and other deprived groups to access to justice” (MJP Year 2 Annual Report). Over the course of MJP, the platform Hakew Sabatili (Access to Justice in Bamanankan) completed relevant studies and “received specialized training in assisting women who face discrimination in land acquisition, inheritance, and employment, whilst continuing to advocate for the sustainability of services to GBV survivors” (MJP Year 6 Annual Report). Respondents lauded the benefits of the knowledge exchange opportunities with other CSOs provided through this collaboration network. As one CSO respondent highlighted, “there has been a lot of change with MJP because it has allowed us to collaborate with other organizations. That is why I USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 13 mentioned the Hakew Sabatili network. It was through MJP that we joined the Hakew Sabatili [access to justice] framework. So, it also allowed us to look elsewhere. That is, to exchange with other [CSOs]. There is a draft bill on the fight against slavery. So, we are in the process of seeing how to set up an advocacy commission. Since there is no political will, to bring the authorities at the highest level for the adoption of this law at the assembly level.” CSO respondents noted four main areas of impact and improvement as a result of MJP activities: impact on peoples’ live (18 responses, 4/8 respondents); improved organizational performance (16 responses, 4/8 respondents); improved organizational capacity (15 responses, 3/8 respondents); and deeper relationships with core stakeholders, such as local authorities, commissions, and the private sector (13 responses, 3/8 respondents). One CSO respondent believed that people’s lives were impacted by greater knowledge of their legal rights: “Absolutely, our main task is training because we help the communities to understand, to know the law and to seize what they could know to practice the law in an alternative way. I even titled one of our projects ‘community and law in the daily life,’ therefore we make a lot of sensitization and a lot of information because the project was based on the fact that people do not know the law and the state is not in a position to inform the people within the communities.” Another CSO respondent highlighted the importance of working on behalf victims of GBV, noting: “Within the framework of MJP, the forums that we organize are also used as a place for advocacy. Because, with MJP, we have understood that issues of [GBV] are not only issues of behavioral change, but also issues of decision-making. Questions of decisions on the part of elected officials, questions of decisions on the part of administrative authorities in charge of promoting and defending rights. From this point of view, the roundtables, the forums that we organize, and even the other much broader sessions that we have …make us go further towards the authorities such as the ministers, such as the chiefs of cabinets, and of the different ministries. The [MOJ], the Ministry for the Promotion of Women, so that, in any case, through their decisions, we can influence behaviors, provisions, and the mechanisms that are there. And [those] who are there to defend the rights of these people affected by [GBV] .” Regarding organizational performance and capacity, CSOs mentioned training and monitoring and evaluation templates that assisted in capacity-building. One CSO respondent noted: “…they advised us how to do monitoring and evaluation. During the workshop, they gave us all the techniques on how to do monitoring and evaluation. They gave us the procedures and cited the different types of monitoring and evaluation. Now, they told us how we should behave in the field to do monitoring and evaluation and have all the data. So, it was a great help. This is the first time I have done this as a teacher. The training that I received was well given. Without this training, I would have had difficulties in the field. This training in monitoring and evaluation has helped me in many ways.” A CDRA respondent highlighted the organization capacity and performance, stating: “During the period in which I started working with these organizations, they obviously expanded the range of services they offer to citizens in direct legal services. Today [these CSOs] offer communities a proximity service through the paralegals, and also with all the activities, especially mediation that they carry out.” A CSO respondent discussed the connection with organizational performance and capacity, highlighting this 14 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV was “[because] we have received a lot of training from MJP, and this capacity-building has allowed us to improve our way of doing things. Not only in terms of reporting, but also in terms of monitoring and evaluation. I noticed that the monitoring and evaluation tools were periodically filled out at each session. So, this has been an important contribution of MJP through the trainings.” Finally, CSOs noted the engagement with other core stakeholders and deepened relationships was a key impact, with one CSO respondent mentioning, “There are doctors, but there are also religious and traditional leaders and heads of families who come to the discussion groups [on legal rights]. This means that today, it is no longer meetings between women but mixed meetings where women and men participate in these information and awareness sessions.” Another CSO respondent stated, “Our services were quite responsive in putting in place a rapid response mechanism to the various problems that were reported. We used community radio and community relays extensively to disseminate information. We have established a partnership with elected officials and police stations to strengthen ties and to be able to act when needed.” Overall, many respondents believed there was significant success from MJP activities and/or working with MJP (15 responses, 4/8 respondents). One CSO respondent highlighted, “…really it is a great success for the project. That is to say that the project at this level has worked and it has reduced forced marriages and violence. So, we can say that it is a great success for the project, and thanks to this project today people know their rights. And if they are victims of such and such things they know where to go. So, this is a positive contribution of the project.” Despite these encouraging testimonials, all respondents also indicated there were at least some challenges to implementing MJP activities and/or working with MJP (34 responses, 8/8 respondents). One CSO respondent noted the administrative challenges of the project, stating “the work we did with MJP was in line with our skills and availability, but the hardest part was the feedback, meetings, and field visits from MJP. It's like after every activity we should have feedback or meetings. We spent more time in meetings than in activities.” Another respondent said, “…there were challenges, but the main challenge that was mentioned to us was the limited scope of the project. During the session of the authorities, the local councilor alerted us that in his district…there are many communes; the project is only interested in three of them, while in each of these communes the same phenomenon of human trafficking exists. And the challenge was to see how to extend the project to other areas not covered by the project and in which the same phenomenon exists.” Additionally, a large amount of respondent feedback centered on bottlenecks related to program implementation as a result of community tradition, traditional authorities, and citizen awareness. As one CDRA respondent noted, “…it is the weight of tradition and religion that has strongly limited the interventions because the issues…are very sensitive in some communities. These include women's rights and access to justice. It is complicated in these communities to intervene on issues that are taboo and sensitive.” A CSO respondent mentioned that the primary challenge from their perspective was “how to create the confidence of the people in justice. This is the important challenge because the people do not trust the justice system. They prefer to go to marabouts than to go to justice because women tell us that the poor are always wrong.” Respondents also indicated there were challenges external to MJP entirely that hindered implementation of programming: “the challenge is great because it is multifaceted; there is the financial challenge, there is the distance between the communities and the courts, there is USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 15 also the lack of knowledge of the law, and all this is included in a challenge that is the challenge of legal information and legal aid” (CSO respondent). Notwithstanding these areas for improvement, CSO respondents emphasized their strong desire to continue working with MJP and even suggested several areas where they would like to see MJP support expanded (22 responses, 6/8 respondents). These suggestions included expanding to additional communes, expanding programming to additional sectors (e.g., anti-corruption), and establishing a framework for engagement with court officers and the bar association. As described by one CSO respondent, “I think that we need to create a framework for dialogue with the organizations that benefit from the MJP and the court officers and with the bar association. Because we work most often with lawyers. Lawyers are taboo. People are suspicious of them. If we could create a constant framework of discussion with the bar, with the auxiliaries of justice, and even with the minister. If we could have this framework, it would make us really happy. We will be able to see continuously how we can improve our way of working. And better equipped in terms of information for access to justice.” Notably, no respondents indicated they would like to cease programming completely. Summary of findings on MJP support to CSOs: CSO respondents noted mediation and paralegal support, GESI integration, and various forms of TA and trainings as the primary intervention received from MJP CDRAs, with only a few mentioning grant management as a form of support received. CSO respondents believed MJP’s support was helpful not only for capacity-building and stakeholder coordination, but also for citizen engagement and impact through such as legal assistance, media campaigns, direct outreach, and training sessions for CSOs on sensitive topics like GBV and human trafficking. As a result of these activities, CSO respondents noted an increase in organizational reach, though some respondents believed this was paired with an increase in administrative workload. CSO respondents also mentioned an increase in the provision of legal support as a result of working with MJP, with many believing the platform Hakew Sabatili (Access to Justice in Bamanankan) was pivotal in this expansion. CSO respondents noted four main areas of impact and improvement because of MJP activities: impact on peoples’ lives, improved organizational performance, improved organizational capacity, and deeper relationships with core stakeholders, such as local authorities, commissions, and the private sector. Overall, CSO respondents believed MJP’s activities were successful and assisted in improved citizen’s access to legal knowledge and gender integration. However, many CSO respondents noted challenges in implementation, chiefly administrative workload, community tradition, traditional authorities, and citizen awareness. Despite these challenges, CSO respondents emphasized their strong desire to continue working with MJP if changes included expanding to additional communes, expanding programming to additional sectors (e.g., anti-corruption), and establishing a framework for engagement with court officers and the bar association. CDRA AND CSO-LED TRAINING TO COMMUNITY LEADERS ON DISPUTE RESOLUTION Due to the disconnect of the formal and informal justice system, many Malian communities have reduced awareness of legal rights and knowledge of pathways to access the formal justice system. MJP’s activities centered on building these linkages while bolstering improvements in the informal justice system through CDRA and CSO-led trainings for community leaders on mediation and dispute resolution. Community leaders reported receiving trainings in topics such as Malian justice and citizen rights, mediation and reconciliation, and local government engagement (74 responses, 13/16 respondents) and also support for GESI work, such as through trainings on sexual assault, human 16 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV trafficking, and intimate partner violence (43 responses, 10/16 respondents). One leader described the mediation and dispute resolution training and its benefits as follows: “I learned a lot from this training because every day you will see cases of disputes. Whether it is between neighbors, or between spouses. So, in these situations, you have to do what you can to resolve these disputes. I learned about the consequences of female circumcision, early marriage, youth migration, and human rights.” Another area of focus highlighted by leaders were GESI trainings provided by Women in Law and Development in Africa/Mali. As one respondent described: “…I learned a lot during this training. In this training we talked enough about women's rights and children's rights in our country. We also did a lot of exercises on these themes.” Respondents also indicated capacity-building for community leaders as a key factor within programming, with one CDRA respondent saying, “…they had to train community and religious leaders on GBV, human rights, and descent-based slavery. They have been trained on this issue, and for a while…they even created a network of community leaders and every two months, these leaders meet here at the office to discuss the problems they have encountered and the solutions they have found. This is to allow each other to refer to these approaches in order to manage cases on the spot.” Several leaders reported that disputes within their communities had decreased since they began receiving training from MJP, a pattern they attribute to their improved conflict resolution and mediation skills. As one community leader respondent explained, “The number of cases has decreased significantly. It has diminished so that the more you share your knowledge, with each incident, people will remember what you have done and said in your previous interventions.” Another community leader respondent reported that “…land conflicts have decreased a lot, thanks to everything we have learned through this project and these trainings. I have been able to manage many cases of land conflict, and it has been well￾limited here in this village.” Community leaders also indicated feeling confident as a result of this decrease to not only address cases as they arose moving forward but also acknowledging fewer cases occur generally, with one community leader respondent even stating, “In any case in my neighborhood I can confirm, the number has decreased significantly. We all live in perfect symbiosis, despite the fact that there are isolated cases, and even if these cases resurface, it is enough for [me and my advisors] to intervene so that it calms down right away.” Another community leader respondent mentioned, “There is hardly any conflict today, but in the past, we have had to deal with it, but in these moments, it is very rare. There are just maybe isolated cases. We all live in perfect social cohesion and symbiosis around the common interest which is peace, reconciliation, and social cohesion.” Furthermore, respondents believed that MJP’s activities improved mediation practices (34 responses) and had an impact on people’s lives (19 responses, 8/16 respondents). One community leader respondent noted, “I have learned that even beyond the training, even in the talking rooms, the one who listens is the one who understands best. It taught us to know how to deal with any problem.” Another respondent mentioned: “The training allowed me to understand many things. Because I knew nothing about domestic violence. Because for me, the woman was at the mercy of the man, and he does with her what he wants. The training sufficiently addressed the issue of women's rights. For example: women who are victims of rape always feel isolated and continue to feel ashamed among their peers. So, she is a woman who needs support and help from others to overcome this ordeal. I understood this thanks to the training and when faced with such a situation I will know the appropriate measures to take.” USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 17 Community leaders believed MJP’s training activities were successful at providing them with skills for mediation and reconciliation (14 responses, 9/16 respondents), with one respondent saying, “We have learned the strategies that allow us to intervene between two people in conflict. In case of failure following the intervention of traditional leaders, we know what attitude to adopt towards the parties in conflict.” However, a greater number of community leader responses noted challenges in implementing lessons learned from MJP trainings (24 responses, 12/16 respondents). One community leader respondent believed “the main challenge is to make the population aware of what ‘law’ and ‘justice’ are, their usefulness and their importance in society. What is the real role of justice to make them understand that it is there for everyone, not only for the wealthy. And that it is there to ensure justice between people, to fight against impunity, corruption, and injustice to name but a few.” Yet another respondent stated, “The first difficulty is ignorance, the fact that people do not know the legal procedures to access justice. This situation often results in people spending all their money without winning their case. Therefore, it is necessary to give information to the people so that they know all the penal procedures.” Respondents provided mixed feedback about improvements to justice service delivery overall directly resulting from MJP activities. Some respondents noted some improvements, such as one community leader respondent stating, “The training has had a great influence and in a positive way, we are able to handle cases easily. The training has awakened us a lot and we also knew that traditional justice had its place in the management of conflicts.” Still others believed there were no improvements, such as one respondent who highlighted, “We [believed] that you were going to bring about change, if you ask me if we can maintain this change that has not even come, I don't understand.” Overwhelmingly, respondents requested the continuation of programming (42 responses, 13/16 respondents), with adjustments such as greater women’s participation, the expansion of programming into adjacent sectors (e.g., orphaned children), continuing legal education rather than one-off trainings, greater involvement of youth, and greater civil society engagement. One community leader requested that “women…be more involved in the training. Because women need to be trained and understand all these concepts.” Another community leader requested continuity of trainings, saying “The lack of continuity of the trainings given the importance of the theme [shows that] these NGOs must organize several trainings. If it was up to us, [we’d] organize monthly trainings and [then] at each period…come to see and ask…what worked and what did not.” No leaders indicated a preference for ending MJP’s activities, which underscored the value they place on MJP’s support. Leaders also used the interviews as an opportunity to express their support for the traditional justice bill. One community leader respondent indicated: “This bill is a very good first step once again for all of Mali, especially for the North. Given that injustice and impunity are some of the factors of all these conflicts it will be a solution then to rectify a little the draw. And to make traditional justice become a part of formal justice. Informal justice can in no way compete with formal justice, on the contrary, it can only be a support. These community leaders are in contact with the grassroots and can make the transition and help the formal justice in these tasks. So, I would say that there are a lot of advantages to giving traditional justice more legitimacy, which is really very beneficial for all parties.” Another leader highlighted that “the advantage of [the bill] is that it will allow people to have confidence in the traditional leaders in the community, and they will say that if the traditional leaders decide a case, 18 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV it is as if the court had done it…This makes traditional leaders of paramount importance…So, it facilitates the work in the court.” Summary of findings on MJP support to Community Leaders: Community leaders reported receiving trainings in topics such as Malian justice and citizen rights, mediation and reconciliation, local government engagement, and support for GESI work through trainings on sexual assault, human trafficking, and intimate partner violence. Several leaders reported that disputes within their communities had decreased since they began receiving training from MJP, a pattern they attribute to their improved conflict resolution and mediation skills. Community leaders also believed that MJP’s activities led to improved mediation practices and had an impact on people’s lives, stating the increase of knowledge through trainings and awareness of GESI work was critical for providing services to their communities. Community leaders requested the continuation of programming with some adjustments, including increased women’s participation, expansion of programming scope into adjacent sectors (e.g., orphaned children), continuing legal education rather than one-off trainings, greater involvement of youth, and greater civil society engagement. ACCESS TO JUSTICE TRENDS FROM THE WJP’S ROL INDEX This section summarizes trends in access to quality justice outcomes using quantitative data from the WJP’s ROL Index. Drawing on data from more than 138,000 surveys and 4,200 legal experts and covering 139 countries, the ROL Index measures how the ROL is perceived and experienced around the world in a highly-standardized manner across countries. Of particular interest to this evaluation are the civil justice and criminal justice sub-indices. The civil justice sub-index captures the degree to which civil justice is accessible and affordable, free of discrimination, free of corruption, not subject to unreasonable delay, effectively enforced, and free of improper government influence. The criminal justice sub-index measures the degree to which the criminal justice system is timely and effective, impartial, free of corruption, free of improper government influence, and operates in adherence to due process and the rights of the accused.2 2 For further details on the ROL Index, visit: https://worldjusticeproject.org/rule-of-law-index/. Last Accessed October 19, 2022. USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 19 Figure 1: Mali's Ranking on the WJP's Civil Justice Index (of 139 countries) 60 70 80 90 100 110 120 130 140 2019 2020 2021 Overall civil justice ranking People can access and afford civil justice Civil justice is free of discrimination Civil justice is free of corruption Civil justice is free of improper government influence Civil justice is effectively enforced Figure 1 displays Mali’s country ranking for civil justice indicators from 2019, the earliest year for which data are available, to 2021. Of the 139 countries covered in the index, Mali’s overall ranking civil justice ranking fell from 106 in 2019 to 119 in 2021. In absolute terms, Mali’s civil justice score fell marginally, from 0.43 (out of a total score of one) to 0.42. In 2019, Mali ranked in the bottom quartile on three measures of criminal justice: free of corruption, free of improper government influence, and accessibility and affordability. By 2021, Mali’s rank on these three indicators had not improved. Indeed, across all five constituent indicators, Mali’s global ranking either fell or stagnated. Figure 2: Mali's Ranking on the WJP's Criminal Justice Index (of 139 countries) 60 70 80 90 100 110 120 130 140 2019 2020 2021 Overall criminal justice score Criminal adjudication system is timely and effective Criminal justice system is impartial Criminal justice system is free of corruption Criminal justice system is free of improper government influence Due process of the law and rights of the accused Figure 2 displays Mali’s country ranking for criminal justice indicators from 2019 to 2021. On the overall score for the criminal justice index, Mali’s rank fell from 121 to 126 between 2019 and 2021. In absolute terms, Mali’s score was stagnant, moving 0.28 (out of one) to 0.29. Overall, the trends reported in Figure 1 and Figure 2 indicate that the accessibility and quality of civil and criminal justice in Mali is among the lowest in the world and has remained for much of MJP’s tenure. But it is not possible to attribute these patterns to the MJP. Relative to the scale of the justice sector in Mali, MJP is a small program, and it is likely that national-level trends are driven by broader social, economic, and political forces outside the control of any one program. 20 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV EQ2: MJP’S CONTRIBUTION TO IMPROVED INSTITUTIONAL CAPACITIES OF KEY JUSTICE SECTOR INSTITUTIONS HUMAN RESOURCE MANAGEMENT SYSTEM AND TRAINING OF CENTRAL SERVICES STAFF AT MOJ, DNAJ, DNAJS Effective and efficient human resource management is essential to the functioning of the Malian justice system. Recognizing this importance, MJP prioritized improving human resource management systems within the MOJ, DNAJ, and DNAJS. Judicial and administrative staff from these justice institutions, which are considered to be the backbone of the Malian justice system, reported receiving extensive training from the MJP. As one respondent from the DNAJ described, “thanks to the support of the MJP, we were provided with training on how to evaluate performance within a court. This training session was extended to several members of our division. Following the training, we went on Missions to Koutiala and Kati. Furthermore, numerous trainings are being organized in different places.” Respondents further reported that these trainings followed a collaborative and progressive approach and were complemented by tailored TA and follow-up support. “The various MJP trainings have helped us a lot in this context because through these trainings, we have understood that we are short of staff. We have understood that there are problems of access to justice, and we have identified these problems, and this has allowed us to make recommendations, including the operationalization of certain jurisdictions” (HR ID 01). MJP launched an institutional diagnosis of the courts, and it was observed that there was a lack of personnel in contrast with the magnitude of the tasks, as well as a dysfunction in terms of the distribution of the workload between the aforementioned staff. This diagnosis allowed the MJP to organize training sessions in HRIS and Occupation and Skill Forecasting (Gestion Prévisionnelle des Emplois, des Effectifs et des Compétences) To train the executives of the Central Services and court officials in human resources management and court management, “MJP provided four training sessions on performance auditing, compliance auditing, risk mapping, as well as court review methodology and verifications,” reported one official from the DNAJ. Moreover, other training sessions were specifically made for the administrative staff of the court clerk for file management and archiving, with “the first module covered the functioning of the court clerks, the second one covered archiving, and the sessions took place in Mopti in March 2022” (DNAJ Official). According to our interviews, the most pronounced impact of MJP’s training was in the area of personnel management, and in particular, the efficient utilization of scarce staff. As one official explained, “despite our lack of staff, the little we had during the training enabled us to manage the few staff, the few human resources we have more effectively.” The training sessions also helped improve the distribution of tasks on the judicial staff and made it possible to launch internal tools to ensure proper collection of information, monitoring, and completion of tasks. “We have established a file, a set of questionnaires, … so the president, the prosecutor, the clerks, all the agents, are questioned through this same set of questions. And this allowed us to identify the number of staff in each jurisdiction, whether in terms of magistrates, clerks, or contractual support staff. This also allowed us to evaluate our staffing structure more effectively to understand how it functions or to propose a more beneficial organization," reported one official from the DNAJ. USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 21 According to respondents, the development of internal human resource tools under MJP’s direction significantly contributed to strengthening the managerial capacities of courts in beneficiary jurisdictions. It also encouraged the initiation of internal measures to ensure the continuity justice service delivery. As one official from the DNAJ explained, “the highlight of the training was forecasting [staff turnover]. Forecasting was very important, because we are in the field of justice. There is mobility almost every year, so magistrates or clerks who are going to retire, we have to identify them first, and prepare the succession in any case.” This respondent further explained that the forecasting tools introduced by MJP enabled them to streamline their recruitment process. MJPs human resources management activities significantly improved staff management within the DNAJ, MOJ, and DNAJS. Respondents highlighted in particular the value of a developing a human resources manual to establish personnel policies for judicial and non-judicial staff. As explained by one official: “there was no human resources procedure manual at the department level when we arrived … but with the support of MJP, we developed a tailor-made HR procedure manual.” Although appreciative of the improved HR management procedures introduced by MJP, respondents were also clear-eyed about the challenges they continue to face. Respondents noted that MJP cannot address the shortage of judicial personnel: "to be effective, we really more staff, not just better management of existing staff” reported one official. Another explained how "We often have difficulties in getting the necessary personnel to accomplish certain tasks. Often, we have the staff, but no qualified personnel. So, this management, honestly, it is not to say that it is bad, but in any case, it is struggling to be effective. We truly suffer from a staffing crisis, especially the few we have, which are not qualified. That's why we often have to bring in people who are not even from the structure, especially trainees." Limited staff also limited the degree to which MJP-led reforms were actually implemented. Reporting on the implementation of the HR management procedures, for example, an official reported that “normally, implementation of an updated procedures manual would not be a problem, but in the present case, we lack the staff to popularize the various courts to this updated manual and to train them so that they can put it into practice.” TECHNICAL ASSISTANCE ON THE DEVELOPMENT, IMPLEMENTATION, AND MONITORING OF ANNUAL PERFORMANCE PLANS MJP’s annual performance programming process centered around strategic planning. Describing the results of this process, an MJP official explained how staffers “now know how to develop a strategy to accomplish strategic planning, they now know how to define socio-economic objectives, service quality objectives, management efficiency objectives, and they also know how to define performance indicators that make it possible to evaluate their plans of activity; and as of 2018, the Ministry of Justice has been preparing its state budget, balancing it, and issuing the accountability reporting.” A member of the Justice Sector Planning Unit explained how his team uses the annual performance and how it has helped his organization with strategic planning: “while developing our annual performance program, we identify which activities contribute to fulfilling each of our objectives. And each year, we define, update, and improve the indicators. If we discover that an indicator is not performing well, we change it and redirect our activities accordingly.” Once MJP’s performance plans are developed, attention turns to implementation, the most important part of which is the budgeting process. “Every year, MJP organizes budget-focused capacity-building 22 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV workshops for directors and the managers,” explained an official from the Justice Sector Planning Unit. These budgetary training sessions covered topics such as budget drafting, implementation, risk mapping, monitoring, and reporting. In addition to these training sessions, MJP developed a set of evaluation and IT tools (web portal and internet) to assist with the annual performance program. As explained by one official, “the application allows us to do data collection on a regular and sequential basis; the software reduces the time of developing program budget revisions.” According to respondents, MJP’s annual performance plan procedures have become adopted by key justice sector institutions. As one official explained, “now all the actors have understood and assimilated the annual performance plan process, the different concepts, and the importance of context.” Officials further explained that the annual budget planning process is well-established: “currently, we can set a budget, and I think those in the Ministry have a meeting, all of the structures of the Ministry meet in order to plan the activities and the MJP usually plays part in that.” According to the officials we spoke with, these procedural improvements to annual budgeting trickled down to the court level, with "numerous justice sector leaders and the public prosecutor’s offices received training regarding setting and controlling their budgets.” Officials also praised the value of these processes for helping them avoid common risks in financial management: “the training allowed all to detect many risks that a financial manager can run into; especially at the level of contracting, budget preparation, and material accounting. We evaluated all these risks. We suggested solutions together in the presence of heads of the divisions; the training allowed us to recognize the risks that we can encounter in our respective areas of expertise.” Beyond improvements to annual planning, officials reported that MJP’s training in this sector improved their staff capacity: “the training allowed us to build the capacity of our agents, it also gave us some executive-level management skills” reported one official. Our interviews identified one core threat to the continued implementation of the annual performance plans post-MJP. In particular, officials reported that the process of consultation, transparency, and participatory decision-making accompanying the annual performance plans is likely to be met with resistance from high-level officials and managers within the DNAJ and DNAJS, who tend to prefer more hierarchical decision-making processes. Expressing skepticism about the impact of the annual performance plan, one official reported “there haven’t been any changes, because … the heads of the program who hold the power of authorization or those responsible for operational budgets are always given top priority. … as of now, the former budget administrators of the Directorate of Finance and Materials (DFM) and others with that authority continue to do the same thing despite us saying there is a need for a new management dialogue regarding the program budget.” In light of these concerns, it may be appropriate for MJP to start from already-technical training and consider transition periods with accompanying measures targeted at people and administrative dynamics (e.g., thematic training sessions on leadership, teamwork, and ethics). TRAINING AND MENTORSHIP OF DNAJ STAFF ON MONITORING COURT PERFORMANCE The DNAJ plays a central role in the administration of justice and the management of district and regional courts. Accordingly, MJP has focused its attention on improving the institutional capacity of this key justice sector institution. To this end, MJP provided training and mentorship services for executives USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 23 of the DNAJ. The main topics covered in these sessions was the budgetary process, court management, human resource management, risk mapping, the functioning of the court clerks, and archiving. According to respondents, several beneficial impacts resulted from these training and mentorship activities. First, in terms of capacity-building, MJP’s support made it possible for DNAJ managers to assess the staffing needs at lower-level courts and develop data collection tools to track these staffing needs. As one manager described, “we developed a database where we record the number of recorded cases and their nature, the number of staff, which of them are on leave, what they lack in terms of training, etc. These tools are all essential and they have facilitated the missions of the DNAJ.” Another manager reported “the tools used by the MJP for data collection regarding the justice system are beneficial.” MJP not only introduced tools to address key staffing challenges at lower-level courts but also provided support on learning how to apply these tools. As one manager explained, “MJP played a role that consisted of organizing training workshops so that the people in charge of developing these notices could have it in writing that everything in the notices must be applied; for instance, sorting in chronological order the judgments that have been rendered, the cases resumed, and the chronological order should be done every month because notices arrive monthly.” This organizational capacity-building approach instituted by MJP has helped alleviate some major obstacles regarding short-staffing. Moreover, thanks to the planning and evaluation tools initiated with the assistance of the MJP, respondents reported significant improvement in the work of the DNAJ. In the context of the courts, “the training sessions in court management and program budget taught us how to manage a court in terms of staff and the contents of budgetary headings for better control and monitoring,” reported one DNAJ manager. As part of its support to the DNAJ, MJP also supported the implementation of several renovations and provided improved equipment. As one manager explained, “the courts were repainted by MJP, the courtroom was redone, and the gate was transformed into a gatehouse; so, a lot of things were done.” The support of MJP also allowed for more renovations or construction of rooms such as an evidence room safeguarding court documents. Reflecting on the improved reception and orientation offices, one DNAJ official reported that “the MJP played a significant part in reinvigorating the reception and orientation office.” Respondents also praised the MJP’s contribution to improved information technology, which they hope will lead to more effective archiving. Describing his hope for continued support in this area improvements, a DNAJ manager explained how “with electronic archiving, we could store the entire case history of each court … we have to move from paper to electronic archiving; it allows for longevity of the archives, and it protects them from numerous factors, such as bad weather.” Officials also cited electronic archiving’s ability to facilitate faster access to information “through just a click, without having to ruffle through papers.” As for areas for improvement in MJP's activities, respondents complained of the lack of flexibility in its programming plans: “well, what I see as a weakness is the lack of flexibility. When MJP made the decision to renovate some jurisdictions, I think the contact had already been awarded and everything. So, when I came in and took a look at the specifics, I noticed that some things were prioritized over others; and I wanted to go back and review what we really needed to classify as priority, but unfortunately, that was 24 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV not possible,” explained one respondent. Another noted that “the lack of flexibility in implementing actions makes it difficult to adapt or change actions once the process has begun.” EQ3: MJP’S CONTRIBUTION TO IMPROVED FRAMEWORKS AND PROCESSES FOR ACCESSING JUSTICE REVISIONS TO THE PENAL CODE AND CRIMINAL PROCEDURE CODE The Penal Code and the Code of Criminal Procedure in Mali are characterized by their long-standing history in the Malian context, which is further impacted by external factors, such as conflict and violence, and political and institutional developments. MJP assisted in the review and revision of this penal code, which aligned with a strong desire by legal practitioners and human rights associations to enable Mali to adapt its penal legislation to the current context. MJP hoped that assistance in the penal code review would “…have a decisive impact on criminal justice and…contribute to strengthening defense rights, decongesting currently overcrowded prisons, speeding up justice proceedings, [and] strengthening investigation resources for Specialized Units against terrorism and corruption which [would] be completely autonomous courts” (MJP Year 4 Annual Report). The revised Penal Code and Criminal Procedure Code was the subject of several discussions before the Permanent Legislative Commission (CPL). However, despite the best efforts of MJP, the revised Penal Code and Criminal Procedure Code has not yet been adopted. According to MJP’s Year 6 Annual Report, the primary barriers to its adoption were the socio-political crisis in Mali which lead to the appointment of a new Minister of Justice, as well as competing priorities within the Legislator. The revision of the penal code and the criminal procedure represents a major challenge for Malian Justice in light of the issues at stake in the security context of the country and the needs for protection and respect of human rights. Within this context, a Ministry of Justice respondent explained the background for MJP’s efforts to reform the Penal code: “it was not only necessary, but it was fundamental because these texts are outdated, so it was necessary to update them, especially given there are many forms of criminality that have emerged while most of our texts date back to more than twenty years practically.” The respondent continued, “It is importantly necessary to revise it, it is even indispensable in order to both bring ourselves up to date in relation to the treatment of new forms of crime, and to bring ourselves in line with everything that is being done in the world, particularly in our sub-regional area.” Prior to MJP, progress towards revising the criminal procedure code had stagnated. As one MOJ respondent noted, “… these are texts that have been under revision for several years. I remember that for example for the code of criminal procedure, it must be five years now. But since the MJP came to take charge of it, we have made extraordinary progress, and we have just finished with the national validation workshop.” MJP collaborated with the Ministry of Justice to establish the permanent legislative commission for penal code and criminal procedure code revisions. One MOJ official involved in these revisions explained, “all these revisions are done with the expert advice of MJP … there is a permanent framework that is set up within the, and the MJP is a strategic partner of this commission. Moreover, all the revised texts have been with the contribution and the experts of the MJP.” USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 25 MJP completed revisions in three stages: i) the drafting of a diagnostic report by an MJP consultant, ii) enrichment of these texts by the various actors during exchange workshops, and iii) validation through validation workshops attended by diverse groups of stakeholders. These steps were technically and financially supported by MJP. Respondents noted were carried out in an inclusive manner with broad participation of different judicial actors, partners, and civil society (5 responses). As one MOJ respondent described, “what is remarkable is that the participation was not only from people of the Ministry of Justice, judges, and the Office of the Prosecutor General (OPG); on the contrary, it was broadly inclusive. There were people from associations, international organizations, the United Nations Office on Drugs and Crime (UNODC), the United Nations Multidimensional Integrated Stabilization Mission in Mali (MINUSMA), civil society actors, the International Committee of the Red Cross (ICRC), and so on. So, everyone comes and debates in all directions and that gives more value to our texts, because they are more inclusive in their elaboration.” Still another MOJ respondent believed the revision process “has gone very well because we, ourselves, have contributed with practical elements, and because we are the ones in the field who are best positioned to identify the weaknesses of our texts. And together with the MJP expert, we make proposals [and find solutions].” In terms of the substance of the revisions, an official involved in the process explained that the revisions are about “a better reorganization of justice,” as well as “major innovations that will allow us to be more in line with modern standards through the elimination of certain practices, and the fostering of other better practices.” In regard to the nature, the definition, and the penalty of offenses, the revised penal code will allow prosecutors “to do their work better and to arrive at facts based in technical and scientific evidence, so it will revolutionize investigations in certain areas, such as terrorism and transnational organized crime or economic and financial offenses.” As to the formal framework for revisions, “the great innovation in Mali, for us, is not only the conformity with the outside, but also the elimination of the courts of assizes. Very soon, the courts of assizes will be eradicated. The supreme courts will be the only ones to deal with crimes, and this is based on the principle of respecting the two￾tier proceedings, which was not respected before.” The revised Penal Code proposes important changes for the courts. As one respondent explained, “there will be criminal divisions at the level of the Supreme Court. The courts of the communes of Bamako will judge the assassins, whereas they were judged only at the level of the court of appeal and by the court of assizes. The principle of two-tier proceedings will also be respected. There will also be a procedure called expedited proceedings, in which the claim is filled today and the judgment is made the same day. The maters of amnesty are also treated” in the revisions. Respondents expect these changes will lead to positive change in various judicial areas, particularly in the area of the rights of the accused. “Once adopted, the Code of Criminal Procedure will allow for the strict respect of the right of defense. The rights of defense will be sufficiently protected. There will be a guarantee of the presumption of innocence, less pretrial police custody, and less preventive detention. This will allow for a rigorous emphasis on the rules of criminal procedure because the rules of procedure will be clearly transcribed” explained one official. Generally, respondents requested a continuation of MJP activities in penal code revisions. However, this was not without a request for a modification of program activities or acknowledgement of challenges with implementation, with one respondent stating, “there is the challenge of information and awareness; this is a huge challenge. Malians are not very impregnated with justice. I think that there is an immense effort to be made at this level. We can make the best texts in the world but if they are not explained, it 26 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV is the same thing. So today, we are taking up the challenge of reform. We have to make the reforms accepted. We have to make the reforms understood. For the next five years, we must focus on awareness, information, and communication about justice.” Other respondents expressed frustration at the slow pace of revisions and disappointment at being invited to participate only in the validation workshop rather than the entire revision process. As one respondent mentioned, “…the great weakness is the slowness in the revision of code of criminal procedure and the penal code. The law on judicial reorganization took a long time and it is only now that I hear that we are invited for national validation meetings…this is a major weakness that the MJP must address.” Conclusion MJP assisted in the review and revision of this penal code, which aligned with a strong desire by legal practitioners and human rights associations to enable Mali to adapt its penal legislation to the current context. MJP completed revisions in three stages: i) the drafting of a diagnostic report by an MJP consultant, ii) enrichment of these texts by the various actors during exchange workshops, and iii) validation through validation workshops attended by diverse groups of stakeholders. These steps were technically and financially supported by MJP. Respondents noted were carried out in an inclusive manner with broad participation of different judicial actors, partners, and civil society. Respondents believed these revisions allowed for important changes to legal text that would allow for innovation and a formal framework, improving rigor for criminal procedure. Overall, respondents requested a continuation of programming with a greater consideration from MJP of the barriers to reforms, such as “awareness, information, and communication about justice.” REVISIONS OF CENTRAL SERVICES UNDER THE JUSTICE SECTOR ORIENTATION AND PROGRAMMING ACT The Malian authorities worked to transform the justice sector as expressed in the Act on the Orientation and Programming of Justice (2020-2024) and with the assistance of MJP. This Act aims to adapt and strengthen central services as well as to enhance citizens' access to justice, thereby emphasizing its importance and justification as an objective for contributing to the improvement of the justice sector. The reform of central justice services is part of the broader Orientation and Programming Act, an initiative that is led by the GOM but supported by MJP. The objective of these reforms is to promote good governance in the justice sector. “Through the work of the permanent legislative commission, the Ministry of Justice set up a working group, in charge of the revision of the texts relating to the creation of the various central services, ... and all the work of this working group there, was also taken in charge by MJP,” described one respondent involved in the process. An important milestone achieved with the support of MJP is that of the Programming and Orientation Act, which was approved and implemented. According to respondents, the implementation of this Act has led to significant changes in the status of central services. As one government respondent described, “the revision of the Acts is among the greatest advances in law in the last 60 years because there were many offenses that were ignored by the previous codes.” Another government respondent explained how “the first institution to benefit from the Orientation Act is the DNAJ. With the Act of Orientation, its visible impact is the improvement of the working conditions of the personnel of DNAJ. So, the judges have better wages today, which contributes to the development of human resources.” Another government respondent praised the Orientation Act’s impact on human resource capacity, reporting USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 27 that “this Orientation Act has allowed for an annual recruitment of staff that would not otherwise be possible.” Government respondents appreciated MJP’s strategy of working through government agencies, rather than around them. As one government respondent explained, “we have always appreciated MJP’s indirect support delivered through the agencies, because projects that try to provide direct support never succeed … in a country where we do not always have the means to implement our policies, you must put money into the administration, support it in a specific area, because it is the administration that knows its priorities.” Respondents noted challenges remain in awareness, human resources, and related infrastructure to ensure revisions can be implemented. As one government respondent noted, “…the lack of personnel is the main cause of the slowness in the treatment of judicial files, especially in the interior of the country.” If these challenges were addressed, respondents believed a continuation of activities would lead to successful programming. A government respondent summarized, “…the challenge is education; people need to be trained to understand the laws that govern them…The problem of access is not only geographical, but it is intellectual and financial…The cost of justice must be reduced by developing legal aid to allow those who cannot afford it to access it….When you act on the financial, geographical, and intellectual situation, you have everything.” Conclusion An important milestone achieved with the support of MJP is that of the Programming and Orientation Act, which was approved and implemented. According to respondents, the implementation of this Act has led to significant changes in the status of central services. Respondents appreciated MJP’s collaboration with government agencies for support in revisions of central services. However, respondents believe challenges remain in awareness, human resources, and related infrastructure to ensure revisions can be implemented. If these challenges were addressed, respondents believed a continuation of activities would lead to successful programming. LEGAL AID BILL ADVOCACY According to respondents, the main activity of MJP as it relates to the legal aid bill was to convene a working group composed of advocates from legal aid groups, DNAJS officials, and corporate representatives. Members of this working group participated in workshops to discuss the current legal aid bill, brainstorm areas for improvement, and identify pathways for passing, funding, and implementing a revised legal aid bill. During these workshops, a consensus emerged that the existing legal aid bill had major shortcomings. As one CSO official described, “when you take the law on legal aid in its current format, it does not take into account many important aspects. Even the eligibility criteria are not clearly defined. There is a real need to revise the bill” (CSO official). Another key limitation identified with the existing legal aid bill, which was passed in 2001, was that it did not provide for a sustainable source of funding. “Apart from the state budget and what MJP is doing, I don't know of any other source of income in relation to legal aid” (Corporate official). Moreover, the process for determining who is eligible for legal aid is “so opaque that it is not even possible” for most Malians to access (CSO official). Recognizing these deficiencies, members of the legal aid working group drafted revised legal aid text for discussion, feedback, and validation. The revised text and associated “vision” for how legal aid could 28 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV work had two major components. The first was a more transparent mechanisms for determining eligibility, wherein eligibility will be determined by a national agency for legal aid on the basis of referrals from local courts and a clear and transparent eligibility determination process (CSO official). The second was an initiative to persuade corporations to contribute to legal aid as part of their Corporate Social Responsibility (CSR) efforts. In terms of concrete progress towards passing a revised legal aid bill, respondents reported that this effort is still at a very early stage. “We held a workshop to validate the draft text,” described one respondent, “but as for the validation, it is now at the level of the Council of Ministers. As far as I am concerned, I am not aware of the validation of this text. I am not aware of the adoption of the draft law by the Council of Ministers” (CSO official). “The process is still ongoing” reported another respondent, “this remains a very preliminary project that is very far away” (CSO official). More broadly, “the discussions have gone a long way but the actual implementation has not progressed at all” (Corporate official). After the conferences, “a consensus was reached regarding the definition of legal aid, the structure of the managing agency, the committee members and the board members, which would include members of the private sector” (Ibid). But the details on the modality and type of funding remain to be worked out, along with the mobilization and pilot phase of the project (Ibid). Asked whether there were any weaknesses to the legal aid bill revision process, a respondent stated that “the weakness lies in the will of the authorities. Because it takes political will to implement this text. … this should not be just another text. We make texts to implement them. As beautiful as a text is, if it is not implemented, it is useless” (CSO official). Respondents also communicated some skepticism towards the viability of using CSR funds to underwrite legal aid. According to respondents, the “pitch” made to corporate CSR representatives was that contributing to legal aid financing will contribute to the state-led resolution of disputes and conflicts within communities, contributing to a broader climate of peace so that “the companies will have fewer constraints to carry out their activities” (Ibid). Whether this logic withstands the scrutiny attending corporate CSR funding decisions remains an open question. Although progress towards actually passing and implementing a revised legal aid bill has been limited, this is not unexpected given the often slow pace of legislative reform efforts and their dependence on broader political dynamics outside the control of MJP. This hard reality was not lost on respondents, who communicated overwhelmingly positive feedback about MJP’s support while simultaneously recognizing the lack of hard progress in terms of ‘results’. As one respondent explained, “even if the text is not adopted, we can at least say that things are in very good way ... as this text goes through the DNAJS, where MJP is a privileged partner … there has been a vast amount of revision to legal texts and MJP is really a partner of size at the side of the Ministry of Justice” (DNAJS official). Another respondent praised the constructive, methodical, and participatory manner in which MJP went about its work: “In my opinion, the MJP is a responsible project and everything we had to do with it was a well-developed methodological approach” (Corporate official). The respondent continued, “at every stage of the business, they ask us and they included us … I have greatly appreciated MJP's actions, which are carried out in a collegial, inclusive and participatory manner” (Ibid). Members of civil society were especially appreciative of MJP because MJP hosted for in which they could engage with authorities on an equal footing: “[Participation at MJP’s workshops] went well because it was a space of free and open discussion where one does not cut off because you are of the civil society when a magistrate or a USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 29 general prosecutor wants to speak. Ordinary people participated in the commission on equal terms … so I found this workshop to be really positive and appreciated the opportunity to share [our CSOs] modest vision of the march of justice in Mali” (CSO official). Respondents also appreciated that sustainability and local ownership post-MJP was taken into account from the outset. Describing discussions from the workshops, a respondent reported that leveraging CSR funds was a direct response to the questions of sustainability (CSO official). Another respondent described the smooth and well-planned process by which ownership of MJP-led legal aid bill activities would be transferred to local actors: “when the MJP left, we set up an interim administration. The conclusions of the validation workshop of the law on legal aid would be carried forward by a [as yet to be created] national agency for legal aid. We have also discussed how to accelerate the activities in relation to the modalities of contribution of funds. It was within the framework of the CSR of the companies so that there is a voluntary contribution of the private actors to achieve sustainability” (CSO official). In terms of areas for improvement and/or consideration in future programming, respondents highlighted two core areas: localization and prioritization. From a localization perspective, respondents humbly requested that future legal aid consultants be Malian nationals, highlighting their local knowledge and the benefits to local capacity-building (CSO official, DNAJS Official, Corporate Official). In terms of prioritization, respondents questioned the decision to prioritize revising the legal aid bill rather than improving implementation of existing legal aid bill (DNAJS official, CSO official). TRADITIONAL JUSTICE BILL ADVOCACY MJP worked with officials from the DNAJS to draft a traditional justice bill that would define and clarify the role of traditional authorities in the justice sector. Respondents report that the process by which MJP drafted this bill was collaborative, thoroughly researched, and well-grounded in the Malian context. Respondents appreciated MJP’s efforts to host workshops with core stakeholders and sponsor trips to field sites to understand the realities of traditional justice rural Mali (MINUSMA official, DNAJS officials 2 and 4). Describing one of these visits, a respondent reported that “in the three regions where we conducted our field visits, we were able bring traditional justice actors and formal sector actors around the same table to lead the discussions. We talked about the similarities and complementarities of their work under the common goal regulating and resolving conflict. We discussed how each set of actors could contribute to the role of calming the social climate” (DNAJS official). The result of this collaborative process was a draft traditional justice bill. In 2021, MJP brought this bill to an inter-ministerial committee composed of general secretaries from different ministries for review before transmission to the Council of Ministers. To the disappointment of MJP, the inter-ministerial committee determined that the bill would grant certain adjudicatory powers to traditional authorities and therefore would require a constitutional amendment to enact the bill. This determination effectively killed the prospects for the enactment of a revised traditional justice bill. Respondents were similarly disappointed by this determination. They highlighted the potential benefits that likely would have resulted from the enactment of the Bill, including a stronger and more legitimate role for traditional authorities in dispute resolution (DNAJS officials 2, 4 and 5), clearer jurisdictional boundaries between the formal and traditional sectors (CSO official), and increased access to justice (DNAJS officials 2 and 4). The adoption of this bill would have “democratized” the distribution of justice 30 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV delivery in Mali, reported one respondent. Another pointed out that "in many localities, there are already cadis and traditional authorities who render justice in places where state judges do not have access” (DNAJS official). This respondent further argued that “the adoption of this text would have strengthened and enhanced the role played by these traditional authorities in the distribution of justice,” which in turn would have “improved the functioning of the Malian justice system because the justice system is overloaded” (Ibid). Respondents communicated only two critiques about the role of MJP in promoting the traditional justice bill. The first is about the advisability of trying to “formalize what is inherently informal” (Penal Code Respondent 11). The respondent continued: “I am going to disappoint all the partners here, I am not optimistic about this law. We cannot formalize what is informal. The traditional justice sector gives results because it is informal; why would we want to formalize it? Formalization is to please a few. It is not for the general interest of Malians” (Ibid). The second critique of MJP’s work on the traditional justice bill relates to positionality, and whether foreign actors should be driving legislative agendas in Mali. According to this respondent, the role of the Malian state in the design of legal texts “should rightfully be a matter of national sovereignty” (DNAJS official). It would be regrettable “if foreign actors come to dictate what we should do and how we should do it,” because “the law is really a matter of national sovereignty” (Ibid). “The Malian state must be more involved in the process of reviewing and adopting legal texts rather than abandoning them to the partners” (Ibid). Notwithstanding these perspectives, the process by which MJP advocated for reform of the traditional justice bill was widely praised by respondents and could serve as a model for future programs to emulate. MJP conducted thorough background research, convened workshops in which diverse groups of stakeholders could voice their views and opinions, and drove progress forward by drafting revised legal text and iteratively soliciting feedback from concerned stakeholders. As one respondent summarized “MJP is a good thing because it allowed us to move forward. The [DNAJS] had ambitions that it did not have the means to implement. But MJP helped to open these various projects and drive progress forward, particularly in the area of revising legal texts” (DNAJS official). EQ4: MJP’S IMPACT ON CORRUPTION This section attempts to analyze MJP’s impact on corruption in the Sikasso region. Unlike the rest of the report, the analysis relies primarily on quantitative public opinion data on corruption, rather than on original qualitative data collected in this study.3 MJP’s anti-corruption activities were concentrated in the Sikasso region and organized around three primary objectives: (i) reduced corruption along the Sikasso trade corridors; (ii) implementing a reporting system for roadside corruption; and (iii) education and public awareness to empower citizens to combat corruption. To achieve these objectives, MJP conducted a wide variety of activities. First, MJP worked to strengthen the operational performance of grantees operating Citizen Advocacy Offices and the advocacy and liaison activities of the Sikasso Platform for the Free Movement of Persons and Goods (Platform). MJP also supported Watch Cells in the Sikasso trade corridor, which carried out more than 700 interventions to defend travelers against corruption and harassment in 2019 alone. The system for reporting roadside corruption was less 3 Following guidance from USAID/Mali, we prioritized EQs 1, 2, and 3 for the qualitative interviews, given the finite scope and budget for this evaluation. USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 31 developed. According to MJP’s Year 6 Annual Report, the KAKOFO mobile reporting app was developed through a hackathon during Year 4 of the program. The app reached more than 2,000 users in its first few months, but this auspicious start was followed by a long period of inactivity.4 In Year 6, MJP re-launched the app and re-registered 150 long-lost users; from these users, a total of 2 complaints were received by CAOs. Alongside these activities, MJP provided anti-corruption investigative training to journalists, supported grantees to deploy mystery shoppers at checkpoints, organized anti-corruption caravans and marches in focal communities, distributed anti-corruption leaflets and stickers, published anti-corruption messages on billboards, and provided grants to leading Sikasso radio stations to sensitize citizens to corruption and report on anti-corruption issues. To evaluate the impact of these activities on corruption (EQ4), the ET drew on data from the Afrobarometer, a pan-Africa public opinion center that has conducted eight nationally-representative public opinion surveys in Mali since 2001. Drawing on data from the six most recent survey rounds - 2005, 2008, 2012, 2015, 2018, and 2020 - we focus on two primary outcomes: whether respondents report paying a bribe to police in the past year and whether respondents believe “most” or “all” of police are corrupt. We focus on police corruption because the police are the primary perpetrators of corruption along trade corridors in Mali. Police also serve as the “face of the state” and directly or indirectly interact with citizens more frequently than any other state institution. In addition, police corruption is measured in a consistent and comparable manner across all six of the most recent survey rounds. Estimating the impact of a regional-level intervention is not straightforward. A simple pre versus post comparison could be misleading because factors other than MJP could (and likely do) account for changes in corruption overtime, such as changing economic conditions, political trends, or technological innovations. A slightly more sophisticated approach would be to compare corruption trends in Sikasso relative to the rest of Mali, to examine whether corruption declines in Sikasso relative to the rest of Mali. The advantage of this approach is it accounts for country-level, time-varying confounders that affect Sikasso and the rest of Mali, such as country-level economic or political trends. The disadvantage is that economic or political trends specific to Sikasso could still lead to a unique, Sikasso-specific corruption trend, confounding the comparison to the rest of Mali. Figure 3 and Figure 4 show the results of this analysis. In Sikasso, the proportion of residents who paid a bribe to the police in the past year increased from 2015 to 2017 before declining in 2020; meanwhile, police bribery increased in the rest of Mali through 2020 (Figure 3). A similar pattern is observed for the proportion of residents who believe that “all” or “most” police are corrupt (Figure 4). These trends cannot be interpreted as causal evidence of impact, however, because trends for both outcomes follow a unique trend in Sikasso relative to the rest of Mali prior to MJP, which indicates that Sikasso was on its own trajectory even before MJP. 4 According to MJP’s Year 6 Annual Report, two core challenges to the success of the application were i) delays in responding to complaints, and ii) poor management of the CAOs by the partner grantee. 32 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV Figure 3: Proportion of Respondents Who Paid Bribes (Afrobarometer 2005-2020) Figure 4: Proportion of Respondents View Police as Corrupt (Afrobarometer 2005-2020) Another approach to estimating the impact of MJP on corruption outcomes is to analyze corruption trends by region to assess whether trends in any one region match Sikasso prior to the intervention. If trends in any one region match that of Sikasso for several periods prior to MJP, then it may be reasonable to assume these parallel trends would continue in the absence of MJP, and hence, any divergence after the introduction of MJP is attributable to the impact of MJP. This analysis is reported in Figure 5 and Figure 6. Unfortunately, every region of Mali appears to be on a unique trajectory in terms of levels of police corruption throughout the panel period, preventing effective comparison. USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 33 Figure 5: Proportion of Respondents Who Paid Bribes by Region Figure 6: Proportion of Respondents View Police as Corrupt by Region In Annex IV, we confirm the uniqueness of Sikasso’s corruption trends through a two-way fixed effects, difference-in-differences style analysis using pre-intervention treatment effect estimates as a “placebo” test to demonstrate that Sikasso followed a unique corruption trajectory even before the start of MJP. In a final effort to identify a valid comparison group that would allow us to make inferences about the impact of MJP on corruption, we employ the synthetic control method, following Abadie, Diamond, and Hainemueller (2010). As Abadie (2021) describes, the “synthetic control method is based on the idea that, when the units of observation are a small number of aggregate entities [such as states or regions], a 34 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV combination of unaffected units often provides a more appropriate comparison than any single unaffected unit alone” (Abadie 2020, 393). To this end, the synthetic control method uses a weighted combination of non-MJP comparison regions to create a “synthetic” Sikasso that matches Sikasso’s trends prior to MJP. If a “Synthetic Sikasso” whose trends match those in Sikasso prior to MJP can be created, then corruption outcomes in Synthetic Sikasso can be compared to corruption outcomes in actual Sikasso post-MJP and interpreted as evidence of MJP’s impact, under the assumption that parallel trends between Sikasso and synthetic Sikasso would have continued in the absence of MJP. Figure 7 and Figure 8 shows the results of the synthetic control analysis. The results indicate that the synthetic control algorithm was unable to identify a weighted combination of control (i.e., non-Sikasso) districts that closely match Sikasso’s pre-MJP trend.5 As a result, even the synthetic control approach to estimating MJP’s impact is not viable.6 Ultimately, we are unable to reliably estimate MJP’s impact on corruption in the Sikasso region. While the proportion of Sikasso residents who paid bribes to the police increased by five percentage points between 2015 and 2017 before declining by three percentage points in 2020, we are unable to attribute these trends to MJP rather than broader social, political, or economic dynamics. The proportion of residents who believe that “most” or “all” of police are corrupt increased by five percentage points between 2015 and 2017 before declining by 20 percentage points by 2020, but we are also unable to attribute this trend to MJP. In the future, credibly estimating the impact of USAID programming will require incorporating evaluation design considerations into the implementation of Activities from the outset. Experimental and quasi-experimental approaches in which implementation is concentrated in certain districts, communities, or locations and withheld from other “comparison” locations are especially promising avenues to credibly evaluate the impact of programs. This approach requires onboarding evaluators early-on in the activity lifecycle. 5 For police bribery, synthetic Sikasso is a weighted combination of Koulikoro (.62 weight), Ségou (.21 weight) , and Gao (.17 weight). For perceptions of police corruption, Sikasso is a weighted combination of Kayes (.41 weight) and Mopti (.59 weight). 6 There are several reasons why the synthetic control approach may not have been a viable method in this instance. First, corruption outcomes may have been too volatile relative to the impact of the MJP program. As Abadie (2021) describes, “small effects will be indistinguishable from other shocks to the outcome of the affected unit, especially if the outcome variable of interest is highly volatile” (Abadie 2021, 409). Another key limitation is the limited number of comparison regions at just eight, including Bamako, making it difficult for the algorithm to find a suitable combination of comparison regions to serve as synthetic Sikasso. USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 35 Figure 7: Proportion of Respondents Who Paid Bribes in Sikasso vs. Synthetic Sikasso Figure 8: Proportion of Respondents View Police as Corrupt in Sikasso vs. Synthetic Sikasso EQ5: SUSTAINABILITY To what extent will MJP’s beneficial impacts persist after the project’s end in December 2022? It is impossible to answer this question definitively without follow-up data, as our analysis is necessarily 36 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV based on respondents’ conjectures about the future, as well as our considered reading of the reform context. For MJP’s grant and training support for CSOs, officials were emphatic about their desire to continue doing the work that MJP has supported them in doing. But the prospect of discontinued support loomed large in their responses. As one official stated frankly, “we really hope that there will be another phase of MJP. An MJP 1, 2, and 3” (CSO Official). When pressed on what his organization would do if MJP did not continue, the official conceded, “certain activities will assuredly stop. For example, the issues of legal aid; for that, we must have resources” (Ibid). Officials from several other organizations were adamant that they would manage to continue the activities they began under MJP even after MJP ends, though they remained uncertain about where the funding for these activities would come from. For MJP’s training of community leaders in mediation and dispute resolution, leaders consistently reported that the improvements achieved under MJP will be at risk when MJP’s support ends. As one leader explained, “To be able to maintain these changes, we need to retrain people, as I said at the beginning, even if it's two or three times a year. You need to come and give us refresher courses. This will allow us to recall what you have taught us.” Other leaders highlighted the lack of consistency in training events as a key barrier to sustainable change: “the lack of continuity of the trainings has been a real barrier… these NGOs must organize several trainings. If it was up to us to organize monthly trainings and that at each period they come to see and ask us what worked and what did not.” Another leader expressed the view that “as long as we continue with the training, the information, the sensitization, we will be able to maintain these behavioral changes in the long term and prevent many cases of conflicts.” Officials also expressed concern that MJP’s contribution to improved capacity at key justice sector institutions will be reversed when MJP ends. These concerns centered around two key issues. The first concern was that advances in staff training will be lost due to staff turnover and retirement: “… as you know, there are many staff who have retired and there are new ones who come in who must be trained. But now, if MJP has to leave, I’m not sure there’s any other project that will step in” (DNAJ official). The second concern was that trained staff will simply forget what they’ve learned if training isn’t continuous. For both of these reasons, officials argued that trainings and workshops should be made permanent and conducted on a regular basis. Asked whether he expects the Ministry of Justice to takeover MJP’s capacity building activities, a DNAJ official replied flatly, “they haven't done it before. And even if they have, it's not enough.” RECOMMENDATIONS We organize our recommendations according to three programming categories: programs designed to increase access to justice, programs designed to build the institutional capacities of justice sector institutions, and programs designed to promote legislative reform in the justice sector. All recommendations are aimed at USAID/Mali and its implementing partners. Recommendations for increasing access to justice This evaluation finds encouraging albeit provisional evidence that MJP’s support for CSOs helped CSOs expand their services within the communities they serve. Based on this finding, we recommend that USAID/Mali and its IPs continue to invest in training and technical assistance for CSOs working in the USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 37 justice sector. While doing so, implementers should take care to minimize the administrative burdens involved so that CSOs can focus their scarce resources on service provision, as this was a common complaint among CSO respondents in this study. To increase the sustainability of these programs, USAID/Mali and its IPs should consider training CSOs in fundraising, and/or referring them to other donors and INGOs interested in their services, so they may continue to fund them. This evaluation also finds encouraging albeit provisional evidence that MJP’s support for training community leaders in mediation and dispute resolution reduced disputes in beneficiary communities. Based on these findings, we recommend that USAID/Mali and its IPs continue to train community leaders on mediation and dispute resolution. While doing so, implementers should prioritize regular, repeated trainings over “one-off” training events, even if it means reaching fewer communities. Alongside these efforts, USAID/Mali and its IPs should seek to generate more definitive evidence regarding the effectiveness of CSO support and community leader training. To this end, USAID/Mali and its IPs should design future programming in such a way as to allow for rigorous evaluation. This includes collecting before vs. after information on justice outcomes in beneficiary communities and employing experimental or quasi-experimental, comparison-group evaluation designs to generate credible evidence about the effectiveness of these programs in Mali. Recommendations for building the institutional capacities of justice sector institutions This evaluation finds that MOJ, DNAJ, and DNAJS officials who received training from MJP on human resources management, annual performance plans, and monitoring of lower-level courts report improved internal staff capacities, better management, and stronger internal oversight. Officials also appreciated the close working relationships developed by MJP over the course of the project. Based on these findings, we recommend that USAID/Mali and its international partners continue to support these key justice sector institutions with tailored technical assistance in the future. Recommendations for promoting legislative reforms in the justice sector According to officials interviewed for this evaluation, MJP successfully drove progress forward on key legislative initiatives pertaining to the justice sector. Moreover, officials expressed high praise for the collaborative and inclusive manner in which MJP went about revising legal texts and building support for legislative reform. Based on these findings, we recommend that USAID/Mali and its IPs continue to advocate for important legal reforms following MJP’s model of inclusive, participatory, and determined advocacy. 38 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV ANNEX I: REFERENCES Abadie, Alberto. "Using synthetic controls: Feasibility, data requirements, and methodological aspects." Journal of Economic Literature 59.2 (2021): 391-425. Abadie, Alberto, Alexis Diamond, and Jens Hainmueller. "Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program." Journal of the American statistical Association 105.490 (2010): 493-505. USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 39 ANNEX II: DESK REVIEW MJP aims to support the GOM in its provision of more efficient, effective, and competent justice sector institutions that are accountable and responsive to the needs of Malians. MJP’s activities, in turn, are anticipated to contribute to USAID’s Development Objective 1 (2015-2020 Country Development Cooperation Strategy) of improving the public trust in government, particularly through the consistent and fair application of the ROL. 7 Activities broadly fall under one of three components: C1: Enhance the Quality and Efficiency of Services Provided by Justice Sector Institutions. C2: Increase Immediate Access to Justice. C3: Increase Access to Justice and Reduce Corruption along the Sikasso-Koury and Sikasso-Heremakono Trade Corridors. Access to Justice (EQ1) MJP recognizes that improvement of GOM justice sector institutions (see EQ2 below), while critical, are insufficient to result in increased access to justice for average citizens. Component 2 of MJP aims to address barriers in affordability and access through low-cost legal assistance; strengthening of civil society organizations, legal aid groups and law clinics; and advocacy campaigns to increase knowledge and demand for legal services. Except for advocacy campaigns, which lagged behind Project targets until recently, MJP has met or exceeded most output metrics under C2. However, several key initiatives to connect the formal and informal sectors are currently stalled, including revisions to the Legal Aid Bill and passing of the Traditional Justice Bill (see EQ3 below). In addition, external metrics indicate uneven progress towards increased access to justice over the period of MJP. The HiiL Justice Needs and Satisfaction survey implemented in 2014 and again in 2018 (3 years into MJP) Malians are increasingly taking action to resolve their legal problems through an informal or formal dispute resolution mechanism (from 70 to 87 percent). However, the proportion of Malians reporting the problem was completely or partially resolved stayed roughly the same (dropping from 48 to 47 percent). The WJP’s ROL Civil Justice Index8 from 2019 – 2021 indicates improvements in access to alternative dispute resolution mechanisms (0.48 to 0.51) and affordability (0.40 to 0.44) from 2019 to 2021. Notwithstanding these improvements, the overall civil justice index remained largely unchanged (from 0.43 in 2019 to 0.42 in 2021) due to setbacks in areas of enforcement, freedom from discrimination, and undue government influence.9 Institutional capacity of key justice sector institutions (EQ2) & Framework and processes of access to justice (EQ3) Through Component 1, MJP aimed to improve the performance of central services and courts to effectively administer justice and increase public confidence in the formal justice system. This has included improving the management of human, material, and financial resources through tools and trainings; the review and revisions of legal text; training on data collection and performance monitoring of courts by central services; case management improvements; training of law students, lawyers, 7 The MJP Activity Monitoring and Evaluation Plan (2020). Project Document. 8 https://worldjusticeproject.org/rule-of-law-index/country/2019/Mali/Civil%20Justice/ 9 The Criminal Justice Index was also largely unaffected overall, increasing from 0.28 in 2019 to 0.29 in 2021. 40 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV prosecutors, court personnel, and police; and Quick Impact Projects aimed at improving citizen trust and confidence in the justice system. MJP’s records indicate significant progress on several indicators linked to Component 1 of its program, including averaging 7.2 public policies per year that were introduced, adopted, repealed, changed, or implemented consistent with citizen input over years 2-6 of the program (MJP7 Indicator). Annual reports also note an average of 192 legal institutions and associations and 800 justice service personnel per year supported by MJP assistance and trainings (MJP2 and MJP3 Indicators).10 However, the effect of Component 1 on trust in government is less clear. According to Afrobarometer data, the percentage of citizens who report having no or little trust in the courts (a Component 1 indicator for MJP) rose from 56 percent to 64 percent from 2014 to 2020. Courts remain the least trusted institution covered in the survey. In addition, the number of citizens reporting that most or all judges or magistrates are corrupt rose from 56 to 62 percent during that same period.11 While the recent decline in public confidence in justice sector institutions is notable, it is impossible to determine whether the decline would have been worse in the absence of MJP. However, the evaluation presents an opportunity to provide a nuanced understanding to the potential reasons underlying the change. This may include acknowledgement that activities under C1, including re-drafting of laws and procedures and training of critical staff, is likely to be insufficient in achieving large scale change in public perception due to Mali’s complex operating environment.12 Corruption (EQ4) Finally, Component 3 of MJP was aimed at reducing corruption on transport corridors through targeted legal services, advocacy, and public information campaigns.13 MJP’s Activity Monitoring and Evaluation Plan frames this anti-corruption component as contributing to USAID/Mali’s development objective of inclusive agriculture growth and funding primarily came from other sectors including agriculture (Feed the Future), economic growth, and anti-corruption according to discussion the ET held with USAID in late-2021. USAID noted that it is an open question whether anti-corruption should sit under mechanisms like MJP and this question, coupled with the external funding streams, means this component is unlikely to continue in future iterations. It was therefore agreed with USAID that C3 is not a primary focus of this evaluation. Sustainability (EQ5) Sustainability of programming like MJP is a key question. Program documentation notes that lack of funding remains a key issue to the future implementation of not only MJP’s grants to CSOs but also larger national initiatives. For example, Project documentation notes lack of funding was one of the reasons for the failure of the 2001 Legal Aid Law. The Project is therefore taking several steps to support program sustainability, from securing pledges by the Malian private sector institutions to co-fund legal assistance through corporate social responsibility, to training grant supported CSOs in proper reporting and accountability to secure funding from other donors in the future. These external funding 10 We exclude Y1 from these calculations as targets and expectations in Y1 are not always defined. 11 https://afrobarometer.org/publications/resume-des-resultats-enquete-afrobarometer-round-8-au-mali-2020-0 12 US Institute of Peace. (2011). Criminal Justice Reform in Post-Conflict States – A guide for Practitioners. 13 The MJP RFTOP. (2015). Public Documents. USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 41 streams are seen as critical at this time given the low ability to pay for legal services of most of the population and the lack of sufficient budgetary support to the Ministry of Justice.14 While these steps are productive, it is unclear to what extent they will be enough to sustain MJP’s improvements overtime. This evaluation presents an opportunity to revisit committed and potential funding streams, speaking to both CSOs, donors and private sector companies to assess steps taken and potential remaining barriers for support. 14 The MJP Y6 (2021) Report. 42 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV ANNEX III: KEY INFORMANT INTERVIEWS CSO KII PROTOCOL Demographic Information Date: Start time: End time: Interviewer name: Primary notetaker name: Respondent name: Respondent Representing Institution: Respondent Position/Title: Sex of respondent: INTRODUCTION 1. Please describe your role at [CSO] and how long you have been on working there. 2. Please describe the support MJP has provided your organization. (Prompt – Financial, technical, advisory, mentorship, coordination, etc.) ACCESS TO JUSTICE (EQ1) 3. Does your organization provide direct legal services to the community? If yes, what type of services are provided (e.g., paralegal support, mediation, referrals). Can you provide a specific example? a) Probe: Has the organization’s caseload increased, decreased, or stayed the same during the period of MJP? And why? b) Probe: Has the type of customers you are servicing changed or stayed the same? And why? c) Probe: Has the way you provide legal services changed or stayed the same? And why? Can you provide specific examples? i. Probe: Are you conducting or receiving more referrals now? From or to whom are the referrals going? 4. Does your organization provide community-wide awareness sessions, trainings, forums, or radio broadcasts? What is the content of these activities and who do they target? a) Probe: Were you providing these sessions prior to MJP? i. If yes, has your organization’s administration of these sessions, the format, the content, or the participants changed in anyway? How has MJP influenced these changes (e.g., through training, mentorship, financial support)? ii. In not, what are the reasons for conducting sessions at this time and how had MJP informed the format, content, or participants of these sessions? 5. Does your organization provide individually targeted training or mentorship (e.g., to community leaders)? What is the content covered in these trainings? a) Probe: Were you providing this support to individuals prior to MJP? USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 43 i. If yes, has your organization’s training or mentorship changed in anyway? How has MJP influenced these changes (e.g., through training, mentorship, financial support)? ii. In not, what are the reasons for training and mentorship at this time and how had MJP informed the format, content, or participants of these sessions? 6. Are there any other activities your organization conducts which are aimed at increasing access to justice in Mali that we have not already discussed? If yes, please describe. a) Prompt: Did MJP enable your organization to conduct these activities? How so? 7. MJP aimed to support CSOs on administrative matters including grants management, and monitoring and evaluation. Has your organization made any changes to improve its reporting and financial practices during the period of MJP? Why or why not? Please provide a specific example. 8. We have discussed many ways MJP aimed to support organizations like yours. However, were there any challenges or limitations to MJP approach or support provided? How were those challenges addressed or are they ongoing? [If not mentioned in initial response] Can you provide specific examples? (EQ5) SUSTAINABILITY 9. Do you expect achievements under MJP to be sustained over time? Why or why not? Is there evidence we can point to? (e.g., sustainability plan, committed funding) 10. When MJP ends will you continue with the activities you started under the program, expand them, or will you stop some activities? Which activities and why? 11. Has MJP affected how you will work with donors in the future? Why or why not? (Prompt – technical support on record keeping, reporting, etc.). 12. What do you consider to be the main challenge in improving access to justice for the average Malian citizen over the next five years? 13. Where do you believe future programming should focus? WRAP-UP 14. Is there anything else you would like to mention or discuss about the MJP, or access to justice in Mali? 44 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV CDRAS KII PROTOCOL Demographic Information Date: Start time: End time: Interviewer name: Primary notetaker name: Respondent name: Respondent Representing Institution: MJP Respondent Position/Title: CDRA Respondent Location/CSO Assignment: Sex of respondent: INTRODUCTION 1. Please describe your role as a CDRA. What are main responsibilities and who do you work with? (How long have you been in this role)? (EQ1) ACCESS TO JUSTICE 2. Have the CSOs you work with increased or altered the direct legal services they provide to the community (e.g., paralegal support, mediation, referrals) during MJP? Can you provide an example of this change? If not, why has this not changed? a) Probe: What is your role (if any) in supporting the provision of direct legal services? (e.g., working with paralegals, coordinating referrals.) 3. Have the CSOs you work with started, increased the number of, or altered their provision community-wide awareness sessions, trainings, forums, or radio broadcasts during MJP? Can you provide an example of this change? If not, why has this not changed? a) Probe: What is your role (if any) in supporting these community wide events? 4. Do the CSOs you work with provide individually targeted training or mentorship (e.g., to community leaders)? Has this type of support changed during MJP? Why or why not? a) Probe: We understand you also provide direct training and mentorship to communities, particularly traditional authorities. Can you describe this support? (Prompt- What is the main goal, how/when/where is support provided) What have been the main achievements of this activity? What have been the main challenges? 5. Do you provide CSOs with any administrative support, including reporting, monitoring, and evaluation? If so, can you provide an example of support you have provided recently in this area? 6. Is there any other support you, or MJP (i.e., Bamako office) provide to CSOs or the community which is aimed at increasing access to justice in Mali that we have not already discussed? If yes, please describe. USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 45 7. Were there any challenges or limitations to the support you could provide? How were those challenges addressed or are they ongoing? [If not mentioned in initial response] Can you provide specific examples? (EQ5) SUSTAINABILITY 8. Do you expect achievements under MJP, either within the CSOs your work with or within the community, to be sustained over time? Why or why not? Is there evidence we can point to? 9. When MJP ends do you believe the CSOs you work with will continue with the activities they started under MJP, expand them, or will they stop some activities? Why? 10. Has MJP affected how these CSOs will work with, or be able to receive funding from, other donors in the future? Why or why not? (Prompt – technical support on record keeping, reporting, etc.). 11. What do you consider to be the main challenge in improving access to justice for the average Malian citizen over the next five years? 12. Where do you believe future programming should focus? WRAP-UP 13. Is there anything else you would like to mention or discuss about the Mali Justice Project, or access to justice in Mali? 46 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV COMMUNITY LEADER KII PROTOCOL Demographic Information Date: Start time: End time: Interviewer name: Primary notetaker name: Respondent name: Respondent Position/Title: Sex of respondent: INTRODUCTION 1. Please describe your role in providing access to dispute resolution or justice in your community. 2. I understand that you received training and/or mentorship from [name of CSO or CDRA]. Can you tell me about your experience during this training and mentorship? a) What topics did the training and mentorship cover? (Prompt – family resolution, land disputes, GBV, referrals to the formal justice system, etc.) b) What were some of the most important things that you learned through the training/mentorship? c) Was there anything about the training or mentorship that could have been improved? (EQ1) ACCESS TO JUSTICE 3. How has your work with [name of CSO or CDRA] influenced the way you manage conflicts and disputes in your community? a) Probe: specific examples of changes the leader has enacted as a result of the training/mentorship b) Probe: Has their caseload increase, decreased, or stayed the same? And why? 4. Cases involving land disputes are often brought to traditional leaders like yourself. Can you tell me about a recent land dispute that came to your attention? How did you handle this case? a) Probe: How did your mentorship/training influence how you handled the case? b) Probe: Did you refer the case to the police or courts? Why or why not? 5. Another type of dispute that traditional leaders often handle are domestic disputes such as divorce, separation, inheritance, or regarding the care of a child. Can you tell me about a recent domestic dispute that came to your attention? How did you handle this case? a) Probe: How did your mentorship/training influence how you handled the case? b) Probe: Did you refer the case to the police or courts? Why or why not? 6. Sometimes, criminal cases such as theft or instances of violence may be brought to traditional leaders like yourself. Can you tell me about a recent criminal case that came to your attention? How did you handle this case? a) Probe: How did your mentorship/training influence how you handled the case? USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 47 b) Probe: Did you refer the case to the police or courts? Why or why not? 7. Basic human rights principles assert that everyone, regardless of their sex, age, ethnicity, religion, political views, etc., has the right to have his or her case heard and determined fairly. Has MJP training/mentorship affected how you or other traditional authorities understand and apply these principles? Can you tell me about a recent case (adjudicated by you or someone else) where you saw a change in how this was applied? 8. Have you trained other members of your community in the skills you learned from [name of CSO or CDRA providing support]? If yes, who did you train, what topics did you cover, and what outcomes came from that training? (EQ3) FRAMEWORK AND PROCESSES OF JUSTICE 9. A Traditional Justice Bill has been proposed to the Malian Government. What have you heard about this Bill? 10. Let me tell you about the Bill. The Bill has two proposals. First, it wants to formerly recognize the role of traditional leaders like yourself in mediating family disputes like intermarriage conflicts, inheritance disputes, child custody, and other civil disputes between neighbors and community members on land, property, etc. Second, it wants to strengthen coordination and the referral network between the traditional justice system and formal institutions like the courts and police. What do you think about this proposed Bill? a) What do you see as its potential benefits? b) What do you see as its potential costs or consequences? (EQ5) SUSTAINABILITY 11. Speaking about the changes you enacted since you received your training/mentorship, such as the [insert changes mentioned by respondent earlier in this interview] – do you expect you will be able to sustain these changes in the future? a) Probe: What are some of the barriers to sustaining these improvements in the future? 12. What do you consider to be the main challenge in improving access to justice for the average Malian citizen over the next five years? 13. Where do you believe future programming should focus? WRAP-UP 14. Is there anything else you would like to mention or discuss about the training/mentorship you received, or about access to justice in Mali in general? 48 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV CAPACITY-BUILDING - MOJ, DNAJ, DNAJS, INFJ AND COURT STAFF KII PROTOCOL Demographic Information Date: Start time: End time: Interviewer name: Primary notetaker name: Respondent name: Respondent Representing Institution: Respondent Position/Title: Sex of respondent: INTRODUCTION 1. In what capacity have you interacted with the Mali Justice Project? ([If none] Can you tell me about any of the activities MJP has supported?] (EQ2) BUILDING CAPACITY OF KEY JUSTICE SECTOR INSTITUTIONS 2. [Strengthening of Human Resource Systems Sub-activity:] a) Were you part of the development of the HRIS? This may include study visits (to Belgium), validation workshops, working group participation, revision of job descriptions, and development of training modules. i. Probe: How do you feel the development process went? What were the strengths and weaknesses? Did you learn anything through the process that you can apply to other components of your work? Please provide specific examples. b) Were you trained on HRIS and the planning of staff, jobs, and skills (Gestion Prévisionnelle des Emplois, des Effectifs et des Compétences – GPEEC)? If yes, how does HRIS/GPEEC: i. Clarify roles and responsibilities within your institution? ii. Ensure adequate and appropriate staffing of key roles? iii. Allow you to proactively anticipate/forecast staffing needs for proactive resourcing? iv. Reach any other objectives? c) Overall, how effective have HRIS and GPEEC been at addressing your agencies’ HR needs? i. Probe: Please provide specific examples of challenges that HRIS/GPEEC’s addresses. 3. [Annual Performance Plans Sub-activity:] a) Tell us about the support you received from MJP regarding your annual performance plans? b) Can you provide specific examples of changes that were enacted because of this support? i. Prompt – How did MJP’s training influence the process for developing an annual performance plan? USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 49 ii. Prompt – How did MJP’s training influence the budgeting and allocation of resources between programs/activities? iii. Prompt – How did MJP’s training influence the monitoring of progress towards plan implementation? 4. [Court Monitoring Visits Sub-activity:] a) What is the objective of the Court Monitoring Missions? (e.g., provide reliable information on human resources/staffing, evaluate workload and performance, recommend improvements) b) How did support provided by MJP (tools, on-the-job training, etc.) help reach those objectives? Can you provide some specific examples? i. Prompt – Were you conducting/receiving monitoring missions prior to MJP? If so, how are they different now? Please provide specific examples. c) What did you learn/what recommendations resulted from your most recent monitoring mission? Are you aware of any action that has been taken on those recommendations? Please provide specific examples. 5. Are you aware of any other ways that MJP has improved the systems and operational capacity at your institution? What were those improvements? 6. Are there any weaknesses to MJP approach or support provided? How could those weaknesses be addressed? (If respondent states there are no weaknesses, ask if there were any challenges, they are aware of.) 7. To what extent did the public (the average Malian) benefit from MJP’s support for [respondent’s Agency/Ministry]? What evidence do you see that the public is aware of these improvements or has benefited from them? (e.g., increased demand for services, higher calls for accountability/transparency) 8. [If not already addressed] Whether the public is aware or not, has the work of MJP altered how the average Malian interacts with justice sector institutions or experiences justice service delivery? Why or why not? What components of MJP are the most effective at increasing access to quality justice services for Malians? Which are the least? (EQ5) SUSTAINABILITY 9. Do you expect achievements under MJP to be sustained over time? Why or why not? Is there evidence we can point to? 10. What do you consider to be the main challenge in improving access to justice for the average Malian citizen over the next five years? 11. Where do you believe future programming should focus? WRAP-UP 12. Is there anything else you would like to mention or discuss about the Mali Justice Project, or access to justice in Mali? 50 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV REVISIONS OF LEGAL TEXT – MEMBERS OF WORKING GROUPS Demographic Information Date: Start time: End time: Interviewer name: Primary notetaker name: Respondent name: Respondent Representing Institution: Respondent Position/Title: Sex of respondent: INTRODUCTION 1. MJP has supported the development, review, and revision of several legal codes and texts. Can you tell me which ones you are familiar with and what you understand to be the objective of the revisions (i.e., why were revisions (or development) necessary)? (Examples include: Penal Code, the Code of Criminal Procedure, Justice Sector Orientation and Programming 2020-2024, Legal Aid Bill, Traditional Justice Bill). 2. What has been your role in the revision/review of texts? How were you recruited for this role? 3. How was your experience participating in the review? (Did you find the review constructive? What did you learn through the process? Did you feel like your voice was heard?) (EQ3) FRAMEWORK AND PROCESSES OF ACCESS TO JUSTICE [For those participating in revisions to the Penal Code]: 1. During your collaboration with MJP, what progress has been made towards revising the Penal Code and getting these revisions approved? 2. If approved: a) What impacts will the revised Penal Code have on [insert reference to justice sector frameworks/processes targeted by the revised Penal Code]. b) How do you expect these impacts will be experienced by the average Malian? 3. If not yet approved: a) Once approved, what impacts will the revised Penal Code have on [insert reference to justice sector frameworks/processes targeted by the revisions to the Penal Code]. b) How do you expect these impacts will be experienced by the average Malian? 4. Are there any weaknesses to the revised Penal Code, and how can those be addressed? [For those participating in revisions to the Code of Criminal Procedure:] USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 51 1. During your collaboration with MJP, what progress has been made towards revising the Code of Criminal Procedure and getting these revisions approved? 2. If approved: a) What impacts will the revised Code of Criminal Procedure have on defense rights, expediating justice proceedings, and strengthening investigation resources? b) How do you expect these impacts will be experienced by the average Malian? 3. If not yet approved: a) Once approved, what impacts will the revised Code of Criminal Procedure have on defense rights, expediating justice proceedings, and strengthening investigation resources? b) How do you expect these impacts will be experienced by the average Malian? 4. Are there any weaknesses to the revised Code of Criminal Procedure, and how can those be addressed? [For those participating in review of central texts under the Justice Sector Orientation and Programming Act 2020-2024:] 1. During your collaboration with MJP, what progress has been made towards revising the constitutive texts for DNAJ and other central services (DNAJS, DNAPES, INFJ, CPS, DFM, ISJ) under the Justice Sector Orientation and Programming Act 2020-2024 and getting these revisions approved? 2. What impacts will the revised Justice Sector Orientation and Programming 2020-2024 have on confirming the role, autonomy, and resources of DNAJ and other central services? 3. How do you expect these impacts will be experienced by the average Malian? 4. Are there any weaknesses to the revised Justice Sector Orientation and Programming Act 2020- 2024 and the text review process, and how can those be addressed? [For those participating in revisions to the Legal Aid Bill:] 1. During your collaboration with MJP, what progress has been made towards revising and passing the Legal Aid Bill? 2. If approved: a. What impacts will the Legal Aid Bill have on legal aid in Mali? b. How do you expect these impacts will be experienced by Malians without the means to defend themselves? 3. If not yet approved: a. Once approved, what impacts will the Legal Aid Bill have on legal Aid? b. How do you expect these impacts will be experienced by Malians without the means to defend themselves? 4. Are there any weaknesses to the Legal Aid Bill, and how can those be addressed? 52 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV a. [If not mentioned] Is there adequate funding for legal aid in Mali? What evidence either supports or refutes the idea of adequate resources? [For those participating in revisions to the Traditional Justice Bill:] 1. During your collaboration with MJP, what progress has been made towards developing and passing the Traditional Justice Bill? 2. Once approved, what impacts will the Traditional Justice Bill have on the role that traditional authorities have in managing and resolving disputes? 3. How do you expect these impacts will be experienced by the average Malian? 4. Are there any weaknesses to the Traditional Justice Bill, and how can those be addressed? (EQ5) SUSTAINABILITY 1. What do you consider to be the main challenge in improving access to justice for the average Malian citizen over the next five years? 2. Where do you believe future programming should focus? WRAP-UP 1. Is there anything else you would like to mention or discuss about the Mali Justice Project, or access to justice in Mali? USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 53 ANNEX IV: RESPONDENT LIST For the analysis of MJP’s support to CSOs, the ET randomly sampled eight individuals from the following organizations: ● ARCV ● AJCAD ● APDF ● APROFEM ● ENDA Mali ● GARDL ● PROMODEF ● WILDAF For the analysis of CDRA and CSO support to community leaders, the ET randomly sampled ten community leaders from the following districts: ● Dioila ● Komoguel 1(Mopti) ● Koulikoro-Ba ● Koulikoro/Kayo ● Koulikoro/Massala ● Toguel(Mopti) ● Toya (Alafia) For the analysis of human resource management and training of central services staff at MoJ, DNAJ, and DNAJS, the ET randomly sampled 6 individuals from the following organizations: ● DNAJ ● DNAPES ● MJP ● TGI de Koutiala ● TGI de Koulikoro ● TGI de Ségou For the analysis of TA on the development, implementation, and monitoring of annual performance plans, the ET interviewed five individuals from the following organizations: ● Checchi Consulting ● CPS ● DFM ● DNAJ ● DNAPES For the analysis of training and mentorship of DNAJ staff on monitoring court performance, the ET randomly sampled 5 individuals from the following organizations: 54 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV ● DNAJ ● INFJ ● TGI de Bamako For the analysis of revisions to the penal code and criminal procedure code, the ET randomly sampled 5 individuals from the following organizations: ● Bamako Children's Court ● Bar of Mali ● National Directorate of Judicial Affairs and Seal ● National Human Rights Commission ● National Integrated Programme to Combat Transnational Crime ● Specialized Judicial Center For the analysis of revisions of central services under the justice sector orientation and programming act, the ET randomly sampled 4 individuals from the following organizations: ● Ministry Delegate to the Prime Minister in charge of Institutional Reforms ● Ministry of Crafts, Tourism, and the Hotel Industry ● Ministry of Justice (MoJ) ● Ministry of Justice and Human Rights For the analysis of revisions of legal aid bill advocacy, the ET randomly sampled 3 individuals from the Malian Bar Association. For the analysis of revisions of traditional justice bill advocacy, the ET randomly sampled 4 individuals from the following organizations: ● DNAJS ● MINUSMA USAID.GOV DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT | 55 ANNEX V: TWO-WAY FIXED EFFECTS ANALYSIS OF CORRUPTION OUTCOMES This section presents the results of a two-way fixed effects analysis of the impact of MJP’s anti￾corruption activities in Sikasso, controlling for survey round and district. As a placebo test to test whether pre-intervention trends in corruption outcomes in Sikasso match those of other regions, we include Sikasso lags for the pre-intervention periods – 2008, 2012, and 2015, in our estimation equation. The results, reported in Figure 9 and Figure 10 indicate the Sikasso’s controlled-differenced differed from the rest of Mali even before the start of MJP, and that this difference was variable year-to-year. It is therefore not feasible to credibly estimate the impact of MJP’s anti-corruption work in Sikasso. Figure 9: Sikasso vs. rest of Mali over time (police bribe, two-way fixed effects analysis) 56 | DRG-LER II TASKING N056 MALI JUSTICE PERFORMANCE EVALUATION: FINAL REPORT USAID.GOV Figure 10: Sikasso vs. rest of Mali (police corruption, two-way fixed effects analysis) U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT 1300 Pennsylvania Avenue, NW Washington, D.C. 20523