This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the 
University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency 
for International Development or the United States Government. 
DRG IMPACT EVALUATION OF 
USAID/MALI JUSTICE PROGRAM 
MEMORANDUM ON INITIAL RESULTS 
FROM ENDLINE SURVEY
March 2019 
Prepared under Contract No.: GS-10F-0033M / AID-OAA-M-13-00013, Tasking N032
DRG LEARNING, EVALUATION, AND 
RESEARCH ACTIVITY
DRG IMPACT EVALUATION OF 
USAID/MALI JUSTICE PROGRAM 
MEMORANDUM ON INITIAL RESULTS 
FROM ENDLINE SURVEY
(MARCH 2019)
Prepared under Contract No.: GS-10F-0033M / AID-OAA-M-13-00013, Tasking N032
Submitted to: 
Brandy Witthoft, USAID COR 
Submitted by: 
Matthew S. Winters 
Jeff Conroy-Krutz
Contractor: 
NORC at the University of Chicago 
4350 East West Highway, 8th Floor 
Bethesda, MD 20814 
Attention: Renée Hendley 
Tel: 301- 634-9489; E-mail: Hendley-Renee@norc.org
DISCLAIMER 
The authors’ views expressed in this publication do not necessarily reflect the views of the 
United States Agency for International Development or the United States Government.
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | i 
TABLE OF CONTENTS 
I. SUMMARY 1
II. IMPACT EVALUATION IMPLEMENTATION 3
III. ANALYSIS 7
(1) Awareness In Lower-Population Villages 13
(2) Awareness In Villages With CSO Reports On Meetings 16
(3) Treatment Effects For Knowledge Index 17
IV. DISCUSSION 18
(1) Limited Reach Of The Information Sessions 19
(2) Limited Time Between Baseline, Campaign, And Endline 21
(3) Non-Adherence To Randomization Schedule 21
(4) Disjuncture Between Description of Meetings Used By Csos And Question 
Wording 22
(5) Improper Survey Data Collection 22
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | ii
LIST OF FIGURES 
Figure 1. Reported Attendance at First Theme Training Session. From 44 CSO reports. 
Depending on CSO, the first training session might either have been one of two 
for the first theme or else it was a combined double session for the theme. ........ 7
Figure 2. Distribution of 2009 Population in Study Villages. Population data from 2009 
census. ..................................................................................................................................... 14
LIST OF TABLES
Table 1. Number of Villages in the Study by CSO Partner. ......................................................... 4
Table 2. Planned and Actual Deployment by CSO Partner. Dates for meeting conduct 
reflect incomplete reporting from the CSOs. ................................................................. 6
Table 3. Awareness of public information meeting. From endline household survey. “To 
the best of your knowledge, have there been any public information meetings in 
this village in the last six weeks?” ....................................................................................... 9
Table 4. Awareness of a public information meeting about the judicial sector. From 
endline household survey; asked only among those who said that there had been 
a public information meeting (i.e., “yes” answers in Table 3). “Can you tell me if 
any of these public information meetings addressed the topic of the judicial 
system or inheritance rights in Mali?”................................................................................ 9
Table 5. Attendance at public information meetings about the judicial sector. From 
endline household survey; asked only among those who said that there had been 
a public information meeting and that it was about the judicial sector (i.e., “yes” 
answers in Table 3 and then in Table 4). “Did you attend any of these meetings 
about the judicial system or inheritance rights in Mali?” ............................................ 10
Table 6. Awareness of public information meeting among village elites. From endline elite 
survey. “To the best of your knowledge, have there been any public information 
meetings in this village in the last six weeks?”............................................................... 11
Table 7. Awareness of public information meeting among village heads. From endline elite 
survey. “To the best of your knowledge, have there been any public information 
meetings in this village in the last six weeks?”............................................................... 11
Table 8. Awareness of public information meeting among women’s group leaders. From 
endline elite survey. “To the best of your knowledge, have there been any public 
information meetings in this village in the last six weeks?”........................................ 12
Table 9. Awareness of public information meeting among youth group leaders. From 
endline elite survey. “To the best of your knowledge, have there been any public 
information meetings in this village in the last six weeks?”........................................ 12
Table 10. Awareness of a public information meeting about the judicial sector among 
village elites. From endline elite survey; asked only among those who said that 
there had been a public information meeting (i.e., “yes” answers in Table 6). 
“Can you tell me if any of these public information meetings addressed the topic 
of the judicial system or inheritance rights in Mali?” ................................................... 13
Table 11. Awareness of meetings in villages with 2009 population below 1,000. From 
endline surveys. Population numbers come from 2009 census data and therefore 
do not precisely reflect current population levels. ...................................................... 15
Table 12. Awareness of meetings in villages with population below 1,000 (2009/2018 data). 
From endline surveys. Population numbers come from 2009 census data plus 
reports from village heads for villages where 2009 census data did not exist. ..... 16
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | iii
Table 13. Awareness of meetings in treatment villages for which we have CSO reports. 
From endline surveys. .......................................................................................................... 16
Table 14. Questions Included in Knowledge Index. ....................................................................... 17
Table 15. Treatment Effects for Knowledge Index. Difference-in-means test for 12-item 
knowledge index across respondents from treatment and control villages. ......... 18
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 1 
I. SUMMARY 
As part of the DRG Learning, Evaluation, and Research (DRG-LER) Activity, USAID 
requested NORC to design and implement an impact evaluation of USAID/Mali’s Justice 
Sector Program. 
The purpose of the USAID Justice Program in Mali is to support the Government of Mali’s 
goal of more efficient, effective, and competent Malian justice sector institutions that are 
accountable and responsive to the needs of Malians. The project aims to work with the 
formal and informal justice sectors by improving the institutional capacities of key justice 
sector institutions such as the National Direction of Justice Administration (DNAJ). 
As with other current USAID/Mali programming, a central objective of the program is to 
increase citizen confidence in state institutions, which has been low since the 2012 coupe 
helped to reveal significant corruption and underperformance in government institutions, 
including within the court system and other justice sector institutions. In the context of 
USAID/Mali work with DNAJ to improve its performance and the performance of courts 
that it oversees, the impact evaluation seeks to understand how informing Malians about 
reforms and planned improvements in justice sector institutions affects their attitudes 
toward and interactions with state institutions. In particular, does information about formal 
sector institutions change people’s willingness to engage with those institutions? 
Specifically, primary research questions include: 
 Does providing information and transparency of justice sector reforms increase 
citizens’ trust in the legal system?
 Does providing and transparency of the court system lead to increased use by 
citizens of formal legal institutions? 
 Does providing information about judicial performance have additional effects 
beyond just providing general information? 
Based on initial ideas developed during the DCHA/DRG Learning Clinic held March 23-27, 
2015, in Dar es Salaam, Tanzania, and further developed during two scoping trips to Mali 
which took place between July 27 and August 7, 2015 and between May 21 and May 25, 
2018, the evaluation team for this assessment devised an impact evaluation of a public 
information campaign conducted by four civil society organizations (ADENORD, APDF, 
CRADE, and WILDAF) supported through the USAID/Mali Justice Project. The information 
campaign consisted of two pairs of meetings focused on the subjects of (a) the overall 
organization of the formal judicial sector, and (b) inheritance law. MJP and the CSOs 
developed the curriculum for the meetings based on the MJP-produced document Justice 
Explained to Malians: 100 Fact Sheets for Understanding the Law (La justice expliquée aux 
Maliens: 100 fiches pratiques pour comprendre le Droit). 
For the impact evaluation, 187 villages listed by the four CSOs were randomly assigned to 
either receive (treatment) or not receive (control) the information campaign after a baseline 
evaluation survey was conducted and before an endline evaluation survey. 
The survey firm GISSE conducted a baseline household survey in all 187 villages over the 
period September 17 – October 8, 2018. The information campaign led by the CSOs began 
immediately following the conclusion of survey enumeration in the 94 villages assigned to 
treatment. GISSE conducted a non-panel endline household survey in the same 187 villages 
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 2 
over the period November 7-24, 2018. At both baseline and endline, GISSE surveyed 10 
households per village. In addition to the household surveys, GISSE collected baseline and 
endline panel data from village elites, aiming to survey up to four elites per village. In nearly 
all villages, GISSE interviewed the village head, a women’s group leader, and a youth group 
leader at both baseline and endline. 
In both the household and elite endline surveys, we asked a set of questions about 
respondents’ awareness of public information meetings. In the overall household data, we 
find no difference between treatment and control areas in terms of awareness of meetings held in 
the villages in the last six weeks. For one of the four CSOs, there is a statistically significant 
difference in the wrong direction (i.e., respondents in control villages are more likely to 
report the presence of public information meetings). When we ask the respondents who 
said that they were aware of meetings whether those meetings concerned the judicial 
sector, we find that most respondents think that they were not about the judicial sector, 
although respondents in treatment areas are somewhat more likely to say that they were. 
Among village elites, we find an increased likelihood of responses that there were public 
information meetings in treatment villages relative to control villages. In the overall data, 
elites in treatment villages are 10 percentage points more likely than elites in control villages 
to say that there had been public information meetings in the last six weeks in their village. 
Nonetheless, the total percentage of elites in treatment villages saying that there had been 
meetings is only 62 percent. When looking only at the village heads, these numbers are 
essentially identical: 65 percent of village heads in treatment villages said that there had been 
an information campaign, and this is 10 percentage points greater than the proportion saying 
that in control villages. As with the household data, estimates continue to be in the wrong 
direction (i.e., they suggest a greater prevalence of information meetings in control villages) 
for one of the four CSO partners. When asked whether the information meetings were 
about the judicial sector, village elites who previously said that there had been a meeting 
were equally likely to say that it was about the judicial sector across treatment and control 
villages, and the proportion of elites saying that the meetings were about the judicial sector 
is only 18 percent. 
These results are unchanged if we look at the set of villages with the smallest estimated 
populations, where we would ex ante expect to see greater elite or mass public awareness 
of the information campaign. 
We explore the most basic possible outcome of the information campaign – a change in 
knowledge levels – and find no difference on a 12-item knowledge index between treatment 
and control villages in an intent-to-treat analysis. 
Given these disappointing findings about awareness of the information campaign and the 
spread of the basic information contained in the campaign, we conclude that we are unlikely 
to find additional evidence of the effectiveness of the information campaign. We review a 
set of possible explanations for why we do not observe more awareness of the information 
campaign: (1) limited reach of the information sessions; (2) limited time for information 
diffusion; (3) non-adherence to the randomization schedule; (4) a disjuncture between the 
description of the meetings used by the CSOs and our survey question wording; and (5) 
improper survey data collection. We are not able to identify a single explanation for the 
lack of campaign awareness and the lack of impact. We are most persuaded by the first and 
third possibilities: that the information sessions were more limited in scope than what the 
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 3 
evaluation team had envisioned in designing the evaluation and/or that the CSOs did not 
strictly adhere to the randomization schedule. 
This memorandum proceeds by (1) describing the impact evaluation process, (2) presenting 
in more detail the results described in this summary, and (3) providing some possible 
interpretations of the results. 
II. IMPACT EVALUATION IMPLEMENTATION 
Members of the USAID/Mali mission and a member of the evaluation team jointly developed 
the idea to conduct an impact evaluation of an information campaign in the context of the 
Mali Justice Program during the DCHA/DRG Learning Clinic held March 23-27, 2015, in Dar 
es Salaam, Tanzania. The evaluation team undertook an initial scoping trip from July 27 - 
August 7, 2015, meeting with government and civil society representatives in order to 
better understand the state of the formal justice sector in Mali and to think about 
appropriate measurement of the effectiveness of an information campaign. At the time of 
this initial scoping trip, USAID was in the process of awarding the MJP contract. The 
evaluation team presented the research design to USAID/Mali mission leadership and 
submitted an impact evaluation design to NORC on August 25, 2015, and then submitted a 
revised version on October 14, 2015. 
After the MJP contract was awarded to Checchi and Company Consulting, one member of 
the evaluation team traveled to Bamako on January 14, 2016, for a meeting with 
representatives from the MJP and USAID/Mali staff. 
Because of subsequent deterioration of the security situation in Mali and the evacuation of 
USAID/Mali personnel, it was unclear if the impact evaluation would proceed. 
In the spring of 2018, conversations among the evaluation team, USAID/Mali personnel, and 
MJP staff resumed, culminating in a second scoping trip over the period May 21-25, 2018. 
During this scoping trip, the evaluation team met with USAID/Mali personnel, MJP 
personnel, and representatives from the four CSO grantees that were to conduct the 
information campaign. Subsequent to this trip, the evaluation team circulated revised impact 
evaluation designs on July 16, 2018, and September 11, 2018. 
NORC issued an RFP for data collection in June 2018. The evaluation team and 
representatives from NORC discussed the relative merits of the seven bids submitted in 
response to the RFP. NORC contracted GISSE in August 2018 to conduct the survey data 
collection. 
In early June 2018, MJP provided the evaluation team with four lists of villages – one from 
each of the participating CSO grantees. Table 1 describes the number of villages in each 
CSO’s list. MJP and the evaluation team agreed on an implementation timeline in which the 
evaluation team would select half of the listed villages to be treatment villages; the survey 
firm would conduct a baseline survey in all villages; the CSOs would immediately conduct 
the information campaign in treatment villages only; the survey firm would conduct an 
endline survey in all villages; and the CSOs would only then proceed to conduct the 
information campaign in the control villages. 
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 4 
Members of the evaluation team traveled to Mali for the period September 9-22, 2018, 
meeting with representatives from MJP and working with GISSE to finalize the survey 
instruments and train the survey enumerators. 
In developing the enumeration plans, GISSE discovered eight village names on the initial lists 
that either did not correspond to a single extant village or else would not be appropriate 
for the study.1
 MJP consulted with the CSO partners and obtained replacement villages for 
all eight entries on the original list of villages. 
Table 1. Number of Villages in the Study by CSO Partner.
CSO PARTNER (REGIONS)
NUMBER OF 
VILLAGES IN 
INITIAL LIST
NUMBER OF 
VILLAGES 
REPLACED
NUMBER OF 
VILLAGES 
ASSIGNED TO 
TREATMENT
ADENORD (Taoudénit) 14 3 7
APDF (Kayes; Koulikoro: Dioila cercle) 122 1 62
CRADE (Gao) 25 0 13
WILDAF (Koulikoro: Kati and Koulikoro cercles) 26 4 12
Total 187 8 94
GISSE began enumeration of the baseline survey on September 17, 2018. In each village, 
GISSE used a random walk method, dividing the village into two areas, using a landmark in 
each area as a starting point, and then using a daily direction and skip pattern to select 
households. At the household level, the enumerator solicited a list of residents age 18 or 
older and then randomly selected either a male or a female eligible respondent, alternating 
between genders from house to house. 
The evaluation team conducted the random assignment of villages to treatment on October 
2, 2018. To reduce the risk of spurious correlation between treatment status and village 
location, the evaluation team stratified treatment assignment on cercle (the geographic unit 
below the region). The information campaign was to be implemented only by one CSO in 
each cercle in the study. Therefore, this stratification also ensures a balance of treatment 
and control villages within the lists provided by each CSO. 
GISSE concluded baseline household survey enumeration on October 8, 2018, and baseline 
elite survey enumeration on October 9, 2018. Of 2,006 households approached, 
 
1
 In Taoudénit, one village was actually a salt mine with only male workers in residence, another was a 
duplicate name for a village already on the list, and a third was no longer inhabited. In Koulikoro, one village 
name corresponded to two different villages; this same issue existed in Kati, while another two village names 
from Kati did not correspond to existing villages. One village in Kayes was a duplicate name for a village 
already on the list. 
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 5 
enumerators obtained lists of potential respondents at 1,984 (99%), began 1,923 interviews, 
and completed 1,903 interviews (96% of contacted households).2
 
The MJP distributed the randomization list to the CSOs and asked them to develop 
deployment plans. MJP provided deployment plans for the CSOs over the period October 
11-17, 2018. The evaluation team checked that the deployment plans followed treatment 
assignment. In response to questions raised by the evaluation team about the number of 
information sessions per village, CRADE submitted a revised deployment plan. Likewise, the 
evaluation team identified a number of ways in which ADENORD’s plan deviated from the 
revised village list, leading ADENORD to revise its deployment plan. The formatting of the 
APDF document made some of the training dates unclear, and APDF submitted a revised 
document. 
As the CSOs conducted the information sessions, they sent reporting sheets to MJP. In 
some cases, the CSOs passed along the sign-in sheets from the meetings, listing the 
date/time, location, and participants for each training session; in other cases, they sent 
summary documents. MJP passed the reporting sheets that it received along to the 
evaluation team. 
The evaluation team has reporting sheets for 52 of the 94 villages assigned to treatment. For 
ADENORD, CRADE, and WILDAF, we have some information for all of the villages 
assigned to treatment; for APDF, we have information only for 20 out of the 42 villages 
assigned to be treated by APDF. 
For 45 villages, the reporting sheets provide information on all four sessions that were 
supposed to take place (although in a number of cases two thematic sessions were 
combined into a single meeting, a deviation from the initial implementation plan for the 
information campaign).3
 For the seven villages in Koulikoro cercle to be covered by 
WILDAF, the reporting sheets provide a number of attendees but provide neither a 
breakdown by the four sessions that were to be held nor a date on which any sessions took 
place. 
Even within the reporting sheets that did provide full information, the evaluation team 
identified a number of discrepancies. While CRADE provided reporting information for all 
48 sessions run in the 12 villages assigned to treatment under its purview, 18 of these sheets 
had a mismatch between the total number of participants listed on the form and the number 
of signatures on the form. Sign-in sheets for one village were initially sent with attendees 
listed in order by age, and then revised sheets were sent. For WILDAF reporting, there 
were discrepancies between summary sheets that the CSO submitted and attendance forms: 
attendance numbers do not match, often by large margins, and the dates are not the same 
on the summary sheet as on the attendance forms. 
The evaluation team also has another eight reporting sheets – all provided by APDF – that 
do not appear to correspond to treatment villages. One of these sheets is for a village 
 
2
 As the list of villages included in the study changed while GISSE was conducting the baseline survey, the 
baseline survey includes data from one village not included in the final list of villages participating in the 
information campaign. 
3
 For two villages, the names are approximate matches, and we are not certain that they are the same villages 
that were included in the deployment plan. 
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 6 
assigned to the control condition, representing a clear violation of the randomization. The 
other seven villages were not included in the original village listing that APDF sent and 
therefore not included in the deployment plan. As mentioned in the footnote above, an 
additional two reporting sheets have village names that are approximate matches, and the 
evaluation team is considering these as correctly treated villages. 
In response to some of the reporting sheets, the evaluation team asked questions about the 
implementation of the training sessions. As mentioned above, the evaluation team 
discovering a number of cases where two training sessions were conducted within a single 
meeting. The evaluation team discovered another case where the names of attendees from 
one training session were exactly repeated in the reporting sheet for another training 
session. CRADE described this latter instance as “a mix-up from the officer filling up the 
form.” 
In trying to understand the surprising results that we present below, the evaluation team 
requested on December 13, 2018, that MJP collect additional information about 
implementation of the information campaign from the CSOs. MJP provided responses from 
APDF and CRADE on December 22, 2018. We refer to this information in the final section 
of the report. 
Based on the incomplete set of reporting sheets, the earliest information session took place 
on October 14, 2018, and the last took place on November 3, 2018. For the 44 first￾session meetings on the first theme where we can identify the date, the reporting sheets 
indicate that between 9 and 56 people attended the sessions; the median was 23, and the 
average was 29. The distribution is summarized in Figure 1. As many of the sessions 
specifically recruited women, the average number of women in attendance is 18. Seven 
sessions featured only women and no men at all. Only one session featured exclusively 
men. 
Table 2. Planned and Actual Deployment by CSO Partner. Dates for meeting 
conduct reflect incomplete reporting from the CSOs.
CSO Partner (Regions)
Earliest 
Planned 
Meeting
Latest 
Planned 
Meeting
Earliest 
Conducted 
Meeting
Latest 
Conducted 
Meeting
ADENORD (Taoudénit) 10/19/18 11/3/18 10/22/18 11/3/18
APDF (Kayes; Koulikoro: Dioila 
cercle)
10/19/18 11/3/18 10/31/18 11/2/18
CRADE (Gao) 10/14/18 10/31/18 10/14/18 10/31/18
WILDAF (Koulikoro: Kati and 
Koulikoro cercle)
10/16/18 10/29/18 10/21/18 11/1/18
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 7 
Figure 1. Reported Attendance at First Theme Training Session. From 44 CSO 
reports. Depending on CSO, the first training session might either have been 
one of two for the first theme or else it was a combined double session for the 
theme.
GISSE began collection of the endline surveys on November 7, 2018, shortly after the 
completion of the last information session. Household survey enumeration lasted through 
November 24, 2018, and elite survey enumeration lasted through November 27, 2018. At 
endline, of 1,973 households approached, enumerators obtained a list of eligible 
respondents at 1,939 (98%) and obtained 1,872 respondents from those households (96.5% 
of contacted households). 
GISSE provided all survey data to the evaluation team by November 27, 2018. The 
evaluation team checked the GPS coordinates recorded by the survey tablets against maps 
of Mali and found that they were consistent with where survey enumeration was supposed 
to take place.4
 
For the village elite data, GISSE was able to recontact all but six of the original 673 elites in 
the survey (< 1% attrition). Of the 667 endline interviews, 24 (3.6%) were conducted by 
phone. There was perfect consistency between baseline and endline data in terms of the 
socio-demographic characteristics of these elites (e.g., they all reported the same age, 
education, religion, and language at both baseline and endline). This perfect consistency is 
somewhat surprising – normally one would expect some over-time variation in responses 
(e.g., about age or language spoken at home) attributable to respondent uncertainty or data￾entry errors. 
III. ANALYSIS 
 
4
 In the case of 10 villages in Taoudénit, the surveys were conducted on paper because of concerns about the 
use of electronic tablets in the villages. The GPS coordinates for those surveys therefore correspond to the 
location of data entry rather than the location of survey enumeration. 
10 20 30 40 50 60
Adults Attending First Theme Training Session
Attendance at Training Session
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 8 
Upon receiving the data, the first set of analyses that the evaluation team conducted used a 
set of questions asked at the end of the endline surveys at both the household level and the 
elite level. The first of these questions asked, “To the best of your knowledge, have there 
been any public information meetings in this village in the last six weeks?” Among the set of 
respondents who answered in the affirmative, a follow-up question asked, “Can you tell me 
if any of these public information meetings addressed the topic of the judicial system or 
inheritance rights in Mali?” And among the set of respondents who answered in the 
affirmative to that question, another follow-up question asked about their own attendance 
at the meetings: “Did you attend any of these meetings about the judicial system or 
inheritance rights in Mali?” We did not collect data about attendance at meetings in other 
sectors. 
The rates of answering the initial question in the affirmative were far lower than expected in 
the treatment villages. As can be seen in Table 3, 33 percent of respondents in treatment 
villages said that there had been a public information meeting in the village in the last six 
weeks. While on the face of it, one-in-three people being aware of a public information 
campaign might not be a bad outcome, this number is indistinguishable from the proportion 
of respondents in control villages reporting that there had been a public information 
meeting. Table 3 also shows the levels of awareness in treatment and control villages across 
the four CSOs. The differences for CRADE and APDF are in the expected direction but are 
relatively small. The difference for ADENORD is in the wrong direction (i.e., nearly twice 
as many respondents from control villages report that there had been a public information 
meeting in the last six weeks), and the distribution of responses for this CSO only is 
statistically significantly different across treatment and control villages. 
Among the set of respondents – in both treatment and control villages – who said that 
there had been information meetings, we asked them if the meetings had been about the 
judicial sector or inheritance rights. Overall, only 18 percent of the respondents in the 
treatment villages who said that they had heard about information meetings say that those 
information meetings were about the judicial sector. This is higher than the proportion of 
respondents in the control villages who say that they heard about meetings, and the 
distributions are different from each other to an extent that meets marginal levels of 
statistical significance. Across the different CSOs, we see that there is a significant 
difference for respondents in APDF’s control and treatment villages in their likelihood of 
saying that the meetings of which they were aware were about the judicial sector, although 
the proportion of respondents in treatment villages identifying the information meetings 
with which they were familiar as having been about the judicial sector is only 11 percent. In 
the villages covered by CRADE, respondents familiar with information meetings were more 
likely to say that they had to do with the judicial sector, but we find this is true for both 
treatment and control villages, such that the patterns of responses to this question are not 
distinguishable from one another across the two sets of villages. In Taoudénit, the region 
covered by ADENORD, respondents are more likely to say that the meetings that they 
were aware of were about the judicial system, although the small number of villages in this 
region means that the distributions are not statistically distinguishable across the treatment 
and control villages. 
In Table 5, we explore whether or not those respondents who had said that they were 
aware of meetings that were about the judicial sector had attended those meetings. In 
treatment villages, 56 percent of the 57 respondents who told us that they were aware of 
meetings and that those meetings had been about the judicial sector said that they attended 
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 9 
the meetings. In control villages, this proportion is less at 43 percent, yet because of the 
very small numbers of respondents answering this question, those differences are not 
statistically significant in the overall data. 
Table 3. Awareness of public information meeting. From endline household 
survey. “To the best of your knowledge, have there been any public information 
meetings in this village in the last six weeks?”
OVERALL
CRADE
(GAO)
APDF
(KAYES AND 
KOULIKORO)
WILDAF
(KOULIKORO)
ADENORD
(TAOUDÉNIT)
C T C T C T C T C T
Yes 0.32
(N=301)
0.33
(N=309)
0.28
(N=37)
0.33
(N=39)
0.31
(N=187)
0.34
(N=212)
0.34
(N=44)
0.32
(N=41)
0.47
(N=33)
0.24
(N=17)
No 0.55
(N=514)
0.53
(N=496)
0.42
(N=54)
0.38
(N=46)
0.62
(N=373)
0.57
(N=352)
0.44
(N=57)
0.38
(N=50)
0.42
(N=30)
0.69
(N=48)
Don’t 
Know
0.12
(N=116)
0.14
(N=136)
0.30
(N=39)
0.29
(N=35)
0.07
(N=40)
0.09
(N=57)
0.22
(N=29)
0.30
(N=39)
0.11
(N=8)
0.07
(N=5)
Χ
2 p < 0.38 p < 0.78 p < 0.10 p < 0.37 p < 0.01
Table 4. Awareness of a public information meeting about the judicial sector. 
From endline household survey; asked only among those who said that there 
had been a public information meeting (i.e., “yes” answers in Table 3). “Can 
you tell me if any of these public information meetings addressed the topic of 
the judicial system or inheritance rights in Mali?”
OVERALL
CRADE
(GAO)
APDF
(KAYES AND 
KOULIKORO)
WILDAF
(KOULIKORO)
ADENORD
(TAOUDÉNIT)
C T C T C T C T C T
Yes 0.12
(N=37)
0.18
(N=57)
0.49
(N=18)
0.51
(N=20)
0.03
(N=5)
0.11
(N=23)
0.02
(N=1)
0.10
(N=4)
0.39
(N=13)
0.59
(N=10)
No 0.81
(N=245)
0.74
(N=228)
0.32
(N=12)
0.33
(N=13)
0.91
(N=171)
0.82
(N=173)
0.95
(N=42)
0.88
(N=36)
0.61
(N=20)
0.35
(N=6)
Don’t 
Know
0.06
(N=19)
0.08
(N=24)
0.19
(N=7)
0.15
(N=6)
0.06
(N=11)
0.08
(N=16)
0.02
(N=1)
0.02
(N=1)
0.00
(N=0)
0.06
(N=1)
Χ
2 p < 0.07 p < 0.92 p < 0.01 p < 0.34 p < 0.12
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 10 
Table 5. Attendance at public information meetings about the judicial sector. 
From endline household survey; asked only among those who said that there 
had been a public information meeting and that it was about the judicial sector 
(i.e., “yes” answers in Table 3 and then in Table 4). “Did you attend any of 
these meetings about the judicial system or inheritance rights in Mali?”
OVERALL
CRADE
(GAO)
APDF
(KAYES AND 
KOULIKORO)
WILDAF
(KOULIKORO)
ADENORD
(TAOUDÉNIT)
C T C T C T C T C T
Yes 0.43
(N=16)
0.56
(N=32)
0.33
(N=6)
0.40
(N=8)
0.20
(N=1)
0.83
(N=19)
0.00
(N=0)
0.25
(N=1)
0.69
(N=9)
0.40
(N=4)
No 0.57
(N=21)
0.44
(N=25)
0.67
(N=12)
0.60
(N=12)
0.80
(N=4)
0.17
(N=4)
1.00
(N=1)
0.75
(N=3)
0.31
(N=4)
0.60
(N=6)
Χ
2 p < 0.23 p < 0.68 p < 0.01 p < 0.58 p < 0.17
In Table 6, we look at the elite endline data, beginning with the question about whether or 
not any public information meetings had taken place. In these data, we do find more 
variation in awareness of information meetings. In the treatment villages, 62 percent of our 
elite sample answer that there had been meetings; in the control villages, however, there is 
still a high level of 51 percent of elite respondents saying that there had recently been a 
public information meeting. For CRADE and APDF, this difference is somewhat larger at 14 
and 16 percentage points, respectively, and among the elites from the villages within APDF’s 
domain, the distributions of responses are statistically distinguishable from each other. As in 
the household-level data, among ADENORD villages, there appears to be more awareness 
of information meetings in control villages. 
When we look only at village heads (Table 7), women’s group leaders (Table 8), or youth 
group leaders (Table 9), the results are generally parallel: in the overall data and for CRADE 
and APDF, there is always more awareness of public information meetings in the treatment 
villages. For WILDAF, among the village heads, the findings are reversed, with more 
awareness of information meetings in control villages, and this unexpected pattern remains 
consistent in ADENORD’s villages across all three types of elites.
When we ask those elites who said that they were aware of information meetings about 
whether or not those meetings had to do with the judicial sector, we find approximately 
equal proportions of respondents in control and treatment villages saying that they were 
(Table 10). And the levels of respondents saying that they were referring to judicial sector 
information meetings is quite low in both APDF and WILDAF’s areas, although much higher 
in CRADE’s area and somewhat in between in ADENORD’s area. These findings exactly 
parallel those found in the household surveys (Table 4). 
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 11 
Table 6. Awareness of public information meeting among village elites. From 
endline elite survey. “To the best of your knowledge, have there been any 
public information meetings in this village in the last six weeks?”
OVERALL
CRADE
(GAO)
APDF
(KAYES AND 
KOULIKORO)
WILDAF
(KOULIKORO)
ADENORD
(TAOUDÉNIT)
C T C T C T C T C T
Yes 0.51
(N=169)
0.62
(N=211)
0.63
(N=33)
0.77
(N=36)
0.42
(N=87)
0.58
(N=128)
0.72
(N=36)
0.78
(N=39)
0.59
(N=13)
0.38
(N=8)
No 0.45
(N=149)
0.35
(N=119)
0.27
(N=14)
0.19
(N=9)
0.56
(N=115)
0.40
(N=87)
0.22
(N=11)
0.20
(N=10)
0.41
(N=9)
0.62
(N=13)
Don’t 
Know
0.03
(N=11)
0.02
(N=8)
0.10
(N=5)
0.04
(N=2)
0.01
(N=3)
0.02
(N=5)
0.06
(N=3)
0.02
(N=1)
0.00
(N=0)
0.00
(N=0)
Χ
2 p < 0.02 p < 0.33 p < 0.01 p < 0.56 p < 0.17
Table 7. Awareness of public information meeting among village heads. From 
endline elite survey. “To the best of your knowledge, have there been any 
public information meetings in this village in the last six weeks?”
OVERALL
CRADE
(GAO)
APDF
(KAYES AND 
KOULIKORO)
WILDAF
(KOULIKORO)
ADENORD
(TAOUDÉNIT)
C T C T C T C T C T
Yes 0.56
(N=52)
0.65
(N=61)
0.77
(N=10)
0.83
(N=10)
0.47
(N=28)
0.63
(N=39)
0.77
(N=10)
0.69
(N=9)
0.57
(N=4)
0.43
(N=3)
No 0.43
(N=40)
0.35
(N=33)
0.23
(N=3)
0.17
(N=2)
0.52
(N=31)
0.37
(N=23)
0.23
(N=3)
0.31
(N=4)
0.43
(N=3)
0.57
(N=4)
Don’t 
Know
0.01
(N=1)
0.00
(N=0)
0.00
(N=0)
0.00
(N=0)
0.02
(N=1)
0.00
(N=0)
0.00
(N=0)
0.00
(N=0)
0.00
(N=0)
0.00
(N=0)
Χ
2 p < 0.31 p < 0.69 p < 0.14 p < 0.66 p < 0.60
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 12 
Table 8. Awareness of public information meeting among women’s group 
leaders. From endline elite survey. “To the best of your knowledge, have there 
been any public information meetings in this village in the last six weeks?”
OVERALL
CRADE
(GAO)
APDF
(KAYES AND
KOULIKORO)
WILDAF
(KOULIKORO)
ADENORD
(TAOUDÉNIT)
C T C T C T C T C T
Yes 0.38
(N=35)
0.51
(N=48)
0.54
(N=7)
0.67
(N=8)
0.27
(N=16)
0.44
(N=27)
0.54
(N=7)
0.77
(N=10)
0.71
(N=5)
0.43
(N=3)
No 0.57
(N=52)
0.46
(N=43)
0.23
(N=3)
0.25
(N=3)
0.73
(N=43)
0.53
(N=33)
0.31
(N=4)
0.23
(N=3)
0.29
(N=2)
0.57
(N=4)
Don’t 
Know
0.05
(N=5)
0.03
(N=3)
0.23
(N=3)
0.08
(N=1)
0.00
(N=0)
0.03
(N=2)
0.15
(N=2)
0.00
(N=0)
0.00
(N=0)
0.00
(N=0)
Χ
2 p < 0.19 p < 0.60 p < 0.05 p < 0.27 p < 0.28
Table 9. Awareness of public information meeting among youth group leaders. 
From endline elite survey. “To the best of your knowledge, have there been any 
public information meetings in this village in the last six weeks?”
OVERALL
CRADE
(GAO)
APDF
(KAYES AND
KOULIKORO)
WILDAF
(KOULIKORO)
ADENORD
(TAOUDÉNIT)
C T C T C T C T C T
Yes 0.51
(N=47)
0.66
(N=61)
0.54
(N=7)
0.75
(N=9)
0.46
(N=27)
0.67
(N=40)
0.69
(N=9)
0.77
(N=10)
0.57
(N=4)
0.29
(N=2)
No 0.47
(N=43)
0.28
(N=26)
0.38
(N=5)
0.17
(N=2)
0.54
(N=32)
0.28
(N=17)
0.23
(N=3)
0.15
(N=2)
0.43
(N=3)
0.71
(N=5)
Don’t 
Know
0.02
(N=2)
0.05
(N=5)
0.08
(N=1)
0.08
(N=1)
0.00
(N=0)
0.05
(N=3)
0.08
(N=1)
0.08
(N=1)
0.00
(N=0)
0.00
(N=0)
Χ
2 p < 0.03 p < 0.48 p < 0.01 p < 0.89 p < 0.28
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 13 
Table 10. Awareness of a public information meeting about the judicial sector 
among village elites. From endline elite survey; asked only among those who 
said that there had been a public information meeting (i.e., “yes” answers in 
Table 6). “Can you tell me if any of these public information meetings 
addressed the topic of the judicial system or inheritance rights in Mali?”
OVERALL
CRADE
(GAO)
APDF
(KAYES AND 
KOULIKORO)
WILDAF
(KOULIKORO)
ADENORD
(TAOUDÉNIT)
C T C T C T C T C T
Yes 0.18
(N=30)
0.18
(N=39)
0.58
(N=19)
0.61
(N=22)
0.03
(N=3)
0.09
(N=12)
0.08
(N=3)
0.05
(N=2)
0.38
(N=5)
0.38
(N=3)
No 0.80
(N=135)
0.79
(N=167)
0.39
(N=13)
0.36
(N=13)
0.97
(N=84)
0.88
(N=113)
0.92
(N=33)
0.92
(N=36)
0.38
(N=5)
0.63
(N=5)
Don’t 
Know
0.02
(N=4)
0.02
(N=5)
0.03
(N=1)
0.03
(N=1)
0.00
(N=0)
0.02
(N=3)
0.00
(N=0)
0.03
(N=1)
0.23
(N=3)
0.00
(N=0)
Χ
2 p < 0.99 p < 0.96 p < 0.09 p < 0.55 p < 0.30
(1) AWARENESS IN LOWER-POPULATION VILLAGES 
As the extent to which the visibility of the information meetings depends on the size of the 
villages in the study, we repeated these analyses looking only at the smallest villages in the 
sample. The most recent comprehensive population data in Mali come from the 2009 
census. Given the turmoil that the country has seen, there are good reasons to be skeptical 
of these numbers, but they are the most comprehensive numbers that we have to work 
with. The census contains population numbers for 172 of the 187 villages in the study. As 
might be expected, nine of the missing data points are from Taoudénit, a region in the north 
of the country formed out of Timbuktu in 2012; three are in Gao, one in Kayes, and two in 
Koulikoro. For those villages where the 2009 population data were not available, GISSE 
contacted the village heads to get population numbers, giving us a secondary dataset that 
contains a mix of 2009 census population numbers and 2018 village-head-reported 
population numbers. We primarily subset the data based on the 2009 census numbers but 
also use the mixed data below. 
Figure 2 shows the distribution of villages by their 2009 population. Around half of the 
villages – 44 percent – had a 2009 population of less than 1,000 residents; one out of five 
had a 2009 population of less than 500 residents. 
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 14 
Figure 2. Distribution of 2009 Population in Study Villages. Population data 
from 2009 census.
In Table 11, we rerun several of the analyses above using only villages with a 2009 
population below 1,000 people. The results are unchanged. In the household data, there is 
no meaningful distinction between treatment and control villages in terms of awareness of 
public information meetings, and relatively few people (although twice as many in treatment 
villages) say that the meetings of which they were aware were about the judicial sector. 
Among elites, there is a greater distinction between treatment and control villages in terms 
of the likelihood of elites saying that there had been a public information meeting in the last 
six weeks, yet among those who said that there were meetings, only around one in 10 say 
that the meetings were about the judicial sector, and meeting-aware elites in the control 
villages are as likely as meeting-aware elites in the treatment villages to say that the meetings 
were about the judicial sector. 
In Table 12, we expand the scope of the “small villages” subset slightly and report the 
results for the first four outcome variables using villages with a population estimate of less 
than 1,000 residents based either on the 2009 data or else reports from the village head.5
 
For the household data, the results parallel those based on using only the 2009 census data 
to distinguish small villages: there is no difference in awareness of public information 
meetings between treatment and control villages but a distinction in the likelihood of 
reporting that the meeting was about the judicial sector among those who said that they had 
 
5
 Note that GISSE updated the population data for two villages in Taoudénit that were listed in the 2009 
census. For Agouni village in Bouzebha cercle, the population was updated from 1,500 to 1,750, and so it 
enters into neither Table 11 nor Table 12. For Tindjambane village in Taoudénit cercle, the population was 
694 in the 2009 census, whereas the village head reported it as 1,815 to GISSE; data from this village, 
therefore, is included in Table 11 but not Table 12. 
0
20 40 60 80
100
Frequency
0 5000 10000 15000 20000 25000
Pop 2009
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 15 
been aware of a meeting. In this subset, we also see greater distinction in meeting 
attendance among those who report having been aware of a judicial sector meeting. This 
difference is driven by one village from Taoudénit which is found in the first table but not 
the second and where four respondents reported attending the meeting. 
Table 11. Awareness of meetings in villages with 2009 population below 1,000. 
From endline surveys. Population numbers come from 2009 census data and 
therefore do not precisely reflect current population levels. 
HOUSEHOLD 
DATA: 
AWARENESS 
OF ANY 
MEETING
HOUSEHOLD 
DATA: 
MEETING WAS 
ABOUT THE 
JUDICIAL 
SECTOR
HOUSEHOLD 
DATA: 
ATTENDED 
MEETING
ELITE DATA: 
AWARENESS 
OF ANY 
MEETING
ELITE DATA: 
MEETING 
WAS ABOUT 
THE JUDICIAL 
SECTOR
C T C T C T C T C T
Yes 0.31
(N=115)
0.33
(N=126)
0.06
(N=7)
0.13
(N=16)
0.71
(N=5)
0.88
(N=14)
0.41
(N=50)
0.58
(N=74)
0.12
(N=6)
0.11
(N=8)
No 0.62
(N=228)
0.57
(N=219)
0.90
(N=103)
0.83
(N=105)
0.29
(N=2)
0.13
(N=2)
0.56
(N=68)
0.40
(N=51)
0.88
(N=44)
0.88
(N=65)
Don’t 
Know
0.07
(N=27)
0.09
(N=36)
0.04
(N=5)
0.04
(N=5)
0.00
(N=0)
0.00
(N=0)
0.02
(N=3)
0.02
(N=3)
0.00
(N=0)
0.01
(N=1)
Χ
2 p < 0.41 p < 0.22 p < 0.35 p < 0.05 p < 0.70
VILLAGE 
HEADS: 
AWARENESS 
OF ANY 
MEETING
VILLAGE 
HEADS: 
MEETING WAS 
ABOUT 
JUDICIAL 
SECTOR
WOMEN’S 
LEADERS: 
AWARENESS 
OF ANY 
MEETING
YOUTH 
LEADERS: 
AWARENESS 
OF ANY 
MEETING
C T C T C T C T
Yes 0.54
(N=20)
0.66
(N=25)
0.05
(N=1)
0.04
(N=1)
0.33
(N=12)
0.39
(N=15)
0.35
(N=13)
0.62
(N=23)
No 0.46
(N=17)
0.34
(N=13)
0.95
(N=19)
0.96
(N=24)
0.64
(N=23)
0.58
(N=22)
0.62
(N=23)
0.32
(N=12)
Don’t 
Know
0.00
(N=0)
0.00
(N=0)
0.00
(N=0)
0.00
(N=0)
0.03
(N=1)
0.03
(N=1)
0.03
(N=1)
0.05
(N=2)
Χ
2 p < 0.30 p < 0.88 p < 0.86 p < 0.04
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 16 
Table 12. Awareness of meetings in villages with population below 1,000 
(2009/2018 data). From endline surveys. Population numbers come from 2009 
census data plus reports from village heads for villages where 2009 census data 
did not exist. 
HOUSEHOLD 
DATA: 
AWARENESS 
OF ANY 
MEETING
HOUSEHOLD 
DATA: 
MEETING WAS 
ABOUT THE 
JUDICIAL 
SECTOR
HOUSEHOLD 
DATA: 
ATTENDED 
MEETING
ELITE DATA: 
AWARENESS 
OF ANY 
MEETING
ELITE DATA: 
MEETING 
WAS ABOUT 
THE JUDICIAL 
SECTOR
C T C T C T C T C T
Yes 0.31
(N=110)
0.31
(N=134)
0.03
(N=3)
0.17
(N=23)
0.33
(N=1)
0.74
(N=17)
0.41
(N=48)
0.54
(N=77)
0.08
(N=4)
0.14
(N=11)
No 0.62
(N=223)
0.60
(N=257)
0.93
(N=102)
0.78
(N=105)
0.67
(N=2)
0.26
(N=6)
0.56
(N=66)
0.44
(N=63)
0.92
(N=44)
0.84
(N=65)
Don’t 
Know
0.08
(N=27)
0.09
(N=40)
0.05
(N=5)
0.04
(N=6)
0.00
(N=0)
0.00
(N=0)
0.03
(N=3)
0.02
(N=3)
0.00
(N=0)
0.01
(N=1)
Χ
2 p < 0.63 p < 0.01 p < 0.16 p < 0.12 p < 0.44
Table 13. Awareness of meetings in treatment villages for which we have CSO 
reports. From endline surveys. 
HOUSEHOLD 
DATA: 
AWARENESS 
OF ANY 
MEETING
HOUSEHOLD 
DATA: 
MEETING 
WAS ABOUT 
THE JUDICIAL 
SECTOR
HOUSEHOLD 
DATA: 
ATTENDED 
MEETING
ELITE DATA: 
AWARENESS 
OF ANY 
MEETING
ELITE DATA: 
MEETING 
WAS ABOUT 
THE JUDICIAL 
SECTOR
C T C T C T C T C T
Yes 0.32
(N=301)
0.29
(N=151)
0.12
(N=37)
0.32
(N=48)
0.43
(N=16)
0.52
(N=25)
0.51
(N=169)
0.64
(N=119)
0.18
(N=30)
0.28
(N=33)
No 0.55
(N=514)
0.54
(N=280)
0.81
(N=245)
0.63
(N=95)
0.57
(N=21)
0.48
(N=23)
0.45
(N=149)
0.34
(N=63)
0.80
(N=135)
0.71
(N=84)
Don’t 
Know
0.12
(N=116)
0.17
(N=89)
0.06
(N=19)
0.05
(N=8)
0.00
(N=0)
0.00
(N=0)
0.03
(N=11)
0.03
(N=5)
0.02
(N=4)
0.02
(N=2)
Χ
2 0.05 0.01 0.42 0.03 0.13
(2) AWARENESS IN VILLAGES WITH CSO REPORTS ON MEETINGS 
We also find generally consistent patterns when we limit the definition of a treated village to 
the set of villages for which we received reports from the CSOs. In Table 13, the treatment 
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 17 
group is defined as villages assigned to treatment for which we received a report from the 
CSO saying that a training related to the first of the two themes had taken place on a 
specific date. The control group remains the same. (Note that this alternative definition of 
the treatment group undoes the randomization in the sense that we do not know what the 
appropriate set of counterfactual units in the control group is (i.e., the villages in the control 
group for which the CSOs would have returned reports had they been assigned to 
treatment).) As in the previous tables, household respondents report public information 
meetings at equal rates (with around 30 percent in both groups saying that there had been 
such meetings in the past six weeks). 
Among those who report awareness of public information meetings in the last six weeks, 
those in the treatment villages are significantly more likely to say that they were about the 
judicial sector (32 percent versus 12 percent, as compared to 18 percent the total set of 
control villages in Table 4). The proportion of those who report that the meeting about 
which they were aware was a judicial sector meeting and said that they attended the 
meeting is approximately the same in this subset of treatment villages as in the overall set of 
treatment villages (52 percent in Table 13 versus 56 percent in Table 5).
Among elites, the patterns are similar. There is no difference in awareness of public 
information meetings across this subset of treatment villages as compared to the control 
villages: the 64 percent of village elites who say that there was a meeting in the last six 
weeks is about the same as the 62 percent in the overall treatment sample (Table 6). These 
elites, however, are somewhat more likely to say that the meetings were about the judicial 
sector: 28 percent in the trimmed treatment sample in Table 13 versus 18 percent in the 
overall treatment sample in Table 10. 
(3) TREATMENT EFFECTS FOR KNOWLEDGE INDEX 
To begin looking at outcomes, we examine the outcome most likely to have been influenced 
by the information meetings: respondents’ basic knowledge about the judicial sector and 
inheritance law. Table 14 lists the 12 questions based on the information sessions that were 
including in the endline survey. The correct answer is included in brackets. Enumerators 
recorded if the respondent provided a correct answer, an incorrect answer, said that they 
did not know, or refused to answer the question. We created a simple additive index of 
correct answers to these questions, giving respondents one point for each correct answer. 
Table 14. Questions Included in Knowledge Index.
1. Can you tell me the name of the court that has the final say over decisions made by all of the courts in 
the Malian court system? [Supreme Court]
2. Thinking about the Malian court system, do you know what the principal of collegiality is? [panels of 
[three] judges try cases]
3. Can you name any of the courts with special jurisdictions that exist in the Malian court system? [any of 
commercial courts, administrative courts, courts for children, courts of labor, or military courts]
4. If a business wanted to challenge the amount of taxes that it was paying, do you know to what type of 
court it would go? [administrative courts]
5. Is there a difference between a magistrate and a judge? [no]
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 18 
1. Can you tell me the name of the court that has the final say over decisions made by all of the courts in 
the Malian court system? [Supreme Court]
6. Who represents you in a court of law if you bring a case to court? [a lawyer]
7. Who keeps the records of the court? [the clerk (greffier)]
8. In what court can you appeal rulings made by the Court of First Instance? [Court of Appeals]
9. If a person dies, and there is a dispute among their relatives over the inheritance, where will the court 
with jurisdiction over the case be located? [in the place where the deceased had his home or in the 
place where the deceased died]
10. What are some grounds for a court excluding a challenge to an inheritance? [the petitioner killed the 
person who gave the inheritance or the petitioner has committed slander against the deceased]
11. What's the name of the document in which one describes how they want their belongings to be 
distributed after they die? [a will]
12. What law determines issues of inheritance? [the Family Code]
Table 15 reports a difference-in-means test across treatment and control villages for the 
knowledge index. Respondents in treatment villages performed no better (and perhaps 
somewhat worse) on the 12-item index than did respondents in control villages. Both had 
quite low levels of knowledge. One-third of the sample answered none of the questions 
correctly; another one-third answered only one question correctly. Even at the high end of 
the scale, respondents who scored well on the scale were evenly distributed between 
treatment and control villages: among the 68 people who scored 10 or better, 35 were 
located in control villages and 33 in treatment villages. 
Table 15. Treatment Effects for Knowledge Index. Difference-in-means test for 
12-item knowledge index across respondents from treatment and control 
villages.
KNOWLEDGE INDEX (0…12)
Treatment 1.63
(0.08)
N=941
Control 1.75
(0.08)
N=931
Difference -0.13
(0.11)
p < 0.28
IV. DISCUSSION 
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 19 
The results about the prominence of the public information campaigns are disappointing. 
The evaluation team designed the impact evaluation with the expectation that the 
information campaign would be a prominent community event and that the information 
relayed to the attendees would additionally diffuse into the community. None of the results 
reported in this memorandum suggest that this was the case. Survey respondents in control 
villages were generally as likely as respondents in treatment villages to report that there had 
been public information meetings in the six weeks before the endline survey. While elites in 
treatment villages were significantly more likely to report that there had been public 
information meetings, the difference was not as great as we would have anticipated ex ante. 
In towns with populations on the order of 1,000 residents or less, we would have expected 
nearly all of the village heads to be aware of the MJP information campaign sessions. 
The initial results are even more surprising when we look at the proportion of people 
reporting that there was a public information campaign about the judicial sector. Although 
the proportion is consistently greater – at least among household respondents – in the 
treatment villages as compared to the control villages, it is still only 18 percent of those who 
report being aware of a public information meeting (or about 6 percent of the overall 
sample in the treatment villages) who say that the meetings were about the judicial sector. 
This proportion is similar among village elites with 18 percent of those who report being 
aware of a public information meeting reporting that the meeting was about the judicial 
sector. (Since a higher proportion of village elites report any meeting awareness, this means 
that around 12 percent of village elites in treatment villages say that there were 
informational meetings about the judicial sector). When we look only at villages with 
populations below 1,000 (according to the 2009 census), we also find similar results among 
the elites. 
These findings about awareness run in contrast to those typically found in studies of 
information campaigns in similar settings (e.g., Gottlieb 2016 in Mali). The evaluation team 
expected the information meetings (a) to be arranged with the village head and other village 
elites, (b) to be widely publicized, and (c) to be well attended. That we do not see evidence 
of this being true and that we do not find effects of village-level assignment to the 
information campaign condition on meeting awareness makes it unlikely that we will identify 
impacts of the information campaign on other outcome variables. 
What might explain these null findings? We propose five possibilities: (1) limited reach of 
the information sessions; (2) limited time for information diffusion; (3) non-adherence to the 
randomization schedule; (4) a disjuncture between the description of the meetings used by 
the CSOs and our survey question wording; and (5) improper survey data collection. 
Exploring each in turn, we are not able to identify a single, dominant explanation for the lack 
of awareness about the information campaign and the lack of impact on the knowledge 
index. We think that the most likely explanations are provided by the first and third 
possibilities: that the information sessions were more limited in scope than what the 
evaluation team had envisioned in designing the evaluation and/or that the CSOs did not 
strictly adhere to the randomization schedule. 
(1) LIMITED REACH OF THE INFORMATION SESSIONS 
One possible explanation for the results presented here is that the information sessions 
simply had a very limited reach. If the information sessions were not widely publicized and 
the individuals who attended them appeared in the endline survey sample in only limited 
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 20 
numbers, then we would not observe significant differences between treatment and control 
villages in terms of awareness of the information sessions. Based on the reporting that we 
received from the CSO partners, attendance at the information sessions ranged from 
around a dozen attendees to 56, with a median attendance of 23. 
We would expect this explanation to apply in large communities. Only 10 percent of the 
study villages had 2009 populations above 5,000 – the median community in the study had a 
2009 population of around 1,200 people. While populations plausibly have increased since 
the 2009 census, we nonetheless believe that the majority of our study sites are small, rural 
villages where we would expect word to spread about information meetings and where we 
would expect the content of those information meetings to spread among a set of 
households that likely are closely connected in terms of day-to-day life. 
Given that the majority of communities in the study are small villages, we are surprised, in 
particular, that the levels of awareness of the information meetings were not higher among 
village elites. Looking at the set of 75 villages with 2009 populations below 1,000, only one 
village head in the treatment condition (and one village head in the control condition) 
reported that the had been public information meetings about the judicial sector held in the 
past six weeks in their village (Table 11). Were the information meetings organized in such 
a way that the village heads either did not know that they were happening or else did not 
know their topic? 
One possibility is that the attendees at the information sessions were a select set of 
residents with whom the CSOs had a prior connection. If this is the case, we might not 
expect awareness of the meetings or awareness of the information discussed in the 
meetings to be particularly widespread, since the meetings might be seen as typical activities 
for those who attended them, and their content might not be seen as something of any 
particular interest to non-attendees. 
We asked MJP to gather additional details from the CSOs about the conduct of the 
information campaign. APDF and CRADE returned detailed responses to our questions. 
Their responses were in line with our initial expectations and not with the results of the 
survey. 
APDF described how “many people participated [in the information sessions] without being 
invited” and implied that there was high community awareness about the meetings: “Certain 
localities were not very populated and considering the limited number [of information 
sessions], everyone wanted to know what was said.” APDF’s comments also implied that 
there should have been relatively little confusion between their meetings and any other 
meetings: “for the majority of people, it was their first time participating in any such 
meeting.”
CRADE likewise described procedures involving widespread mobilization for the meetings: 
“The villagers are informed about the meetings by paralegals and village councilors going 
door-to-door, at the mosque, and in meetings of women’s and youth groups.” CRADE 
talked about “passerbys who, out of curiosity, want to know what is going on or what is 
being said” joining the meetings and noted that attendance was greater during the second 
meetings because “people come just from what was said to them about the first session.”
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 21 
CRADE also emphasized the need to work with village elites “because no action can be 
undertaken or carried out without going through the village chief.” APDF also talked about 
village elites attending the trainings. 
The ways in which the two CSOs answered the evaluation team’s questions correspond to 
the evaluation team’s ex ante expectations. Yet, as described above, there was no 
significant difference in awareness between treatment and control communities in the areas 
covered by these two CSOs for household respondents. For village elites, on the other 
hand, there was greater awareness of information meetings in treatment villages and greater 
awareness that these meetings were about the judicial sector (although for treatment 
villages covered by APDF, it is less than one out of 10 village elites who said that the recent 
information session of which they were aware was about the judicial sector). 
(2) LIMITED TIME BETWEEN BASELINE, CAMPAIGN, AND ENDLINE 
Because these information campaign activities were timed for the end of the implementation 
year for the MJP, the MJP requested that the information meetings follow shortly after the 
baseline survey and that the endline survey follow shortly after the information meetings. 
Perhaps with additional time after the information meetings, awareness of the meetings and 
the information contained within them would have spread more widely among village 
residents, and we would have observed more awareness and more informational effects in a 
survey enumerated at a later date. 
We are somewhat skeptical of this explanation. The evaluation team anticipated that 
conducting the endline survey in a short timeframe after the information meetings would 
make it easier to observe differences across treatment and control villages in terms of 
awareness and that meeting attendees would be more likely to retain the information from 
the meetings, making it easier to observe significant treatment effects on the knowledge 
index. As described above, we did not anticipate that we would find relatively low levels of 
awareness of the meetings in treated villages. 
(3) NON-ADHERENCE TO RANDOMIZATION SCHEDULE 
As described above, we received reporting from the CSOs for only around half of the 
villages assigned to treatment, although the reports that we are missing are all from APDF 
villages. We also received reports about information sessions having happened in villages 
not included in the study and in one control village. If the CSOs did not adhere to the 
randomization schedule, we might expect to see patterns like those that we find in the 
household data question about awareness of public information meetings. 
That there was greater awareness of public information meetings among village elites in 
treatment villages and that public information meetings about which people were aware 
were more likely to be perceived as having been about the judicial sector in treatment 
villages relative to control villages suggests that villages assigned to treatment did, in fact, 
receive public information meetings to a greater extent than control villages did. 
We nonetheless, as described above, find it surprising that these numbers are not higher. 
We expected a priori that many elites – and certainly nearly all village heads – would be 
aware of the information sessions, as we expected that the CSOs would need to contact 
the village heads in order to conduct the information sessions (see the comments by 
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 22 
CRADE above). As posited above, if the CSOs conducted these information meetings in 
the context of ongoing routine activities, then they might not have been recognized as the 
kind of unusual public events that we had conceptualized them as when designing the 
evaluation, but the comments from APDF and CRADE above suggest that this was not the 
case: that many meeting attendees were having their first interaction with the CSO’s 
programming. 
(4) DISJUNCTURE BETWEEN DESCRIPTION OF MEETINGS USED BY CSOS AND 
QUESTION WORDING 
Were the information campaign meetings described and conducted in such a way that the 
question wording that we used (i.e., “public information meetings [réunions d'information 
publique]”) in the survey questions simply did not cue the MJP-sponsored activities in the 
minds of respondents? This might explain why respondents in treatment and control 
villages have an equal probability of saying that there was an information meeting, while 
those who say that there was an information meeting are then somewhat more likely in the 
treatment communities to say that the meeting was about the judicial sector. (Although 
among elites, those who say that there was a meeting are equally likely to say it was about 
the judicial sector in either treatment or control villages, and only one-out-of-10 does so.) 
No concerns were raised by GISSE during the enumerator training about these questions. 
After data collection was complete, we contacted an expert in Bambara linguistics to review 
the question; we also made contact with a Songhai speaker through this initial reference. 
These experts pointed to some ambiguities in the question wording but thought that the 
enumerators would have been able to make the intent of the question clear with proper 
tone and context. 
In the Bambara version, the question might be translated as “In the last six weeks, was 
meeting information given here in (this) town? Do you remember?” where the phrase might 
mean information from a meeting or information about a meeting. The Bambara expert said 
that this was not a commonly used expression but that it was not unlikely that people would 
make the connection between the phrase and the types of meetings conducted under the 
information campaign. 
In the Songhai version, the Songhai speaker translated “public information” as “literacy,” but 
noted that the word can mean “comprehension,” “clarification,” or “explanation” as well. 
He said that he guessed that we were asking about a community meeting. 
Part of the low rates of awareness, therefore, may be due to poor question wording, but 
even if this is the case, we would expect to see treatment effects insofar as some portion of 
the population is correctly understanding the questions. In addition, even if the awareness 
questions suffer from faulty wording and the information meetings happened at scale, we 
would still expect to see treatment effects on the knowledge questions, which we do not. 
(5) IMPROPER SURVEY DATA COLLECTION 
If the survey data were not collected in the assigned locations, the null results that we find 
here would not be surprising. The surveys were enumerated on tablets equipped with GPS, 
and we have checked the survey locations against detailed maps of Mali, and the 
enumeration coordinates match with what we would expect, except for cases where the 
USAID.GOV Tasking N032-DRG Impact Evaluation of USAID/Mali Justice Program | 23 
enumerators could not use the tablets and had to enumerate the surveys on paper (as 
happened in Taoudénit). GISSE, the survey firm, also was active in helping us diagnose the 
original list of villages for villages that no longer existed or that had unusual populations. 
We received no indication during survey preparation, enumerator training, survey 
enumeration, and data processing that they were not taking their work seriously.