IMPACT EVALUATION 
USAID/SOMA UMENYE ACTIVITY
AID-696-TO-16-00002
International Business & Technical 
Consultants, Inc. (IBTCI)
Edward Jay Allan, IBTCI, Project Director
Magda Raupp, IBTCI, Team Leader
Ali Protik, Mathematica Policy Research, Methodologist
Kenn Ndirangu, Incisive Africa, Data Manager
Nikita Ramchandani, Mathematica Policy Research, Research Analyst
April 26, 2018 1
IMPACT EVALUATION QUESTIONS
• 1)To what extent are changes in Kinyarwanda reading outcomes 
for students in Grades 1-3 attributable to the Soma Umenye 
activity (as a package)? 
• 2) To what extent has classroom instruction changed as a result 
of Soma Umenye? 
• 3) What is the evidence that improved student reading outcomes in 
Kinyarwanda lead to improved opportunities for students to succeed in 
schooling?
• 4)To what extent are the improvements in the systematic capacity for 
early grade reading instruction a result of the Soma Umenye activity?
April 26, 2018 2
Random Assignment
District
Phase 3 
Control
Phase 2 
Non-Study Group
Phase 1 
Treatment
*Big sectors have more than the average number of schools in the district. 
Small sectors have less than the average number of schools in the district.
Small Sectors*
Phase 3 
Control
Phase 2 
Non-Study
Group
Phase 1 
Treatment
Big Sectors*
 Sectors assigned to phases randomly
• In each district
• In the presence of district officials and sector representatives
April 26, 2018 3
Phase 3 
Control
168 Sectors
Phase 2 
Non-Study Group
112 Sectors
Phase 1 
Intervention
136 Sectors
Impact Evaluation Sampling
Intervention
School = 1 school/sector
Student = 18 
students/grade/school
Teacher = 1 teacher/school
Random 
sample
Intervention
School = 1 school/sector
Student = 18 
students/grade/school
Teacher = 1 teacher/school
Random 
sample
April 26, 2018 4
What and How We Measured
• P1 reading skills measured by the EGRA adapted 
to speakers of Kinyarwanda and to Rwanda’s 
educational scenario. 
• Teacher instructional behavior in the 
Kinyarwanda class measured by the Lesson 
Observation Protocol.
April 26, 2018 5
TREATMENT CONTROL TOTAL
SCHOOLS 136 168 304
STUDENTS 2,455 3,011 5,466
MALES 1,229 1,523 2,752
FEMALES 1,226 1,488 2,714
HEAD
TEACHERS
122 146 268
TEACHERS 132 155 287
LESSON 
OBSERV.
129 152 281
Data Sources
April 26, 2018 6
Characteristics of the Sample
April 26, 2018 7
19.7
30.4
72.7
31.9
11.5
5
15.7
64.3
20.2
31.3
76.5
31.4
9.5
5.2 15
62.9
0
10
20
30
40
50
60
70
80
90
Piped water Electric bulb Mobile phone Bicycle Motorcycle Car Television Radio
Treatment Control
EGRA Results: Timed Sub-tests 
April 26, 2018 8
22.19
12.11
4.51 4.98
19.41
11.19
3.72 4.27
0
5
10
15
20
25
Letter Identification (MAX 100
clpm)
Syllable identification (MAX 100
csilpm)
Familiar Word Reading (MAX 50
cwpm)
Reading Fluency (MAX 38 orf)
 Treatment Control
April 26, 2018 9
EGRA Results: Untimed Subtests
0.6 0.6
3.9
3.7
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Treatment Control
Reading Comprehension (MAX 5 questions) Listening Comprehension (MAX 5 questions)
3.35%
96.65%
80% correct (183 students)
correctly answered 4 out of 5
coprehension questions)
Not proficient readers (5,274
students)
April 26, 2018 10
Percentage of Proficient Readers 
Impact on Male and Female Students 
April 26, 2018 11
24.48
20.58 19.92
18.26
13.54
11.20 10.70 11.17
5.03 4.18 4.00 3.27
5.73 4.89 4.25 3.65
Females T Females C Males T Males C
clpm (max 100) csilpm (max 100) cfwpm (max 50) orf (max 38)
Impact on Repeaters and Non-Repeaters
April 26, 2018 12
24.32
20.79 21.35
18.88
12.39
11.55 12.01
11.05
4.72 4.21 4.81
3.54
5.44 4.72 4.43 4.09
Repeaters T Repeaters C Non-Repeaters T Non-Repeaters C
clpm (max 100) csilpm (max 100) cfwpm (max 50) orf (max 38)
Soma Umenye Impact on Rural Students
April 26, 2018 13
21.79
11.70
4.37 4.81
18.74
9.77
3.52 3.99
clpm (max 100) csilpm (max 100) cfwpm (max 50) orf (max 38)
Treatment Control
IMPACT EVALUATION QUESTION 
# 2
To what extent has classroom 
instruction changed as a result of 
Soma Umenye? 
April 26, 2018 14
Lesson Observation Scores
April 26, 2018 15
Total lesson 
observation raw 
score 20.07 17.88 .45**
Lesson preparation raw
score
3.01 3.02 .19
Teaching methodology 
raw score
5.5 5.34 .27
Literacy technical skills
raw scores
4.28 4.23 .06
Assessment raw score 3.81 2.85 .96*
Use of teaching 
materials raw score
3.47 2.54 .92**
DATA SUPPORTED 
CONCLUSIONS
April 26, 2018 16
# 1
Participation in the Soma Umenye activity 
has the potential to cause P1 students to 
improve their basic reading skills, particularly 
in letter and syllable recognition and familiar 
word reading. 
April 26, 2018 17
# 2
The number of words correctly read per 
minute in connected text was not sufficient 
to allow P1 students to understand what 
they read. This is the key reason for the very 
low percentage of students able to answer 
the comprehension questions.
April 26, 2018 18
# 3
Male and female students in treatment 
schools benefitted equally from the Soma 
Umenye activity and both groups scored 
higher than their counterparts in the control 
schools.
April 26, 2018 19
# 4
One interpretation of the large concentration of scores 
at the upper limit of the distribution in the listening 
comprehension sub-test (average score of 3.91 out of a 
possible 4.0) is that even P1 students who have not yet 
learned to decode have oral language, vocabulary, and 
comprehension skills that allow them to answer 
questions correctly. It is also possible that the sub-test 
was too easy and produced a ceiling effect.
April 26, 2018 20
# 5
While participation in Soma Umenye may 
contribute to improved student academic 
achievement and/or improved efficiency down 
the road, no impact was observed in the rates of 
tardiness and absenteeism during the first year 
of implementation.
April 26, 2018 21
# 6
The low scores on the EGRA sub-tests that measure 
pre-reading skills (on the average 20 letters 
recognized out of 100; 16 syllables out of 100; and 
fewer than five familiar words out of 50) suggest that 
students are not receiving sufficient focused 
classroom instruction and practice to master these 
skills and become fluent readers by the end of P3. 
Fluent readers as defined by the EGRA Toolkit (2nd edition, March 2016) “Students are considered fluent readers if 
they read the entire passage in one minute or less and can answer 80 percent of the reading comprehension questions 
correctly.” (3.2 out of 4 questions in this particular case).
April 26, 2018 22
# 7
It is encouraging that the instructional 
behavior of the 129 teachers who received 
Soma Umenye training was half a standard 
deviation higher than that observed for the 
152 control teachers.
April 26, 2018 23
# 8
Learning is limited by tardiness and the high 
rate of absenteeism. Over half of the P1 
students self-reported having been absent for at 
least one day the previous week – and by actual 
count against the class rosters, over 20 percent 
of enrolled students were absent the day their 
class was assessed.
April 26, 2018 24
RECOMMENDATIONS
April 26, 2018 25
# 1
In the early grades, especially in P1, focus 
on pre-reading or foundational skills—
phonological awareness and alphabetic 
principles, coupled with practice in 
applying these skills to the decoding of 
syllables and words. 
April 26, 2018 26
# 2
Revisit the EGRA tool used to assess 
students’ Listening Comprehension 
skills.
April 26, 2018 27
# 3
Focus on early identification of slow or 
immature learners and provide extra 
assistance to avoid leaving students 
behind. The repetition rates observed at 
P1 should be seen as unacceptable.
April 26, 2018 28
# 4
Focus on pre-service training and on what 
future teachers are learning and 
practicing while still at teachers’ 
preparation colleges. Large scale in￾service training is not a sustainable model 
once project funds are no longer available. 
April 26, 2018 29
# 5
Make the development and 
implementation of benchmarks and 
reading standards for grades P1 – P3 a 
priority shared by students, teachers, head 
teachers and community members.
April 26, 2018 30
QUESTIONS 
and 
COMMENTS
April 26, 2018 31