QUALITY 
ASSURANCE 
PROJECT
Center for Human Services • 7200 Wisconsin Avenue, Suite 600 • Bethesda, MD 20814-4811 • USA
EVALUATION REPORT
The Niger QAP/BASICS Joint Project
Final Evaluation
Stephane Legros, MD, MPH
Youssef Tawfik, MD, MPH
Xavier Crespin, MD, MPH
Mamoudou Djingarey, MD, MPH
Elizabeth Goodrich
Hany Abdallah, MHS
The Niger
QAP/BASICS Joint Project
Final Evaluation
Quality Assurance Project
7200 Wisconsin Avenue, Suite 600
Bethesda, MD 20814-4811
301-654-8338, FAX 301-941-8427
www.qaproject.org
© Center for Human Services (CHS), 2000
Acknowledgements
The evaluation team is indebted to the staff of the QAP/
BASICS project for their technical assistance throughout
the mission and for the secretarial help, transport, and
equipment they made available throughout the mission.
The team also appreciates the willingness of the local
provincial and district health authorities in the Tahoua
and Dosso Departments to be available at all times.
The evaluation team acknowledges the support provided
by the USAID Mission in Niamey in funding this final
evaluation and thanks the Agency for providing the
Quality Assurance Project with the opportunity to
contribute to the success of the QAP/BASICS project in
the Tahoua and Dosso Regions of Niger. The evaluation
team’s work was supported through the Center for
Human Services (USAID Contract No. HRN-C-00-96-
90013) and the BASICS project (USAID Contract No.
HRN 6006-C-00-3031-00).
Recommended citation
Legros, S., Y. Tawfik, X. Crespin, M. Djingarey, E.
Goodrich, and H. Abdalah. 2000. The Niger QAP/
BASICS Project: Final evaluation. Evaluation Report
1(1). Published for the United States Agency for
International Development (USAID) by the Quality
Assurance Project (QAP): Bethesda, Maryland.
About this series
The Evaluation Report series presents the findings,
recommendations, and lessons learned of completed
quality assurance (QA) evaluations in countries with
long-term QA programs. An electronic copy of this
publication may be found at qapdissem@urc-chs.com.
Table of Contents
List of Tables and Figures .............................................................................................................................................. iv
List of Appendices and Appendix Maps, Tables, and Figures.......................................................................................... v
Abbreviations and Acronyms......................................................................................................................................... vi
Abstract ....................................................................................................................................................................... viii
Executive Summary ....................................................................................................................................................... ix
I. Introduction ............................................................................................................................................................. 1
1. QAP/BASICS Project Activities......................................................................................................................... 1
2. Pre-Project Assessment....................................................................................................................................... 2
3. Results Summary ................................................................................................................................................ 2
4. Cost Recovery .................................................................................................................................................... 3
II. Final Evaluation: Objectives of the QAP/BASICS Project ....................................................................................... 4
1. Evaluation Background ....................................................................................................................................... 4
2. Evaluation Summary ........................................................................................................................................... 5
3. Evaluation Method ............................................................................................................................................. 5
4. Evaluation Report Structure ................................................................................................................................6
5. Limits of the Project Evaluation .......................................................................................................................... 6
III. Institutionalization of the QA Concept (TAP Objective Number 1) .......................................................................... 7
1. Standards: Background ....................................................................................................................................... 7
2. Implementing and Monitoring Standards ............................................................................................................ 8
3. Quarterly Meetings ........................................................................................................................................... 10
4. QA Cycles and Process Improvement ............................................................................................................... 11
5. Process Design/Redesign .................................................................................................................................. 15
6. Results Analysis................................................................................................................................................ 15
7. Results from Case Studies................................................................................................................................. 17
8. Results from Focus Groups ............................................................................................................................... 21
IV. QAP/BASICS: Applying QA Measures to Introduce IMCI (TAP Objective Number 2) ........................................ 22
1. Development and Communication of IMCI Guidelines..................................................................................... 22
2. Local Implementation ....................................................................................................................................... 23
3. Applying QA Measures to Introduce IMCI ....................................................................................................... 25
V. Disseminating the Techniques and Results of QA ................................................................................................. 31
1. Dissemination during QAP ............................................................................................................................... 31
2. International Dissemination of QAP/BASICS ................................................................................................... 31
3. QA/IMCI Dissemination ................................................................................................................................... 31
4. Niamey Conference, October 26-28, 1998 ........................................................................................................ 31
VI. Costs of the QAP/BASICS Project Activities......................................................................................................... 32
1. Approach .......................................................................................................................................................... 32
2. Basic QA and IMCI Training ............................................................................................................................ 33
3. Training of Trainers .......................................................................................................................................... 34
4. Team-Based Problem Solving ........................................................................................................................... 34
5. Supervision and Coaching Activities................................................................................................................. 35
6. Supervision Checklist ....................................................................................................................................... 37
7. Recommendations Relating to Supervisory and Coaching Costs ....................................................................... 37
8. Quality Meetings .............................................................................................................................................. 37
9. Cost of Sustaining Essential QA Activities........................................................................................................ 38
VII. Conclusions/Recommendations ............................................................................................................................. 38
1. Conclusions ...................................................................................................................................................... 38
2. Lessons Learned ............................................................................................................................................... 39
3. Main Factors Limiting QA ................................................................................................................................ 40
4. Recommendations ............................................................................................................................................ 40
Appendices ................................................................................................................................................................ A-1
iv
List of Tables and Figures
Table 1-1. Baseline Indicators: Tahoua Department (SNIS 1993) versus Year 2000 National Objectives...................... 3
Table 1-2. Key Data from the QAP/BASICS Project .................................................................................................... 4
Table 2-1. TAP Expectations Compared to Activities ................................................................................................... 5
Table 2-2. Timetable of Major Healthcare Developments ............................................................................................. 6
Table 3-1. Existing Standards for Patient Management and CSI Administration ........................................................... 7
Table 3-2. Supervisory System Strengths and Weaknesses ............................................................................................ 9
Table 3-3. Strengths and Weaknesses of the Supervision Checklist ............................................................................. 10
Table 3-4. Examples of Problem-Solving Activities.................................................................................................... 11
Figure 3-1. Review of the Quality Assurance Cycle ..................................................................................................... 11
Table 3-5. Workers Trained in All QA Techniques (1993–1998) ................................................................................ 11
Table 3-6. Individuals Trained in Basic QA by Year and Center ................................................................................. 12
Table 3-7. Number of Cycles per Year and Percentage Abandoned ............................................................................. 13
Table 3-8. Cycle Distribution According to PMA Activity.......................................................................................... 13
Table 3-9. PMA Indicators: 1993 to 1997 (Tahoua Department; in Percentages) ........................................................ 16
Table 3-10. Comparison of QA CSI and Control CSI ................................................................................................... 17
Figure 3-2. Factors Influencing or Interacting with the CSI Center’s Utilization (Tahoua Department) ......................... 17
Figure 3-3. Evolution of Prenatal Consultation Indicators within a 5-Km. Radius
(Tahoua Department; 1993 to 1996) .......................................................................................................... 18
Figure 3-4. Evolution of Infant Consultation Coverage Indicators within a 5-Km. Radius
(Tahoua Department; 1993 to 1996) .......................................................................................................... 18
Figure 3-5. Evolution of the Rate of Family Planning Utilization within a 5-Km. Radius
(Tahoua Department; 1993 to 1996) .......................................................................................................... 18
Figure 3-6. Evolution of the CRENA Drop-Out Rates within a 5-Km. Radius
(Tahoua Department; 1993 to 1996) .......................................................................................................... 19
Figure 3-7. Evolution of the BCG Coverage Rate for Infants up to 11 Months ............................................................. 19
Table 4-1. Training Courses: Dates and Results.......................................................................................................... 23
Table 4-2. MEG Stocks before and after Cost Recovery (Konni) ................................................................................ 23
Table 4-3. MEG Stocks before and after Cost Recovery (Boboye) ............................................................................. 24
Figure 4-1. Utilization of Services in Konni District (1997 and 1998; January 1 to June 30) ........................................ 24
Figure 4-2. Utilization of Services in Boboye District (1996, 1997 and 1998; January 1 to June 30) ............................ 24
Figure 4-3. Utilization of Services in the Dosso CSI (1997 and 1998, January 1 to June 30) ........................................ 25
Figure 4-4. Utilization of Services in Badaguichiri CSI (First Three Quarters of 1997 and 1998)................................. 25
Table 4-4. Fee Structure in CSI Badaguichiri .............................................................................................................. 26
Table 4-5. Results of ERPAs Conducted in 1997 and 1998 (in Percentages) ............................................................... 27
Table 4-6. Strengths and Weaknesses of the ERPA ..................................................................................................... 28
Figure 4-5. IMCI Forms at Three CSIs in the Boboye District ..................................................................................... 29
Figure 4-6. IMCI Forms at Three CSIs in the Konni District (September and October 1998) ....................................... 29
Figure 6-1. Training Costs for QA and IMCI (US Dollars) .......................................................................................... 33
Figure 6-2. Breakdown of QA Training Costs (Percentages) ........................................................................................ 33
Figure 6-3. Breakdown of IMCI Training Costs (Percentages) ..................................................................................... 34
Table 6-1. Average Incremental Cost of Training Trainers in QA ................................................................................ 34
Table 6-2. Average Total Recurrent Cost of QA Problem-Solving Activities (Estimated; US Dollars)......................... 35
Table 6-3. Effects of Various Factors on Average Cost of a Problem-Solving Cycle .................................................... 35
Table 6-4. Cost of Supervisory Checklist (US Dollars) ............................................................................................... 37
Table 6-5. Estimated Average Cost of QA-Related Meetings (US Dollars) ................................................................. 37
Table 6-6. Average Annual Cost of Essential QA and IMCI Activities (Tahoua, US Dollars) ...................................... 38
Table 7-1. Recommendations ..................................................................................................................................... 41
v
List of Appendices and Appendix Maps, Tables, and Figures
Appendix A: Niger Overview ..................................................................................................................................... A-1
Map A-1. Niger ................................................................................................................................................ A-1
Table A-1. Ethnic Groups in Niger..................................................................................................................... A-2
Figure A-1. Diagram of the Healthcare System in Niger ...................................................................................... A-3
Table A-2. Development Indicators.................................................................................................................... A-3
Table A-3. Epidemiological Data ....................................................................................................................... A-4
Table A-4. Immunization Rates (1997) .............................................................................................................. A-4
Appendix B: Visit Schedule and Comments ................................................................................................................ A-5
Table B-1. Schedule of Visits of the Evaluation Team ........................................................................................ A-5
Table B-2. Comments on Health Facilities Visited by the Evaluation Team ........................................................ A-6
Appendix C: Utilization of Services ............................................................................................................................ A-7
Table C-1. Utilization of Services in Konni ........................................................................................................ A-7
Table C-2. Utilization of Services in Boboye ..................................................................................................... A-7
Table C-3. Utilization of Services at Urban CSI Dosso Toudobou ...................................................................... A-7
Table C-4. Utilization of Services at Dosso CSI of Badaguichiri ........................................................................ A-7
Appendix D: Analyses of IMCI Form Utilization ........................................................................................................ A-8
Table D-1. Analysis of IMCI Forms in Three CSIs in Boboye ............................................................................ A-8
Table D-2. Analysis of IMCI Forms in Three CSIs in Konni .............................................................................. A-8
Appendix E: Questionnaire Used by the Evaluation Team ........................................................................................... A-9
Appendix F: Details of QA Training ......................................................................................................................... A-14
Table F-1. Number of QA-Trained Health Providers by District ...................................................................... A-14
Table F-2. QA Training Sessions (Dates, Types, Locations, Number Trained) ................................................. A-14
Appendix G: Summary of Problems Selected and Outcomes: Tahoua and Dosso CSI Centers (1993-98) .................. A-15
Appendix H: Costs of QA Activities ......................................................................................................................... A-16
Table H-1. In-Country Costs Associated with the Joint QAP/BASICS Program (US Dollars) .......................... A-16
Table H-2: Details of Training Costs Associated with Quality Assurance (in CFA)........................................... A-16
Table H-3: Breakdown of Training Costs Associated with IMCI (in CFA)........................................................ A-17
Table H-4: Details of Estimated QA Training of Trainer (TOT) Cost ............................................................... A-17
Table H-5: Cost Components of Team-Based Problem-Solving Techniques (US Dollars) ................................ A-17
Table H-6: Labor Costs Associated with a Standard Problem-Solving Cycle (US Dollars) ............................... A-17
Table H-7: Annual Cost of Coaching Activities (1997: US Dollars) ................................................................. A-18
Table H-8: Annual Cost of Supervision Activities (1997: US Dollars) .............................................................. A-18
Table H-9: Cost of Coaching Activities (US Dollars) ....................................................................................... A-19
Table H-10: Annual Cost of Supervision Activities (US Dollars)........................................................................ A-19
Table H-11: Cost of Travel: Round-Trip between CSIs and District Headquarters.............................................. A-20
vi
Abbreviations and Acronyms
Acronym Abbreviation Translation (where appropriate)
ASV Agent de Santé Villageois Village Health Worker
BASICS Basic Support for Institutionalizing Child Survival
CCM Combined Case Management
CHD* Centres Hospitalier Départmentaux Departmental Hospital Centers
CimeFor Circonscription Médicale de Formation et de Recherche Medical Division for Training and Research
CN Consultation Nourrisson Infant Consultation
CNSF* Centre National de Santé Familiale National Family Health Center
CPN Consultation Prénatale Prenatal Consultation
CRENA Centre de Récupération et d’Education Nutritionnelle Nutritional Training and Rehabilitation Center
Ambulatoire
CRQ Conseil Régional de Qualité Regional Quality Council
CSI* Centre de Soins Intégrés Integrated Healthcare Center
DDS* Direction Départmentale de la Santé Regional Health Directorate
DS* District Sanitaire Health Care District
EAQ Equipe d’Assurance Qualité Quality Assurance Team
ECD Equipe Cadre de District District Supervision Team
ENSP* Ecole Nationale de Santé Publique National School of Public Health
EPS Education pour la Santé Health Education/Training
ERPA Evaluation Rapide de la Performance des Agents Rapid Performance Assessment of Health Workers
ESV* Equipe de Santé Villageoise Village Health Care Team
IMCI Integrated Management of Childhood Illnesses
LANSPEX* Laboratoire National de Santé Publique d’Expertise National Public Health Laboratory
MEG Médicaments Essentiels Génériques Essential Generic Drugs
MPH* Ministry of Public Health
ONPPC* Office National des Produits Pharmaceutiques National Office for Pharmaceutical and Chemical
et Chimiques Products
PEV Programme Elargi de Vaccination Extended Immunization Coverage
PF Planning Familial Family Planning
PMA Paquet Minimum d’Activités Basic Package of Services
PSF Pharmaciens Sans Frontiéres Pharmacists Without Borders
vii
Acronym Abbreviation Translation (where appropriate)
QA Quality Assurance
QAP Quality Assurance Project
SMI Santé Maternelle-Infantile Mother and Child Health
SNIS* Système National d’Information Sanitaire National Health Information System
SPT Stratégies Plainte-Traitement Complaint/Treatment Strategies
SRO Traitement de Réhydration par Voie Orale Oral Rehydration Solution
STD Sexually Transmitted Disease
VAD Visite à Domicile House Call
VAT Vaccin Antitétanique Antitetanus Vaccine
WHO World Health Organization
* These entities are described more fully in Appendix 1, Section 2
viii
Abstract
This document reports on the final evaluation of the
joint Quality Assurance/Basic Support for Institutional￾izing Child Survival (QAP/BASICS) project in Niger,
West Africa. The evaluation took place in from October
25 to November 18, 1998.
The Quality Assurance Project (QAP) began introduc￾ing methods and tools for quality monitoring and quality
improvement of district healthcare services in the
Tahoua Department of Niger in 1993. In 1996 quality
assurance was extended to the Boboye District in the
Dosso Department to support the introduction of
Integrated Management of Childhood Illnesses (IMCI:
a program supported by the World Health Organization
[WHO] to improve and standardize healthcare services
for children). QAP and BASICS merged in Niger in
1997, and the resulting joint project supported the intro￾duction of IMCI in the Konni District in Tahoua.
The evaluation team consisted of an expert from QAP,
Center for Human Services; a specialist from BASICS;
two from the Niger Ministry of Public Health, Niamey;
and one from the U.S. Agency for International Devel￾opment (USAID)/Niamey. The team visited two
regional health headquarters, six district health head￾quarters, and 13 integrated health centers (CSIs). The
team interviewed health officials and international part￾ners, reviewed all relevant documents, and attended the
October 1998 International QA Conference in Niamey,
where the joint project’s outcomes were presented.
The goals of the evaluation were to: (a) assess the
degree of QA institutionalization and project achieve￾ments against anticipated results and (b) summarize
lessons learned from introducing IMCI in a QA envi￾ronment. The team found that:
■ The joint project achieved over 90 percent of its
objectives as detailed in its Technical Action Plan
(TAP) and, in some cases, even exceeded
expectations
■ By taking advantage of the presence of QAP and
BASICS in Niger, a unique opportunity was realized
for USAID and its national partners. The merger
reduced costs, and benefits were also realized locally,
regionally, nationally, and internationally by applying
QA measures to support the introduction of IMCI
■ The QAP/BASICS project has trained 400 health
workers in quality assurance skills. At the time of the
evaluation, over 77 quality improvement teams had
formed and had worked on about 120 problem-solv￾ing cycles in areas such as prenatal care, family plan￾ning, immunization coverage, and nutritional
rehabilitation
■ The project introduced a number of management
tools that contribute to the support of IMCI. The new
management framework used by healthcare workers
and their managers provides indicators and a new
environment for managing health organizations by
allowing health workers to change their behavior and
improve their performance. Staff motivation and
interest increased through team activities and the task
follow-up process
ix
Executive Summary
This document reports on the final evaluation of the
joint Quality Assurance/Basic Support for Institutional￾izing Child Survival (QAP/BASICS) project in Niger,
West Africa. The evaluation took place in from October
25 to November 18, 1998.
The evaluation occurred after several changes in
Niger’s healthcare system were implemented: From
1993 to 1998, the Quality Assurance Project (QAP)
worked in the Tahoua Department of Niger to improve
the quality of healthcare. This included the introduction
of methods and tools for quality monitoring and quality
improvement of district healthcare services.
In 1995, an independent program called BASICS
(Basic Support for Institutionalizing Child Survival)
supported the Ministry of Public Health (MPH) in
introducing Combined Case Management (CCM) for
children in primary healthcare facilities. CCM contrib￾uted to the revision of national policies for diarrheal
diseases, acute respiratory infection, and malaria, and
assisted in reducing missed opportunities to identify
sick children.
In 1996, QAP extended quality assurance (QA) meth￾odologies to the Boboye District in the Dosso Depart￾ment to support the introduction of the Integrated
Management of Childhood Illness (IMCI), a program
developed by the World Health Organization (WHO).
In 1997, the U.S. Agency for International Develop￾ment (USAID) merged QAP and BASICS activities in
Niger to support the introduction of IMCI in a QA
environment, beginning in the Konni District in the
Tahoua Department.
The evaluation team consisted of an expert from QAP,
Center for Human Services; a specialist from BASICS;
two from the MPH; and one from the USAID/Niamey.
The goals of the evaluation were to: (a) assess the de￾gree of QA institutionalization and QAP/BASICS
project achievements against anticipated results and (b)
summarize lessons learned from introducing IMCI in a
QA environment.
At the start of the evaluation, the team attended the
USAID-funded International QA Conference in
Niamey, where the results of the QAP project were
presented. Afterwards, they reviewed all relevant
documents from the project; visited two departmental
healthcare centers, six healthcare districts, and 13 inte￾grated healthcare centers, interviewing individuals from
all the centers, as well as QAP, BASICS, and other
international staff. The team also conducted two focus
groups of women who used the healthcare facilities.
The following summarizes some of the joint project’s
most significant achievements. Greater detail is in the
report.
Development of Standards
Various and numerous standards have been imple￾mented in Niger since the early 1990s, some with QAP/
BASICS support. The evaluation team discovered that
it was difficult for health workers to manage the high
number of clinical standards, which include child
healthcare; family planning; clinical procedures for
tuberculosis, malaria, and malnutrition; Complaint/
Treatment Strategies; and, later, IMCI.
Some of the clinical protocols that were being used
had different recommendations for the management of
illnesses, depending on which organization created
them and how. This caused confusion and lack of com￾prehension on the part of practitioners. QAP assisted
the Nigerien healthcare system in developing a standard
management procedures manual for the integrated
healthcare centers (CSIs).
While CCM was controversial when it was first intro￾duced in 1995, three years before IMCI, the evaluation
team concluded that CCM had paved the way for
IMCI.
Supervision
The project created trained supervision teams that can
provide technical support to health facility staff. There
has been limited success in achieving regular supervi￾sory visits of district staff by the department-level
Health Directorate (DDS).
QAP/BASICS also created a comprehensive supervi￾sion checklist. While this tool needs to be streamlined,
it helps in conducting all-day supervisory visits. It
brought about an important behavior change among
supervisors: They accepted the checklist despite its
weightiness, and its use improves health worker perfor￾mance.
The concept of coaching facilitated the transformation
of supervision from the traditional authoritarian style to
a supportive problem-solving approach.
x
Quarterly Meetings
Quarterly meetings of two to three days’ duration were
established to permit a regular exchange of information
between different levels in the department or district.
This allows the teams to regularly present their work
and results. The quarterly meetings facilitate: (a) good
communication between department staff and the
District Health Management Team (DHMT, staff who
have taken oversight of QA activities as part of their
regular responsibilities), (b) benchmarking1 by the
problem-solving teams, and (c) ongoing training
through feedback provided at the meetings. The meet￾ings also increase health workers’ motivation.
Problem Solving
Though the problem-solving technique is rarely focused
on clinical skills, it has increased the ability to identify
problems and find solutions within healthcare staff in
Tahoua and Dosso. Collecting, using, and analyzing
data and brainstorming became common practices at the
health facility and district levels. The problem-solving
approach and specific QA tools (such as the selection
matrix and flowchart) are now well known. The project
encourages health workers to address problems at the
local level first, using available means.
The focus on the patient/client changed many habits.
Problem-solving sessions now target users’ needs,
requests, and satisfaction levels. However, provider
satisfaction is not truly considered in attempts to
improve quality.
An improvement made possible through the joint
project was the development of a management
framework with appropriate tools and indicators. This
provides a new environment for managing health struc￾tures and facilitates improvements in health worker
behavior. The motivation and consideration of staff
have increased both from team problem-solving activi￾ties and from the sense that their ideas and efforts will
receive follow-up. QAP/BASICS has created or con￾tributed to teamwork, morale, and motivation in Niger’s
health system.
Process Design/Redesign
The team observed tentative success in process design/
redesign, particularly with regard to offering all
healthcare services during a single visit and in the
implementation of postnatal consultations. However,
this is an area where opportunities for improvement
abound, and strengthening this aspect of QA in Niger
would significantly and positively impact the delivery
of healthcare.
Cost Recovery
Cost recovery has clearly succeeded in improving the
availability of essential drugs at health facilities. Since
its introduction, there is ample evidence of fewer stock￾outs and higher average monthly consumption rates.
However, cost recovery has been associated with at
least temporary drops in health facility utilization rates,
including visits by children less than five years old.
Implementation of IMCI and Related
Issues
The team found many areas that need improvement for
IMCI to succeed. It learned that the IMCI working
group had decided not to provide chronometers or
routinely use IMCI forms (because of inadequate finan￾cial resources). The evaluation team suggests that the
working group reconsider this decision, because the
lack of these essential items, especially IMCI forms,
could be detrimental to IMCI’s success.
In addition to the IMCI form, health workers have to
complete four other records for each sick child: the
National Health Information System (SNIS) daily tally
notebook, the Daily Register Book, the Cost Recovery
Register and the Card of Care. These multiple records
contribute to health workers’ inability to complete the
IMCI form for each child.
The team also learned that health workers and caretak￾ers have difficulty complying with IMCI standards
regarding the referral of very sick children and the need
for patients to return to the health facility for follow-up
visits. In addition, both the Konni and Boboye district
hospitals are not prepared to act as effective referral
centers due to a lack of equipment.
Furthermore, limited exit interviews with mothers sug￾gested the need for improvements in the explanations
they receive regarding the use of prescribed medicine
and nutritional counseling.
Problems in the cold chain are eroding IMCI’s benefits.
While health staff are capable of checking children’s
vaccination status, children identified as needing vacci￾nations are not being vaccinated immediately in many
health facilities.
xi
The Rapid Assessment of Health Worker Performance
(ERPA), implemented by the QAP/BASICS project,
was designed to monitor healthcare worker perfor￾mance with IMCI. It proved valuable in improving
performance through the timely feedback of results.
However, using it is time-consuming, and sustaining
routine supervisory visits will be challenging, let alone
this additional evaluation activity.
Cost Analysis
The evaluation team conducted an analysis of the
project activities’ recurrent costs. The joint QAP/
BASICS project in Niger presented an opportunity to
estimate and understand—at a programmatic level—the
resources required to launch and operate a quality man￾agement system throughout an entire regional
healthcare system. The team conducted a cost analysis
to determine the cost of operating and sustaining
various QAP/BASICS activities essential to a QA and
IMCI program. These activities include:
■ Training in QA and IMCI methodologies
■ QA training of a local cadre of trainers
■ Implementation of team-based problem solving at
health facilities
■ Supervision and coaching, including the implementa￾tion of a supervision checklist
■ District and regional QA meetings
On average, the annual cost of sustaining these essential
QA and IMCI activities for a district the size of Tahoua
was projected to be approximately 3.1 cents per capita
in direct costs. The team also determined estimates of
individual essential QA/IMCI activities.
Findings and Recommendations
Assessing the degree of QA institutionalization and
lessons learned from introducing IMCI in a QA
environment, the team found that:
■ The joint project achieved over 90 percent of its
objectives as detailed in its Technical Action Plan
(TAP) and, in some cases, even exceeded
expectations
■ By taking advantage of the presence of QAP and
BASICS in Niger, a unique opportunity was realized
for USAID and its national partners. The merger
reduced costs, and benefits were also realized locally,
regionally, nationally, and internationally by applying
QA measures to support the introduction of IMCI
■ The QAP/BASICS project has trained 400 health
workers in quality assurance skills. At the time of the
evaluation, over 77 quality improvement teams had
formed and had worked on about 120 problem￾solving cycles in areas such as prenatal care, family
planning, immunization coverage, and nutritional
rehabilitation
■ The project introduced a number of management
tools that contribute to the support of IMCI. The
management framework provides indicators and a
new environment for managing health organizations
by allowing health workers to change their behavior
and improve their performance. Staff motivation and
interest increased through team activities and the task
follow-up process
The evaluation team developed 56 recommendations
and organized them according to government level—
ministry, department, and district—and further orga￾nized them according to area, such as IMCI, coaching,
and sustainability. They appear in Table 7-1 at the
conclusion of the report. (Some recommendations are
repeated in the table because they affect more than one
level.) To highlight some recommendations relating to
specific areas of study by the evaluation team (such as
improving IMCI and reducing training costs), several
recommendations also appear in the discussions of
those specific areas.

1
The Niger
QAP/BASICS Joint Project
Final Evaluation
1
 For background information on Niger, see Appendix A.
I. Introduction
1. QAP/BASICS Project Activities
Quality assurance (QA) has gained widespread accep￾tance as a necessary goal of healthcare systems, particu￾larly in the context of health sector reforms that seek to
deliver cost-effective, equitable, high-quality services to
priority populations. While many countries have articu￾lated goals relating to healthcare quality, the develop￾ment of structures and processes to support QA activi￾ties at the service delivery level is often limited.
1.1 QAP Scope and Objectives
The U.S. Agency for International Development
(USAID) created the Quality Assurance Project (QAP)
in 1990 to improve the quality of health, population,
and nutrition services in developing countries through
technical support to service provider institutions,
USAID Missions, and field-based cooperating agen￾cies. The Center for Human Services (CHS), the non￾profit affiliate of the University Research Co., LLC
(URC), has operated QAP since its inception.
In its first five years, QAP developed national and
provincial quality assurance programs and stimulated
QA interventions at the national, regional, and local
levels in 16 countries and conducted training activities
in another eight. During its second five-year phase,
QAP is demonstrating the cost-effectiveness of QA
interventions by establishing sustainable QA programs
at the country level, undertaking operations research,
and establishing accreditation programs and other
regulatory mechanisms.
1.2 Tahoua QA Project
The Tahoua QA project began in April 1993.
Through this project, USAID/Niger2 sought to
provide the government of Niger with both techni￾cal and operational assistance to improve the deliv￾ery of critical primary healthcare services in one
demonstration region by institutionalizing and
integrating a QA system. A QAP resident advisor
directed in-country activities and worked directly
with the Tahoua Health Director, his staff, and
district healthcare personnel to accomplish project
objectives.
The Tahoua project was designed with four main
objectives:
■ Identifying priority programs and determining
the resources necessary to deliver a basic pack￾age of services (PMA)
■ Introducing a QA system of management to
healthcare delivery through QA training, clarify￾ing and communicating clinical and management
standards, and monitoring and implementing a
process for preventing and correcting problems
■ Improving the quality of family healthcare
services by strengthening core support services,
increasing effective coverage of the population
with preventive services, and improving case
management of the most prevalent conditions
threatening the well-being of women and
children
■ Demonstrating the effectiveness and feasibility
of quality management strategies by assessing
the application of QA interventions, analyzing
their impact, and disseminating findings
2
1.3 The BASICS Project
The Basic Support for Institutionalizing Child Survival
(BASICS) project is an international public health
project started in 1993 and ending in 1999. USAID
funded BASICS with the purpose of providing technical
leadership to reduce child mortality and illness world￾wide. The project applies cost-effective child survival
interventions in six programmatic areas:
■ Sustaining immunization programs
■ Integrated Management of Childhood Illness (IMCI)
■ Strengthening the link between nutrition and health
■ Promoting and sustaining healthy behaviors
■ Improving techniques for monitoring and evaluation
■ Establishing public/private partnerships
The BASICS technical agenda is driven by a model—
the Pathway to Child Survival—that identifies a wide
range of factors, inside and outside the home, that play
a critical role in a child’s well being. The Pathway rec￾ognizes the essential role of mothers as a primary source
of disease prevention and early disease recognition. It
reveals that the child’s survival depends on a number of
factors, including the mother’s ability to adopt certain
behaviors, such as breast-feeding and the vaccination of
their children. It highlights the mother’s key role in the
early detection of child sickness and care seeking. It
emphasizes the importance of the quality of care
received and health workers’ ability to detect danger
signs and take correct and quick action.
BASICS-supported programs in over 30 countries use
the Pathway to Child Survival to support policies and
interventions aiming at improving mothers’ and commu￾nity behaviors, improving the quality of care provided
by public and private health providers, and building the
capacity of health systems and local communities to
sustain child survival interventions.
BASICS supported the Niger’s Ministry of Public
Health (MPH) in applying Combined Case Management
(CCM) for children in primary healthcare facilities.
CCM contributed to updating national policies for diar￾rheal disease, acute respiratory infection, and malaria. It
assisted in reducing missed opportunities to identify sick
children and was a prelude to the introduction of IMCI.
1.4 Description of QAP/BASICS Activities in
Niger
Since early 1997, the joint QAP/BASICS project inter￾ventions in Niger focused on the possibility of revising
the approach to the improvement of child healthcare
services at the district level. The Konni District
(Tahoua) and the Boboye District (Dosso) were
selected for QA/IMCI project interventions on the basis
of the availability of essential drugs and the interest
shown by District Supervision Teams (ECDs).
To reinforce the institutionalization of the QA concept
and develop and test a model plan to improve the inte￾grated management of childhood illnesses, the follow￾ing activities were implemented:
■ Strengthening district-level management capacity
■ Implementing cost-recovery systems in both districts
■ Training in basic QA
■ Developing methods and tools for quality, training,
supervision, and handbook evaluations
■ Adapting and communicating standards
■ Monitoring and evaluating progress
2. Pre-Project Assessment
There was no baseline assessment of the Tahoua or
Boboye healthcare systems before QA activities began,
but it is known that even minimal thresholds of appro￾priate standards were not in place. Norms and standards
were unknown to the health workers. Regional supervi￾sory competence was lacking, and there were no ECDs
in most districts. Healthcare workers could receive
training only along vertical programs—there were no
regular meetings for the exchange of information and
problem solving. In addition, many of the health centers
did not have the essential equipment to implement the
PMA, and cleanliness in most facilities was largely
unsatisfactory. Table 1-1 presents several baseline
indicators that were used to measure improvements in
outcomes during the QA interventions.
3. Results Summary
The global results expected by USAID for this program
are that the “child survival research project be com￾pleted and the results provided to the other African
countries.” The QAP/BASICS project has therefore
concentrated its efforts on reaching the following two
objectives:
3
Table 11
Baseline Indicators: Tahoua Department (SNIS 1993)
versus Year 200 National Objectives
Baseline
Indicators
Heath Care Coverage
Curative services utilization rate
Immunization coverage for measles
Immunization coverage for BCG
Fixed return rate for DTCP3/DTCP1
Family planning services utilization rate
CN coverage rate (0 to 5 km)
CPN coverage rate (0 to 5 km)
Recovery rate—CRENA
Withdrawal rate—CRENA
Tahoua
(%)
26
30
24
35
63
5
58
63
14
15
National
Objectives (%)
45
50 urban, 30 rural
100
100
100
7
75
75
Not provided
Not provided
BCG and DTCP3/DTCP1 are immunizations recognized as essential to child health
CN: Infant Consultation
CPN: Prenatal Consultation
CRENA: Nutritional Testing and Rehabilitation Center
3.1 TAP Objective Number 1
Institutionalize quality assurance in two districts in
Tahoua and one in Dosso. Strengthen the ECDs’ man￾agement capabilities; revitalize problem-solving teams;
support and strengthen cost-recovery activities in the
Konni, Illéla and Boboye Districts.
The objective was reached or exceeded in every case:
■ ECDs and health committees are in place and operat￾ing in two districts
■ There is a qualified trainer in each of the nine ECDs
■ All the target districts have monitoring and evaluation
plans that are reviewed every six months
■ Health workers and the community use the QA con￾cept and tools for planning and problem solving in
the rural health centers in all target districts
■ The QA concepts manual of standards and tools has
been tested and implemented in other African
Francophone countries
■ Supervisory and quarterly meetings are held in the
target districts
■ A management system has been implemented to
organize the MEG (essential generic drugs) inventory
and routine support activities
3.2 TAP Objective Number 2
Review, test, and distribute in Niger and the surround￾ing countries a model to improve the management of
childhood illnesses (IMCI) at the district level. Improve
the health workers’ skills in IMCI. Disseminate in Niger
and the surrounding region the materials produced and
lessons learned from the QA/IMCI approach.
This objective was also reached or exceeded in every
case:
■ Performance standards for IMCI are communicated
and implemented by district supervisors and health
workers
■ Tools and rapid evaluation methods are available to
ensure quality of medical care and identify problems
in child survival services
■ ECDs trained in supervisory techniques have
improved their supervision skills
■ Health workers in the target districts are familiar with
QA lessons learned and IMCI
■ The case studies, lessons learned, development of
standards, procedural improvements, activity analy￾ses, and all other QA documentation can be dupli￾cated in Niger and other African Francophone
countries
■ The national protocols for IMCI include lessons
learned and procedures from the QAP/BASICS
project
■ An international conference was held in the project’s
second year to disseminate lessons learned from the
approach that combined QA and CCM techniques in
the field
The evaluation team established that the joint project
contributed to the development of several elements that
support quality healthcare in Niger. These are pre￾sented in Table 1-2.
4. Cost Recovery
Cost recovery in primary healthcare was implemented
in the Boboye District in 1993. A prepayment system
(a mix of revenues from local taxes and consultation
fees) was initially implemented. In late 1997, this sys￾tem was changed, having been found to be insufficient
and inefficient, to a system where patient fees are the
4
only source for cost recovery. The Konni District imple￾mentation began early in 1998 under the new system.
Fees for services were set based on the average cost of
visits and people’s ability to pay. The intent of cost
recovery is to replace essential drugs, including IMCI￾designated drugs, and minor supplies such as forms and
other documents. In addition, revenues are to be used to
cover the salary of a stockkeeper, who is selected with
community involvement.
Revenue that each integrated healthcare center (CSI)
collects is sent to the district level and deposited in a
common bank account. CSIs obtain their drug supply
from a district level pharmacy through a monthly order.
The pharmacy then charges the cost of the monthly
order to the cost recovery account. District stocks at the
pharmacy are replenished from a central stock at the
National Office for Pharmaceutical and Chemical Prod￾ucts. (Initial stock was provided to districts through a
grant from different donors.)
The National Program of Support for Primary
Healthcare established a data collection system on cost
recovery activities. With this system, the facilities can
calculate the average cost of a prescription and the
recovery rate, among other indicators.
II. Final Evaluation:
Objectives of the QAP/
BASICS Project
1. Evaluation Background
The goals of the joint QAP/BASICS project final
evaluation were to:
■ Evaluate the level of success with respect to the
anticipated results as detailed in the Technical Action
Plan (TAP)
■ Interpret these results to determine lessons learned
from this experience
■ Make recommendations that will help the project
expand into other departments in Niger and neighbor￾ing countries that are interested in introducing IMCI
in a QA environment
Development of the QAP/BASICS project 1997–98
TAP was guided by the USAID/Niger’s objectives,
which are to:
■ Increase the use of healthcare services for mother and
child health
■ Reduce infectious diseases and medical hazards
■ Improve control measures for HIV/AIDS
QAP/BASICS aspects that were evaluated included:
■ Communication of standards and compliance with
procedures of clinical and nonclinical standards
■ Use of QA methods
■ Efficiency, coverage, and increase of local efforts for
QA
■ Assessment of health worker skills
■ Cost analysis of the main activities
■ Results observed and lessons learned from the
healthcare viewpoint
■ Program monitoring and management
Table 12
Key Data from the QAP/BASICS Project
Elements Supporting Quality Healthcare Quantity or Percentage
People trained in basic QA techniques 400, including 10 qualified trainers
People trained in IMCI 39, including 7 trainers
Health Centers with at least 2 IMCI-trained agents 14 out of a total of 18
Problem-solving teams formed 77
Completed or active problem-solving cycles 120
Regional quality council 1
Subregional quality councils in place 9
Quarterly meetings held at the
Regional Health Directorate
Quarterly meetings held at the district level 108
Supervisory teams formed 9
Department and district supervision operations held 116
Health centers fully equipped in conformity
with standards
Participation in International Society for Quality
Assurance in Healthcare (ISQuA) meetings, 1993–98
QA conferences held 2 (1995, 1998)
Cost of a basic QA training session $US 230 per worker
Cost of an IMCI training session $US 430 per worker
Cost of supervision $US 28 per visit
Cost of coaching $US 18 per visit
4 times
85%
17
5
Table 21
TAP Expectations Compared to Activities
2. Evaluation Summary
The QAP/BASICS project’s TAP
was used as a foundation for the
evaluation. Table 2-1 highlights the
activities that met and often exceeded
the objectives and subobjectives in
the TAP.
3. Evaluation Method
The evaluation team consisted of five
members, including four doctors who
specialize in public health (two from
the United States and two from Niger
who were appointed by the MPH)
and a health program assistant from
USAID. The team worked for three
weeks following this schedule
(Details are in the Schedule of Visits,
Appendix B):
First week in Niamey:
■ Participation in an international
conference on QA and the man￾agement of childhood illnesses
■ Prepare for field visits
Second week in Tahoua:
■ Field trips and visits in the Tahoua
Department: DDS, Tahoua,
Abalak, Illéla, Konni, Madaoua
Districts
Third week in Dosso:
■ Field trips and visits in the Dosso
Department: DDS, Dosso and
Boboye Districts
■ Complete final report
■ Deliver draft report and debrief the
Secretary General of the MPH
At the beginning, the team had access
to all documentation and reports
produced by the project. The team
also received communications/pro￾ceedings from the International QA
Conference.
The team divided into two smaller
teams to interview all personnel
involved in the QAP/BASICS project
Objective 1
Institutionalize the
QA concept in 2
districts in Tahoua
and 1 district in
Dosso
Subobjectives
Reinforce the ECD’s
management
capabilities
Activity and Status
Three ECDs were trained in QA/IMCI in Konni and Illéla (Tahoua)
and in Boboye (Dosso). Community health management committees
were established and are operational in these districts.
There are at least 4 qualified QA trainers in each of the 9 ECDs; for
example, there are 5 QA trainers in Konni, and 4 each in Illéla and
Boboye. For IMCI, there are 3 trainers in Boboye and 4 in Tahoua.
The target districts have 3-month supervision plans and monthly
coaching plans for the CSIs. However, department-level supervision
of the districts is conducted every 3 months in Dosso and only twice
a year, at best, in Tahoua.
The health workers and community use QA concepts and tools for
planning and solving problems at the CSIs in 2 target districts.
However, the evaluation team observed the community’s irregular
involvement in certain places. There were 120 active problem￾solving cycles relating to the PMA. Another 27 cycles were started
but ceased before resolving the problem.
The QA concept standards manual and tools were tested and
released in Mali, the Ivory Coast, and Burkina Faso but have not yet
been implemented.
Management committees were established in all the CSIs in Konni,
Illéla, and Boboye, with a stockkeeper recruited by the community
and paid through cost recovery. Once a month, the management
committee inventories the essential generic drug stock at the CSI
level and prepares purchase orders. Each month, the stockkeeper
deposits the funds collected in a community bank account.
The supervisory visits and quarterly meetings are conducted and
organized every 3 months in the 3 target districts.
Revitalize the
problem-solving
teams
Support and reinforce
cost recovery
activities in Konni,
Illéla and Boboye
Objective 2
Review, test, and
disseminate in
Niger and West
Africa a model to
improve the
management of
childhood illnesses
(IMCI) at the district
level
Subobjectives
Improve health
workers’ performance
in IMCI evaluation,
treatment, and
counseling
techniques in the
target area
Activity and Status
Workers from 18 CSIs in Konni and Boboye were trained in IMCI.
IMCI forms are available to help them understand comprehensive
care management for sick children. Standards were communicated
and implemented though training modules by national and district
supervisors.
ERPAs were conducted in October 1997 in Konni and Illéla and in
June 1998 in Konni, Illéla and Boboye. Feedback from the ERPAs
was systematically provided to workers.
Three ECDs trained in supervisory techniques were able to
effectively improve their performance with the introduction of new
tools, such as the supervision checklist. It has been tested and is
being validated.
The health workers in Konni and Boboye are familiar with the QA
lessons learned and were recently trained in IMCI. Staff in Illéla
have not yet received this training.
The case studies, lessons learned, development of standards,
improved procedures, activity analyses, and all other QA
documentation are being duplicated in Niger in the (nonproject)
departments of Zinder, Diffa and Tillabéry and in eight Francophone
African countries.
The system to implement national protocols for IMCI includes
lessons learned and beneficial procedures from the QAP/BASICS
project, such as supervisory and problem-solving systems, data
analysis and processing, the ERPA, and cost recovery management
techniques.
An international conference on QA was held in Niamey from
October 26–28, 1998, and brought together more than 200
participants from 8 countries in the region: Benin, Burkina Faso,
Burundi, Ivory Coast, Guinea-Conakry, Mali, Senegal, and Togo.
Disseminate in Niger
and the region the
materials produced
and lessons learned
from the QA/IMCI
approach
6
Table 22
Timetable of Major Healthcare Developments
at the QAP, BASICS, and department levels and at the
districts and CSIs. This included:
■ Two DDSs: Tahoua and Dosso
■ Six districts: Tahoua, Abalak, Illéla, Konni, Madaoua
and Boboye
■ 13 CSIs
The team interviewed approximately 60 people, cover￾ing all levels of the healthcare system, using a, multi￾level questionnaire of about 110 questions covering
different aspects of QA (see Appendix E). The team
designed the questionnaire at the beginning of the mis￾sion and revised it with use.
In Illéla and Konni the team organized two focus
groups of women who had used health services (Chap￾ter III, Section 9). Case studies were also prepared
(Chapter III, Section 8). To validate overall results and
proposed recommendations, the team presented a pre￾liminary report to the MPH General Secretary and other
partners on the last day of the evaluation.
The evaluation team focused on:
■ Documenting the benefits of QA institutionalization
and integrated aspects of the joint project
■ Documenting the aspects that helped strengthen
health worker qualifications and increase their moti￾vation
■ Evaluating how the different tools and standards
were introduced and how they have helped integrate
the QA and IMCI approaches
4. Evaluation Report Structure
This report opens with an introduction that focuses on
the health system in Niger as it has been impacted by
QAP, BASICS, and IMCI. This chapter, Chapter II,
more specifically views the two QAP/BASICS objec￾tives and other issues that were studied by the evalua￾tion team. It also outlines the team’s methodology.
Chapter III focuses on QA institutionalization (TAP
Objective 1) in general in the Tahoua and Dosso
Departments, taking into consideration all technical
aspects, including problem solving, tools, teamwork
and productivity, and recommended and implemented
solutions. It presents an analysis of results observed
from QAP/BASICS in 1997–98; compares two CSIs
(one with QA and one without); and provides tables
showing changes in service utilization rates, three case
studies, and the results of two focus groups.
Chapter IV focuses on QA and IMCI operational
research (TAP Objective 2) that began in 1997 in
Konni and Boboye. It responds to two important ques￾tions: Did IMCI achieve its objectives using QA tools
such as problem-solving methods? How did the QA
methods make the IMCI training more effective?
Chapter V presents the dissemination of QA activities
throughout Niger and in West Africa and beyond.
Chapter VI details the costs of QAP/BASICS activities
in Niger; this work asked two main questions: What is
the cost for each intervention, such as QA and IMCI
training, supervision and coaching? And how can con￾tinued financial support be assured? (Cost details are in
Appendix H.) Finally, Chapter VII has the evaluation
team’s conclusions and recommendations.
5. Limits of the Project Evaluation
Considering the time allotted to the team (two weeks in
the field), the total duration of the QA project (five
years), and the QAP/BASICS project (two years), the
evaluation does not claim to have made an exhaustive
analysis of project performance and results. Rather, the
evaluation first gives a broad view of QA accomplish￾ments and then highlights the joint QA/IMCI experi￾ence in the Konni and Boboye districts. The results
analysis is made cautiously because of the interaction
of many factors and other projects that may have influ￾enced the results.
QA
QA introduced in Tahoua at the CSI and
district levels
QA training begins
QA training of CSI supervisors
QA conference in Tahoua
QA training continues
QA introduced in Boboye
Training of trainers
Start of joint QAP/BASICS project
Introduction of supervisory checklist
1st ERPA
2nd ERPA
International QA conference in Niamey
Evaluation
3rd ERPA
Year
1993
1994
1995
1996
1997
1998
BASICS and Other
Cost recovery introduced in limited
areas
BASICS introduced
CCM introduced in Boboye and Say
Districts
Cost recovery starts to be introduced in
Konni, Illéla and Boboye
Cost recovery continues to be
introduced in Konni, Illéla, and Boboye
IMCI introduced in Konni and Boboye
7
Table 31
Existing Standards for Patient Management and
CSI Administration
Department
Unit
Child Health
and
Development
Available Standards
and Guidelines
Child Health Care/ Family
Planning (SMI/PF)
Handbook of Clinical
Procedures for TB, malaria
and malnutrition cases
Complaint/Treatment
Strategies (SPT)
Combined Case
Management
IMCI
Standard management
procedures manual
(administration, finance
organization and health
policy)
Source
MPH
MPH
MPH
BASICS/MPH
WHO/MPH/BASICS
QAP/MPH
WHO: World Health Organization
Year
1993
1995
1993, revised
in 1998
1995
Mid-1998
1996, revised
in 1998
CSI
Management
III. Institutionalization of the
QA Concept
(TAP Objective Number 1)
1. Standards: Background
Achieving customer satisfaction regarding service qual￾ity requires compliance with standards and eligibility
criteria, yet the evaluation team discovered that Nigerien
health workers had difficulty managing all the clinical
standards in place (see Table 3-1). In addition, some of
the clinical protocols differ depending on the model,
which leads to confusion and lack of comprehension on
the part of practitioners.
1.1 Development of Standards
At the national level, the MPH developed a document
that specifies Mother and Child Healthcare/Family
Planning (SMI/PF) standards.
At the Tahoua Department level, a team established an
annual SMI/PF action plan, ensured follow-up of the
different SMI/PF programs, helped the DDS elaborate
and implement its action plans, ensured ongoing staff
training, and collected and analyzed district-level data.
Handbooks that stressed the correct clinical procedures
were developed for the case management of malaria,
malnutrition, and tuberculosis, and to help correctly
determine a child’s age. Refresher training needs were
identified to strengthen health workers’ capabilities to
comply with standards.
Since 1993, there have been guides on Complaint/Treat￾ment Strategies (SPT) of the algorithmic type that are
used in all health facilities. These guides help not only to
standardize treatment but also to streamline the use of
drugs. They are periodically reviewed, taking into
account instructions from vertical programs and the
users’ viewpoints. The last SPT review took place in
late 1998. Its effectiveness is easy to understand: It is an
older model and many health workers are still using it.
The DDS and the districts in Tahoua Department pre￾pared and distributed a document, “Standardized Man￾agement Procedures Manual,” for the CSIs, coaches,
and supervisors at all the department’s health facilities.
This simple and practical summary guide covers several
essential aspects of healthcare service delivery, such as
administration, finance, standards, relationships with the
community, healthcare organization, and health policy
in Niger. QAP contributed to its production and, by
training health workers, to distribution to the CSI level.
It was revised in 1998 when IMCI was being intro￾duced, which added to some existing confusion.
1.2 Communication of Standards
Standards are usually communicated during quarterly
meetings and supervisory visits. Initially, QAP had no
plan to establish specific training for the implementa￾tion of national standards. The IMCI training that began
in August 1998 was based on WHO guidelines and was
an eleven-day, full-time course. BASICS-Niger
financed IMCI training sessions in Niger.
1.3 Recommendations on Developing and
Communicating Standards
The evaluation teams recommends that at the national
level, Niger should:
■ Develop an official list of existing and compatible
standards and guides for SMI and CSI management
■ Determine which types of standards and protocols
should be replaced with more adaptable models
■ Release this list for use as a reference
8
I believe that supervision is an excellent
opportunity for personnel improvement.
A District Health Management Team Member
2. Implementing and Monitoring
Standards
In 1993, the DDS decided to develop a set of follow-up
indicators based on data provided by the SNIS, the
PMA, and Niger’s healthcare priorities. QAP partici￾pated actively in the development and regular system￾atic use of these indicators by health workers.
2.1 Supervision
Before QAP began in 1993, supervisory activities were
either nonexistent on the regional level or conducted
through MPH vertical programs. Six months after the
project began, the newly established Regional Quality
Council (CRQ) identified supervision as the main
vehicle to introduce and strengthen QA and improve the
quality of healthcare.
A survey brought to light problems hindering supervi￾sion in the Tahoua Department, including:
■ Insufficient standards and procedures for supervisory
activities
■ Lack of supervisory skills and lack of task delegation
by the chief doctor
■ Inadequate logistics
■ Lack of coordination between the different hierarchy
levels
■ Lack of qualified personnel
■ Insufficient activity planning
■ Inadequate time on the part of the chief doctor who
was the only supervisor
Faced with this situation, QAP asked the DDS in
Tahoua to improve the supervisory system. The first
quarterly department meeting took place in Konni Octo￾ber 9–11, 1993, and a supervisory policy was estab￾lished. A document was developed listing the following
essential elements:
■ Definition of supervision
■ Objectives of supervision
■ Methodology
■ Supervisory organization and planning
■ Supervisor’s profile and functions
■ Composition of the supervisory team at each level
■ Frequency and duration
■ Activities to supervise
■ Writing standards for the supervisor’s report
Once the supervisory policy was established, QAP
encouraged the DDS to create a supervision system.
Some measures taken to reach this objective were:
■ Staff was trained in supervisory techniques. The
project now includes trainers as well as supervisors.
Each healthcare district (DS) has at least four supervi￾sors who are ECD members
■ A local management team was organized in each
district. These teams formed the basis of the ECDs.
Out of the eight DSs in Tahoua, six complete teams
are now in place with two doctors, a manager, a com￾municator, and an epidemiologist. Only the DSs in
Abalak and Illéla (both in Tahoua Department) and
Boboye (Dosso) have incomplete teams. All the
ECDs have been trained in QA, coaching, and super￾visory techniques
■ Health worker training in technical equipment and
inventory management was begun. QAP first estab￾lished an inventory of the entire existing equipment in
the CSI, and then compared it with technical require￾ments before ordering necessary supplies. A stock
management system for the equipment was developed
and includes inventory and distribution. The team
found a complete technical inventory in all the
healthcare facilities it inspected
■ The car pool management system was strengthened.
(However, it remains limited because of logistical
problems that are hindering the supervisory system in
both Tahoua Department and Boboye District.)
■ Management standards and a PMA have been drafted,
adopted, and disseminated. In all the health training
centers inspected (in Tahoua Department and Boboye
District), the team found a large stock of documents
on standards (SMI/PF, SPT, IMCI, etc.)
9
Table 32
Supervisory System Strengths and Weaknesses
Strengths Weaknesses
High health worker motivation Insufficient strategic support from the head office
Frequent identification of problems Frequent breakdown of supervisory vehicles
and problem solving at the local level
Better communication between Lack of response to some requests expressed at
the different levels of the health the CSI level
care system
Adequate frequency of supervision Irregular frequency of supervision of DS by
of CSI by the ECD the DDS
ECD’s involvement Insufficient feedback from the DDS to the DS
Timely correction of practices Insufficient archiving and documentation system
Having a specific tool: the checklist
2.1.1 The Supervisory System
The project has worked collaboratively with MPH staff
at the departmental and district levels to establish a
structure for the supervision system. The department is
expected to make supervisory visits to the districts
every six months, and the districts are expected to make
such visits to the CSIs every three months. While
department staff have not been successful in fully
implementing their supervisory visits, the district staff
have. In addition, several information collection visits
by both levels have occurred and may well be function￾ing at least partly as supervisory visits.
Interviews with CSI health staff revealed improvements
from supervisory visits. They reported that the visits
have helped them acquire the needed technical and
financial support from the district staff, which contrib￾uted to improved operations and greater staff confi￾dence and motivation.
2.1.2 Challenges Facing the Supervisory System
The most obvious challenge to the supervisory system
is its sustainability after the project ends. The project
has financed supervisory training and transportation
costs. Unless this responsibility is transferred to other
health partners, it is unlikely the system can continue.
The project has started a dialogue with the MPH and its
international partners to facilitate transferring the finan￾cial responsibility to other donors. While a number of
donors have expressed interest, a specific plan has not
yet emerged.
The evaluation team solicited health workers’ ideas on
how to ensure the supervision system’s financial
sustainability after the project ends. A number
suggested using some cost recovery funds for transpor￾tation costs. This idea is worth exploring, but the
evaluation team recommends caution because the goal
of cost recovery is to fund the cost of essential drugs.
Adding other costs could overburden the system,
generate pressure to raise consultation fees, and thereby
cause reductions in the use of health services.
The other aspect of the supervision system’s
sustainability is its technical sustainability: the staff’s
capacity to maintain the system after the project ends.
The evaluation team found the staff’s technical capacity
reassuring at the district level, especially in Konni.
However, there is always the risk that trained staff will
transfer to other posts, leaving gaps in the supervision
system. The MPH’s commitment to QA could be cru￾cial in ensuring that trained health staff are not trans￾ferred into districts without a QA system in place,
which would likely result in loss of effectiveness. Also,
QA training must continue both to maintain the skills
and motivation of trained staff and to train new staff.
2.1.3 Strengths and Weaknesses of the
Supervisory System
2.1.4 Recommendations for Improving the
Supervisory System
To improve the supervisory system:
■ The frequency of supervision must be followed
closely at each level of the healthcare system. The
institutionalization of a follow-up system for recom￾mendations and solutions during supervision at the
CSIs must be quickly implemented at the departmen￾tal level. A debate on continuous supervision, in
general, and possible co-financing by cost recovery
must begin soon.
■ All ECD members must receive supervisory training
■ Regular evaluation of completed supervisions must
be organized
■ The DS must have well-maintained vehicles at its
disposal to be able to complete all planned supervi￾sory activities. (These vehicles need not be used
exclusively for this work.)
2.1.5 Supervision Checklist
Following recommendations from the November 1997
evaluation of the supervisory system in the Tahoua
Department, a supervision checklist was developed,
modified, and tested for six months in the DS. The idea
for the checklist came from one that a BASICS consult￾ant had designed for the Madagascar health system.
10
This tool includes numerous sections that help assess
the overall PMA services. It includes sections on IMCI,
prenatal consultation, maternity, postnatal consultation,
infant consultation and the Nutritional Training and
Rehabilitation Center, extended immunization cover￾age, family planning, STD/AIDS consultations, and an
exit interview to use with women clients. Cost recovery,
QA, community participation, equipment, supplies and
procurement, log books, monthly reports, and posters
are covered.
The checklist was to be evaluated in December 1998
(after the end of the QAP/BASICS project); however, it
applies only to the DS supervision of the CSIs. Another
two-page checklist is being developed for supervision
of the districts by the DDS. Users at various health
training centers provided the comments presented in
Table 3-3.
Table 33
Strengths and Weaknesses of the Supervision Checklist
Strengths Weaknesses
The checklist is easy to archive. The present checklist is too long.
Supervision is made easier with the It takes too long to complete, especially the
checklist, even by a supervisor who is section on the patient interview. It requires at
not trained in supervisory techniques. least 2 supervisors to conduct a complete
supervision of a center in a single day.
The QA section is simple to use and The cost recovery section does not match the
helps follow the teams’ work. cost recovery systems in place in the Tahoua
Department (initiative taken by Bamako in
Madaoua, Pharmacists Without Borders system
in Abalak).
The most important sections apply to It can be tested only for on-going activities on
 IMCI and community participation. the day of supervision; using the entire
document would require more than a day.
The checklist is a good guide but it is not
easy to use.
A Project Supervisor
3. Quarterly Meetings
3.1 Purpose
Quarterly meetings were established to provide oppor￾tunities to reflect on and exchange information between
the different health training centers within one depart￾ment or district. Before QAP started in 1993, meetings
were irregular or nonexistent. Starting in August 1993,
with the DDS agreement of Tahoua, QAP decided to
focus on the quarterly meeting system as previously
established for supervision.
Quarterly meetings are held at two levels:
■ Department coordination meetings of all district head
doctors and department-level staff
■ District coordination meetings of the CSI managers
of a given district with their ECD
A record of each meeting, including important resolu￾tions and recommendations, is forwarded to the external
health facilities and the head office. Problems discussed
during these meetings pertain to all aspects of the PMA,
health center administration, and management. Solu￾tions to a number of problems show how important
these meetings are for the problem-solving process to
work. Table 3-4 provides an overview of identified
problems in two districts by activity and suggested
solutions.
3.2 Main Accomplishments
Quarterly meetings facilitate communication and coop￾eration between the department and district supervisory
staff. Specifically, they generate the following benefits:
■ Systematic review of changes in the PMA and SNIS
follow-up indicators, which helped develop a data￾base on PMA activities and QA cycles, which
fostered a better exchange of information and data
collection
■ Communicating the positive experiences from one
district or CSI to another within the same department
or district
■ Disseminating the Tahoua and Dosso Departments’
experiences to other Nigerien departments when
representatives from other departments attended
■ Adoption and communication of standards
■ Health worker training and self-evaluation
11
4. QA Cycles and Process
Improvement
4.1 Review
The QA cycle includes ten steps that focus on three
essential elements: quality design (steps 1 through 3),
quality control (step 4), and quality improvement (steps
5 through 10), as shown in Figure 3-1.
The Quality Assurance cycle consists of ten steps that
can be characterized as having three components: qual￾ity design (steps 1 through 3), quality control (step 4),
and quality improvement (steps 5 through 10).
4.2 QA Training at the DDS, ECD, and CSI Levels
The objectives for training seem to have been reached at
the ECD and CSI levels. At each district that the team
visited, all ECD staff members had received at least QA
training (i.e., basic training, coaching, and supervision).
A considerable effort had been made by the entire QAP
team to create a critical QA mass at the department
level. The high number of trained staff partly explains
the quality of teamwork and why less-than-optimal
worker mobility did not impair problem-solving results.
An intensive, ongoing training activity proved to be
both worthwhile and necessary. The length of the basic
QA training course was increased from five to seven
days in 1997, which was appreciated by all individuals
surveyed. The basic QA training manual (especially the
1997 version) is viewed as clear and easy to read.
Overall, 520 health workers were trained. From 1993 to
1996, 320 workers received training in QA, coaching,
and supervision. From 1997 to 1998, 200 workers were
trained in the joint project: 110 health workers (includ￾ing 10 qualified QA trainers) were trained in fundamen￾tal QA techniques; 43 were trained in checklist tech￾niques; and 39 health providers (including 6 trainers)
received IMCI training (see Table 3-5).
■ In the Tahoua Department, training conducted since
1993 led to the creation of 68 problem-solving teams
(over 98 percent of CSIs with trained staff had teams)
■ In Boboye (Dosso), eight problem-solving teams
were trained and formed since 1996 (CSI coverage of
80 percent)
More information on QA training by year and by entity
is shown in Table 3-6.
Table 34
Examples of Problem-Solving Activities
District Activity Problem Solution Meeting Date
Abalak PMA Lack of integration Integrated activities April 1997
Completing Incomplete reports CSI sessions for July 1997
a form and forms form completion
Madaoua Curative Low utilization rate Raising awareness, First quarter 1997
activities availability of MEG
CPN Low consultation Operational Second quarter
rate at 9 months research 1997
Table 35
Workers Trained in All QA Techniques (19931998)
Year Basic QA Coaching Local QA Supervision IMCI Training Instructors
1993 43 0 0 8
1994 0 0 0 20
1995 17 11 0 23
1996 218 in 37 0 0
succession 4 in Boboye
10 in Boboye
1997 23 0 12 0
1998 88 22 refresher 0 43 on the 39 (including
training checklist 6 trainers)
Total 400 52 10 valid 94 39
Figure 31
Review of the Quality Assurance Cycle
Quality Improvement Quality Control
Quality Concept
Step 1: Plan
Step 2: Set Standards
Step 3: Communicate Standards
Step 4: Monitor
Step 5: Identify & Prioritize
Opportunities
for Improvement Step 6: Define Problem
Step 7: Identify Who Will
Address Problem
Step 8: Analyze &
Study Problem
Step 9: Choose & Design Solution
Step 10: Implement Solution
12
QA is still considered as an additional
task to perform.
An ECD Member
Table 36
Individuals Trained in Basic QA by Year and Center
Center 1993 1995 1996 1997 1998 Total
Regional Headquarters of
Tahoua and Reference Centers
DDS 11 1 4 0 2 18
District health center 100304
Tassigui Maternity Reference Center 100203
Health Districts
Abalak 0 0 13 1 5 19
Birni N Konni 5 3 54 2 11 75
Bouza 2 3 30 1 5 41
Illéla 4 3 18 1 9 35
Keita 4 3 23 2 7 39
Madaoua 6 3 18 1 6 34
Tahoua 4 3 18 2 5 32
Tchintabaraden 4 3 15 2 1 25
Other Regions
Dosso DDS and Boboye District — — 4 1 25 30
Care Zinder (district chief doctors — — 3003
and one immunization [PEV]
coordinator)
Health Ministry, National School 000404
of Public Health [ENSP] and
other regions
Other countries 000033
TOTAL 42 22 200 22 79 355
Note: Abalak became a health district in 1996.
During the initial QAP phase (1993–97), QA institu￾tionalization was addressed with training on the QA
concept, supervisory techniques, and coaching. Other
training themes, including IMCI, cost recovery, and a
special session on nutrition and management of tubercu￾losis, started with the joint QAP/BASICS project in
1997.
Moreover, QA teams noticed that some problems were
associated with the lack of training in specific areas.
The MPH therefore gave its support to train workers in
cost recovery and to establish 30-day, ECD training
modules (management, supervision, administration,
monitoring). This intervention from the MPH is a fun￾damental, indirect project result.
Over the course of the QAP/BASICS project, the
Tahoua Department’s CRQ assumed responsibility for
guiding, coordinating, and following all actions relating
to quality improvement of healthcare and services.
4.3 Results and Observations on Mastering the
Problem-Solving Cycle
4.3.1 QA Team Composition
QA teams are usually composed of CSI staff or district
hospital functional units. Their composition and num￾ber differ depending on the CSI/district and can include
four to eight members, with an average of five mem￾bers per team. Community participation varies, depend￾ing on the problem chosen. Most often matrons, female
shift workers, community leaders, and members of the
health community are involved. Only health workers
are trained in QA.
In many cases, the initial team changed significantly,
causing delays in the problem-solving process, espe￾cially in the event of staff transfers. However, in most
cases new workers were trained in QA techniques
thanks to the QAP training efforts (see Section 5.2).
All workers interviewed by the team said that team￾work was beneficial.
4.3.2 Identifying and Selecting a Problem
Most interviewed workers reported that problems were
identified through brainstorming. Many workers used
document analysis with SNIS reports, and most believe
that the problem identified was pertinent and a priority
for the CSI. In 90 percent of cases, the selection matrix
was the most-used QA tool at this stage. It is generally
used to choose between different problems and causes,
but also to choose solutions and different implementa￾tion measures.
Health workers use a scoring pattern in their decision
making: 1= weak; 2= average; 3= significant. Criteria
most often used for selection are frequency of the prob￾lem, its impact on other activities, and its importance
and impact on service utilization.
We systematically use brainstorming and the
selection tree to choose problems to solve.
Member of a Problem-Solving Team
13
Completed and ongoing cycles focus on PMA and
include curative consultation, prenatal consultation
(CPN), infant consultation (CN), immunization (PEV),
nutritional training activities (CRENA), management of
tuberculosis and leprosy patients, and family planning
(PF).
Teams tend to select similar problems, which limits the
range of problems addressed. The evaluation team
questions whether it was necessary to limit the teams’
choices to PMA and restrict them to a limited field of
operation. By doing so, issues such as logistics or health
center management were excluded. On the other hand,
benchmarking on the completed cycles can be con￾ducted very quickly, because problems are common to
all centers (the complete list of problems selected is in
Appendix G).
In the Tahoua CSIs, most teams have already reached
their second or third cycle (see Case Studies Number 1
and 2). The Boboye teams were still identifying or
analyzing problems at the time of the evaluation, be￾cause the QA project began there in mid-1997. Only the
CSI team in Koygolo has put solutions into practice
(see Case Study Number 3).
The number of organized cycles increased from 7 in
1994 to 31 in 1998; 120 cycles were begun and 27
cycles were abandoned (Table 3-7). The reasons for
abandoning cycles are discussed below in Section 4.3.6.
Table 3-8 depicts cycle distribution according to PMA
activity. Note that the highest percentage of completed
cycles pertain to CRENA and family planning.
4.3.3 Describing a Problem
Most of the interviewed teams reported difficulties
describing problems, especially at the beginning of a
cycle. “Wording” was redone many times before a final
version was reached, but with experience teams ac￾quired this skill, especially when they were starting their
second cycle. This became obvious in comparing the
Tahoua District, where most teams had reached their
second or third cycle, to Boboye, where all the teams
were still in their first cycle. In all 13 CSIs visited,
wording was satisfactory and consistent with later prob￾lem solving. Problems were sometimes stated simply,
but always clearly.
4.3.4 Problem Analysis
Overall, problem-solving teams have the techniques
well in hand and use most of the tools taught. The first
cycle helps problem-solving teams become familiar
with the techniques; thereafter, these tools are easily and
Table 37
Number of Cycles per Year and Percentage Abandoned
Completed Cycles
Number Percentage
1994 7 5.88 1 14.20
1995 19 15.96 5 26.30
1996 36 30.25 16 44.40
1997 27 22.68 4 14.80
1998 31 25.21 1 2.80
Total 120 100.00 27 22.50
Interrupted Percentage
Cycles Abandoned Year
Table 38
Cycle Distribution According to PMA Activity
Number Percentage Number Percentage
CRENA 26 21.66 5 4.16 19.23
PF 24 20.00 7 5.80 29.16
CPN 20 16.66 5 4.16 25.00
CN 15 12.50 1 0.83 -0.66
PEV 14 11.66 2 1.60 14.28
Curative 13 10.83 3 2.50 23.00
Delivery 3 2.50 0 0.00 0.00
Hygiene 3 2.50 3 2.50 100.00
Postnatal 2 1.66 1 0.83 50.00
consultation
Total 120 100.00 27
Percentage
Cancelled Activity
systematically used. All teams used the diagramming
process and the cause and effect diagram to analyze
problems and thought they were the most effective QA
tools for this stage. The evaluation team found that the
diagramming process is consistent with the selected
problem. Indeed, well-designed diagrams usually
clarify basic solutions. In Boboye, the team noticed
slight imperfections and hesitations in the typical dia￾grams during a first cycle. Unfortunately, this cycle is
limited to health staff and excludes the larger commu￾nity. Also of concern is that the Pareto and control
diagrams are never used.
Some teams used questionnaires to refine their causal
analysis of a problem. The team concluded that many
teams do not master the questionnaire and data analysis
techniques, delaying the process. On one occasion, a
survey became too complicated and was abandoned,
resulting in cycle interruption. All teams believed that
the identified causes were satisfactory.
Completed Cycles Interrupted Cycles
14
4.3.5. Identifying, Applying, and Analyzing
Solutions
The evaluation team found that the selective criteria
from the basic QA manual (cost, ease of application,
secondary effects, resistance to staff changes, and time
needed) are applied systematically to identify viable
solutions. On the other hand, the analysis of the effect
of the solutions is generally insufficiently documented
and analyzed. The evaluation team found no mention of
a group’s having reviewed its strategy or solutions
because of unsatisfactory results. Groups are inclined to
hasten to a second problem rather than assess the im￾pact of the solution from the previous cycle.
At the district level, results monitoring is inconsistent.
The ECD could state the number of active or completed
cycles at the CSI level but could not indicate whether
solutions were positive, negative, or absent.
Several types of activity have been conducted in the
framework for applying solutions. They include activity
integration, raising population awareness, organizing
medical home visits, and requesting the involvement of
matrons (see Appendix G).
Follow-up indicators developed with the SNIS at the
CSI level to assess the implementation of solutions are
sometimes unused or underused once solutions are
applied.
At the departmental level, the team observed spontane￾ous but disorganized rapid problem-solving practices (a
one-day process, for example) and even the creation of
redesign processes. Some restraining factors, related in
particular to logistics problems, seem to have hindered
these alternative QA initiatives, which are a direct
result of QA practice. Of concern is the fact that staff at
this level are unaware that a thorough results analysis
must be conducted after the solutions are applied.
4.3.6 Difficulties and Constraints Encountered by
QA Teams
Several difficulties and restrictions caused the delays or
cessations in the problem-solving process, including
difficulties using QA tools (defining the problem or
collecting/analyzing data), staff mobility/availability,
lack of coaching, and irregular supervision.
4.3.7 Problem-Solving Process: Conclusions
Problem-solving activities led to the following:
■ A motivating and participatory atmosphere devel￾oped for the health workers. Their morale was high￾est after group activities were structured to use valid
tools for problem solving, and they became more
involved in their work
■ Teamwork development was a new experience in
working methods in Niger. Previously, teamwork
was absent or minimal in the public health sector. A
new team spirit was thus created
■ Teams regularly practiced brainstorming techniques
to analyze and resolve problems
■ Full commitment from workers to solve problems
locally at first (despite minimum community involve￾ment) tended to ensure the continuity of QA tech￾niques
■ Concerns about patients changed doctors’ habits.
Problems were effectively centered on patients’
expectations and satisfaction. Indicators show varia￾tions in utilization and attendance
■ Unfortunately, health worker satisfaction was largely
omitted as a consideration of QA
■ A problem-management framework with appropriate
tools led to a new style of management that, in turn,
improved health workers’ habits and performance.
The new, QA-created health center management
focused on improving the quality of healthcare
Near a CSI where a large, privatized
cement factory was operating, the team did
not think of asking the factory management
to assist them in providing solutions.
A DDS Representative
On the whole, local solutions to problems were found
without significant outside help at both the CSI and
district levels and therefore were easily applied. The
community may be asked to participate (to build a
waiting room or for food contributions for nutrition
presentations, for example) and responds to this kind of
request inconsistently. Community involvement was
effective in Koygolo (see Case Study Number 3), but
remains uncertain in some CSIs (e.g., Malbaza). An
effort must be made to improve community response to
requests.
Districts are often asked for help when solutions are not
found locally, especially for larger resources such as
refrigerator or vehicle maintenance, gas leaks, and
small equipment needs; the districts help when they
can, often with project funds.
15
On the local and departmental level, the evaluation
team observed that the ECDs conducted few or no
problem-solving activities, often because of a lack of
time or initiative. Three major drawbacks are:
■ Lack of regular QA practice to ensure efficient and
skillful field coaching
■ Lost opportunities to solve problems on a regional
scale, involving many CSIs (an average of 60) or
districts (an average of seven or eight), which would
have had an indirect effect on problem solving by the
CSIs
■ Difficulty in attempting to launch another QA tech￾nique, which represents quality designing or rede￾signing for the entire healthcare system
4.4 Team Documentation
The team found that work was often well documented
with flip charts and record books that are well kept and
include minutes of weekly problem-solving meetings.
All QA tools used are logged in both charts and record
books; however, the team could not determine how the
problem-solving teams use the tools afterwards or
when a member misses a meeting. The postcycle use of
such tools should be explained more clearly.
To ensure that QA activities are maintained, the contin￾ued use of flip charts and other writing tools seems to
be an important logistic consideration.
4.5 Coaching System
16
most of the Boboye cycles had not been completed at
the time of the evaluation, results from their problem
solvings were not assessed.)
From 1993 to 1996, the utilization of curative services
improved, rising from 30 percent in 1993 to 37 percent
in 1995. A drastic improvement in the coverage rates
for measles vaccination was reported, rising from 24
percent in 1993 to 83 percent in 1996.
Between 1996 and 1997, however, several key rates
plummeted. The utilization rate of curative services
dropped from 37 percent in 1995/1996 to 29.8 percent
in 1997. BCG coverage rates dropped from 107 per￾cent3 in 1996 to 76 percent in 1997, and coverage rates
for measles vaccination dropped from 83 percent in
1996 to 55 percent in 1997.
An analysis of these trends shows a significant increase
in coverage rates between 1993 and 1996. When cost
recovery with user fees was installed in 1997, some
indicators dropped suddenly. Some factors such as
price adjustments, misunderstanding of the new cost
recovery system, and/or a general exodus from crop
failures may partly explain the decrease.
The team noted, however, that after some time the
number of visits to the centers slowly increased. This,
in turn, may have been due to a growing satisfaction
with healthcare services among the population. This
may have been generated by: (a) permanent availability
of drugs at the CSIs, (b) management of integrated
services, and/or (c) lower medical costs. (Before cost
recovery, prescriptions were written at the CSI, and
drugs were bought from the local pharmacists at a
substantially higher cost that is paid under cost recov￾ery.) Still, it is evident that the average rural family’s
purchasing power does not cover healthcare expenses
for all family members, especially under certain condi￾tions, such as during the rainy season. This was, how￾ever, also true before cost recovery.
The team observed that health authorities did not give
any special provision or exemption to the poor or
needy for healthcare and that cost recovery changed
patients’ behavior: They now demand better quality
healthcare services. The QA system is responding
appropriately to this new demand.
It is often said that it is difficult to measure the impact
of QA and clearly determine its many effects. The team
visited and evaluated a CSI that was not part of the
project. Table 3-10 shows the major differences.
A thorough operational research exercise could be
conducted to compare these two kinds of CSIs and to
analyze, for example, the impact of cost recovery, QA,
and IMCI on the main activity indicators of a specific
center, as well as the satisfaction levels of staff and
Table 39
PMA Indicators: 1993 to 1997 (Tahoua Department; in Percentages)
Indicators Department
Objective
Year
1993 1994 1995 1996 1997
Utilization rate of curative 30 28 37 33 29.8 50
services (total population)
CN coverage rate (0-5 km.) 58 57 69 89 75.6 85
CPN coverage rate (0-5 km.) 63 70 70 85 76.6 85
CRENA recovery rate 14 19 15 14 24.8 25
CRENA rate of dropping out 15 17 14 11 16 10
PF utilization rate (total population) 5 5 7 11 14 7
BCG coverage (0-11 months) 35 60 87 107 76 100
Fixed coverage rate for measles
(0-11 months) 24 37 61 83 55 80
Fixed coverage set rate of DTCP/3
(0-11 months) 63 59 62 66 68 90
Fixed return rate VAT2/1 74 58 80 65 73 90
VAT: Anti-tetanus Vaccine
Source SPIS/DDS/TA
NB: These indicators were calculated from demographic data from the general population
census (RGP 1988) and revised according to the annual population growth rate in the Tahoua
Department and calculation methods recommended by SNIS
Demographic Data:
The number of children of 0 to 11 months represents 4.7 percent of the population
Expected pregnancies represent 52/1000 of the population
Women of childbearing age represent 22 percent of the population
Calculation Methods:
Utilization rate of curative services number of clients / total population of the department
CN coverage rate (0-5 km) = children up to 11 months old newly registered at CN (0-5 km) /
children up to 11 months old (0-5 km)
CPN coverage rate (0-5 km) = newly registered at CPN (0-5 km) / pregnancies expected with
0-5 km range
CRENA recovery rate = number of patients cured at the CRENA / number of children cared for
at CRENA
CRENA drop-out rate = number of drop-out cases at the CRENA / number of children cared for
at the CRENA
PF utilization rate = 2 years of protection / women of childbearing age
BCG coverage rate (0-11 months) = Fixed BCG doses given to children between 0-11 months /
children between 0-11 months (0-5 km)
Measles coverage rate (0-11 months) = measles fixed doses given to children from 0-11
months / children from 0-11 months (0 to 5 km)
DTCP3 return rate = DTCP3 fixed doses given / DTCP1 fixed doses given
VAT2 return rate = VAT2 fixed doses given / VAT1 fixed doses given
3 Some children were vaccinated more than once, for reasons unknown to the evaluation team. Perhaps healthcare workers were unsure of some
children’s vaccination status or doubtful of the effectiveness of any previous vaccination.
17
visitors. The following factors make it difficult to deter￾mine cause and effect and therefore to analyze results
and draw definitive conclusions:
■ The many interactions between the different pro￾grams (cost recovery, QAP, BASICS, etc.)
■ The influence of other projects in the same region
(UNICEF, ALAFIA, etc.)
■ The variations in the length of programs (five years
for QA alone, two months for IMCI)
■ The introduction of standards over a long period of
time
■ The variations in implementation dates (SPT in
1993, CCM in 1995, IMCI in 1998)
■ The absence of operational research focusing on a
control region to establish valid comparison
■ The absence of a baseline study on the state of
healthcare facilities before the QAP/BASICS project
Figure 3-2 graphically displays some of the different
interactions.
Figures 3-3 through 3-7 show the evolution of PMA
indicators over four years in the Tahoua Department.
The 1997 figures are not included because that was the
first year of cost recovery and famine, which both
affected the QA project’s impact.
7. Results from Case Studies
Case Study Number 1:
Postnatal Consultation at the Koufan-Tahoua CSI
(Tahoua DS)
The Koufan-Tahoua CSI is located in the western part
of the city of Tahoua and serves 12,540 inhabitants.
The staff is composed of six managers and three
assistants. The CSI team members used brainstorming
techniques to identify service problems. Three prob￾lems were identified as priority items in the PMA:
■ CPN’s low success rate
■ No postnatal consultation
■ CRENA’s low recovery rate
The decision matrix helped determine the priority
problem: the lack of a postnatal consultations. It was
defined as follows:
Table 310
Comparison of QA CSI and Control CSI
Non-QA, Non-RC
■ Uncertain activity schedule
■ No follow-up indicators
■ No data charts
■ Key statistics unknown
■ Unaware of the SNIS report
■ No idea of the changes in the center
from one year to another
■ Clean premises and wards
■ No drugs; medical prescriptions are
filled by a local pharmacist
■ Inconsistent logistical management
■ No integration
■ Well-organized documents
QA with CR
■ Better organization of services and
activity planning
■ Regular follow-up of activities
performed by centers (attendance,
utilization)
■ Good understanding and handling of
data
■ Data analysis
■ Data on SNIS posted
■ Injection and wound care facility is
clean and well kept
■ MEG available
■ Inconsistent logistical management
■ Strong integration of services
Figure 32
Factors Influencing or Interacting with the
CSI Center’s Utilization (Tahoua Department)
Massive
exodus
Income
linked to
1998 harvest
CSI
utilization
in Tahoua
Purchasing
power linked to
1997 harvest
+
Cost recovery /+
began in
1998
 QAP
since
1993
+
Vertical training
for SMI/PEV,
Malaria, etc.

UNICEF
and ALAFIA
programs
+
CCM in 1995
plus IMCI in
mid-1998
+
+
SPT in
1993


18
Figure 33
Evolution of Prenatal Consultation Indicators within a 5Km. Radius (Tahoua Department; 1993 to 1996)
1993
1994
1995
1996
Departmental
Standards
Percentage of Prenatal Consultations Compared to Births
85
63
70
70
85
Figure 34
Evolution of Infant Consultation Coverage Indicators within a 5Km. Radius (Tahoua Department; 1993 to 1996)
1993
1994
1995
1996
Departmental
Standards
Percentage of Infant Consultations Compared to Births
85
58
57
69
89
Figure 35
Evolution of the Rate of Family Planning Utilization within a 5Km. Radius (Tahoua Department; 1993 to 1996)
1993
1994
1995
1996
Departmental
Standards
Percentage of Women of Reproductive Age Using Family Planning
7
5
5
7
6
“The postnatal consultation records were studied in
March 1998 and revealed no postnatal consultations. No
postnatal visits were recorded from 1995 to March
1998, although this activity is part of the PMA services.
Considering the importance of this activity for mothers
and newborn infants, the situation must be improved to
raise the consultation rate from 0 to 80 percent before
the end of 1998.”
A team was identified to take charge of the problem,
and the situation was studied by first establishing the
initial process diagram. Several shaded areas were iden￾tified and the cause and effect diagram was used to list
the cause hypotheses. Possible causes were then verified
through staff interviews and a questionnaire that was
given to 100 women. Possible causes uncovered
included:
19
Figure 36
Evolution of the CRENA Drop-Out Rates within a 5-Km. Radius (Tahoua Department; 1993 to 1996)
1993
1994
1995
1996
Departmental
Standards
Percentage of Children Dropping out of CRENA Programs
10
15
17
14
11
Figure 37
Evolution of the BCG Coverage Rate for Infants up to 11 Months
1993
1994
1995
1996
Departmental
Standards
Percentage of Children Receiving Vaccinations
100
35
60
87
107
■ Insufficient staff knowledge of postnatal consultation
standards
■ Insufficient public awareness of the existence of
postnatal consultation
■ Lack of integration of this activity in the CSI’s PMA
After reviewing the interviews and questionnaire
results, it was found that insufficient awareness of
postnatal consultation was the main cause for low
attendance. The following solutions were identified:
■ Integrate postnatal consultation in the CSI’s PMA
■ Increase public awareness on the importance of the
activity
As a result, the CSI implemented the following
activities:
■ Increased supervision of postnatal consultation
■ Weekly activity planning
■ Reference to SMI/PF standards
■ Neighborhood meetings
■ Education on individual and public health issues
■ Home visits
Four follow-up indicators were identified:
■ Number of health education/training (EPS) group
sessions held on postnatal consultation
■ Number of home visits
■ Number of women attending postnatal consultations
■ Number of neighborhood meetings conducted
The following results were obtained after these activities
were conducted:
■ 8 out of 12 planned home visits were conducted
■ 36 out of 48 planned EPS sessions were conducted
■ 3 out of 3 planned neighborhood meetings were held
■ 46 of 69 admitted women attended postnatal consulta￾tions
20
The following schedule was adopted for the overall
process:
■ March: Identify and define the problem
■ April/May: Analyze the problem
■ June: Choose solutions
■ July: Implement solutions and follow up
Conclusion: Three-and-a-half months after implement￾ing the solutions, 46 women, or 67 percent of those
expected, had attended a postnatal consultation by end
of September. The team hoped to achieve an 80 percent
attendance rate by the end of 1998.
Case Study Number 2:
The PMA Activity Integration Process in the
Madaoua CSI
The Madaoua CSI serves 163,235 inhabitants with
30,420 within a 5-kilometer radius. It has a maternity
ward, a healthcare ward, and a social services depart￾ment. The staff is composed of 12 managers and 2
assistants.
After a problem-solving session on “CRENA’s high
failure rate” conducted using the QA approach,
integrating activities was identified as one solution to
improve CSI services. Reasons for the high failure rate
were identified as follows:
■ Long waiting time for customers, leading to
frustration
■ Unmotivated healthcare workers
■ Long travel distances to the CSI for some women
■ No service integration, insufficient public awareness
Interviews of 55 customers and healthcare providers
from the center provided information to clarify the
problem. One customer answered, “I prefer that all
necessary services be provided to me during a single
visit rather than being forced to come back another
day.” The team studying the problem reorganized the
center activities by:
■ Creating permanent positions in major service areas
(PEV, CRENA, Curative, CPN, PF)
■ Supplying the necessary equipment
■ Increasing customer awareness of service integration
■ Assigning tasks among identified staff members
A follow-up system was established using the following
indicators:
■ Waiting time
■ Visit time
■ Use of SMI/PF services (CNS, PF, and CPN)
■ Number of services requested
■ Number of services provided
Results obtained were:
■ Waiting time was reduced from 120 minutes to 5
minutes 11 weeks after service integration
■ The length of each visit increased from 2 to 5
minutes during the 11th week
■ CNS services’ use rate increased from 45 percent in
1993 to 85 percent in 1996
■ PF services’ use rate increased from 4 percent in
1993 to 7 percent in 1996
■ CPN services’ use rate increased from 30 percent in
1993 to 85 percent in 1996
Conclusion: Integration as a solution to increase the
use of services, establish a quality environment, and
satisfy customers is clearly demonstrated by the
Madaoua CSI experience.
Case Study Number 3:
Partnering with the Community to Solve
Problems Related to the Quality of Healthcare:
The Koygolo CSI Center (Boboye)
The Koygolo CSI center serves 24,205 inhabitants; cost
recovery was implemented there in 1996.
Several attendance problems were identified and tack￾led during regular Health Committee meetings. These
problems were:
■ Low CPN attendance
■ Underutilization of maternity services
■ Low CNS attendance
■ No observation room
Based on a number of criteria, the Health Committee
decided to identify a single priority problem: the
“underutilization of maternity services.” A study identi￾fied several reasons, all relating to patients’ needs. It
found that women resent:
21
■ Being attended by men
■ High fees (500 CFA for childbirth)
■ Birth registration not being performed
■ Lack of space for relatives who accompany them
Several solutions, some requiring patients’ participa￾tion, were proposed to encourage the use of maternity
services. These were:
■ Build a facility for relatives who accompany patients
■ Use “pay” contributions to build an observation room
for the patients
■ Establish a follow-up committee to monitor actions
taken
Implementing the suggested solutions helped reach the
following results between January and April 1998:
■ An increase in the number of assisted births from 12
to 57
■ A facility (hangar) was built
■ The center collected 100,000 CFA to build an obser￾vation room
Conclusion: Based on the results, it is obvious that
addressing the low use of maternity services had a posi￾tive impact. This successful experience of partnership
with the community should be applied in other
healthcare centers to improve the quality of healthcare.
8. Results from Focus Groups
Focus Group Number 1:
Badaguichiri, Illéla DS Center (Tahoua)
Group composition: 10 mothers who used CN services
Summary of comments made by the group:
Positive Points:
■ An EPS group session is held before the consultation
■ Seating is available in a shady area
■ Entering the room proceeds according to arrival
■ Immunizations are available every day
■ Nutrition recommendations are given before the CN
sessions
■ Regular meetings are held with women for feedback
on CSI activities
■ Family planning services are provided daily. They
understand the importance of contraceptives, espe￾cially the injections
Regarding cost recovery, focus group participants
believe that rates are acceptable and are satisfied with
the medicine supply available at the health center. They
were relieved to learn that they are not required to pay
for prescription medicine.
Negative Points:
■ The long waiting line, which they believe is due to
the large number of mothers visiting the center
■ There is no facility (hangar) to make nutrition pre￾sentations to mothers of malnourished children
Focus Group Number 2:
Konni City CSI (Konni District, Tahoua)
Group composition: 12 mothers using the CN and CPN
centers
Summary of comments made by the group:
Positive Points:
■ EPS group and individual sessions are held before
the consultation
■ Patients receive a warm welcome
■ Immunizations are given daily for children
■ Patients show a good knowledge of cost recovery
■ Patients are satisfied with the medicine supply
available
Negative Points:
■ There is no facility (hangar) to make nutrition
presentations
■ There are no home visits or neighborhood meetings
■ The waiting line is sometimes long
22
IV. QAP/BASICS:
Applying QA Measures
to Introduce IMCI
(TAP Objective Number 2)
1. Development and Communication
of IMCI Guidelines
1.1 Combined Case Management: A Prelude to
Introducing IMCI in Niger
Before QAP and BASICS were merged, BASICS had
provided technical assistance to the MPH in combining
vertical case management approaches for childhood
diseases to minimize missed opportunities to identify
and treat sick children. CCM included the updating and
merging of clinical standards for potentially fatal child￾hood diseases: diarrhea, acute respiratory infection
(ARI) and malaria. It also included verifying a child’s
vaccination and nutritional status and counseling moth￾ers about their malnourished children. CCM training
was introduced in 1995 in Say District (Tillabéry
Department) and Boboye District (Dosso Department).
CCM’s introduction before IMCI was controversial and
raised these issues:
■ Introducing CCM seemed advisable because there
was an obvious need to merge the vertical treatment
approaches of childhood diseases to minimize missed
opportunities
■ CCM’s introduction was thought to be “premature”
because the WHO IMCI materials were not ready.
CCM might jeopardize the introduction of IMCI as it
had in other countries, which involved government
officials in adapting WHO materials and selecting
sites for pilot testing
After interviewing WHO officials at the national,
departmental, and district levels, the evaluation team
concluded that CCM improved the introduction of
IMCI. This conclusion is based on the following:
■ CCM resulted in the updating of national case man￾agement standards for diarrheal disease, ARI, and
malaria, a step required before introducing IMCI
■ CCM involved WHO officials at different levels and
sensitized them to the benefits of replacing vertical
case management of childhood illness with an inte￾grated process. As a consequence, they were better
prepared to accept the IMCI rationale
■ The process of updating and combining case man￾agement standards through CCM resulted in the
creation of a national working group that included
directors (of national programs to control diarrheal
disease, ARI, and malaria) and prominent pediatri￾cians. This same structure is needed for the national
adaptation of the WHO IMCI clinical guidelines
■ CCM introduced the processes of checking the vacci￾nation and nutrition status of all child patients and
counseling mothers of children with low weight for
age
■ CCM’s introduction in Boboye resulted in stimulat￾ing efforts to improve the availability of essential
drugs and supervision before IMCI was introduced
■ CCM contributed to the early identification of the
inconsistencies between the commonly used SPT and
the combined case management approach
1.2 Introducing IMCI in Niger
The QAP/BASICS project played a crucial technical
and managerial role throughout the introduction of
IMCI in Niger.
Planning: The project was an important partner
working with the MPH and WHO in planning IMCI’s
introduction. In February 1997, a working group was
established to adapt WHO IMCI generic standards to
Niger. IMCI was to be piloted in three districts: Konni
(Tahoua Department) and Boboye and Doutchi (Dosso
Department). In addition, the MPH nominated a point
person to coordinate IMCI activities.
A one-year action plan for 1998 IMCI implementation
was developed. Funding for the activities was provided
by the contributing partners, including the MPH, WHO,
the QAP/BASICS project, and others. The project
funded all 1998 workshops for district mangers’ orien￾tation, training of trainers, and training of health work￾ers in the Konni and Boboye Districts.
Orientation: The project participated in the August
1996 National IMCI Seminar in Kollo, which WHO
had organized to orient MPH decision makers, interna￾tional partners, and prominent pediatricians. In 1998,
the project funded two district-level orientation work￾shops in Konni and Boboye.
Adaptation: The IMCI working group, with the MPH
represented by the point person, participated in the
adaptation process. This included reviewing all existing
national standards for vertical programs (such as diar￾rheal disease, ARI, and malaria control) as well as the
generic WHO IMCI guidelines. The adaptation process
resulted in Niger’s adopting WHO IMCI generic guide-
23
lines with some modifications to fit the Niger situation.
Some examples of these modifications are: (a) All of
Niger is a high-risk area for malaria, so the low-risk
section of the WHO guidelines was omitted, (b) the
IMCI working group did not believe parasitic infestation
to be common in Niger, so mebendazole is not system￾atically given to all malnourished children, and (c) the
signs for evaluating dehydration were reduced to four
from six.
Training: In general, the project used training to com￾municate IMCI guidelines and played a key technical
and managerial role in organizing the training events
shown in Table 4-1.
2. Local Implementation
2.1 Merging QAP and BASICS
IMCI is perceived as an efficient way to minimize
missed opportunities to identify and treat childhood
illnesses and save sick children. It focuses on five condi￾tions that cause most child mortality: diarrhea, ARI,
malaria, measles, and malnutrition. WHO introduces
IMCI in developing countries. In the late 1990s,
BASICS was USAID’s main vehicle for introducing
IMCI in selected developing countries, including Niger.
Table 41
Training Courses: Dates and Results
Training Course Dates Results
Regional Training
Course organized by
WHO: Niamey
Regional Training
Course organized by
WHO: Niamey
National Training of
Trainers Course: Niamey
Training of Trainers
Course: Niamey
January 19–30, 1998 18 West African participants,
including 5 Nigeriens, trained in
clinical IMCI
8 participants including 5 Nigeriens,
trained as facilitators and clinical
instructors
15 participants, including 14
Nigeriens, became national trainers
9 Nigeriens trained as trainers: 6 at
national and 3 at regional and
subregional levels
18 Nigeriens (16 at the district level
and 2 at the national level) trained
in clinical IMCI
17 Nigerien district-level health
workers trained in clinical IMCI
June 29–July 3, 1998
July 6–18, 1998
August 4–8, 1998
Training Course for August 10–22, 1998
Health Workers at the
District Level: Niamey
Training Course for
District-Level Health
Workers: Niamey
August 31–
September 12, 1998
The success of IMCI introduction is
enhanced where QA field preparation
precedes it.
Director, WHO Office in Niamey
2.2 The Influence and Impact of Cost Recovery
and Drug Supply Management
The evaluation team was impressed with the Konni
District stockkeepers’ ability to apply basic stock man￾agement principles and calculate and initiate monthly
orders. The project has done a commendable job in
coordinating the stockkeeper training with Pharmacists
Without Borders (PSF, a nongovernmental organization
[NGO] funded by the European Union). The staff’s
ability to keep a “security stock” saved the CSIs from a
national shortage in cotrimoxazole in 1998. After the
introduction of cost recovery in Konni, fewer stock￾outs occurred and the average monthly consumption
increased (see Table 4-2).
The presence of QAP and BASICS in Niger presented a
unique opportunity for USAID and its national partners
to merge the two projects and benefit from applying QA
measures to support the introduction of IMCI. USAID
and its partners envisioned that this merging would have
the following national, regional, and international ben￾efits:
■ In addition to the obvious benefit of reducing opera￾tional costs by combining the two projects, Niger
would benefit from the synergistic effect of applying
QA measures to ensure that IMCI was successfully
introduced. It would also benefit from using better
quality child care standards at the primary healthcare
level through IMCI
■ For the West Africa region and the international com￾munity, it would provide a unique opportunity to pilot
test a new approach of introducing IMCI, one that
starts with strengthening health support systems well
before IMCI is introduced
Table 42
MEG Stocks before and after Cost Recovery (Konni)
Cotrimoxazole: 480 Mg. Chloroquine: 100 Mg. Oral Rehydration Solution
(Box of 1,000 Tablets) (Box of 1,000 Tablets) (SRO) Bags
Before After Before After Before After
(1997) (1998) (1997) (1998) (1997) (1998)
Number of 1 1 3 0 2 0
stock-outs
Duration of 3 mos. 1 mo. 19 days 0 10 mos. 0
stock-outs
Average
monthly 19.6 boxes 39.4 boxes 20.1 boxes 56.4 boxes 84 bags 455 bags
consumption
Indicators
24
Figure 41
Utilization of Services in Konni District
(1997 and 1998; January 1 to June 30)
New Patients
Under 5 Years
Infant
Consultation
Prenatal
Consulatation
17964
15457
17828
11943
8160
6935
1997
1998
Figure 42
Utilization of Services in Boboye District
(1996, 1997 and 1998; January 1 to June 30)
New Patients
Under 5 Years
Infant
Consultation
Prenatal
Consulatation
9464
13510
17099
18133
7422
8852
1996
1997
1998
14196
10238
6272
Cost recovery also resulted in the increased availability
of essential drugs (MEG) in Boboye. Staff said that
before cost recovery “shelves were empty.” (In fact, the
team visited a nonproject CSI [urban CSI in Toudobou,
Dosso] where cost recovery had not yet started and
found the shelves to be empty.) However, the Boboye
staff’s stock-management skills need improvement.
Relevant district- and CSI-level staff need training
similar to that conducted with Konni staff in: (a) keep￾ing stock cards up-to-date and safety stock on hand and
(b) using previous consumption patterns to predict
future demand. In one of the Boboye health facilities
(Fabigi), there was no cotrimoxazole from February to
June 1998. The district-level stock also lacked
cotrimoxazole during that period (see Table 4-3).
The evaluation team made the following observations:
■ The chloroquine and SRO stock forms were not
up-to-date
■ The Boboye cotrimoxazole stock-out was caused by
a national stock-out. However, keeping a district￾level safety stock could have prevented or at least
shortened it
■ First-line antibiotics required for IMCI are generally
available; however, second-line antibiotics are gener￾ally lacking
■ Unlike Konni where the CSI stockkeeper oversees
the drug stock, in Boboye a certified nurse performs
this task in addition to his/her other clinical and
administrative tasks. Konni’s system could probably
be followed in Boboye after staff are trained in drug
supply management
■ While the availability of MEG has improved remark￾ably with cost recovery in both Konni and Boboye,
its impact on the utilization of services should be
carefully monitored. Service use rates before and
after the implementation of cost recovery in both
districts suggest that it caused a decrease in the use of
services. Particularly worrisome are decreases in
utilization by children under five. Comparing usage
during the same quarters in 1997 and 1998 reveals a
33 percent drop in Konni and a 22 percent drop in
Boboye. These and other changes in ulilization are in
Figures 4-1 and 4-2
Table 43
MEG Stocks before and after Cost Recovery (Boboye)
Cotrimoxazole: 480 Mg. Chloroquine: 100 Mg. Oral Rehydration Solution
(Box of 1,000 Tablets) (Box of 1,000 Tablets) (SRO) Bags
Before After Before After Before After
(1997) (1998) (1997) (1998) (1997) (1998)
Number of 0 1 1 1 1 0
stock-outs
Duration of 0 mos. 4 mos. 4 mos. 2 mos. 1.5 mos. 0
stock-outs
Average
monthly 33.1 boxes 14.2 boxes 29 boxes 60.1 boxes 400 bags 572 bags
consumption
Indicators
25
While the utilization of health services could be
affected by many factors, such as population move￾ments and environmental and political factors, the data
suggest a strong association between the introduction of
cost recovery and decreases in utilization. Health facili￾ties that have not yet implemented cost recovery, such
as urban CSI in Toudobou (Dosso), have not witnessed
a drop in the utilization rate (see Figure 4-3).
Figure 4-4 shows utilization rates in Badaguichiri,
where cost recovery has been implemented, and indi￾cates drops in utilization rates similar to those seen in
Konni and Boboye.
2.3 Cost Recovery and Family Income
Service fees vary from one center to another. Table 4-4
shows the fee structure in CSI Badaguichiri (Tahoua).
Comparing these with a family’s average daily income
of about 700 CFA suggests that, for individual cases,
the system seems affordable. However, it could present
a financial burden in epidemics. For example, in the
malaria season, a family could have three or four cases
needing health service, which could cost the family
three to four days’ income.
2.4 Health Information System (SNIS)
The project has greatly succeeded in training health
staff to collect, analyze, and use data, especially at the
CSI and district levels. The evaluation team was par￾ticularly impressed with Konni District staff’s analytical
capacity. They were able to regularly monitor health
service indicators (rates of prenatal visits, neonatal
visits, vaccinations, and family planning visits) and
detect malnutrition, tuberculosis, and leprosy cases.
However, the Boboye District staff’s ability to maintain
and use data could be further developed.
Certain IMCI indicators should be monitored, espe￾cially at the start of IMCI implementation. The SNIS
reports and IMCI forms are good data sources.
3. Applying QA Measures to Introduce
IMCI
Assessing the merger of the two projects also has an
international dimension. In the initial phases of intro￾ducing IMCI in developing countries, the emphasis was
primarily on training. Later, it was found that while
training is essential, it is not enough to ensure IMCI’s
successful implementation. As a result, WHO has
added two components to IMCI training:
Figure 43
Utilization of Services in the Dosso CSI
(1997 and 1998; January 1 to June 30)
1997
4274
6456
3776
6050
1849
1631
1998
New Patients Under 5 Years
Infant Consultation
Prenatal Consulatation
Figure 44
Utilization of Services in Badaguichiri CSI
(First Three Quarters of 1997 and 1998)
264 T1
1997
T2
1997
T3
1997
4181
4292
272
320
308
T1
1998
T2
1998
T3
1998
4359
178
160
New Patients Under 5 Years
Infant Consultation
Prenatal Consulatation
Note: T1 is the first quarter of the year; T2 is the second quarter, etc.
2679
88
88
2073
184
160
3680
210
163
26
■ A support system to ensure the adequate provision of
essential drugs and that health workers’ performance
is maintained through effective supervision
■ A community component to reach beyond health
facilities to promote key child survival behaviors
such as breast-feeding
In Niger, QAP introduced methods and tools for health
services quality monitoring and quality improvement in
the Tahoua Department in 1993. Merging QAP and
BASICS allowed the projects to support the introduc￾tion of IMCI in the Konni District. In addition, in 1996,
quality assurance was extended to the Boboye District
(Dosso) to support the introduction of IMCI.
Due to the national and international importance of the
unique merging of QA methods and IMCI, the evalua￾tion team had a special focus on assessing the impact of
applying quality monitoring and quality improvement
tools to support the introduction of IMCI. Their find￾ings appear in the following sections.
3.1 The Supervision System in Support of IMCI
3.1.1 Supervision/Coaching
In the Tahoua Department, the project succeeded in
reviving the supervision system at departmental,
district, and CSI levels.
The departmental and district-level supervisory teams
are trained in both QA and IMCI, so they are capable of
providing technical support to CSI health workers.
Each district supervisory team includes at least one
“coach” to help CSIs in problem identification and
solving. Coaching is performed in a supportive manner
that replaces the previous authoritarian type of supervi￾sion.
Supervision’s benefits are greater when the results of
supervisory visits are communicated clearly to health
workers. (A copy of the supervisory report is not always
Table 44
Fee Structure in CSI Badaguichiri
Service Fees (CFA)
Curative visit for children 5 years or less 350
Curative visit for children over 5 years or adults 700
Prenatal visits (covering all visits for one pregnancy) 700
Neonatal visits (covering all child vaccination) 350
Exchange rate: US $1: CFA 500
given to CSI staff.) The supervisory staff’s ability to
write their observations precisely with required actions
clearly noted could be improved. In Konni, supervisors
also use a Supervision Notebook, which contains a
summary of each visit, and leave a copy at the CSI.
IMCI quality monitoring and improvement efforts
could be disrupted when a trained supervisor is trans￾ferred out of the districts using IMCI. This risk could
be minimized with the support of decision makers at the
central and departmental levels.
The evaluation was too early to demonstrate
supervision’s impact on maintaining health workers’
skills that were acquired in IMCI training. IMCI was
introduced at the CSI level in August 1998. There were
two rounds of ERPA (Rapid Assessment of Health
Worker Performance, discussed below), but they were
completed before IMCI was introduced. The results of
future ERPAs should demonstrate the supervisory
system’s ability to maintain health workers’ skills.
After the project ends, the supervisory system’s
sustainability is not assured. The MPH’s commitment at
the national and departmental levels and international
partners’ involvement could help sustain the supervi￾sory program.
3.1.2 Supervision Checklist
The evaluation team found an impressive and signifi￾cant behavior change among supervisors in their regu￾lar use of the supervision checklist. Before the project
started, supervisors did not have a checklist to use as
part of a supervisory visit, so they had limited ability to
systematically verify service quality. The project real￾ized that a checklist would help them structure their
visits and maximize their benefits. Concurrently, how￾ever, there was some reluctance to use the new tool.
The integrated supervision checklist is a comprehensive
tool to assist supervisors in systematically evaluating a
CSI’s technical, managerial, administrative, and logisti￾cal abilities. It has evolved from its initial version and
now covers all services that CSIs offer, including
numerous sections on the PMA and IMCI. It integrates
the supervision of QA with IMCI by including a
section, taken mainly from the ERPA, that assures
adequate monitoring of health workers’ IMCI skills. It
also has a section that monitors the MEG stock and the
cost recovery system.
In addition to its value at the CSI level, the checklist
can be used to collect data that could help districts and
higher levels plan and monitor overall health worker
performance with IMCI and other services. It has the
27
capacity to monitor the effectiveness of training and
supervision in developing and maintaining health work￾ers’ skills.
When the evaluation was conducted, the project was
pilot testing the checklist and had a completion target
date at the end of 1998. Health workers and supervisors
said that the checklist is comprehensive but very
lengthy. While supervisors do not need to complete the
entire form in each supervisory visit, the evaluation
team concluded that it should be shortened and/or the
questions should be prioritized in each section.
3.1.3 Rapid Assessment of Health Worker
Performance (ERPA)
The Rapid Assessment of Health Worker Performance
(ERPA) is based on the “Health Facility Quality
Review” methodology. Its objectives are to evaluate
workers’ performance in following IMCI and to gather
information on the QA process. It is conducted through
observation of health workers while they examine
children under five and through exit interviews with
mothers to measure their satisfaction and assess the
workers’ ability to communicate instructions related to
prescribed medicine, danger signs that require them to
bring the child back to the health facility, and effective
nutritional counseling. Using it, supervisors can
monitor health workers’ ability to greet caretakers
appropriately, take the patient’s history, examine chil￾dren according to IMCI standards, classify the disease,
identify treatment, verify the child’s vaccination and
nutritional status, and counsel mothers. Information
from the exit interviews helps in verifying the caretak￾ers’ understanding of messages from health providers.
A pilot ERPA was performed in October 1997 in Konni
and Illéla (which are in Tahoua). A second occurred in
June 1998 in Konni, Illéla, and Boboye (which is in
Dosso). A third was scheduled for December 1998. It
has proven to be an easy and valuable tool to monitor
health worker performance and generate IMCI data for
district- and higher-level managers.
The ERPA informed the ECDs about the quality of
services provided to sick children before first level staff
were trained on the IMCI approach. Repeated every six
months, its continued use informs ECDs of the devel￾opment process of quality evaluation tools and helps in
monitoring workers’ performance and service quality.
The ERPA is comprehensive, easy to use, and capable
of producing valuable data for monitoring health work￾ers’ performance. The team also found that feeding
ERPA results back to health workers resulted in im￾proved performance. Results from the first ERPA were
communicated to health workers, and this feedback
alone seems to have positively influenced their perfor￾mance. Results from the second ERPA showed an im￾provement in several IMCI performance indicators,
such as the ratio of mothers adequately counseled for
the treatment of their sick children. On the other hand,
some indicators did not improve or declined between
the ERPAs (see Table 4-5). The evaluation team inter￾preted this finding as indicating that communicating
standards and ERPA results without IMCI training is
not enough to bring about needed improvements in
clinical skills.
Conducting the ERPA during routine supervisory visits,
however, presents a significant limitation. It will be
very challenging to sustain the routine supervisory
visits along with an additional evaluation activity like
the ERPA. The team suggests exploring ways that
would facilitate conducting ERPAs as part of the super￾visory system.
The team categorized the strengths and weaknesses of
the ERPA as a review mechanism (Table 4-6). As pres￾ently organized, it focuses only on the child health
aspect of health facility services and excludes the evalu￾ation of other health services, but it could easily be
modified to integrate more services.
Table 45
Results of ERPAs Conducted in 1997 and 1998 (in Percentages)
Indicators
Konni Illéla
Children checked for general
danger signs
Children correctly examined
Children correctly treated
Mothers adequately counseled for
the treatment of their sick child
Mothers counseled for signs that
merit bringing their child back
Children having their nutritional
status checked
Mothers who knew at least two
general danger signs
Mothers who know how to give the
prescribed medicine to their sick
child
ERPA 1 ERPA 2 ERPA 1 ERPA 2
(Oct. 97) (June 98) (Oct. 97) (June 98)
n = 78 n = 41 n = 81 n = 32
49 25 51 63
54 28 19 49
81 83 77 78
52 73 50 69
75 83 33 50
93 45 6 16
43 83 79 53
73 83 40 80
28
The ERPA gives me an excellent
opportunity for self-evaluation.
A Project Health Worker
Table 46
Strengths and Weaknesses of the ERPA
Strengths Weaknesses
It is a useful tool to help health workers It is conducted outside the routine supervisory
and supervisors apply the QA process. system, so it could add a burden to supervisors.
It is specific to examining health Health workers are not accustomed to such
performance in IMCI. intense workers’ scrutiny.
It relies on direct observation of health Evaluation is based on observing a limited
workers, so it provides an objective number of cases per health worker.
evaluation of their performance.
It includes exit interviews with The test is a tool framed to assess the clinical
caretakers to verify their level of capabilities of only one individual
understanding and satisfaction.
It is a concise instrument that is easy It is limited to IMCI.
to analyze in a short time.
It is feasible with available human ECD members must be available to manage it.
resources.
Timely feedback of its results to
health workers could result in
improving their performance.
It helps health workers to evaluate
themselves.
It is part of the problem-solving approach.
Results from the two ERPAs show that this methodol￾ogy is applicable at the CSI and DS levels and that it
could also apply to follow-up and supervision for im￾proved healthcare services. The team recommends the
following:
■ Give the ERPA results to surveyed workers as quickly
as possible, the same day if possible or within 15
days. This will help them assess their strengths and
weaknesses. A review of overall performance must be
conducted during quarterly meetings to provide a
general overview
■ Ensure coordination with various operational research
activities and monitor results
■ Give ECDs training in operational research
■ Adapt the quality indicators measured by the ERPA
and integrate them into the SNIS systems
3.2 Problem-Solving Approach
All staff who received IMCI training in Konni and
Boboye had also been trained in QA methods, includ￾ing problem solving. It was clear to the team that all
interviewed staff had mastered the QA tools, such as
brainstorming and the selection matrix. The project has
clearly succeeded in raising the staff’s ability to identify
and solve problems.
Specific to IMCI, the problem-solving approach
enabled staff to be aware of problems with malnour￾ished children and follow their progress to recovery.
Some CSIs in Konni (CSI Konni and CSI Samo)
reported an impressive decrease in the drop-out rate
among malnourished children from treatment at the
Nutritional Training and Rehabilitation Center
(CRENA). However, there seemed to be a variance in
the supervisory team’s effectiveness in assisting CSI￾level health workers in solving problems. For example,
there were long breaks in the cold chain in a number of
CSIs that district supervisors visited repeatedly.
It is important to recognize the limitations of the prob￾lem-solving technique in supporting IMCI. While the
method enables staff to identify certain problems, such
as low utilization of health services, it cannot detect
defects in the staff’s clinical skills. Skill problems can
only be identified through observation by trained
supervisors.
3.3 Findings on IMCI in Konni and Boboye
Because IMCI implementation and service provider
training had been conducted only recently, it was too
early to evaluate the pilot fully. The team did make
several observations, however, as follows:
■ Trained health staff are capable of applying the IMCI
clinical guidelines to examine and treat children.
They are also capable of checking children’s vaccina￾tion and nutrition status
■ The team did not find many of the items needed for
IMCI to be successful. This included IMCI forms in
Boboye and the universal lack of chronometers or
clocks with second hands for counting respiratory
rates. One facility did not have a baby scale (urban
CSI in Konni). The team learned that the IMCI work￾ing group had decided not to provide chronometers
or routinely use IMCI forms to reduce costs (because
of inadequate financial resources). The team suggests
that the working group reconsider this decision be￾cause the lack of these essential items, especially the
IMCI forms, could be detrimental to IMCI’s success
29
Figure 45
IMCI Forms at Three CSIs in the Boboye District
Number of forms studied (116)
Number of children with
one danger sign (11)
Number of recorded transfers (1)
Figure 46
IMCI Forms at Three CSIs in the Konni District
(September and October 1998)
Number of forms studied (122)
Number of children with
one danger sign (15) Number of recorded transfers (5)
■ In addition to IMCI forms, health workers have to
complete four different records for each sick child:
the SNIS daily tally notebook, the Daily Register
Book, the Cost Recovery Register, and the Card of
Care. Having so many records reduces the health
workers’ ability to complete the IMCI forms
■ There is no clear understanding among health staff as
to what to do with the completed forms. The team
had a difficult time tracking completed IMCI forms
at the CSIs visited. These forms could be valuable in
monitoring IMCI implementation, especially in its
early phase: The team recommends that they be com￾pleted at the CSI level and delivered to the district
supervisory team during supervisory visits
■ From its review of 127 completed IMCI forms in
three CSIs in Boboye (Fabigi, Falmey, and
Tonkossarey), the evaluation team found that some
trained health staff need closer supervision to com￾plete the IMCI forms correctly
■ The evaluation team found that only 11 percent of
expected follow-up visits actually occurred. This
suggests that caretakers find it difficult to comply
with IMCI standards that encourage their return to the
health center for follow-up (70 percent of Niger’s
population lives more than five kilometers from a
health facility)
■ A similar discrepancy exists between children who
had a general danger sign and should be referred to a
hospital and those who actually were: Only 9 percent
of children in Boboye with a general danger sign
were referred. It could be that either the health staff
neglected to indicate on the form that they had
referred cases or that the health staff is reluctant to
refer cases knowing that it is hard for caretakers,
lacking transportation, to go to the hospital. (See
Figure 4-5 and Appendix D, Table 1.)
From its review of 142 completed IMCI forms in
three CSIs in the Konni District (CSIs Malbaza,
Yaya, and Dogueraoua), the team found that health
staff are generally capable of following IMCI stan￾dards. However, on 30 percent of the forms, the
nutrition section was not completed. Similar to the
situation in Boboye, the Konni CSIs had difficulty in
complying with the IMCI standard on referral instruc￾tions. Only 33 percent of children with a general
danger sign were actually referred. (See Figure 4-6
and Appendix D, Table 2.) CSI Malbaza referred
most of such cases to a nearby missionary hospital
(Galmi Missionary Hospital).
■ Obviously, caretakers in the Konni District are having
difficulty complying with IMCI standards on follow￾up visits. Of the 92 required follow-up visits, none
was recorded
■ The Konni and Boboye District Hospitals are not
prepared to act as effective referral centers because
they lack needed equipment. For example, the oxygen
mask needs to be replaced in the Konni District hos￾pital. (It has not been able to provide oxygen to pa￾tients for a year.) These hospitals are also
underutilized. In the Boboye District Hospital, only
four children under five were admitted in September
and October 1998
■ Limited exit interviews with mothers suggest a need
for improvement in the instructions they receive on
how to use prescribed medicine: It is given to them
wrapped in a paper with no written instructions.
Counseling and written instructions should be pro￾vided for prescriptions
30
■ Cold chain problems and failures to provide vaccina￾tions are eroding IMCI’s benefits. Out of eight CSIs
in Konni, four have cold chain equipment in disrepair
and many have refrigerators that are over 10 years old
and should be replaced. Cold chain maintenance
support was lacking in the six months prior to the
evaluation, resulting in a serious drop in vaccination
coverage in Konni
■ Health staff are capable of checking children’s vacci￾nation status, but in many CSIs children identified as
needing vaccination are not vaccinated on the exami￾nation day. SNIS district reports show that measles
vaccination coverage dropped from 68 percent in
May 1997 to 26 percent in June 1998
3.4 Recommendations Regarding IMCI
The evaluation team makes the following recommenda￾tions to improve the implementation of IMCI:
■ Ensure that the new IMCI guidelines have replaced
the previously existing case management for child￾hood diseases, such as SPT
■ Provide CSIs with an adequate supply of IMCI forms
and chronometers, and ensure the availability and use
of the baby scales that the project provides
■ Simplify, integrate, and reduce the number of records
required for each sick child (SNIS daily tally note￾book, the Daily Register Book, the Cost Recovery
Register, the Card of Care, and IMCI form)
■ Ensure that completed IMCI forms are kept either at
the health facility or at the district level so the data
can be used as appropriate, especially at this early
stage of IMCI implementation
■ Use data in the SNIS and from the IMCI forms to
monitor selected IMCI indicators at the district level,
such as:
■ Service utilization rate for children under five
years old
■ Percentage of forms completed
■ Percentage of children with one or more general
danger signs
■ Percentage of children with one or more general
danger signs who were referred to the district
hospital
■ Percentage of children who were required to come
back for a follow-up visit
■ Percentage of follow-up visits at the health facility
■ Percentage of children with very low weight for
age or anemia
■ Train stockkeepers in Boboye in the same manner
as was used in Konni, including working with
Pharmacists Without Borders
■ Supervise more closely health workers’ perfor￾mance in completing the IMCI forms, communi￾cating the instructions for prescribed drugs to
mothers, and providing nutritional counseling
■ Dispense drugs to mothers in envelopes with
illustrated instructions
■ Develop, test, and distribute to mothers a card to
facilitate communication during nutritional
counseling
■ Improve referral systems and strengthen child
health services in district hospitals
■ Rehabilitate the cold chain system by renovating
or replacing refrigerators that are over 10 years old
■ Vaccinate children on the same day their need for
a vaccination is discovered
31
V. Disseminating the
Techniques and Results
of QA
1. Dissemination during QAP
The following dissemination activities occurred before
the merger of QAP and BASICS:
■ A QA conference was held in December 1995 in
Tahoua, with over 200 people in attendance. They
came from all over the country to present results
obtained by the problem-solving teams and to
exchange viewpoints with people from other districts
■ The MPH conducted a 10-day mission in May 1998
in the Tahoua District to assess QAP progress. He
visited all the centers involved in QAP, and a national
QA policy was subsequently adopted. In his last state￾ments, he clearly expressed his intent to create a
national QA program, although implementation meth￾ods and financial means are not yet determined
■ The WHO representative visited the project in 1995
■ Staff from other DDS and DS offices were invited to
each quarterly meeting in the Tahoua District. This
generated interest and a demand for QA in different
regions. The World Bank representatives in Niger
have started to show interest in the project. As a
result, the MPH, with a loan from the World Bank,
launched a request for bids in October 1998 for a
large, two-year QA program in the Zinder, Diffa, and
Tillabéry Districts
2. International Dissemination of
QAP/BASICS
QAP staff participated in the following international or
regional conferences and projects:
■ International Society for Quality Assurance in
Healthcare (ISQuA)
■ The 1993 conference in Austria, where staff
received the best presentation award for a QA
experience in a developing country
■ The 1994 conference in Italy
■ The 1995 conference in Canada, where staff gave a
presentation and participated in the poster session
on the Tahoua supervisory system
■ In Burkina Faso staff collaborated with a GTZ
project
■ In Cameroon and Togo, staff introduced QA
elements to the Regional Family Health and AIDS
Prevention and John Snow International project
■ A delegation from the Ivory Coast visited Tahoua for
30 days to study and receive training
■ In Rwanda staff held an awareness workshop and
training seminar with John Hopkins University
■ With WHO/AFRO, staff participated in a consulta￾tion to reproduce the Tahoua experience
■ African experts were invited to Brazzaville in
November 1996 to define national QA strategies and
programs
■ Staff organized the first QA workshop in Niamey for
French- and Portuguese-speaking countries
3. QA/IMCI Dissemination
Project staff participated in the following international
or regional conferences and projects:
■ In Mali a one-week visit for study
■ In Senegal a collaboration with the BASICS regional
office
4. Niamey Conference,
October 2628, 1998
While “international,” the Niamey conference dissemi￾nated information to entities in Niger as well as in West
Africa and worldwide. It was held to explore and dis￾seminate information on various experiences by
healthcare providers in the Tahoua and Dosso Depart￾ments, following the implementation of the joint QAP/
BASICS project. More than 200 participants attended
the conference, representing all levels of the healthcare
system: public administration at the prefecture level;
related sectors and associations; NGOs and projects in
Niger and eight Francophone African countries (Benin,
Burkina Faso, Burundi, Ivory Coast, Guinea Conakry,
Mali, Senegal, and Togo); and bilateral and multilateral
cooperating agencies such as USAID/Washington,
Quality Assurance Project URC/CHS/Washington,
BASICS/Washington and/Dakar, WHO/AFRO in
Harare, WHO/Niger, UNICEF/Niger, and the Belgian
Medical Cooperation. Throughout the conference, staff
from the two departments discussed their experiences
with the QA approach and IMCI.
32
Presentations were conducted during plenary sessions
on the basic concepts and general topics, and in working
groups on issues focusing on specific PMA elements.
The following items were discussed in plenary sessions:
■ Basic concepts of the QA approach
■ Child survival approach
■ QA rationale
■ The problem-solving process using the QA approach:
the Tahoua, Niger, experience
■ IMCI
■ Lessons learned from supervisory activities and
quarterly meetings
■ Implementation of the ERPA to improve the
performance of healthcare workers
■ Using the operational research methodology to
determine client satisfaction
Working groups were organized on the last day to elabo￾rate on priority themes, such as staff motivation, institu￾tionalization of the QA/IMCI approach, and QA project
implementation strategies and tools. The conference
provided abundant documentation on the various inter￾ventions to give a significant amount of information on
the progress of various project activities, lessons
learned, etc.
VI. Costs of the QAP/
BASICS Activities
This chapter provides an overview of the QA costs for
each main activity and the project’s overall operational
costs. Key statistics are provided to help ensure that
QA will continue in the future, first in the two target
districts and then nationally.
1. Approach
The evaluation team conducted a cost analysis to evalu￾ate the operational costs of implementing and maintain￾ing essential program activities. The investigation
focused on costs incurred at the health system’s facility,
district, and regional levels and relied mostly on histori￾cal data tracked by the QAP/BASICS project. The
most comprehensive source was a program report on
costs completed by the administrator for the QAP/
BASICS project. The team also used findings from
interviews with the QAP project manager and other
stakeholders (who were not directly involved in cost
analysis). Where appropriate and where direct data
were lacking, costs were estimated based on reasonable
assumptions. Excluded from this analysis was an as￾sessment of the full cost of technical assistance and the
QA and BASICS project-level costs and overhead, so
cost estimates for operating essential activities do not
reflect the allocation of indirect costs related to the
necessary administrative infrastructure.
The evaluation team presents cost estimates with recog￾nition of their limitations. Government-set per diems
were high compared to devaluating salaries, so person￾nel costs are higher for activities that involve a per
diem (training, supervision, and coaching) than they
are for salary-based activities (problem solving). The
former activities involve travel and consequently
require per diems. In addition, several factors limit the
application of these figures to Niger: government poli￾cies, the high cost of fuel in Niger, and the fact that
spending was influenced by a budget. Lastly, the team
was unable to obtain sufficient data to link the costs of
these activities with their effectiveness.
Given the evaluation’s narrow timeframe, the team
focused on evaluating the costs of core activities of the
QA project and the joint QAP/BASICS project that
were identified as critical in achieving objectives.
These were reduced to:
■ Basic QA and IMCI Training
■ Training of trainers
33
■ Team-based problem solving
■ Supervision and coaching activities (including the
supervisory checklist)
■ District, Regional, and Quality Council QA meetings
■ Cost of sustaining essential QA Activities
2. Basic QA and IMCI Training
Based on project estimates, training constitutes the most
significant cost category relative to the total cost of
essential activities. Over the course of the QAP/
BASICS project, costs for training accounted for nearly
60 percent of the total cost of essential activities (see
Appendix H, Tables 2 and 3).
Methodology: The evaluation team primarily based
cost estimates for training providers and trainers in QA
and IMCI on an analysis of 1997 and 1998 training
costs. While exact estimates of the average costs per
trained individual tended to vary from one session to
another (largely because of each session’s specific
needs), estimates of average training costs could be
derived across multiple sessions. Estimates included the
cost of per diems paid to participants and trainers, hono￾raria for trainers, transportation reimbursements, train￾ing kits and other supplies, and refreshments. In the case
of IMCI training, the team also included the cost of
accommodations for participants and trainers, as well as
training site rental.
Several costs that may be considered in “economic”
or “real” terms were excluded from the analysis. For
instance, not included were the opportunity costs of the
time that participants and trainers (when local personnel
conducted the training) spent training. This stems from
the Nigeriens’ desire to integrate QA principles into the
way that care is delivered so that QA training becomes
part of the healthcare providers’ responsibility and
functioning. Consequently, time costs often cannot be
extracted. In the following analyses, when local staff
provided technical assistance, the cost was assessed
based only on expenses incurred to cover their local per
diem and honorarium; costs associated with the
individual’s time (or salary not directly supported by the
government) were not included. The team made this
adjustment to capture the actual training cost that the
Nigerien health system will most likely have to support
when QA principles are nationally institutionalized. In
this scenario, local trainers would be sufficiently
equipped to meet the training needs of other participat￾ing regions in Niger (and possibly Africa), and QA
would be integrated into the delivery of care.
Figure 61
Training Costs for QA and IMCI (US Dollars)
QA
IMCI
Average Cost per Trainee
$ 230
$ 430
Findings: The cost associated with training one pro￾vider in QA averaged approximately $230 ($33 per
provider per day; see Figure 6-1; Appendix H, Table 2
has details). Similarly, it cost approximately $430 per
provider to train them in IMCI ($39 per provider per
day; see Appendix H, Table 3). The cost difference
between the two training sessions is partly due to the
length of the training (seven days for QA versus 11
days for IMCI) and the accommodation and rental
costs.
The investigation of training costs revealed that the
most significant driver of QA training costs are per
diem charges (over 50 percent) paid to both trainers and
providers, followed by costs associated with training
kits (up to 15 percent). Figure 6-2 shows the proportion
of these and other costs.
Figure 62
Breakdown of QA Training Costs (Percentages)
Honorarium
(5%)
Training Kits & Supplies
(19%)
Transport
(10%)
Local TA*
(7%)
Refreshments
(5%)
Per Diem
(54%)
Based on four training sessions held between February and April, 1998
*TA: Technical Assistance
For IMCI training, cost categories were divided in
roughly the same manner, with training kits constituting
a slightly higher percentage of total costs (about 20
percent; see Figure 6-3).
34
Figure 63
Breakdown of IMCI Training Costs (Percentages)
Other
Materials
(2%)
Training Kits & Supplies
(21%)
Transport
(7%)
Hall Hire
(10%)
Accomodations
(4%)
Per Diem & Honorarium
(56%)
Based on two training sessions held between August and September, 1998
3. Training of Trainers
Creating and maintaining local training capacity to
sustain program goals beyond the project’s lifetime are
critical aspects of a program’s institutionalization.
Knowing this, the evaluation team estimated the incre￾mental cost of training a cadre of local trainers that
would be independently capable of continuing training
activities in either QA or IMCI. QA trainer training
consisted essentially of a week of intensive training
(after basic QA training) and a series of practical appli￾cations/demonstrations that trainees in basic QA were
required to complete to obtain trainer qualification. This
training varied in length and rigor based on the
individual’s perceived needs.
Methodology: To estimate the cost of this training
process, the team focused on the requirements associ￾ated with the initial week-long basic training. In most
cases, the amount included the cost of the per diems
paid to trainees, QA counselors, and training consult￾ants, and transportation and other expenses related to
the training exercises (e.g., materials, supplies, refresh￾ments). These costs are considered “standard” costs.
Other expenses provided on an as-needed basis were
considered “situational” costs (i.e., costs that are not
consistently incurred but were expended if needed for a
given session). These included accommodations for
participants and trainers and rental for the training site.
(Accommodation was often free if space was available
in the health facilities during nonwork hours or if
accommodations were found in the homes of local
residents. If such space was not sufficient or available,
expenses were incurred to rent it.)
Findings: Based on 1998 expenditures for training, the
incremental cost of training a trainer for QA qualifica￾tion was about $260, of which $200 was considered a
“standard” cost and about $60 was driven by situational
training needs (see Table 6-1; Appendix H, Table 4 has
details).
3.1 Training Recommendations
Opportunities to decrease costs should be targeted and
capitalized. They may include:
■ Assuring cost efficiencies, e.g., importing the experi￾ence from other countries of conducting on-site train￾ing (trade-off between the cost of transport for train￾ers versus the cost of accommodations for partici￾pants)
■ Minimizing the cost of training kits
■ At a higher level, changing the criteria for per diem
payment
4. Team-Based Problem Solving
Historically, information on the cost of conducting and
implementing team-based problem-solving activities
has been difficult to collect. One such initiative began
in 1997 and aimed at collecting data on the frequency
of team-based problem-solving meetings, the amount of
time staff spent in the meetings, and the activities that
helped support a team’s problem-solving efforts, includ￾ing the cost of collecting data. The team was able to
develop a cost estimate of instituting team problem
solving in facilities through a retrospective analysis and
estimation.
Methodology: Appendix H, Table 5 provides a sum￾mary of the cost elements of team-based problem
solving with cost estimates. The cost list assumes that
the recommended problem-solving schedule (weekly
meetings of one to two hours for three months) was
followed.
Assessed completely, the cost estimate of conducting
problem-solving activities should include the equivalent
cost of the time of staff who participated in the activity.
This cost is particularly important if that time would
have been devoted to their regular duties (i.e., an oppor￾Table 61
Average Incremental Cost of Training
Trainers in QA
Average Cost per Provider
“Standard” Costs $ 200
“Situational” Costs 60
Total $ 260
$
35
Table 63
Effects of Various Factors on Average Cost
of a Problem-Solving Cycle
$ Factor Impact on Average Cost
Team size and mix + / -
Cycle duration + / -
Level of team activity + / -
“Safety stocking” + +
+ / - = Net effect undetermined + = Increase cost
tunity cost for their time either exists or is perceived).
An average problem-solving team in Niger is composed
of three to seven people, typically a head nurse, assis￾tant nurse, janitor, some mix of traditional birth atten￾dants, and the district chief.
Findings: On average, a team-based problem-solving
cycle cost about $104 (see Table 6-2), ranging from
about $78 to $130 per cycle in recurrent costs. (A one￾time investment of about $14 in a tripod to hold
flipcharts would increase this cost, but the tripod would
likely be used for multiple cycles, rendering this cost
negligible on a per-cycle basis.)
Overall, for data collection and supplies, the most sig￾nificant cost components were the per diems paid to
data collectors (about 30 percent of costs), transport or
fuel costs (about 23 percent) and markers (about 20
percent). Paper-based inputs such as survey forms, flip￾charts, and a record book were separately not a large
component of the total recurrent costs, but combined
they accounted for a little over 25 percent of costs.
The cost of the service providers’ time was, on average,
roughly 50 percent of the total recurrent cost of imple￾menting problem-solving activities. Again applying the
recommended guideline of a standard problem-solving
cycle (i.e., team meets for one to two hours per week for
three months), the average equivalent staff and commu￾nity member time cost is approximately $52 per cycle.
This is influenced as much by the number of representa￾tives from each skill or health provider category as it is
by the compensation level within a specific skill level.
It is reasonable to expect that the actual cost of complet￾ing a team-based problem-solving cycle will vary from
these estimates and may even be higher. A variation will
occur from team to team, depending on several factors.
One factor is the team’s productivity level and the effi￾ciency with which it completes a cycle. (According to
some estimates, 70 percent of teams are semi-active and
only half are active. The average cycle lasts over six
Table 62
Average Total Recurrent Cost of QA Problem￾Solving Activities (Estimated; US Dollars)
Type of Cost
Team member costs $ 52
Data collection and supply
costs (recurrent only) 52
Total $ 104
$
monthsto a year with teams meeting bimonthly.)
Another factor depends on the team’s size, composi￾tion, and the mix of staff and skill levels. Finally, a
culture of “safety stocking” (where teams request more
supplies than necessary) will tend to inflate the actual
consumption of problem-solving supplies. While the
degree of these effects is highly situational, “safety
stocking” nearly always leads to higher costs. Table 6-3
depicts the forces that influence total costs.
4.1 Recommendations to Reduce the Cost of the
Problem-Solving Cycle
One or a combination of the following is recommended:
■ Use less expensive inputs (e.g., use low-cost chalk
and blackboard and transcribe activities into records
rather than more expensive flipchart and markers)
■ Change the length and breadth of the survey forms
(this is currently being done). Shorter and broader
surveys could cover more than one district for econo￾mies of scale, or more comprehensive surveys could
capture more information per survey and reduce the
need for numerous surveys
■ Encourage and enforce data collection on team
activities to better support cost analysis in the future
and, more importantly, to use the information to
identify teams needing help
■ Minimize “safety stocking” of supplies by reassuring
teams that supplies will be replenished should theirs
run out
5. Supervision and Coaching Activities
Together with costs for regional and departmental meet￾ings, supervision and coaching activities constituted a
significant cost category for the QAP/BASICS project’s
field costs. Over the course of the project, over 40
36
percent of essential QA activities focused on supervi￾sion, coaching, and meetings (see Appendix H, Table
7). For supervision and coaching, cost estimates are
based on 70 to 75 percent completion rates of all
scheduled activities in this category; this may actually
underestimate the full cost of supervision and coach￾ing. (The referenced table also compares completion
and consumption rates.)
Caveats Regarding Cost: A caveat regarding costs of
supervision activities in the Nigerien context relates to
the reality of existing supervision activities in most
districts. Traditionally, the cost of QA supervision
would be considered incremental to existing supervi￾sion expenses, but in Niger there was no previous
supervisory system. Hence, any cost in this area results
from a stimulated need and demand for supervision to
support QA activities. In other words, for this function
to occur, investment in labor and the support infrastruc￾ture must be made “de neuve.” The same is true for the
coaching system. To the extent possible, the evaluation
team made adjustments in the methodologies to esti￾mate costs of both activities (see below). Another
shortcoming of this analysis is its limitation in linking
the cost of supervisory and coaching visits with the
quality of those visits. The team was unable to obtain
sufficient data to link the nature/quality of the supervi￾sory/coaching visit and the teams’ performance.
Methodology: Appendix H, Tables 7 and 9 document
the actual costs associated with coaching activities in
eight districts in 1997. Typically, two coaches are
responsible for visiting two centers a day (centers
receive coaching visits monthly) and conducting each
coaching visit. Per diems are typically collected by
coaches upon demonstration that they visited two
health centers a day. In addition to the per diem paid to
each coach, costs include the cost of enlisting a driver
and fuel. Since this activity is specific to QA initia￾tives, a lump-sum fee for maintaining this activity is
allocated quarterly to each district. While difficult to
determine exactly, the degree of cost efficiency4 in
carrying out coaching activities varied significantly
across districts.5
Controlling for the rate of completion of scheduled
coaching visits, the percentage of costs ranged from
close to 50 percent below budget (Illéla) to about 35
percent above (Tchinta).
Appendix H, Tables 8 and 10 document the actual costs
associated with supervisory activities in eight districts
in 1997. Centers are supervised quarterly, and each
supervisory visit is typically conducted by two supervi￾sors. Costs include the cost of enlisting a driver and fuel
for the vehicle. Controlling for the levels of activity and
hence expenditures across districts—which vary con￾siderably—the evaluation team observed that spending
for supervision has generally tended to exceed amounts
budgeted for this activity in all districts, on average by
about 30 percent. This is partly due to an increase in
fuel prices, which was not foreseen when the budgets
were developed.
Findings: The Konni and Tahoua Districts, where the
schedule of coaching visits was successfully completed,
provided the basis for estimating the average cost of
completing a coaching visit: $15 to $20. While per
diems for both drivers and coaches are fixed, a break￾down of total coaching costs reveals that fuel accounts
for 70 to 80 percent of the total. This proportion holds
across all CSIs and districts, as well as for each district,
regardless of the number of CSIs or the average
distance of a CSI from the district headquarters. (See
Appendix H, Table 7 for details.)
An analysis of the supervisory cost per CSI and cost per
visit for four districts where all scheduled visits were
conducted (Tahoua, Konni, Illéla, Madaoua) led to
several findings. While estimates vary by district and
are based on the distribution of CSIs in a district, super￾visory visits to a CSI averaged $20 to $35 each. Unlike
the breakdown of coaching costs, both the cost of total
per diems paid to supervisors and fuel costs are major
drivers of supervisory costs, accounting for roughly 20
to 40 percent and 40 to 75 percent of total costs, respec￾tively. Both the slightly higher rates of per diems for
supervisors and the relative importance of per diems to
fuel costs (supervision visits are less frequent that
coaching visits) partly account for this differeece. This
is supported by the high sensitivity of relative per diem
and fuel costs to variations in the number of CSIs vis￾ited and to the distance traveled to the CSI. Appendix
H, Table 11 has information on the average round-trip
distance to a CSI in a given district and corresponding
fuel costs.
4 Determined as the ratio of resources consumed relative to the amount budgeted versus the number of activities completed relative to the number
scheduled.
5 The difficulty stems from the possibility that idiosyncrasies of the performance reporting process may account for variations in the budget utilization rates
as much as real variations in activity efficiencies. Essentially, because monthly coaching activities are reported quarterly, it is possible that districts under￾or over-report the number of coaching visits completed relative to the actual amounts spent on coaching.
37
Table64
Cost of Supervisory Checklist (US Dollars)
Component Cost Comments
Supervision Tool*
* Cost reflects two copies of the tool, one for the center and one for the supervisor.
Checklist $10 every 6 months Currently covers material for 2 supervisory
visits. Plans are under way to extend
material coverage for up to a year.
Cost per CSI. It is foreseeable that
encouraging re-use of initial binder for
new periods can reduce this cost.
Binder (one time)
Total annual
recurrent cost
Training
Per diem
Transport
$14 per center
$34 per center
$25 per supervisor
$8–17
Compensation for 2-day basic training
course.
Based on average distance traveled of 50
to 100 kms. and fuel mileage rate of US
17 cents per kilometer.
Total annual cost $33–42 per supervisor
6. Supervision Checklist
Because of its fairly recent introduction, the impact of
the supervisory checklist is still being measured, with
early indications of potential benefits. It seemed appro￾priate to assess and make early projections on its costs.
Using the checklist generates two costs: (a) the recur￾rent cost of reproducing the binder containing the forms
and (b) the investment in minimal training for supervi￾sors in using the checklist. Table 6-4 provides a break￾down.
7. Recommendations Relating to
Supervisory and Coaching Costs
The evaluation team proposes the following recommen￾dations regarding the costs associated with supervision
and coaching:
■ Encourage the institutionalization of supervisory
systems, such that QA supervision can be measured
as part of routine supervision, minimizing incremen￾tal QA supervision costs relative to overall QA pro￾gram costs
■ Merge supervisory visits with coaching visits
■ Re-use binders and materials for the checklist over
more visits, so that the recurrent cost of the checklist
per supervisor is reduced (e.g., content of one check￾list is designed to accommodate multiple visits; reuse
checklist storage binders once a set of checklists are
completed for the annual supervision cycle)
■ Link information on the quality and effectiveness of
supervision and coaching with cost
8. Quality Meetings
Methodology: Regular, quarterly meetings of providers
and other stakeholders in the QA process have been
well regarded as having significant impact on the
sustainability and progress of the district and regional
initiatives to promote quality assurance. Consequently,
the team estimated and documented their cost. For this,
the team used historical costs expended to support
meetings, which primarily included per diems, transport
reimbursements, and refreshments.
Findings:
District Meetings: District meetings typically are
attended by eight head nurses and ten other individuals
from the district and occur over two days. Historically,
the average cost of quarterly meetings has been allo￾Table 65
Estimated Average Cost of QA-Related
Meetings (US Dollars)
Estimated
Average Cost
District Meetings (quarterly) 200
Regional Meetings (quarterly) 3,000
Quality Council Meetings (bimonthly) 120
$
cated a flat rate of around $200 per quarter per district
(see Table 6-5).
Regional Meetings: Regional meetings have histori￾cally been well attended by representatives from differ￾ent levels of the health system. Typically, a meeting
accommodates two representatives from each of the
eight districts implementing QA programs, five indi￾viduals from the departmental office, and three from the
MPH. The cost of regional meetings per quarter
amounts to approximately $3,000, including per diems
paid to participants.
Quality Council: Another practice of general meetings
that has become part of the QA program is the Quality
Council meeting. Typically, this entails the travel and
per diem expenses of about four people from each dis￾trict who meet about six times per year. Costs to support
these have been about $120 per meeting.
38
9. Cost of Sustaining Essential QA
Activities
With regard to the cost of sustaining the essential QA
and IMCI activities that are evaluated in this report, the
evaluation team developed an estimated projection of
the total annual cost (see Table 6-6). For a department
with the size and structure of Tahoua, training, supervi￾sion, coaching and quality meetings, and IMCI training
require an investment of approximately 3.1 cents per
capita. This figure is the equivalent of 7.7 cents per
person within the catchment area of a QA health center.
This projected level of investment capitalizes on the
other QA activities built into the design of the program,
including the development of standards and quality
monitoring. These estimates reflect only the direct costs
and do not include the overhead costs associated with
maintaining a basic administrative infrastructure to
support these activities. This infrastructure would in￾clude a minimal logistical capacity at the department
level, including a regional coordinator, a core local
cadre of trainers, a minimal support of external techni￾cal assistance, and basic investment in supplies and
maintenance of buildings, equipment, and vehicles. The
costs of other QA activities (quality assessment surveys,
developing standards, and incentive and reward sys￾tems) that could be beneficial in supporting these essen￾tial QA activities are also not included.
VII. Conclusions/
Recommendations
1. Conclusions
The merger of QAP and BASICS in Niger led to a new
way of implementing IMCI. This new approach starts
with interventions to support the systems needed for
IMCI rather than with IMCI training. In selected areas,
QA interventions had been implemented at least a year
before IMCI training began and resulted in significant
improvements. There is an important lesson from the
joint project for the West African region and the interna￾tional community. It demonstrated a new way to intro￾duce IMCI, one that starts with strengthening health
support systems well before beginning IMCI training.
The joint project succeeded in bringing many improve￾ments to the healthcare system in the target areas. First,
it revived the supervisory system and created trained
supervisory teams who can provide technical support to
health facility staff. The concept of coaching facilitated
the transformation of supervision from the traditional
authoritarian style to a supportive, problem-solving
approach.
Second, the project introduced a number of tools that
contributed to IMCI’s introduction. ERPA, the assess￾ment tool, proved valuable not only in monitoring
health workers’ performance but also in improving it
through the timely feedback of its IMCI compliance
data. Also, the integrated supervision checklist is a
helpful, comprehensive tool, albeit too long.
Third, the problem-solving tool elevated the health
providers’ ability to identify and solve problems in the
delivery of healthcare services. It is important to recog￾nize, however, that this tool cannot detect defects in
clinical skills, which must be addressed through supervi￾sion. The project has also had a great impact in improv￾ing the staff’s ability to collect, analyze, and use data at
the health facility and district levels.
Table 66
Average Annual Cost of Essential QA and IMCI Activities
(Tahoua; US Dollars)
Supervision/coaching
Supervision - district $ 27 58 centers, 4x/yr $ 6,264
Coaching - health centers 18 58 centers, 12x/yr 12,528
Supervisory checklist1 10 58 centers, 2x/yr 1,160
Training
New staff2
QA $ 2,300 1x/yr $ 2,300
IMCI 10,750 1x/yr 10,750
QA refresher for current staff3 2,300 2x/yr 2,300
Meetings
Regional $ 3,000 4x/yr $ 12,000
District 200 4x/yr 800
Quality Council 120 6x/yr 720
Team-Problem Solving
Data collection and supplies $ 52 63 teams, 2 cycles $ 6,552
Total for QA only $ 44,624
Total for QA amd IMCI $ 55,374
Average Unit Cost Description Total Cost
1 This excludes one-time costs of $14 for a binder and $33–42 for supervisor training in using the
checklist.
2 Basic QA and IMCI training: 10 people at approximately $230 per person (making allowance for
potential personnel turnover rate) for QA and 25 people at approximately $430 per person for IMCI
3 Basic QA refresher: 10 people at approximately $230 per person (may be less per person with
shorter training)
39
Fourth, improved stockkeeping of essential generic
drugs, combined with cost recovery, resulted in lower
cost, more reliably available MEG, which stimulated
the continuity of care and activity integration. The team
measured utilization rates and samples of MEG stocks
and found a slight decrease in utilization rates at the
beginning of cost recovery followed by an increase in
most cases.
The evaluation team found that there are many items
missing that are needed for IMCI to be successful, such
as IMCI forms and the respiratory rate counter. (Baby
scales were also missing sometimes.) The team sug￾gests that the national IMCI working group reconsider
its decision to cut back these tools because of their
cost. Their absence, especially the IMCI forms, could
be detrimental to IMCI’s success.
 In addition to the IMCI form, health workers have to
complete four different records for each sick child
(SNIS daily tally notebook, the Daily Register Book,
the Cost Recovery Register, and the Card of Care). It is
difficult for health workers to complete this paperwork,
so the forms should be integrated to reduce and sim￾plify recordkeeping.
Problems in maintaining the cold chain are eroding
IMCI’s benefits. This has a twofold downside. First,
children are receiving ineffective vaccines, unbe￾knownst to health providers. Second, statistics from
IMCI reports are incorrect, since staff are reporting
vaccinations that are not effective. Some investment in
new refrigeration and maintenance could solve these
problems.
While healthcare providers are capable of checking a
child’s vaccination status, children identified as need￾ing a vaccination frequently do not receive it on the day
of the examination. Since people frequently live far
from their health centers, it is especially important to
provide this service when the opportunity arises.
Results show that health workers and caretakers are
having difficulty complying with IMCI standards re￾garding both the referral of very sick children to other
healthcare facilities and the need to return to the CSI
for follow-up visits. This difficulty, exacerbated by
Niger’s dispersed population, presents a challenge not
only to the health delivery system but also to IMCI
standards. Further adaptation could be needed to adjust
the requirements for follow-up visits to better suit the
dispersed population.
The team discovered some troubling aspects of
healthcare services delivery in the target areas. Both
district hospitals in Konni and Boboye are not prepared
to act as effective referral centers due to lack of equip￾ment and trained staff. This means that children are not
receiving appropriate care and that the facilities cannot
follow the IMCI guidelines. Furthermore, the exit inter￾views with mothers indicated that providers should
improve their communications regarding prescriptions
and nutrition.
Looking broadly, the project enabled Niger to take a
leadership role in introducing IMCI in the West African
region. Participants from many West African countries
have participated in IMCI training in Niger, and Niger’s
experience in QA and IMCI was presented at a number
of regional conferences and meetings. Considering the
large number of trained agents and trainers, prepared
materials, existing documentation, and the success of
the international conference in Niamey in October
1998, the project’s replication in Francophone Africa
should be rapid. Further, the project’s benefits could
spread to other continents. The supervision system’s
launch with the efficient use of the supervision check￾list, ERPA management and analysis, and the coaching
methods were important experiments that have been
field tested and should benefit other QA projects in
Latin America and Asia.
2. Lessons Learned
Several aspects of the joint project experience should
serve as guideposts for those who will implement simi￾lar endeavors in the future. First and foremost, the
evaluation determined that the QA approach helped
improve the integrated management of childhood ill￾nesses by creating favorable conditions, such as activity
integration, a referral system, and regular follow-up of
activities.
General experience showed that discussion opportuni￾ties within the problem-solving cycle generated favor￾able group dynamics and teamwork. The experience￾sharing mechanism is an indispensable part of the
system, and the emergence of teamwork will likely
improve healthcare services in the target areas for a
long time to come.
All staff were able to use the QA tools and generate
satisfactory—sometimes very impressive—results. The
problems identified for problem solving were generally
limited to the PMA, which may be linked to the fact
that teams use SNIS data analyses in determining which
problems to consider.
The integration of PMA activities is a good example of
process improvement that all interviewed field workers
40
noted as a product of QA efforts. An illustration of this
is the integration of immunization into the daily activi￾ties of all the centers visited by the evaluation team.
The average length of the problem-solving cycle was
six months—a minimum of three and maximum twelve.
Regular coaching (comments are below), which is
vigorous in the target areas (especially at the CSI level),
shortens the cycle.
Community participation varies in the target areas
depending on the CSI. Since commitment from the
community leads to longer lasting-solutions, the CSIs
should integrate community involvement into their QA
activities.
The sometimes dramatic improvement in coverage and
utilization rates—along with user satisfaction—follow￾ing problem-solving cycles shows that problems are not
always caused by a lack of means but sometimes by
poor service organization and management problems.
The problem-solving teams’ work generates an optimal
impact when teams are effectively supported by: (a) a
supervision and coaching system that the DS manages
and (b) a district coordination structure, such as the
Quality Regional Council.
Using the problem-solving cycle, the teams have con￾stantly improved the quality of healthcare and services,
although records of progress vary from team to team.
The main problem that they encountered pertains to the
analysis of the problem, specifically data collection and
interpretation.
Regular CRQ meetings at the DDS level and problem￾solving team meetings at the DS and CSI levels are
important for the continuity of QA activities. Informa￾tion sharing by coaches helps resolve priority problems.
Two major problems that may endanger QA continuity
are discontinuing training and follow-up (supervision,
coaching, and quarterly meetings).
3. Main Factors Limiting QA
The evaluation team identified several factors that will
limit the success of QA unless improvements are made.
They include:
■ Personnel mobility and motivation
■ Old refrigerators, maintenance problems, gas short￾ages, unreliable car/motorbike pool
■ Inaccurate and/or out-of-date demographic data
■ Access to healthcare limited by the dispersed nature
of the population
■ Wide use of several versions of clinical standards
confuses health workers
■ Unmet salary payments for health workers (five
months in 1997 and three months in 1998) were often
compensated with QAP per diem payments for super￾vision, coaching, and training activities
■ Time spent away from the center to attend training
sessions. Some workers were away from their centers
for several months during the year when they
attended training organized by MPH or other partners
4. Recommendations
Recommendations are listed in Table 7-1 by level of
government (Ministry, Department or District) and by
area of concern (Norms & Standards, IMCI, Supervi￾sion, Coaching, ERPA, Quarterly Meetings, Problem￾Solving Cycle, Sustainability, and Dissemination/
Documentation). Where there is no recommendation for
a level/concern, the word “none” is entered.
41
Table 71
Recommendations
Recommendation
■ Establish an official inventory of existing, consistent standards and guidebooks on child health and CSI management.
■ Release this inventory as a reference.
■ Ensure that the new IMCI standards effectively replace the previous child care management standards (example: SPT).
■ Improve procurement, especially for IMCI forms, chronometers, and baby scales.
■ Expand the training of managers and health workers in essential generic drug (MEG) stock management techniques.
■ Closely monitor health workers’ performance in terms of completing forms and providing therapeutic and nutritional advice to mothers.
■ Improve the referral system and strengthen district hospitals by establishing standards.
■ Maintain the cold chain by repairing and replacing refrigerators .
■ Pursue efforts to integrate supervision activities.
■ Improve the regional coaches’ performance by providing them with regular, ongoing QA training.
■ Adapt the quality indicators used in ERPA and integrate them into the SNIS support system.
■ Complete the DDS and ECD staff training in operational research.
■ Continue and support these meetings as they provide workers with ongoing training, motivation, and an opportunity to evaluate
themselves and share experiences (benchmarking).
■ Schedule periodic meetings at all levels on the functions of management teams.
■ Complete DDS and ECD staff training in operational research in such areas such as population census techniques.
■ Support the rapid implementation of a QA curriculum in training schools, such as ENSP and the medical school.
■ Support distribution of QA documents and distribute a QA skills and methodologies guide.
■ Encourage local participation in covering costs for fundamental QA activities, such as fuel and food expenses, supplies for quarterly
meetings, and coaching and supervisory activities.
■ Investigate methods of using cost-recovery techniques to finance these essential activities.
■ Explain and show the QA cost/activity system to various sponsors.
■ Establish an ongoing training system to solve staff transportation problems.
■ Organize regular forums to share QA experiences.
■ Ensure coordination of various operational research activities to disseminate results achieved.
■ Establish QA/IMCI operations research with a sample group to use the experiences and progress in Tahoua and Boboye. Service
provider satisfaction should also be studied as it has not been documented.
■ None
■ Improve the archive system for completed IMCI forms to facilitate data analysis and service evaluation.
■ Closely monitor health workers’ performance in terms of completing forms and providing therapeutic and nutritional advice to mothers.
■ Establish a minimum of biannual supervisory visits.
■ Regularly update the supervisory system to take into account changes in health programs, such as IMCI.
■ Reinforce the DDS supervision system in the district by promoting the use of a central supervision checklist for DDS.
■ Simplify the supervision checklist, especially the patient interview section.
■ Institutionalize a follow-up system for requests and suggested solutions from supervisory visits and establish focal points for each
health center at the DDS and district levels.
■ Ensure that all supervisors are trained in coaching techniques.
■ Conduct monthly coaching activities at the department and district levels when there are new cycles.
■ Provide guidance for DDS district coaches to continually update their knowledge.
LEVEL/Area
MINISTRY
Norms & Standards
IMCI
Supervision
Coaching
ERPA/Monitoring
Workers’ Performance
Quarterly Meetings
Problem-Solving Cycle
Sustainability
Dissemination/
Documentation
DEPARTMENT
Norms & Standards
IMCI
Supervision
Coaching
continued on next page
42
Table 71 (continued)
Recommendations
Recommendation
■ Provide surveyed workers with continuous feedback on the ERPA results as quickly as possible so they can use it to improve their
performance.
■ Conduct the ERPA on a semiannual basis.
■ None
■ Conduct regular problem-solving cycles to improve support systems. This will help the central teams retain personnel resources and
prepare for other, higher-level QA techniques, such as process design/redesign.
■ Encourage personnel transfers within departments where QA is already in place.
■ Increase awareness among the other partners involved in the districts on QA/IMCI activities included in the yearly district action plan.
■ Establish a logistics transport system that includes cars in working condition, spare parts, and trained mechanics to ensure the viability
of key QA activities such as coaching, supervision, or transfers required by IMCI.
■ Organize regular forums to share QA experiences.
■ None
■ None
■ Combine supervisory and coaching visits to save time and thus promote optimal supervision.
■ Institutionalize a follow-up system for requests and suggested solutions from supervisory visits and establish focal points for each
health center at the DDS and district levels.
■ Simplify the integrated supervision checklist, especially the patient interview section.
■ Continue coaching meetings so that they can share experiences.
■ Train coaches in data collection, processing and interpretation techniques, given their importance in monitoring activities.
■ Follow rules pertaining to the composition norms of problem-solving teams.
■ Conduct monthly coaching activities at the department and district levels when there are new cycles.
■ Provide guidance for DDS district coaches to continually update their knowledge.
■ Provide surveyed workers with continuos feedback on the ERPA results as quickly as possible so they can use it to improve their
performance.
■ Conduct the ERPA on a semiannual basis.
■ None
■ Systematically encourage better community participation within the problem-solving teams.
■ Conduct regular problem-solving cycles to improve support systems. This will help the central teams retain personnel resources and
prepare for other higher level QA techniques such as quality design.
■ Encourage personnel transfers within departments where QA is already in place.
■ Increase awareness among the other partners involved in the districts on QA/IMCI activities included in the yearly district action plan.
■ Establish a logistics transport system that includes cars in working condition, spare parts, and trained mechanics to ensure the viability
of key QA activities such as coaching, supervision, or transfers required by IMCI.
■ None
LEVEL/Area
DEPARTMENT
ERPA/Monitoring
Workers’ Performance
Quarterly Meetings
Problem-Solving Cycle
Sustainability
Dissemination/
Documentation
DISTRICT
Norms & Standards
IMCI
Supervision
Coaching
ERPA/Monitoring
Worker’s Performance
Quarterly Meetings
Problem-Solving Cycle
Sustainability
Dissemination/
Documentation
A1
Appendix A: Niger Overview
Background Information on Niger
1.1 Geography
Niger is located in the eastern part of West Africa in the
Sahel-Saharan area. It covers 1,267,000 square kilome￾ters and is bordered on the north by Algeria and Libya,
on the south by Benin and Nigeria, on the west by
Burkina Faso and Mali, and on the east by Chad.
The country is divided into seven large regions or
departments (Agadez, Diffa, Dosso, Maradi, Tahoua,
Tillabéry, Zinder), and Niamey and its suburbs, which is
also considered a department. The departments are
divided into 42 districts.
The sites visited by the evaluation team are marked with
a triangle. In the Tahoua Department: Tahoua, Abalak,
Illéla, Konni, and Madaoua. In the Dosso Department:
Boboye.
Algeria
Libya
Chad
Nigeria
Mali
0 50 100 150 Kilometers
0 50 100 150 Miles
National capital
Department capital
Sites visited
Department names are the
same as their capitals
Cameroon
➤
Benin
Burkina
Faso
Tera
Tillabéry
Ouallam
Fillingué
Niamey Baleyara
Dogondoutchi
Dosso
Tapoz
Birni Ngaouré
Tahoua
Illéla
Keita
Bouza
Birni Nkonni Madaoua
Agadez
Bilma
Nguigmi
Diffa
Maradi Zinder
Goure
Magaria
Dakoro
Tessaoua
NIGER
A2
1.2 Population Data
Niger’s population is estimated at about 10 million,
distributed unevenly with most of the population living
on or near the southern border and only a small portion
living in the north. In effect, three-fourths of the popu￾lation live on one-fourth of the land. Agadez Depart￾ment, which covers more than half the territory (56.6
percent) contains only 2.8 percent of the population
while Tillabéry Department (which includes Niamey
[Niger’s capital] and its suburbs) covers only 8 percent
of the territory has 10 times more inhabitants (28 per￾cent) than Agadez. Maradi Department, which covers
only 3 percent of the territory, has 20 percent of the
population. Population densities vary from Tahoua
Department at 11.5 inhabitants per square kilometer
(1.8 million people, 113, 372 square kilometers) to
Dosso Department at 46 inhabitants per square kilome￾ter (1,476,611 people, 31,000 square kilometers).
There are slightly more women than men: Women
represent 50.5 percent of the population.
1.3 Social and Cultural Data
There are eight ethnic groups in Niger; their population
percentages are shown in Table A-1.
In terms of religion, over 95 percent of Nigeriens are
Moslems. Other religions are Christianity and
Animism.
The external services include:
■ Regional Health Directorate (DDS)
■ Departmental Hospital Centers (CHD)
■ National Hospitals
■ Specialized Centers and teaching schools such as
■ The Leprosy Center
■ The National Center for Tuberculosis
■ The Regional Medical Training and Research
Institute (CimeFor)
■ The National Family Health Center (CNSF)
■ The National Public Health School in Niamey
(ENSP)
■ The ENSP in Zinder
2.3 The National Health Information System
SNIS is responsible for collecting and analyzing the
country’s overall health information except data
pertaining to national and departmental hospital cen￾ters. Data are collected through established collection
systems that are reviewed every two years. Data are
then forwarded quarterly from the local integrated
healthcare center (CSI) to the healthcare district (DS),
from the DS to the Regional Health Directorate (DDS),
and from there to the SNIS. A preliminary analysis is
performed at each level before data are forwarded to
the next level.
The SNIS is also responsible for epidemiological
surveillance. The health centers must submit a weekly
epidemiological report to the SNIS on contagious
diseases (cholera, measles, meningitis, diphtheria,
tetanus, yellow fever, and poliomyelitis).
2.4 The Healthcare System
The healthcare system in Niger is organized around
training and public and private healthcare institutions.
It is structured as a three-level health pyramid:
The first level is composed of three echelons and
corresponds to the health district (previous medical
division):
■ At the first echelon, there are village healthcare
teams (ESV) made up of at least two first-aid
workers and two matrons working out of a small
cabin. The ESV members are volunteers
2. Niger’s Healthcare System
2.1. Health Organization
The healthcare system is organized along the country’s
administrative divisions.
2.2 The Ministry of Public Health
The Ministry of Public Health (MPH) is divided into
central directorates and a number of programs, includ￾ing the National Health Information System (SNIS,
attached to the General Secretariat) and external
services.
Table A1
Ethnic Groups in Niger
Ethnic Groups Percentage of the population
Houassa 56.0
Zarma-songhai 22.0
Peulhs 8.5
Touaregs 8.0
Kanouris 4.5
Arabs, Toubous, and Gourmantches 1.0
A3
■ At the second echelon are the integrated healthcare
centers (CSIs), including the former rural dispensa￾ries, the neighborhood clinics, and the medical posts
■ The third echelon, which includes district hospitals
(usually with a dispensary, hospitalization unit,
maternity wing, mother and child welfare unit [SMI]
and medical laboratory). Each health district covers
an administrative district
The second or intermediate level includes reference
centers for the department. This level includes the
Department Hospital Center (CHD) with specialized
care services such as surgery, pediatrics, internal
medicine, and maternity (obstetrics).
The third level (national or central level) includes three
national hospitals, two in Niamey and one in Zinder, as
well as the Niamey Maternité Centrale, Maternité
Poudrière (Maternity Clinic), National Family Health
Center, National Center for Tuberculosis, and Leprosy
Center. These are the country’s referral centers. The
MPH is supported by a supply center for medicine and
equipment, the National Office for Pharmaceutical and
Chemical Products (ONPPC). The country also has a
national laboratory, the Laboratoire National de Santé
Publique et d’Expertise (LANSPEX), which is interna￾tionally known and sponsored by the World Health
Organization.
Average annual Gross Domestic Product US $ 290
Population 9,923,265
Women 50.3%
Women of fertility age 22%
Children < 15 years 45%
Children < 5 years 21.2%
Infants from 0 to 11 months 4.7%
Urban Development rate 153/1000
Fertility rate 232/1000
Fertility index 7.1
Birthrate 53/1000
Gross death rate 19/1000
Child death rate 123.1/1000
Death rate: Children < 5 years 318.2/1000
Maternal death rate 7/1000
Malnutrition rate: Children < 5 years 41/1000
Life expectancy at birth 48 years
Population growth rate 3.3%
Percentage of children in full-time education 27.5%
Literacy rate 14%
Access to health centers
0 to 5 Km., Rural Areas 32%
0 to 10 Km., Rural Areas 50%
Access to drinking water 59%
Table A2
Development Indicators
Figure A1
Diagram of the Healthcare System in Niger
Administrative
Organization
Tahoua
Health
Department
Health
Districts
Integrated
Health Centers
(CSIs)
Technical
Organization
Regional
Hospital
District
Manager
(MD)
District
Hospital
QA Structures at the
Department Level in Tahoua
Quality
Council
(QRC)
District Health
Management Team
(DHMT)
Problem-solving
Team
Regional Level
Subregional Level
Peripheral
Level
Regional Level
Subregional Level
Peripheral
Level
3. Niger’s Healthcare Situation and
Health Indicators
A4
Other diseases also have a significant impact on
public health, including schistosomiasis, leprosy,
dracunculosis, AIDS, tuberculosis, malnutrition due
to protein deficiency, Vitamin A deficiency, and
nutritional anemia.
Poor sanitation and very low immunization coverage
further aggravate this poor health situation.
Table A3
Epidemiological Data
Illness New Cases (Percentages)
Acute respiratory infections 28.45
Malaria 24.58
Diarrheal diseases 10.36
Trauma/cuts/burns 6.94
Conjunctivitis 5.20
Skin diseases 3.96
Oral infections 2.44
Gynecological/obstetric infections 1.83
Urinary tract infections 1.31
Sexually transmitted disease 0.95
Measles 0.88
Meningitis 0.11
Main Causes of Morbidity Reported by Health Authorities (1997)
Table A4
Immunization Rates (1997)
Age Groups Antigens Percentage Covered
Under 1 year old BCG 43.64
DTC P1 45.11
DTC P3 28.47
Measles 41.78
Under 5 years old BCG 10.90
DTC P1 13.11
DTC P 7.24
Measles 17.63
Data Sources:
1. Health Chart from the Republic of Niger: WHO/MPH/1994
2. 1997 Directory of Statistics – Morbidity/DSNIS/SG/MPH
A5
Monday November 2 Prepare the questionnaires
Collect information and documentation
(Sabou and SNIS)
Tuesday November 3 Trip to Tahoua
Regional Health Directorate (DDS)
Préfet: community leader
District chief doctor, District Supervisory
Team (ECD)
2 CSIs: Koufan-Tahoua, Kalfou
(one team per CSI)
Debriefing meeting. Write documents
Wednesday November 4 Abalak (1st team)
Sous-Prefét, community leader
Health management committee
ECD
2 CSIs: Abalak, Tabalak
(one team per CSI)
5 PM: Quality Regional Council
Debriefing meeting. Write documents
Illéla (Second team)
ECD, Sous-Préfet
ERPA
Thursday, November 5 Illéla
Community focus group with women CSI
users in Badaguichiri
2 CSIs: Tajae, Badaguichiri
(one team per CSI)
Departure and night spent at Konni
Debriefing meeting and drafting of
documents
Friday, November 6 Konni (call SG and DPS)
Sous-Préfet, community leader
ECD
2 CSIs: Malbaza, Dogeraoua
(one team per CSI)
Appendix B: Visit Schedule and Comments
Table B1
Schedule of Visits of the Evaluation Team
Saturday, November 7 Madaoua (1 team)
(Morning) Sous-Préfet, community leader
ECD
2 CSIs: Sabon Guida, Azarori with one
team per CSI
Y. Tawfik stays at the Konni CSI for a
group-led discussion with IMCI and ERPA
trainers and supervisors
Return to Tahoua
Sunday, November 8 Consolidate data, write the draft
Monday, November 9 Departure for Dosso
DDS, ECD, 1 urban CSI
Préfet, community leader
Tuesday, March 10 Departure for Boboye
2 focus groups in Falmey, Koygolo with
the women
2 CSIs: Falmey, Koygolo
Wednesday, November 11 Boboye
ECD
Thursday, November 12 Final version of the evaluation report
Prepare meeting for final debriefing/
validation of the mission
Friday, November 13 Presentation of results, comments,
recommendations to SG, DPS, DSF,
ENSP, Medical School faculty
End of mission, return of S. Legros to
Washington
A6
Table B2
Comments on Health Facilities Visited by the Evaluation Team
Identified Problems; Suggested Supervision,
Number of Cycles Solutions and Results Related Costs Frequency,
and Duration Implementation Duration, Report
QA-Trained
Individuals
(#)
Tahoua 10 Secretarial Very infrequent,
DDS Vehicle maintenance report established,
Gas for cold chain 5 trained supervisors
Established a 2-page
checklist for the districts
Dosso 4 No problems were treated Reduced 3 months
DDS attendance in the
beginning
Tahoua 4. All of ECD 3 months
District 2 trained supervisors
Abalak 4. All of ECD CPN completion rate No DDS supervision
District / Malnutrition screening for 2 years
CSI TBC screening
3 cycles, 1 year
Konni 5. All of ECD
District except the
CSI chief
Boboye 3 No problem-solving MEG available 3 months with reports
District / activities with increased written and delivered
CSI frequency
Kalfou 2 Drop-out rate from CRENA Matrons conduct Drop-out rate 3 months, October 5,
CSI 1st cycle, 1 year nutrition presentations falls from lasts the whole day
and home medical 60% to 40%
Low PF attendance calls (VAD)
Starts August 98
Build a hangar
Koufan 3 Negative rate at the Neighborhood 0 to 66% MEG available 3 months, 1 day
Tahou postnatal consultation meetings end of Attendance tripled Received reports
CSI 2 cycles, 3 months VAD September 98 despite an
Weekly schedule awareness problem
 in the beginning
Tabalak 2 Bad service given to the Awareness raising 3 months
CSI malnourished at CRENA Nutrition presentations
1st cycle, 1 year, second Time-table changes
 for the last 3 months
Tajae 2 Low attendance Awareness raising, From 18 to 3 months, August 98
CSI CN ASV involvement 60 %, 25 Received report
1 cycle 6 months children / month
Malbaza 4 Poor welcoming system Benches, chairs, Slightly higher 3 months (8/12/98)
CSI Withdrawal rate from CRENA RDV elimination attendance Supervision log book is
Low CPN rate, 2 completed Activity integration 18% to 0% full
cycles, 1 year and 9 months
1 active cycle
Tsernaoua 2 High rate of withdrawal from Activity integration Increased 3 months, 1 year
CSI CRENA attendance
1 cycle, 6 months MEG available
The community is
getting involved
Falmey 3 Low immunization coverage Has not arrived yet 3 months with full
CSI supervision log book
Fabigi 3 Low rate of screening for ths
CSI malnourished
1 cycle, 3 months
A7
Appendix C: Utilization of Services
Table C1
Utilization of Services in Konni
Indicators 1997 1998
Number of new patients < 5 yrs, 17,828 11,943
Number of new patients, all ages 52,578 30,328
Well-baby clinic 17,964 15,457
Prenatal consultations 8,160 6,935
January 1 to June 30 (SNIS T1 and T2)
Table C2
Utilization of Services in Boboye
Indicators 1996 1997 1998
Number of new patients < 5 yrs, 17,099 18,133 14,196
Number of new patients, all ages 45,297 46,238 33,344
Well-baby clinic 9,464 13,510 10,238
Prenatal consultations 7,422 8,852 6,272
January 1 to June 30 (SNIS T1 and T2)
Table C3
Utilization of Services at Urban CSI Dosso
Toudobou
Indicators 1997 1998
Number of new patients < 5 yrs, 3,776 4,274
Number of new patients, all ages 9,000 9,341
Well-baby clinic 6,050 6,456
Prenatal consultations 1,849 1,631
1997–98 from January 1 to June 30 (SNIS T1 and T2)
Table C4
Utilization of Services at Dosso CSI of Badaguichiri
1997 1998
T1 T2 T3 Total T1 T2 T3 Total
Number of new patients, all ages 4,181 4,292 4,359 8,832 2,679 2,073 3,680 8,432
Well-baby clinic 264 272 178 714 88 160 210 458
Prenatal consultations 320 308 160 788 184 163 435 782
January 1 to September 30 (T1, T2, and T3). SNIS figures. T1 is the first quarter of the year, T2 the second, etc.
Indicators
A8
Appendix D: Analyses of IMCI Form Utilization
Table D1
Analysis of IMCI Forms in Three CSIs in Boboye
Number Percent Proportion
Forms Studied 127
Forms Completed Correctly During the First Visit/ Follow-up Visit 118 93% 118/127
Children with a General Danger Sign 11 9% 11/127
Transfers Reported 1 9% 1/11
Forms with the Nutrition Section Completed 112 88% 112/127
Underweight Children 11 10% 11/112
Children with Serious Anemia 29 26% 29/112
Notes:
Fabigi, Falmey, and Tonkossarey: September and October 1998
Number of children under 5 admitted to the Boboye District Hospital in September and October 1998: Four
Distance between the CSI and the hospital in Boboye: Fabigi: 25 km.; Falmey: 85 km.; Tonkossarey: 40 km.
Table D2
Analysis of IMCI Forms in Three CSIs in Konni
Number Percent Proportion
Forms Studied 142
Forms Completed Correctly During the First Visit/ Follow-up Visit 134 94% 134/142
Children with a General Danger Sign 15 11% 15/142
Transfers Reported 5 33% 5/15
Forms with the Nutrition Section Completed 100 70% 100/142
Underweight Children 33 33% 33/100
Children with Serious Anemia 44 44% 44/100
Notes:
Malbaza, Yaya, and Dogueraoua: September and October 1998
CSI Malbaza transferred four children to the Galmi private hospital (15 km. from Malbaza)
Distance between the CSI and the Konni District Hospital: Malbaza: 33 km.; Yaya: 57 km.; Dogueraoua: 45 km.
A9
QA Training  Problem Solving
Analyze the follow-up forms developed by Saibu.
For District managers:
1. In this district, how many people were trained in quality assurance?
2. How long does the training last and what topics does it cover?
3. Is the duration of the QA training adequate, too long, or too short?
4. How many individuals have actually attended all the steps of the cycle and used the method correctly?
5. How many cycles has the team completed? How many are active now?
For the CSI/coaches/ECD/DDS:
6. How did you organize your problem-solving team?
7. To what extent was the community involved in the process? What role did it play?
8. What type of documents can you show us?
9. How were problems selected? Does this system seem adequate to you? Were they priority problems?
10. How was the problem expressed? Several versions of the problem were provided before the final version was
selected.
11. What data did you use to identify the problem?
12. What tools did you use most among the available QA tools (brainstorming, process diagram, cause and effect
diagram, selection tree)?
13. Which would be the most useful? Explain. Now that you are trained, do you intend to keep using the QA tools even
if you are in an area outside the project area?
14. Do you have a message for the central office to ensure that such tools continue to be used?
15. Was the process diagram useful and did it correspond to the solution applied afterward?
16. What QA manuals do you have?
Appendix E: Questionnaire Used by the Evaluation Team
QA/BASICS Evaluation, Niger
Tahoua, Dosso
Standard Multiple Part, Multilevel Questionnaire
Region, District, ECD, CSI, QA teams
Name of Person Surveyed
DD/District/CSI
Location
Date Beginning/End Time
A10
17. Which is the most useful?
18. Were the causes identified satisfactory?
19. Was the evolution of the teamwork quick, slow, or irregular?
20. Were there any interruptions or disruptions? For what reason?
21. Were there many staff member changes within the team? Why?
22. In the past 6 months, how many health workers left the district after the training session? For what reason? Where
did they go?
23. How does this affect the smooth running of the service and teamwork?
24. How did you adjust to these changes? How were the new participants trained? In the field, accelerated training,
special role of the coach?
25. How do you rate the teamwork? Good, improving constantly, disappointing, bad?
26. How were solutions selected and implemented? Were you satisfied?
27. What is the follow-up mechanism for the implementation of this solution?
28. What are the means used to implement this solution?
29. What type of difficulties did you encounter? How did you solve them?
30. What changes did they lead to? Have these changes lasted?
31. Did this situation generate actual improvements? Did this situation lead to new solutions to problems?
32. Do you think that you have sometimes treated several problems at the same time? Why and how? Was it after one
or 2 prior cycles?
33. How long does a problem-solving cycle last? Less than 6 months? More than 6 months? Why is it so long? Can you
reduce this period now?
Supervision in general:
34. When was the last supervision performed?
35. Is there a supervisory system in place? How often does it occur?
36. Is it followed? Were there many changes in program planning? What were the reasons?
37. Are there at least 2 trained supervisors per district?
38. How was the ECD trained in developing supervisory skills?
39. Do they show an improvement in supervisory techniques?
40. Do you systematically write a supervision report? To whom do you address it?
41. How do you determine if there is an improvement in healthcare quality at the centers because of your supervision?
42. What type of ongoing training were you able to provide to the healthcare staff?
For the CSI
43. Who is your supervisor? Is it always the same person?
44. How long does supervision last? Is it enough or too frequent?
45. What advantages do you get from supervision?
A11
46. What role did supervision play in problem solving?
47. See above.
48. Did you feel you needed supervision? Did you ask for it?
49. See above.
50. Do you receive supervisory reports on a regular basis? What lessons can you draw from them? What decisions did
you take? Give an example.
Supervisory Checklist
51. Which supervisory tool were you using before this supervisory checklist was implemented?
Supervisory checklist
52. What supervisory tool did you use before this checklist?
53. Why did you decide to use this supervisory checklist?
54. How do you use the supervisory checklist? Do you use the entire checklist, part of the checklist? What parts do you
use most?
55. Do you use the part of the checklist on Quality Assurance always, often, regularly, never? What does it help you
measure?
56. Was it hard to adjust to it in the beginning?
57. What elements do you prefer? What elements don’t you like?
58. What advantages do you get from it?
59. How can you improve the supervisory checklist?
60. What type of improvements did the supervisory checklist bring you?
61. Do you use this supervisory checklist to assess the training at the IMCI? Always? Regularly? Never?
62. What does it help you measure?
Support systems
Coaching/ Facilitation
For the CSI:
63. Is the coach a significant element of the QA system?
64. What role does coaching play in problem solving?
65. How does the coach’s presence change the performance of the problem-solving team?
66. Are there any story books available for the QA teams? Why not?
67. Does the coach visit you at the same time as the supervisor?
Documentation systems
68. What types of documents do you have? Which are the most useful? Why?
69. Do you wish to have other types of documents?
A12
70. Do you have easy access to documents?
71. Is the quantity of available documents sufficient?
72. Are the format and layout of documents convenient?
73. Are the documents easy to understand? Are they complicated?
74. What are the problems with the documentation?
75. Do you think that you have too many, enough, too few, or no documents?
Monitoring
76. What type of support do you have to collect data?
77. How do you conduct analysis at your level, with what methods and specialized support?
78. Do you have in-depth knowledge of the data and related indicators?
79. What lessons do you draw from the data analysis?
ERPA
80. How did ERPA facilitate the work of work teams?
81. How does ERPA improve your performance?
82. Are there any limitations with respect to ERPA implementation in the district?
83. Is ERPA a source of motivation for you?
Quarterly meetings
84. What do you think about the weekly meetings? What do they bring you?
85. What lessons do you take from these meetings?
86. What do you think about the organization and agenda of these meetings?
87. Do you think they should be continued, and how often should they be held?
88. How can they be made more efficient?
89. How are these meeting documented?
90. What do you think about the presence of observers from the central level at these meetings?
Training
91. Does the training meet your expectations?
92. Is the length of training sufficient?
93. Would you rather follow a training course or field training with a supervisor/ coach?
94. How are the current training sessions different from the previous ones?
95. When was your last training session?
96. Do you feel you need refresher training?
A13
97. Did you contribute to or were you consulted in planning training or training needs?
98. Do the training programs answer all needs?
Standards
Adaptation, communication, compliance
At the central level (MPH, OMS)
99. How were healthcare standards of care developed or adapted in Niger?
100. Who was involved in the development of standards?
101. Are the users involved or consulted?
At the regional and district level:
102. What is the most cost-effective means to communicate standards?
103. What is the best format to use to remember the standards easily?
104. Do you think there are too many standards to handle?
105. How did you go from CCM standards to IMCI?
Expansion
106. How were the lessons learned from the QA/BASICS project released in at least 2 African countries?
107. How were the QA/BASICS equipment and program (management, training, supervision and community) shared
with other providers and program directors?
Miscellaneous
108. Do you have accurate data to evaluate the total population? Has there been a significant change in population (exo￾dus); how large?
109. Did the cost recovery have an impact on the use of your center? How?
110. Cite 2 major problems that are a threat to the continuity of the QA activity in your district.
111. How do you assess the impact of other partners’ projects on the QA activity?
112. What impact did the cost recovery have on the QA activities? Positive, negative impact on MEG, on recovery rate,
on general attendance?
113. Did the strikes disturb the development of activities at the district/CSI level? If they did, was the disturbance signifi￾cant, slight, non-existent?
114. How do you intend to self-finance and pursue this QA/IMCI activity later? What are the means you can use?
115. Are there any questions that we have not asked you?
Other questions
Comments
A14
Appendix F: Details of QA Training
Table F1
Number of QA-Trained Health Providers by District
Structure Trained MED/DR TSSI/ IDE TSSO/ SF TSAS/ AS TL THA/ AH IC
MPH/ENSP 3 2 1 0 0 0 0 0
DDS TAHOUA 2 0 0 1 1 0 0 0
CHD 3 1 1 0 1 0 0 0
Maternite Tassigui 2 1 0 0 0 0 0 1
DDS Dosso 1 1 0 0 0 0 0 0
DS Boboye 25 1 9 2 6 1 3 3
DS B.Konni 13 0 3 1 2 1 1 5
DS Bouza 6 0 1 1 1 1 0 2
DS Abalak 6 1 2 1 1 0 1 0
DS Keita 9 1 1 2 1 0 3 1
DS Illéla 10 1 4 0 2 1 0 2
DS Tahoua 7 0 1 2 1 0 2 1
DS Madaoua 7 2 3 1 0 0 0 1
DS Tchinta 3 0 2 0 0 0 1 0
MPH RCI 3 0 3 0 0 0 0 0
Other 1 1 0 0 0 0 0 0
Total 101 12 28 10 15 4 10 16
October 1997 to June 1998
Table F2
QA Training Sessions (Dates, Types, Locations, Number Trained)
Dates Sessions Location Trained (#)
June 16-20, 1997 QA modules review Tahoua 11
Training of QA trainers Level 1 (Education) Tahoua 4
August 11-16, 1997 Training of QA trainers Level 2 (Education) Tahoua 12
October 27-November, 1997 Basic training on the QA approach Tahoua 23
February 22-28, 1998 Basic training on the QA approach Tahoua 18
March 4-10, 1998 Basic training on the QA approach Tahoua 18
March 30-April 6, 1998 Basic training on the QA approach Tahoua 15
April 15-22, 1998 Basic training on the QA approach Tahoua 21
July 20-25, 1998 Basic training on the QA approach Tahoua 6
July 29-31, 1998 Coach refresher training Tahoua 22
A15
Appendix G: Summary of Problems Selected and Outcomes:
Tahoua and Dosso CSI Centers (1993-98)
Table G1
Summary of Problems Selected and Outcomes
Districts CSI/Team Problems Level Comments
Abalak Abalak 1-CRENA Dropped out 1995
2-Low CPN completion rate Continued 1997
3-Low MPE detection rate Continued 1998
Ibécétane 1-Mothers of children with diarrhea have not mastered ESS/SRO Continued 1996
(oral rehydration solution) preparation techniques at this CSI center yet
2-Low utilization rate of recovery services Continued 1997
Tabalak 2- Low utilization rate of PF services Analysis 1998
1-Low treatment rate of malnutrition detected at CRENA Continued 1996
2- Low detection rate of problem pregnancies C-effect 1997
B.Konni Team: 1- Low treatment rate of malaria cases Continued 1994
Team: 1- High number of multiple pregnancy cases Questions 1995
2-High withdrawal rate from CRENA Analysis 1997
Team: 1- High withdrawal rate from PF at the Konni CSMI Withdrawal 1995
(mother and child healthcare facility)
Gounfara 1-Poor PF service provided Continued 1996
2-Low treatment rate of MPE cases Continued 1997
3- Insufficient knowledge of CPN time-table Problem definition 1998
Alléla 1-Low CN attendance Continued 1996
2-Cure treatment follow-up procedure not followed by patients Continued 1998
3-Low CPN coverage rate Continued 1997
Yaya 1- Low CRENA recovery rate Continued 1996
2-Low CPN utilization rate Solutions 1998
G.Idder 1-Decrease in the coverage rate for measles Continued 1995
2-Low CRENA recovery rate Continued 1996
Malbaza 1-Poor welcoming of women at CN Continued 1995
2- Tendency to stop CRENA activities Continued 1996
3-Low CPN attendance rate Questions 1998
Dogueraoua 1-Low utilization rate of curative services Dropped out 1996
2-High level of dropping out at CRENA Continued 1997
3- Decrease in the coverage rate for VAT2 and more Evaluation 1998
Tsernaoua 1- 9th month pregnancy consultation Dropped out 1996
2-Rate of dropping out for CRENA Questions 1998
A16
Appendix H: Costs of QA Activities
Table H1
In-Country Costs Associated with the
Joint QAP/BASICS Program (US Dollars)
Phase 1997 1998 Total (9 Months) (12 Months)
Training in QA and IMCI $32,684 $67,472 $100,157
Support (supervision, 35,568 38,643 74,211
coaching and meetings)
Total $68,252 $106,115 $351,572
Training as a percentage 48% 64% 57%
of total essential QA
activities
Supervision, coaching and 52% 36% 43%
meetings as a percentage of
total essential QA activities
Exhange Rate: US $1 : CFA 500
Data are derived from final report on costs presented to the National Conference on
Quality Assurance (Abdurahmane, October 1998)
Table H2
Details of Training Costs Associated with Quality Assurance (in CFA)
Phase 1998 QA Total February March March/April April
Per diem1 907,500 1,338,252 1,261,500 1,107,500 4,614,752
Training kit 315,000 315,000 270,000 360,000 1,260,000
Honorarium2 272,000 135,000 — — 407,000
Transport 218,500 218,500 200,000 200,000 837,000
Supplies 105,000 105,000 90,000 120,000 420,000
Refreshments 89,090 73,500 63,000 87,500 313,090
Technical assistance3 — — 273,000 273,000 546,000
Total 1,907,090 2,185,252 2,157,500 2,148,000 8,397,842
Number of providers 18 18 15 21 72
Cost per provider 105,949.44 121,402.89 143,833.33 102,285.71 116,636.69
Cost per provider (US $) $211.90 $242.81 $287.67 $204.57 $233.27
Exchange rate: US $1 : CFA 500
1 Per diem paid to all providers/participants and trainers
2 “Consultation fee” paid to (local) trainers for extra efforts to prepare and conduct training session
3 Per diem paid to local technical experts for supervision and oversight of training events
A17
Table H3
Breakdown of Training Costs
Associated with IMCI (in CFA)
Training Cost August 19981
Per diem and honoraria 3,899,000
Training kit 1,485,000
Hall rental 720,000
Transportation 505,000
Lodging 300,000
Other copies 150,000
Total 7,059,000
Number of providers 33
Cost per provider 213,909
Cost per provider (US $) $428
Exchange rate: US $1 : CFA 500
1
 Two training sessions combined
Table H4
Details of Estimated QA
Training of Trainer (TOT) Cost
August 1997
CFA US Dollars
Number of providers 11 11
“Standard” Costs
Per diem 291,500 482
Per diem to QA counselors 58,944 97
Per diem to consultants 258,570 427
Refreshments 60,000 99
Supplies1 627,014 1,036
Subtotal 1,296,028 2,141
Average cost per provider 117,821 195
“Situational” Costs
Hall rental2 50,000 83
Accommodation3 330,326 546
Subtotal 380,326 628
Average cost per provider 34,575 57
Total 1,676,354 2,769
Total average cost per provider 152,396 252
Exchange rate: US $1 : CFA 500
1 Average supply cost determined to be about 57,000 CFA on average,
including reproduction of modules, exercises, flip charts, markers and
overheads
2 When incurred, expense averages 10,000 CFA per participant
3 When incurred, expense averages 30,000 CFA per participant
Table H5
Cost Components of Team-Based
Problem-Solving Techniques (US Dollars)
Cost
Category Description per Cycle
Data Collection
Transport Average cost of fuel to operate $12
standard motorcycle for data
collection purposes1
Data collector 2-3 analysts hired for 2-3 days, $8 - $24
compensated at $2 per day
Reproduction Cost of photocopying 30 $3
of survey questionnaires at rate of
US 10 cents per questionnaire
Other Other supplies (e.g., pencils ) Negligible
Supplies
Flipchart Imported flip-chart paper sufficient $6
and stand for one cycle
One-time cost of tripod stand with [$14]
a useful life lasting over multiple one-time
cycles
Markers Supply of 3-4 imported markers $9 - $12
at $3 each
Record book2 Single record book at $8, sufficient $4
for 2 cycles
Total fixed cost [$14]
Total recurrent cost $42 – $61
Exchange rate: US $1 : CFA 500
1 Estimate based on average distance of 80 kilometers traveled per cycle and
fuel mileage reimbursement rate of approximately US 15 cents per kilometer
2 For meeting minutes and coaching instructors
Table H6
Labor Costs Associated with a Standard
Problem-Solving Cycle (US Dollars)
Average Average Cost
Category Monthly Salary1 per Cycle
Head nurse 120 9 – 18
Assistant nurse 80 6 – 12
Janitor 40 – 50 3 – 7.5
Cook 402 3 – 6
Midwives or traditional 15 – 25 15 – 25
birth attendants (1 or 2)3
Total average labor costs $36 – 68.50
Exchange rate: US $1 : CFA 500
1 Based on a 160-hour week
2 Average salary, ranging from income of farmer to average salary of
schoolteacher
3 Estimate determined per cycle, based on an estimated 6 to 10 births per
cycle (3 months) and 500 CFA to 2000 CFA fee charged by midwife or
traditional birth attendant
A18
Table H8
Annual Cost of Supervision Activities (1997: US Dollars)
Average Average Average Average Consumption
Amount Actual Number Visits Visits Cost per Cost per Cost per Cost per Completion Consumption Rate Adjusted
Budgeted Cost of CSIs Scheduled Completed CSI: CSI: Visit: Visit: Rate Rate for Completion
Budgeted Actual Budgeted Actual Rate
Districts Completing Supervision
Tahoua 1,208 1,448 14 4 4 86 103 22 26 100% 120% 120%
Konni 884 1,052 8 4 4 111 132 28 33 100% 119% 119%
Illéla 840 984 10 4 4 84 98 21 25 100% 117% 117%
Madaoua 736 880 7 4 4 105 126 26 31 100% 120% 120%
Average Cost 96 115 24 29
Districts Partially Completing Supervision
Abalak 444 550 4 4 3 111 137 28 46 75% 124% 165%
Bouza 700 434 9 4 2 78 48 19 24 50% 62% 124%
Keita 672 408 7 4 2 96 58 24 29 50% 61% 121%
Tchinta 784 250 5 4 1 157 50 39 50 82% 103% 127%
Average Cost 103 96 26 33
Exchange rate: US $1 : CFA 500
District
Table H7
Annual Cost of Coaching Activities (1997: US Dollars)
Average Average Average Average Consumption
Amount Actual Number Visits Visits Cost per Cost per Cost per Cost per Completion Consumption Rate Adjusted
Budgeted Cost of CSIs Scheduled Completed CSI: CSI: Visit: Visit: Rate Rate for Completion
Budgeted Actual Budgeted Actual Rate
Districts Completing Coaching
Konni 1,784 1,482 8 12 12 223 124 19 15 100% 83% 83%
Tahoua 2,384 2,534 14 12 12 170 211 14 15 100% 106% 106%
Average cost 197 167 16 15
Districts Partially Completing Coaching
Madaoua 1,148 626 7 12 9 207 52 17 10 75% 43% 58%
Keita 1,296 349 7 12 9 185 29 15 6 75% 27% 36%
Bouza 1,232 712 9 12 6 137 59 11 13 50% 58% 116%
Abalak 1,040 322 4 12 6 260 27 22 13 50% 31% 62%
Tchinta 1,880 1,244 5 12 6 376 104 31 41 50% 66% 132%
Illéla 1,760 469 10 12 6 176 39 15 8 50% 27% 53%
Average cost 215 90 18 15
Exchange rate: US $1 : CFA 500
District
A19
Table H9
Cost of Coaching Activities (US Dollars)
Main- Percentage Percentage Percentage
tenance per Diem: per Diem: Main- Total
Fee Coaches Driver tenance Fee
Tahoua 14 $ 252 $ 70 $ 1992 $ 80 $ 2,394 11% 3% 83% 3% 100%
Tchinta14 5 180 50 1800 80 2,110 9% 2% 85% 4% 100%
Konni 8 144 40 1560 80 1,824 8% 2% 86% 4% 100%
Illéla 10 180 50 1512 80 1,822 10% 3% 83% 4% 100%
Madaoua 7 126 35 1248 80 1,489 8% 2% 84% 5% 100%
Bouza 9 162 45 1008 80 1,295 13% 3% 78% 6% 100%
Keita 7 126 35 888 80 1,129 11% 3% 79% 7% 100%
Abalak 4 72 20 912 80 1,084 7% 2% 84% 7% 100%
TOTAL $1,242 $345 $10,920 $640 $13,147
Percentage of TOTAL 9% 3% 83% 5% 100%
Amounts based on estimation and not actual costs; percentages may not total 100 due to rounding
Assumes fuel mileage rate of 25 liters per 100 kilometers traveled, at rate of CFA 340 per liter, CFA 500: $1
Because of the distance between health centers in Tchinta, per diems are calculated at one visit per day
1 Two coaches’ per diem is CFA 1500 per two health centers visited
2 Driver’s per diem is CFA 1500 per health center visited (CFA 500 : US $1)
CSIs Per Diem: Per Diem: Fuel Total Percentage
(Number) Coaches1 Driver2 Cost Cost Fuel Cost
Table H10
Annual Cost of Supervision Activities (US Dollars)
Percentage Percentage
 per Diem: per Diem: Total
Supervisors Driver
Tahoua 14 560 168 664 1392 40% 12% 48% 100%
Illéla 10 400 120 416 936 43% 13% 44% 100%
Konni 8 320 96 504 920 35% 10% 55% 100%
Abalak 4 160 48 600 808 20% 6% 74% 100%
Bouza 9 360 108 336 804 45% 13% 42% 100%
Tchinta 5 200 60 520 780 26% 8% 67% 100%
Madaoua 7 280 84 304 668 42% 13% 46% 100%
Keita 7 280 84 296 660 42% 13% 45% 100%
TOTAL $2,560 $768 $3,640 $6,968
Percentage of TOTAL 37% 11% 52% 100%
Exchange rate: US $1 : CFA 500
Amounts based on estimation and not actual costs; percentages may not total 100 due to rounding
1 Two supervisors’ per diem is CFA 2500 each per health center visited (CFA 500 : US $1)
2 Driver’s per diem is CFA 1000 per two health centers visited
3 Assumes fuel mileage rate to 25 liters per 100 kilometers traveled at a cost of CFA 340 per liter
CSIs Per Diem: Per Diem: Fuel Total Percentage
(Number) Supervisors1 Driver2 Cost3 Cost Fuel Cost
A20
District/CSI Distance Total Fuel Cost Rounded Fuel Costs
(Round-trip in kilometers) CFA CFA Dollars
Tahoua 980 83300 83000 $ 166
Bambeye 60 5100
Takanamat 140 11900
Barmou 60 5100
Tebaram 184 15640
Moggar 90 7650
Taza 130 11050
Affala 50 4250
Kalfou 66 5610
Samo 100 8500
Wadata 0
Kouffan Ta 0
Eddir 100 8500
Illéla 770 65450 65000 130
Tajae 56 4760
Bagaroua 220 18700
Dangona 108 9180
Badaguichiri 32 2720
Yama 70 5950
Kaoura 124 10540
Sahiya 160 13600
Konni 746 63410 63000 126
Tsernaoua 28 2380
Allela 134 11390
Dogueraoua 90 7650
Gounfara 264 22440
Malbaza 66 5610
Guidan Idder 50 4250
Yaya 114 9690
Madaoua 610 51850 52000 104
Arzerori 20 1700
Sabonguida 50 4250
Ourrno 200 17000
Bangui 130 11050
Takorka 80 6800
Manzou 130 11050
Table H11
Cost of Travel: Round-Trip Between CSIs and District Headquarters
A21
District/CSI Distance Total Fuel Cost Rounded Fuel Costs
(Round-trip in kilometers) CFA CFA Dollars
Table H11 (continued)
Cost of Travel: Round-Trip Between CSIs and District Headquarters
Bouza 450 38250 38000 76
Deoule 50 4250
Babankatami 50 4250
Gradoume 50 4250
Tama 40 3400
Tabotaki 80 6800
Karofane 40 3400
Tadoupta 140 11900
Keita 498 42330 42000 84
Tamaske 44 3740
Ibohaman 40 3400
Gadamata 80 6800
Garhanga 130 11050
Fararat 90 7650
Insafari 114 9690
Abalak 430 36550 37000 74
Chadawanka 160 13600
Ibecetane 100 8500
Tabalack 170 14450
Tchinta 880 74800 75000 150
Tassara 240 20400
Tillia 260 22100
Kaou 110 9350
Telemces 270 22950
Exchange rate: US $1 : CFA 500
A22
Notes